Amazing work! But deeply frustrating on the lack of reproducibility and how far off this is from a proper SDLC.
How can I inspect exactly how agents were prompted? How can I reproduce this setup and try out my own AI setup? Am I missing a link to a repo somewhere?
These evals probably match how many people are using AI, but its not the full package of how we know software development needs to be done, and not how I do it with AI. The closest would be "Audit" which scored the highest when using xhigh- that actually incorporates a review cycle- something we know is the most important part of the software development process for code correctness (design/specification is not as much of an issue in this problem since the task is to write code against an existing spec). However, we don't know what instructions they have in their "Audit".
I would love to benchmark my own flow [1] if I can be given their exact problem. What it does is (assuming there is already a solid spec)
* plan with expensive model. Review the plan.
* implement with cheap model. Review for spec compliance and code quality.
* Reviews are done adversarially from the expensive model with a fresh context.
* ensure that verifications (automated or manual) are performed.
For non-trivial changes, the review and verification process almost always catch significant issues.
The workflow does use TDD. I do find useless tests being written and I need to dig into this part of the workflow a lot more, so its great to see that aspect of this article. My experience writing software has taught me that code must be written to be easy to test, but not necessarily done TDD style.
As someone who also spent a bunch of time benchmarking various techniques, this is as good as you can do without publishing a formal versioned benchmark suite that you want to maintain and run forever at immense cost to yourself. If you actually go to benchmark your own flow as you mentioned, you will quickly run into like a dozen problems that discourage you from publishing.
- Are you sure that temperature and other nondeterminism isn't affecting your output?
- Are you sure you're not being routed through an A/B test at this moment?
- Are you sure there's not a bug affecting the model at this moment?
- Are you sure that you picked the right model and effort level?
- Are you sure that your result generalizes across providers?
- Are you sure that you set up the correct level of sandboxing and the agent can't e.g. look at a sister directory or git history in the current directory for answers?
- Are you sure that the agent isn't leaking answers in memory or its conversation history?
- Are you sure that tool calls aren't somehow affecting results?
- Are you sure that your results are robust, i.e. you see the same results with mild tweaks to the prompt?
- Are you comfortable keeping your blog post live when your results are invalidated next week with the next model launch?
And that's just a quick list off the top of my head.
I personally decided that it wasn't worth it, I'm glad that Dan decided to publish his. Frankly I think we could use a lot more of these "I ran these 2 techniques side by side and here's what I saw" anecdata, because most people who promote prompt techniques can't produce a single prompt they ran twice because they never actually tested it per se.
I am really grateful that he’s publishing as well.
I am just asking for the bare minimum to actually understand what has been tested and for reproducibility of methods- the publication of the prompts. It would take a lot less time than all the guess work analysis write up and be a lot more useful.
Without the prompts the rest of your questions about reproducibility are moot.
This post boils down to, "I tried a bunch of things that didn't work very well". But we don't actually know what he tried. Everything hinges on Dan's prompting skills.
Get them under version control.
I have a git repo with my skills for software development [1]. There is an installer script that symlinks to the skills. Updating the skills on a machine is then just a matter of advancing the git repo. One of the skills comes with some bash scripts, but the rest are effectively just prompts.
Putting project specific skills in projects works well.
I make sure they work by understanding every skill, reviewing pull requests, and testing the end product. The result is rarely perfect, so I am constantly tweaking the skills and how I use AI.
The way to control LLM memory with existing tools now is to ask it to write out a file with all relevant information (this could include explicit retraction instructions). Then clear out the context. Basically /compact.
By default LLMs will start creating a tangled hard to test mess of javascript. But if you ask them to do TDD they will, and if you give them access to Playwright and have them write playwright tests this can all work.
The advantage for the LLM is the same for humans. With the right abstractions, code can be written quicker and more robustly.
To get more hands off with LLMS you really want to have a lot of engineering rigor. A big part of that is a focus on tests. If your abstractions make testing easier, your results will be a lot better. For Javascript the more you can test logic without DOM manipulation and without playwright tests, the better.
I found LLMs to be default to and be pretty good at working with HTMX, but not great (mostly lots of state synchronization issues). The focus on DOM markup of HTMX does not lend itself well to simplifying testing. I don't claim to be an expert at HTMX, but I don't see a separate section on testing in their documentation.
I am trying out the foldkit framework because it makes a lot more testable outside the DOM. Its probably too heavy weight for most humans (and for simple apps- it only seems appropriate for client-side apps), but I will see how well LLMs can work with it.
Great stuff. AGENTS.md is not the ideal place for most of it though. Most of what is shown in this article can go in CODING_STANDARDS.md. The skills that I use find this document when it is needed (writing and reviewing code) so it doesn't pollute context when code is being read.
I also have sub-agent reviews (both of a planning phase and the produced code) that would catch some of these problems and demand revisions. [1]
> - If the prompt indicates that a bug is being fixed, don't write the fix right away. First write the test. Observe it failing. Then write the fix. And observe the test passing.
I always use /tdd [2]. Occasionally it results in some silly tests, but it produces much lower defect code. Its not just for bugs.
You mentioned libkrun in the post, but I am not clear on why you chose not to use it for Mac even if you want to stick with Firecracker on Linux. Why not use libkrun?
I have accomplished code base (small size) migrations with it as well. Currently I do review each PR. For a large code base migration I think the core skills would still work but need a different way of driving it as you have come up with.
If you start with a spec you understand at the beginning then you don't need the LLM to generate high-level information about the changes at review time.
The grilling (grill-with-docs) skills [1] are amazing for ensuring you produce a through spec that covers all the edge cases. The /code-review skill from there helps ensure that the code changes meet the spec.
I use an intermediate detailed plan stage (done by a more expensive model) before implementation. Information from that plan is posted on the PR to give pretty much all the intermediate level context reviewers need.
I do like incorporating the idea of this article into my flow- that the spec and PR context could be presented in a more educational way.
It blocks network access by default, mounts only what you specify, and you can add a customization layer. This is all done in the container itself (srt for network blocking). It doesn't implement a central point for secret sharing, MCP exposure, etc. So it might not have enough features for some but it works well for my needs.
I just found through this thread yoloai which has an apple container backend, so its quite similar using that. My main issue would be that network access is allowed by default. https://github.com/kstenerud/yoloai
Several other projects listed here use libkrun which is an alternate implementation that works with Mac's HVF. smolvm, microsandbox, podman (with likrun backend), gondolin.
If you have taste in writing code, you spend less time on bugs and can implment features more quickly. This is still true with LLMs if you learn how to direct them instead of having them direct you.
Someone else without taste can re-implement what you have done with AI. But it will take them more effort, more time, and more money.
But that's always been the value in writing code. Any MBA can end up producing software to accomplish something by hiring other people. But their company might not get off the ground because of the expense of not having technical competency (or engineers with taste) at their core.
How can I inspect exactly how agents were prompted? How can I reproduce this setup and try out my own AI setup? Am I missing a link to a repo somewhere?
These evals probably match how many people are using AI, but its not the full package of how we know software development needs to be done, and not how I do it with AI. The closest would be "Audit" which scored the highest when using xhigh- that actually incorporates a review cycle- something we know is the most important part of the software development process for code correctness (design/specification is not as much of an issue in this problem since the task is to write code against an existing spec). However, we don't know what instructions they have in their "Audit".
I would love to benchmark my own flow [1] if I can be given their exact problem. What it does is (assuming there is already a solid spec)
For non-trivial changes, the review and verification process almost always catch significant issues.The workflow does use TDD. I do find useless tests being written and I need to dig into this part of the workflow a lot more, so its great to see that aspect of this article. My experience writing software has taught me that code must be written to be easy to test, but not necessarily done TDD style.
[1] https://github.com/gregwebs/skills-sdlc/
reply