1. Specs are subject to bit-rot, there's no impetus to update them as behaviour changes - unless your agent workflow explicitly enforces a thorough review and update of the specs, and unless your agent is diligent with following it. Lots of trust required on your LLM here.
2. There's no way to systematically determine if the behaviour of your system matches the specs. Imagine a reasonable sized codebase - if there's a spec document for every feature, you're looking at quite a collection of specs. How many tokens need be burnt to ensure that these specs are always up to date as new features come in and behaviour changes?
3. Specs are written in English. They're ambiguous - they can absolutely serve the planning and design phases, but this ambiguity prevents meaningful behaviour assertions about the system as it grows.
Contrast that with tests:
1. They are executable and have the precision of code. They don't just describe behaviour of the system, they validate that the system follows that behaviour, without ambiguity.
2. They scale - it's completely reasonable to have extensive codebases have all (if not most) of their behaviour covered by tests.
3. Updating is enforcable - assuming you're using a CI pipeline, when tests break, they must be updated in order to continue.
4. You can systematically determine if the tests fully describe the behaviour (ie. is all the behaviour tested) via mutation testing. This will tell you with absolute certainty if code is tested or not - do the tests fully describe the system's behaviour.
That being said, I think it's very valuable to start with a planning stage, even to provide a spec, such that the correct behaviour gets encoded into tests, and then instantiated by the implementation. But in my view, specs are best used within the design stage, and if left in the codebase, treated only as historical info for what went into the development of the feature. Attempting to use them as the source of truth for the behaviour of the system is fraught.
And I guess finally, I think that insofar as any framework uses the specs as the source of truth for behaviour, they're going to run into alignment problems since maintaining specs doesn't scale.
In my view, Spec-Driven systems are doomed to fail. There's nothing that couples the english language specs you've written with the actual code and behaviour of the system - unless your agent is being insanely diligent and constantly checking if the entire system aligns with your specs.
This has been solved already - automated testing. They encode behaviour of the system into executables which actually tell you if your system aligns or not.
Better to encode the behaviour of your system into real, executable, scalable specs (aka automated tests), otherwise your app's behaviour is going to spiral out of control after the Nth AI generated feature.
The way to ensure this actually scales with the firepower that LLMs have for writing implementation is ensure it follows a workflow where it knows how to test, it writes the tests first, and ensures that the tests actually reflect the behaviour of the system with mutation testing.
Actually, every single word was hand typed. You'll probably notice that in areas where I could improve my grammar. It's understandable that you hit the wall of text and felt a bit dismayed by the length- hence the TLDR at the top and the example repo :)
That's the kicker - you taught it conventions that it follows. Even better (in my view) is to ensure you're getting higher coverage and tests focused on behaviour by getting it to write the tests first.
Even moreso by ensuring it writes "feature complete" tests for each feature first.
Even moreso by running mutation testing to backfill tests for logic it didn't cover.
You'd be surprised - I know I was - you can encode Test-Driven development into workflows that agents actually follow. I wrote an in-depth guide about this and have a POC for people to try over here: https://www.joegaebel.com/articles/principled-agentic-softwa...
I've found the best way to achieve that is to force the agent to do TDD. Better to get it to do Outside-in TDD. Even better to get it to run Outside-in TDD, then use mutation testing to ensure it has fully covered the logic.
I've been able to encode Outside-in Test Driven Development into a repeatable workflow. Claude Code follows it to a T, and I've gotten great results. I've written about it more here, and created a repo people can use out of the box to try it out:
I'm surprised to see the lack of support in the comments. In my take, this article was very well written and goes a long way to describe the refreshing and enlivening aftermath of a peak experience with Psilocybin. It also mirrors the experience of the research subjects at Johns Hopkins.
One study, conducted on terminally ill cancer patients, found that most of them rated it as one of the most important experiences of their lives [1], rating it alongside the birth of their children, their wedding day, etc. Additionally, the depression easing effects were shown to persist up to a year by this study.
I wouldn't be surprised if experiencing one of the most important, enliving, and connective experiences of your life wouldn't go on to reduce depression and anxiety for the rest of your life.
Great to see you at the top of HN today tectonic! I'm at Pivotal in Sydney Australia now. Thanks again for the impromptu interview all of those years ago!