Ask HN: How would you harden AI changes to a 1M-line legacy SaaS before review?
11 comments
Answering your questions from my knowledge level on AI:
Managing AI agents is like running a knowledge business. If you are not an expert enough to understand what agents are doing, there is still large room for errors. Its important to set design decisions early in the development process.
Your questions:
3. I think this is absurd. Agent can generate the test cases, but you still have to verify the tests are actually valid.
4. The architecture document should be the source of truth. SO when a engineer reviews they know what to look for & flag/fix. The comprehensive the document is the better. Version it based on what the prototype carries. — Data models, API design, Services, Roles, etc. You can ask this to AI — What all should a high/low level SDD carry?
5. Just make sure the architecture doc is solid, development is as per the document, ask AI to generate/perform regression test cases and document them as well (pass/fail).
So the engineer knows where things exactly stand.
Thats the best can do
Appreciate thoughts on this.
Managing AI agents is like running a knowledge business. If you are not an expert enough to understand what agents are doing, there is still large room for errors. Its important to set design decisions early in the development process.
Your questions:
3. I think this is absurd. Agent can generate the test cases, but you still have to verify the tests are actually valid.
4. The architecture document should be the source of truth. SO when a engineer reviews they know what to look for & flag/fix. The comprehensive the document is the better. Version it based on what the prototype carries. — Data models, API design, Services, Roles, etc. You can ask this to AI — What all should a high/low level SDD carry?
5. Just make sure the architecture doc is solid, development is as per the document, ask AI to generate/perform regression test cases and document them as well (pass/fail).
So the engineer knows where things exactly stand.
Thats the best can do
Appreciate thoughts on this.
I agree that in perfect world I could have done this exercise with competent engineers and a UX desginer. I know about software development processes and applied that to e.g. testing and hardening, but it is not a substitute for deep technical knowledge.
THe idea behind having a separate agent create the tests is to that then the coding agent couldn't get functionality to work or fix bugs by weakening the test. I believe Bob Martin has talked about this.
Noted about the architecture document - will have to ensure it is as solid as possible. I already run the full CI pipeline that we run for our app + the new ones that the agents developed - would it be good to have additional regression tests?
If you're planning to put it into production anything you do outside of bringing somebody qualified at this point is likely a waste of time. Highly recommend this recent talk by Kelsey Hightower on the subject - https://www.youtube.com/watch?v=A7WFt2JQ5sg
Thanks for the link! You could very well be right that there is not much more that could be done before the review. My reading of what Kelsey said is that agents can produce useful outcomes, it is just that humans need to validate and own them. In other words, I couldn't conclude that this somehow disproves that at least some of the functionalities that were generated in my project, couldn't be (close to) production-grade.
This is more about understanding the big picture and architecture of what's going on (the part about the diagrams / blueprints). One practical thing actually that may be useful to you - try thinking and testing in terms of "what can go wrong" rather than "is this prod-ready".
1. A CI pipeline improves confidence a lot. A document showing how the product adheres to existing decision records, and specific cases where it deviated (and why).
2. Use different prompts and only allow interaction with the product (end-to-end tests).
3. Separate from the tests in the existing 13k lines, have tests where agents or models are only allowed to interact with the product itself.
4. Docs with UML diagrams, common code patterns, and working examples. Diagrams that show where the new system interacts with the legacy one. Record demos.
5. Security, maintainability, performance.
2. Use different prompts and only allow interaction with the product (end-to-end tests).
3. Separate from the tests in the existing 13k lines, have tests where agents or models are only allowed to interact with the product itself.
4. Docs with UML diagrams, common code patterns, and working examples. Diagrams that show where the new system interacts with the legacy one. Record demos.
5. Security, maintainability, performance.
In the past, problems like these were solved with software engineers. Those trained through schooling and experience to be able to read through and understand obscure code conventions and programming quirks.
Their kind has almost gone extinct in this universe...
Their kind has almost gone extinct in this universe...
I hear you. This must have been a long time ago, like last year or something. Those were the times.
PRD? Product Requirements Document?
Pontiac Racing Division https://en.wikipedia.org/wiki/Toyota_Racing_Development
Yes, that is correct
I'd do two things before any code changes:
1. Ask the AI to document the code base; either in a single repo or in a sidebyside like <file>.md; then scale up with larger documents for functionality/submodules etc. This will get you and your team to properly contemplate the monstrosity.
2. Once you have documentation, then you write tests; unit tests, e2e, etc. Make sure everything is passing the tests.
After that, you'll hopefully both gained proper insight and solidified functionality. Then you can port whatever you like.
1. Ask the AI to document the code base; either in a single repo or in a sidebyside like <file>.md; then scale up with larger documents for functionality/submodules etc. This will get you and your team to properly contemplate the monstrosity.
2. Once you have documentation, then you write tests; unit tests, e2e, etc. Make sure everything is passing the tests.
After that, you'll hopefully both gained proper insight and solidified functionality. Then you can port whatever you like.
At this point what stops you from asking AI this question?
But honestly, extensive QA. There are probably more quirks than you could imagine
I have asked the AI, but was hoping for a second opinion, ideally by someone who has experience from doing something a bit similar. Your point about QA is interesting. Bob Martin, doesn't review AI code any more, just does extensive QA (https://x.com/unclebobmartin/status/2080257779395154409). Perhaps the key move would be for me to now focus on QA and try to nudge the engineering review to also emphasize testing over reading code?
Once you no longer "own" the code, does it matter? If you provide a very detail set of specifications and success criteria then you can turn it into a black box. The code is no longer the asset, just the business that uses it. You can use AI to extract the requirements from the existing code and then modify as needed however, I would recommend doing this in two phases: first the conversion, then the enhancements.
I suppose you are right at least what comes to the functionality. Security is the one thing where I am sure that my company will require some peaking under the hood.
idk, I'm of the opinion that you own the code you wrote with AI.
The codebase is 1M+ lines, 15 years old, hosted on Azure, and primarily written in C# and React.
The prototype needs to be available for customer testing in September. No developers were available to work on it full-time, although I could occasionally get help with specific technical issues. An engineer will evaluate the implementation in August and decide how much confidence we can have in the AI-generated code so that we can decide how to “convert” the code to production-grade.
My question is what I can do before August, primarily using AI tools and manual testing, to make the code as robust and reviewable as possible. I want to increase the likelihood that the AI-generated code would be so good that the path to production-grade would be closer to “copy-paste” than building everything again from scratch.
Environment The prototype is being developed in a separate branch, deployed to a separate internal environment, and connected to its own database and schema.
Planning process We interviewed customers in June and turned the resuls into an MVP spec.
The process was approximately: 1. Write a PRD. 2. Use an LLM to convert the PRD into an architecture document, which was reviewed by an architect. 3. Create product designs consisting of screen images and md files containing interaction details ans edge cases with Claude Design. 4. Use an agent to break the work into epics using the PRD and architecture document as guardrails.
The epics were the most granular planning artifacts that received review by me and the architect.
Development process The development flow was intended to run with little intervention: 1. A planner agent converted epics into md story files and Jira stories 2. A coding agent implemented the stories including tests and opened PRs 3. A reviewer agent reviewed the PRs, requested changes, and merged them into the prototype branch
The coding agent polled PRs for review comments and could escalate issues back to the planner.
Agents also had a “stop and ask” list for decisions they were not allowed to make autonomously. I (and a few times an engineer) were involved by resolving those escalations and by manually testing the accumulated changes end to end each day.
Most implementation and initial review were done with Claude-based agents. For riskier PRs, I also used Codex as a reviewer. The second-model review found substantially more relevant issues in the Claude-generated code, but token quotas limited the use.
I had separate refactoring and harden runs for the code as well.
Results The planning and setting up the environment and agentic flow took about two weeks and then the agents built the whole MVP in about two weeks. Size-wise it was 13k lines of functional code + the same amount for tests.
What I would like advice on Assuming that I cannot get substantial developer involvement before the review, how can I increase the likelihood that the code is as close to production-grade as possible?
Here are some of the questions I have been thinking about: 1. What checks or development loops would give the largest improvement in confidence before an engineer reviews the code? 2. How would you use independent agents or models to reduce the risk that the coder and reviewer make the same incorrect assumptions? 3. Should tests be generated by a separate agent from the one that wrote the implementation? 4. What documentation or evidence would make the eventual engineering review faster and more reliable? 5. If you had only a few weeks to improve this prototype before handing it to an engineer, what would you prioritize?