"At the time of writing, about 4% of Bun's Rust code sits inside an unsafe block (~13,000 unsafe keywords across ~27,000 lines / ~780,000 lines), and 78% of those blocks are a single line — a pointer that came from C++, or one call into a C library. I expect this number to go down over time as we refactor from a faithful Zig port (which had no greppable unsafe keyword) to idiomatic Rust, but we are going to continue using C & C++ libraries like JavaScriptCore so it will always have more unsafe than pure Rust projects."
Drilling into the original article where Jarred explained the reasoning behind the change, It's pretty clear that under zig the team was doing things by hand that are automatic in rust.
Humans and agents share one thing: they are both non-deterministic. He talks about the issue of tracking memory lifecycles manually in zig so it can be explicitly freed. As expected, this leads to a long list of bugs where people missed things.
Rust does this automatically. It removes an entire class of errors from his backlog. From an engineering management perspective, this looks like a pretty good trade.
The bonus here is that compiler errors are exactly the kind of deterministic guardrail you need to put around coding agents. Claude works really well if you give it a way to test for correctness and "make it compile" is a pretty good target.
There's a general version of this: the artifact you expose plus the test you run on it. Deterministic tests turn stochastic output into a hard guarantee. Wrote it up here if useful: https://michael.roth.rocks/blog/verification-surface/
Agents are fast and powerful, and that is a double-edged sword. The volume you can produce is breathtaking, but they make far too much for you to review every line of output. Personally, I focus my limited attention on the places that give the biggest levers and put deterministic guards throughout.
First, recognize that most of what you want done is actually a sequence of steps, and those steps produce intermediate artifacts. And you can either review or gate artifacts as they flow through the process. The earlier you look, the more leverage you have.
I make software, which follows these steps: make a plan with tasks, make a design for each task, code each task, run the tests. I spend a lot of time on the plan, because mistakes here expand drastically once they hit code.
But it's not just all on me: I write scripts and use tools to make guarantees about the artifacts. For a plan, for example, I have a script that makes sure it has the correct structure. I also use another agent to review it according to my personal specification.
Every artifact has this, because the reliability of the final product of the pipeline is actually composed from all those checks.
You have to have a multi-layered approach. This is the "Swiss cheese" model of prevention: individual layers may have holes, but if none of the holes in the stack line up, nothing makes it all the way through.
I approach this by thinking of verification surfaces. Each one is a layer.
For SQL, I have a tool that the agent can call to run queries. (I actually keep a range of tools: for known queries the SQL is baked into the tool and only results come back; I also have a general SQL tool. Prefer the former; deploy the latter only if you must.)
The general SQL tool itself is multi-layered. The raw SQL produced by the agent is a verification surface, as are the parameters used in the tool call.
There are plenty of existing utilities that validate SQL and define its claims, scope, and if it mutates in a structured way. My tool runs those against the SQL as a verification surface and it hard gates some requests.
It also verifies the parameters. It does this in two ways: at a shallow level, it just makes sure they are valid and sane, so it can fail fast if the LLM does the wrong thing.
But it also does a deeper check, where it calls out to supporting microservices to verify a) permissions embedded in the request and b) consistency of the request with the rest of the system.
Since you asked about challenges in prod: if you are not careful, the LLM can write wildly inefficient queries. Some of those made it through and I had to go back and tweak the prompts to give enough context to have it tune properly.
My suggestion: think through the verification surfaces, figure out what you can verify deterministically, and use that as gates in your tool.
I break this up into two parts: assets and workflow state. All assets go in the repo, this is the context that all agents read to be able to understand what we're doing and how we do it.
The workflow state is the release train with all the tasks and dependencies. I use a personal MCP I wrote to store all this in a DB, out of band. I previously had .md files that did this, but I needed 1) a guaranteed structure and 2) hard enforcement of the structure when the agents change things.
My general approach is to have a conversation with Claude Code about what I want and how to do it, in the context of the repo. I typically start with a context-priming exercise where I ask it how something works now. Then we talk about the approach and I have it describe what it is going to do. I iterate until I agree and think it properly captures it correctly.
Then I have it build out the release train to do what we agreed in my tool. The tool itself has structured docs that it cannot change that describe the workflow, so I have it read that first. I review the releases, then have it burn down the tasks.
Once it finishes with a release, I use this prompt to keep everything up to date in the repo:
"Please review all the relevant .md files and compare them to what you know now. If you can improve them please do so. Use modular documentation, with concise CLAUDE.md files at the appropriate places in the dirtree that point to full docs in .md files which you can read as needed."
This works for me: the immutable stuff is in the tool as docs, critical state is also in the tool with an enforced interface, and the repo has all the living context.
The biggest strength of "would this get merged" is that it is actually a compound property: does it work, does it match convention, is it maintainable. And of course: does the reviewer actually want to take ownership of it. A single score has to average across these disjoint axes, which is probably why you needed so many rubrics per problem.
What is the shape of the failures? When a model loses points, do they cluster on correctness? On convention? I run an autonomous pipeline and I can handle model shortcomings, but this kind of detail tells me what I need to shore up.
Yes, at some point AI will be fully integrated into society so that there are entire autonomous sections. But society doesn't move at the same pace as technology.
The output of agents has to have economic value, and for the foreseeable future this means someone is going to have to buy something.
Right now, it is humans who ultimately make the economic decision. Even if you have fully autonomous agentic organizations selling to each other, there are one or more humans at the end of the of the chain who agree to exchange money for value.
Science Fiction writers have envisioned futures where some currency other than money is used to track value, but so far as I can tell we are nowhere near moving to anything like that.
Let me ask you this: would people accept a nuclear power plant whose safety control software is vibe coded? Would a CFO of a public company sign off on financial statements created entirely by AI?
I'm fully into agentic coding, and I'm actively studying agent reliability. I can make all kinds of deterministic guarantees about agentically produced code, but no, I would not accept either of these, or many other examples.
We write code (or we used to, before AI), so we naturally value that. But the code is one small part of a deployed system, and this has always been the case. Numerous studies have shown that writing the code is actually the cheapest part.
All the most important things that you need to know about what makes the code correct, both before it is written and after it is deployed, are not in the specs. They come from walking around and talking to people. Looking them in the eye, sussing out what their real requirements are, and figuring out how to address their concerns with empathy.
Until LLMs can do that, I'm not worried. Let them write the code, that's the least important part.
The easy decision is to just go with the biggest SOTA model you can afford.
But this overlooks the other critical part of getting the most out of these things: the harness. I run an autonomous plan/design/code/build/test pipeline with agents using my own orchestrator. Different models are better at different stages, and I use LLMs to judge the output between them. Not everything needs Opus 4.8.
The harness provides both the scaffolding to get the right things into the model, and the right things out. But it also lets you dictate which model does which work.
It's the pipeline, not the model, that gets you quality at a given token budget.
Some of it can be busywork, but for me the intermediate artifacts (plans, design docs, etc) serve a real purpose: they create a verification surface where you can check that the agent is creating the right thing before it goes all the way. It's exactly the same reason we created short sprints: if the team misunderstood the requirements and built the wrong thing, you only lost a sprint. We lost months of work when we did waterfall because the product did not match what the customer had in mind.
I have deterministic and stochastic tests that run on each artifact. For those that have a high risk of "not the right thing", I manually review the artifacts. But if it's bog standard I just rely on the auto-gates to reject and get the agent to retry the artifact.
This gets me a high-volume pipeline that yes uses a lot of tokens, but at the same time doesn't overwhelm me. I only deal with things that genuinely need my attention. That's worth it for me, and not busywork.
I've been specializing in distributed systems for nearly 35 years. I've read your work, and it's shaped my thinking. When you say you have a person in mind when you write, I am that person. Thank you for what you've done.
I don't think this replaces you. The hard part of reliability is understanding the failure modes in the context of the business. No one has unlimited time or money, we always have to make tradeoffs. Only experienced humans have both the ability to interrogate the stakeholders and a vision broad enough to understand what to pursue versus what to give up.
Tools like this make the grind part of the job easier. They do not replace the holistic view you need to be able to confidently tell someone "worry about X, do not worry about Y".
Definitely stacks. The thing that made it clear for me was being explicit about the stages, and where/what you can verify with a guardrail, or gate. I wrote up the framework I use here: https://michael.roth.rocks/research/trust-topology/
Being explicit about the space between the stages is critical, because that's your enforcement point.
Yes, "guardrails" is a squishy term. But it gets clearer if you ask what transition is being guarded.
Some of this is inside the model, like topic refusals. Forge sits at the tool call level.
My personal workflow uses guardrails at the SDLC level: I have a standard pipeline (plan, design, code, build, test). I use gates between each stage, and the right composition leads to a much higher quality in the final product.
Also worth mentioning that gate failures are given to the agent that produced the artifact, so it has a chance to fix it. That means that I don't have to review obviously wrong output.
I fully agree with the idea: above a model capability threshold, the power comes from the harness far more than the model. Engineers can get tremendous power from learning how to do CICD and automation. If you view the models and agentic code pipelines as a natural evolution of this, you see the benefit.
I did a quick look at the content, and it seems verbose and AI generated but conceptually OK. I learn by tinkering, not a good fit for me, but if you learn by reading, maybe this is for you.
My view is that human time is more precious than computer time. If something can be automated, then automate it. I don't lint code by hand, I get the linter to do it. Similarly, LLMs expand the list of things that computers can do. That's what you get from the harness, however you learn to do it.
I have the same experience. I've been running sequential agents in my own harness that is a standard SDLC pipeline (plan, design, code, build, test). It has gates between each stage to control quality.
The big benefit of automating this for so long is that I have lots of data. I analyzed it and found that I can change the models out without much of a change in the output quality.
For one-off tasks, where there is no harness and you're just YOLOing with the TUI, yes, big difference. You need a harness.
The pipeline controls the quality far more than the model, empirically.
The "blurring" framing makes Simon's tension sound intrinsic when it is actually structural. Vibe coding and agentic engineering aren't on a continuum. They're distinguished by the process.
Engineering is always about a defined process. We follow it to produce predictable artifacts that meet the specifications. Even though code is somewhat "squishy" in that it is an art just as much as a science, it still has to meet the spec.
This has always been true, even before agents started writing code for us. We've all dealt with spaghetti code because of undisciplined practices. That's exactly why we came up with the standard SDLC process: plan, design, code, test, deploy. Repeat.
The part people seem to forget about when looking at this is the space between the steps: the gates. We review the artifacts produced at each stage. If the reviewer does not approve, the engineer has to fix it until it passes. True for human coders, doubly true for agentic coders.
Agentic engineering still follows the process. Artifacts are now cheap to produce, which means we have to adjust it so we don't overwhelm the humans in the loop. For me, this means augmenting my review step with agentic reviewers to catch the dumb stuff. It only escalates to me when either a) it passes clean or b) there is something that genuinely needs my experience.
The list in the article looks like verification practices. Document intent, develop taste, find the hard stuff, etc. It assumes that when code is cheap the bottleneck shifts to knowing whether what you generated is actually right.
e2e tests can do a lot, but in my experience it's not enough. By the time the test fails you've already burned a generation cycle on an artifact that came from a flawed spec or design. I've gotten more mileage from having checks at stage boundaries (standard SDLC: plan, design, code, test). We all know the earlier you catch the mistake, the cheaper the fix.
The "implement to learn" is the same idea: you need to know enough about both where you want to go AND the path to get there to guide the agents to a proper implementation. You have contact with the world, both the users and the operational considerations that come from running software. Agents do not. We do the same thing with spikes, but now our spikes become far more sophisticated.
Code being cheap doesn't remove verification, it moves it earlier.
https://michael.roth.rocks github.com/mrothroc