From harness? Because people expect a squishy set of things from a harness that is different from what I end up building. I end up with a rigid internal structure that the harness uses in-turn (tests with clear error messages, tools, etc.) and a matched rigid external structure that drives the turn tracking progress and deterministically handles the overall progress. You could call that whole thing a harness but that makes the definition muddy and hard to talk about. So scaffold or skeleton seems more appropriate. The harness constrains the agent. The matched endoskeleton and exoskeleton gives it structure.
I’m definitely on the deterministic code train as well. All of my success for long running tasks has been around wrapping the agentic harness (cc, codex-cli, etc.) in a deterministic workflow with deterministic gates. We need a name for this outer layer. In my mind that is the true harness because it constrains the agents failure mode. I think flow engineering has been proposed. Maybe it’s the agentic exoskeleton?
For me it was earlier this year when I started dusting off some old stalled projects and had an agent work on them. In a few days I:
* Built a clone of the Alpha Zero implementation[1] my team built at oracle
* Ported my hobby NES emulator from javascript to rust[2] (this actually took less than 30 minutes and worked on the first try)
* Implemented all of the lessons from the C++ Grandmasters Challenge (which eventually led to a complete c++ compiler[3])
The thing that flipped the switch was using it to build things that I actually put sweat-equity in to previously. I knew how hard these things were to build, so it landed in a way that other projects had not.
I've been experimenting quite a bit with long-horizion agentic coding[1] and I have also noticed that agents seem to perform worse when forced into certain architectural patterns. I have found that is a bit better when including the constraints along the way instead of adding them after the fact. There seems to be a side-effect I have been calling "calcification", where a pattern starts appearing in the codebase and the agent follows the pattern to the point where it dominates the context and becomes self-reinforcing. This could potentially be a strength or a weakness for existing code bases depending the codebase quality. I will have more insights on this soon as more from-scratch runs conclude that include architectural guidance from the beginning.
I think there is a flow in most organizations from:
llm -> prompt -> result
llm -> prompt + prompt encoded as skill -> result
llm -> prompt + deterministic code encoded as skill -> result
I do think prompting to generate code early can shortcut that path to deterministic code, but we're still essentially embedding deterministic code in a non-deterministic wrapper. There is a missing layer of determinism in many cases that actually make long-horizon tasks successful. We need deterministic code outside the non-deterministic boundary via an agentic loop or framework. This puts us in a place where the non-deterministic decision making is sandwiched in between layers of determinism:
deterministic agentic flows -> non-deterministic decision making -> deterministic tools
This has been a very powerful pattern in my experiments and it gets even stronger when the agents are building their own determinism via tools like auto-researcher.
This is actually a very interesting moment to potentially overcome network effects, because more and more code is going to be written by agents. If a crdt approach is measurably better for merging by agent swarms then there is incentive to make the switch. It also much easier to get an agent to change its workflow than a human. The only tricky part is how much git usage is in the training set so some careful thought would need to be given to create a compatibility layer in the tooling to help agents along.
I share the concern based on the current productivity expectations but I did stumble across something recently which makes me feel a lot better. There was a change in the tax code that coincides with the beginning of the post-pandemic layoffs[1]. This was changed back last month by the BBB which likely means a bunch of new R&D spend for big tech. I think this is why we are seeing intense M&A activity and if we can keep the AI hype under control it will probably lead to new hiring as well.
I am by no means an AI skeptic. It is possible to encode all sorts of things into instructions, but I don’t think the future of programming is every individual constructing and managing artisan prompts. There are surely some new paradigms to be discovered here. A code locking interface seems like an interesting one to explore. I’m sure there are others.
I'm really baffled why the coding interfaces have not implemented a locking feature for some code. It seems like an obvious feature to be able to select a section of your code and tell the agent not to modify it. This could remove a whole class of problems where the agent tries to change tests to match the code or removes key functionality.
One could even imagine going a step further and having a confidence level associated with different parts of the code, that would help the LLM concentrate changes on the areas that you're less sure about.
Heroku is turning the 12 factor manifesto into a community project and modernizing it. I posted my thoughts in a blog[1], and I'd love to hear what other people think!
This is something I’ve been arguing for for a while[1]. I called it a “Framework Knowledge Base”. I think it needs to go a bit further and include specific code examples, especially for newer bits that are not in the training set. Ultimately RAG or even fine tuning might be better than a system prompt.
[1]: https://devops.com/the-rise-of-coding-assistants-superchargi...
I've followed Caleb on YouTube for a while, due to his MtG content. He is PhD in Optical Sciences that makes some very interesting AI art. He has created a new card game with hand curated AI art and some interesting rules aimed at solving a bunch of the problems around existing collectible card games. He wrote an essay[1] about his design goals which is fascinating.
[1] https://calebgannon.com/2023/07/08/the-making-of-algomancy/
Just reran the benchmarks from 10 years ago, python is only 37X slower than C on the benchmark now, and the go version is running faster than the C version. Python still has big productivity wins of course...
Python is extremely slow for some tasks. I was surprised to discover how slow when I ran some benchmarks, despite having used python for many years at the time. It has been improving lately, but here is a blog post I made on the topic quite a few years ago that has some interesting comparisons: https://gist.github.com/vishvananda/7a2f1942d0e9ffff4093
This is definitely an intriguing line of thinking. I too am constantly appalled by the complexity we introduce into the things we build, but I suspect a lot of it has to do with human issues that can't be solved by better technology. That said, I'm very curious to see what tractor ends up looking like.
I'm surprised that neither the documentary nor the review gets into the legacy of OpenStack. I may be biased, but it seems to me that a huge amount of the success of kubernetes is directly attributable to OpenStack.
First, OpenStack paved the way for a bunch of companies to invest real money in working together to compete with AWS. Second, there was massive turnover in ~2013 in open source contributors from OpenStack to Kubernetes. I wouldn't be surprised if a good 50% of the kubernetes community was inherited directly from OpenStack.
In this particular case, some of the runs from iFit instructors are actually quite good, and it cool that it adjusts the speed and incline to match the instruction. Probably not worth the extra $$$$ but it is pretty cool. But now I also want to be able to watch regular videos. I usually walk outdoors for an hour a day to get my 10,000 steps in, and the Chicago winter makes that tough, so I'm thinking an hour of walking on the treadmill while i catch up on my favorite shows might be a good substitute.
I hadn't been following the "Right to Repair" movement too closely until I discovered that NordicTrack recently locked their users out of Privileged Mode[0], which makes it virtually impossible to install 3rd party software like Netflix on my treadmill. I just want to watch a video while I exercise!