#1 - the tricky part there is in scenarios from a few AI Native teams. There often are a multiple agents rolling out linked changesets to a bunch of documents on behalf of controlling humans. Eg updating compliance policy, and references and change log and current procedures at the same time.
So changesets have to be atomic across multiple documents and semantic (so that agents can resolve the changes). Weak per-document versioning isn’t enough here.
#4. Nice! Same story, but also virtualizing ripgrep, find and tree (plus MD-aware outline mode). With that setup even agents with weaker local models (eg runnable on DGX Spark) can solve complex tasks in the Agentic Commerce domain.
Personally I’ve been trying very hard to migrate away from git+Obsidian project setup according to the OpenAI Harness Engineering. It works wonderfully in Codex Desktop.
The only gotcha - I want to share knowledge bases with the team in a way that is:
(1) versioned (a la git, not Notion)
(2) usable from any chat (a la MCP)
(3) basic access controls for team setup.
(4) works through the interface that optimizes accuracy and token use across agentic architectures and LLMs.
Funnily enough, 4 is the easiest one (I have a platform for agent training and verification where I publish fun challenges for agents in simulated worlds around agentic commerce and personal OSes. With 98M agentic interactions recorded, that is already enough information for tuning)
Working on benchmark arena for AI agents with my wife.
We grab interesting business problems, turn them into fun challenges for hundreds of AI engineers to find the best architecture for. Insights are shared back with the community.
It is a fun learning process with unexpected scaling challenges.
I built a platform to learn how to build personal AI agents and test them with fast feedback. It is free for individuals and small teams.
Platform deterministically generates tasks, creates environments for them, observes AI agents and then scores them (not LLM as a judge).
We just ran a worldwide hackathon (800 engineers across 80 cities). Ended up creating more than 1 million runtimes (each task runs in its own environment) and crashing the platform halfway.
104 tasks from the challenge on building a personal and trustworthy AI agent are open now for everyone.
I liked NixOS pre-LLM era, since it allowed me to manage a couple of servers in a reproducible way. Ability to reboot back to a stable configuration felt like magic.
Nowadays I love it, since I can let Codex manage the servers for me.
“Here is the flake, here is nix module for the server, here is the project source code. Now change all of that so that wildcard certificates work and requests land through systemd socket on a proper go mux endpoint. Don’t come back until you verify it as working”
I’m working on a platform to run a friendly competition in “who builds the best reasoning AI Agent”.
Each participating team (got 300 signups so far) will get a set of text tasks and a set of simulated APIs to solve them.
For instance the task (a typical chatbot task) could say something like: “Schedule 30m knowledge exchange next week between the most experienced Python expert in the company and 3-5 people that are most interested in learning it “
AI agent will have to solve through this by using a set of simulated APIs and playing a bit of calendar Tetris (in this case - Calendar API, Email API, SkillWill API).
Since API instances are simulated and isolated (per team per task), it becomes fairly easy to automatically check correctness of each solution and rank different agents in a global leaderboard.
Code of agents stays external, but participants fill and submit brief questionnaires about their architectures.
By benchmarking different agentic implementations on the same tasks - we get to see patterns in performance, accuracy and costs of various architectures.
Codebase of the platform is written mostly in golang (to support thousands of concurrent simulations). I’m using coding agents (Claude Code and Codex) for exploration and easy coding tasks, but the core has still to be handcrafted.
Quite the opposite. Context caching requires state (K/V cache) close to the VRAM. Streaming requires state. Constrained decoding (known as Structured Outputs) also requires state.
I think there are two different layers that get frequently mixed.
(1) LLMs as models - just the weights and an inference engine. These are just tools like hammers. There is a wide variety of models, starting from transparent and useless IBM Granite models, to open-weights Llama/Qwen to proprietary.
(2) AI products that are built on top of LLMs (agents, RAG, search, reasoning etc). This is how people decide to use LLMs.
How these products display results - with or without citations, with or without attribution - is determined by the product design.
It takes more effort to design a system that properly attributes all bits of information to the sources, but it is doable. As long as product teams are willing to invest that effort.
There is a team of 5 people that are passionate about their indigenous language and want to preserve it from disappearing. They are using AI+Coding tools to:
(1) Process and prepare a ton of various datasets for training custom text-to-speech, speech-to-text models and wake word models (because foundational models don't know this language), along with the pipelines and tooling for the contributors.
(2) design and develop an embedded device (running ESP32-S3) to act as a smart speaker running on the edge
(3) design and develop backend in golang to orchestrate hundreds of these speakers
(4) a whole bunch of Python agents (essentially glorified RAGs over folklore, stories)
(5) a set of websites for teachers to create course content and exercises, making them available to these edge devices
All that, just so that kids in a few hundred kindergartens and schools would be able to practice their own native language, listen to fairy tales, songs or ask questions.
This project was acknowledged by the UN (AI for Good programme). They are now extending their help to more disappearing languages.
None of that was possible before. This sounds like a good progress to me.
It is actually funny that current AI+Coding tools benefit a lot from domain context and other information along the lines of Domain-Driven Design (which was inspired by the pattern language of C. Alexander).
A few teams have started incorporating `CONTEXT.MD` into module descriptions to leverage this.
Agreed. AI is just a tool. Letting in run the show is essentially what the vibe-coding is. It is a fun activity for prototyping, but tends to accumulate problems and tech debt at an astonishing pace.
Code, manually crafted by professionals, will almost always beat AI-driven code in quality. Yet, one has still to find such professionals and wait for them to get the job done.
I think, the right balance is somewhere in between - let tools handle the mundane parts (e.g. mechanically rewriting that legacy Progress ABL/4GL code to Kotlin), while human engineers will have fun with high-level tasks and shaping the direction of the project.
This is why there has to be "write me a detailed implementation plan" step in between. Which files is it going to change, how, what are the gotchas, which tests will be affected or added etc.
It is easier to review one document and point out missing bits, than chase the loose ends.
Once the plan is done and good, it is usually a smooth path to the PR.
It works really nice with the following approach (distilled from experiences reported by multiple companies)
(1) Augment codebase with explanatory texts that describe individual modules, interfaces and interactions (something that is needed for the humans anyway)
(2) Provide Agent.MD that describes the approach/style/process that the AI agent must take. It should also describe how to run all tests.
(3) Break down the task into smaller features. For each feature - ask first to write a detailed implementation plan (because it is easier to review the plan than 1000 lines of changes. spread across a dozen files)
(4) Review the plan and ask to improve it, if needed. When ready - ask to draft an actual pull request
(5) The system will automatically use all available tests/linting/rules before writing the final PR. Verify and provide feedback, if some polish is needed.
(6) Launch multiple instances of "write me an implementation plan" and "Implement this plan" task, to pick the one that looks the best.
This is very similar to git-driven development of large codebases by distributed teams.
Blog: https://abdullin.com LinkedIn: https://linkedin.com/in/abdullin/
BitGN Agent Arena: https://bitgn.com/