Gitar runs multiple specialized AI agents on every code change. They review code, fix CI failures, execute custom repository rules as workflows, and respond to developer feedback in-thread. That's easily 50-100 LLM calls per PR, and complex ones can hit 500+. We tried swapping Claude for Kimi K2.5 at 1/5th the price over a weekend.
Three things bit us: finish_reason semantics differ between "compatible" providers, the model retried identical failing tool calls instead of adapting, and provider failover invalidated prompt caches on both sides.
We built our own harness from the ground up to account for this
Secrets come from aws secret manager and never injected into env directly.
Each part of the agentic workflow only gets the secrets it needs injected. Agent can see env var names but not the values (our harness masks them) . We also mask any attempts to output to stdout/files.
This keeps the agent architecture simple with env vars that all agents can operate on as it locally. Prompt injection attempts will only yield masked values
Feature flags are great for safely releasing features fast.
As you add more of them though, they add tech debt and make the code harder to reason about. Developers are rarely motivated to clean them up after rollout.
I was leading such a team at previous employer focused on building integrating various tools to turn this tool soup into something palatable.
The main challenge is the increasing complexity of software and processes that necessitate such tools as the engineering team grows. Usually teams building internal tools are seen as cost centers and don't get the same funding as product teams which eventually leads to teams buying more than build to save on future maintenance.
I feel this advice has some merit when dealing with oss projects, systems/backend or small teams.
It tends to break apart when working with large teams or when working on mobile apps where one can't just rollback a change easily after it's shipped. The descriptions need to be more detailed and capture lot more information.
For example, our team would require all mobile devs to add information about feature flags for each change to turn the feature off if things go wrong. This also places additional review burden to make sure the flag covers all the new code introduced and does not interact in a bad way with other existing feature flags
I feel like the number of reviewers should be predicated more on how well the reviewers understand the relevant part of the code review.
Most code review tooling is too coarse in the sense that it expects a small number of reviewers to know the full context of the change or has too many reviewers that slow down things.
Tools to show relevant parts of the change to specific reviewers and potentially break apart a large change into smaller ones automatically will go a long way especially in large scale codebases or when team sizes are large enough that the full context is not understood by everyone on the team
For many use cases, cold-blooded software is not viable. We need better tools to automate and remove the tedium involved in upgrading dependencies or modernizing codebases to protect against ever evolving threats and adapt to changes in the ecosystem
Huge fan of the Arc browser. The UX is really smooth and intuitive. It also declutters UI and provides some nice hints.
I usually go to the url box to copy the url. one day, it suggested me to use a keyboard shortcut instead based on how often I did it. Little things like this that are built-in to the experience without having to install plugins and make sure they work together is a breath of fresh air.
Have you considered using something like a treesitter grammar? It could solve the editor specific uses cases like highlighting and even linting as it creates asts that are more amenable for a language server implementation
I setup codeql locally to do some simple taint analysis recently. I found the tooling very powerful but quite complex to use. Updating the codeql database after making code changes was quite slow as well which made it harder to do incremental analysis.Can't imagine the complexities involved with quantum computing code analysis.