This is neat! It tackles a boring but real problem with agents. When you have too many tools, the model gets confused. This is a good search box for its tools instead of dumping everything into the prompt. Benchmarks is amazing!
Cool! I watched the demo. The concept makes sense. Quick thought: free-text answers might be high friction for adoption. Not sure if you'd consider various verification method, but something like multiple choice questions or "highlight the relevant code chunk" could be lower effort for the developer, and easier to evaluate programmatically too.
When working on an existing project, I always take a close look on what got changed by LLM. Sometimes I explicitly tell it which line to change and nothing else. Like what most people found, LLM sometimes, if not always, creates redundant over-engineering work trying to be useful, while most of time it actually breaks something, e.g. code quality and readability. In new projects, I focus more on plan, spec, test scheme. I review how modules are set up and sometimes functions, but pay less attention on implementation details.
Curious if there's a demo/screenshot on what questions developers should expect and what acceptable answers look like. Checked out the repo but didn't find the showcase.