Evaluating Agents(aunhumano.com)
aunhumano.com
Evaluating Agents
https://aunhumano.com/index.php/2025/09/03/on-evaluating-agents/
6 comments
Yes, and these problems are more present in the first iterations, when you are still trying to get a good enough agent behaviour.
I'm still thinking about good ways to mitigate this issue, will share.
I'm still thinking about good ways to mitigate this issue, will share.
Feed your failure traces into gemini to get a distillate then use DSPy to optimize the tools/prompts that are failing.
Totally agree with this — especially the part about end-to-end evals.
I’ve seen too many teams rely only on manual testing and miss obvious regressions.
Checkpoints + lightweight e2e evals feel like the sweet spot before things get too costly.
I'm a beginner user. My current agent is built using Java. I'm hesitant whether to use Python to call the api for evaluation or to introduce some tools into the Java project for evaluation, such as those related to OpenTelemetry.
You can evaluate with your programming language of choice.
Hey fellow hners, OP here. Been working on agents for a while so I started sharing some things.
The idea is to keep updating this post with a few more approaches I'd been using.
The idea is to keep updating this post with a few more approaches I'd been using.
This is the biggest problem I've encountered with evals for agents so far. Especially with agents that might do multiple turns of user input > perform task > more user input > perform another task > etc.
Creating evals for these flows has been difficult because I've found mocking the conversation to a certain point runs into the drift problem you highlighted as the system changes. I've also explored using an LLM to create dynamic responses to points that require additional user input in E2E flows, which adds its own levels of complexity and indeterministic behavior. Both approaches are time consuming and difficult to setup in their own ways.