Every company I work with evals is on the agenda, but often takes so long to get started or it got removed again, just spinned up the first skills in the article, if this works well it’s gonna hit home, thnx for the finding and awesome work guys
We had so many successfull stories with the LangWatch MCP server, an MCP integration that brings agent evaluation infrastructure directly into Claude Code, Cursor, and any MCP-compatible environment. That i had to share some of the successes here:
The problem it's solving: teams building AI agents are fully in their coding assistant, but evaluation still requires logging into a separate platform, learning a new UI, and context-switching. The MCP closes that gap. What you can do from within your editor:
Ask your AI assistant to instrument your existing code with LangWatch tracing (it fetches the docs, adds imports, wraps functions with @langwatch.trace())
Generate simulation-based agent tests using Scenario — describe the behavior in plain English and it writes the pytest file
Search and inspect live traces from your project without touching the dashboard
Version and sync prompts to LangWatch's registry
Query cost/latency analytics in natural language
Set up LLM-as-a-judge evaluators that can gate CI/CD
Three real-world cases from the blog post:
A PM at an HR/payroll platform generated 63 agent test scenarios across 11 categories (happy paths, edge cases, wage tax mutations) in a single Claude conversation — no code written by hand.
A Senior AI Engineer migrated an entire Langfuse implementation to LangWatch in one session: Claude read the existing integration, rewired tracing, converted Jinja prompts to versioned YAML, and scaffolded model benchmarking notebooks comparing GPT-4o, Gemini, and Anthropic models.
A Dutch government AI team (LangGraph, multi-agent grant assessment system) used the MCP to build a full testing pyramid: end-to-end scenario tests, model comparison notebooks, and CI-gated quality evaluators before they'd written a single line of eval code themselves.
Curious if others are building MCP-powered eval workflows. The self-instrumenting agent angle (agents setting up their own observability while being built) is something we've been exploring and it gets weird fast.
Clawdbot has been exploding in usage over the past weeks, so we ran a hackday around it at LangWatch and quickly hit a familiar problem: great agent behavior, zero visibility into what was actually happening.
This weekend contributors from LangWatch, Orq, and Red Hat collaborated to add native OpenTelemetry instrumentation to OpenClaw, including support for the OTEL GenAI semantic conventions.
You can now trace agent execution steps, LLM calls, tool usage, and token-level costs in a standard OTEL pipeline instead of custom logging.
We open-sourced Scenario, a tiny framework to simulate and test AI agents by using another AI agent — much like how self-driving cars are tested in controlled environments before real-world deployment.
The idea: if you wouldn’t deploy an autonomous vehicle without simulation, why would you deploy an AI agent without pressure-testing it first?
Scenario lets you:
Describe multi-turn user flows (e.g. “book a flight and cancel it”)
Set success criteria ("user got confirmation + refund info")
Have one agent simulate the user, testing how another agent performs the task
Debug regressions, track failures, iterate on behavior
It’s like writing unit tests, but for conversations.
Why? Traditional evals fall short when testing multi-step flows, memory handling, tool use, or goal alignment. Inspired by simulation in autonomous vehicle testing, where cars are exposed to rare, edge-case situations at scale, Scenario allows similar validation loops for AI agents.
Instead of hoping your support bot or agent workflow handles complexity… you simulate it:
Wow, they were really one of the first companies I noticed in this space, always heard very great feedback on the founder when speaking to prospects. All the best in the upcoming journey.
If there are any users/customers around here, we provide full migration support from any platform (but also Humanloop :)) contact @ (langwatch.ai)
LLM evaluations are tricky. You can measure accuracy, latency, cost, hallucinations, bias... but what really matters for your app? Instead of relying on generic benchmarks, build your own evals --> focused on your use case, and then, bring those evals into real-time monitoring of your LLM app. We open-sourced LangWatch to help with this..
How are you handling LLM evals in production?
Excited to introduce LangWatch, the tool designed for developers working with LLMs. It allows you to experiment with DSPy optimizers in a simple way and monitor the performance of LLM features in your projects.
Key features include:
DSPy optimizer integration: Test various optimization algorithms to enhance your LLM's efficiency.
Monitor and visualize the performance metrics of different LLM features over time.
Open Source: Contributions are welcome to help improve and expand the tool's capabilities.
Feedback, suggestions, and contributions are highly appreciated.
Awesome to see more opensource tools in this space. In transparency we'r building the oss tool https://github.com/langwatch/langwatch which is tool for tracing and monitoring your LLM features and open telemetry is supported as well. Monitoring is key to any team building LLM-features, and still much can be done in this field. What i believe in is the power of optimizing when understanding your performance with these solutions. For ex we're using DSPy optimizers. Curious towards your thoughts int this too! Congrats on the launch and all the best!