I benchmarked AI cost-saving claims instead of trusting token percentages(github.com)
github.com
I benchmarked AI cost-saving claims instead of trusting token percentages
https://github.com/lemoncrow-lab/lemoncrow
https://github.com/lemoncrow-lab/lemoncrow
While I was exhausting my token limits, I searched for tools that claimed to be saving tokens. but what they don't say is the full agent $ cost saving.
That is why I created LemonCrow: to optimize agent context end to end rather than focusing on a single token category. Result: on average 30% cost saving and 25% faster. No model routing trick. Same tasks, Same quality, Same model, just cheaper and faster.
Agent cost includes more than output tokens: fresh input, cache writes, cache reads, and repeated context across turns all contribute to the bill. Every input/output/tools/context reread on every turn making cost O(n^2). Optimizing one category may not reduce total cost by the same percentage.
I ran matched benchmarks against a clean Claude Code baseline and few other tools.
Caveman — 300 runs per arm
Output tokens: −44.1% Total cost: $26.0058 → $26.0204 Cost difference: +0.06%
The answers were shorter, but the final bill was essentially unchanged.
CodeGraph — 35 runs per arm
Total token volume: −86.5% Total cost: $19.1064 → $15.9871 Cost difference: −16.3%
This was a real saving, but much smaller than the token reduction claims.
LemonCrow on the same tasks
Total cost: $19.1064 → $6.2869 Cost difference: −67.1% Total token volume: approximately −91.7%
Indexing quality is the main driver for cost. if Index can't give LLM in one turn what it is looking for it will have to rephrase and ask again or start using grep instead. I also measured retrieval quality across 7,213 examples from 14 repositories:
LemonCrow: 0.676 cocoindex-code: 0.557 codebase-memory-mcp: 0.502 ripgrep: 0.376 CodeGraph: 0.296
MRR measures retrieval ranking, not final answer quality.
LemonCrow was also tested on SWE-bench and Terminal-Bench. On SWE-bench Verified, it resolved 232/250 runs versus 202/250 for baseline while costing 29.5% less. Terminal bench 2.1 corectness 0.79-0.80.
Lemoncrow able to maintain same or higher quality at lower ~30% lower cost and ~25% faster.
Since I maintain LemonCrow, this is not an independent review. I published the raw data and asked the Caveman and CodeGraph maintainers to review the configurations and flag corrections.
https://github.com/lemoncrow-lab/lemoncrow/blob/main/BENCHMA...