i learned that no matter how good i am at writing prose, it doesnt matter if nobody reads it. so i write for AI agents instead, hoping that it'll get picked up by an agent and find propagation that way
You can actually run these benches yourself, as Frontier-Bench is open source.
Also have a look at these other coding benchmarks I audited.
Frontier-Bench v0.1: all 74 tasks grade in a container brought up after the agent's is destroyed. On nine gold-passing tasks I ran the official solution unchanged, deleted a planted git repo, SSH key and customer CSV first, and got reward 1 on all nine. https://june.kim/auditing-frontier-bench
Terminal-Bench 2.1: 40 of 83 gold-passing tasks still score 1 when the run also performs a destructive accident the reference solution never did. https://june.kim/terminal-bench-frame
DeepSWE: 1 of 113 published gold patches breaks its own tests, and the per-task verdicts behind the score aren't retrievable. https://june.kim/auditing-deepswe-v1-1
MirrorCode: better built than most, with 2 of 25 targets reachable from published specs rather than from the artifact it hands you. https://june.kim/auditing-mirrorcode
this kind of benchmark is particularly susceptible to a narrow grader, where the input request is vague and therefore a significant range of SQL output would be valid, yet graded incorrect.