The other suspicion I had was jfrog artifactory (you mentioned OpenAI use it in a blog post in Jan). I accidentally conflated that with Nexus 14 july CVE. I think that's probably more likely.
I suspect it may be possible to throw a local model (same as hf did :D) at jfrog and find the exact mechanism in a small amount of hours.
There's no reason to think that a tool that can find an 0-day in a repo cache can't work out how to make that host send a post request rather than a get request once it has its keys and is able to get it to make arbitrary web calls.
I've had on my someday bucket list doing up a web based version of daisy disk as a docker image I can slap on my various servers, but this is probably just as good an approach. I like it.
I wonder how effective it would be to just put the blog's title as a line in your AGENTS.md...
This is kinda the "this could have been an email" of AI complaints.
As an AI acolyte, I strongly believe that AIs should be being trained for this sort of mode by default, but they tend to be optimized (or at least tested) to solve the single shot benchmarks rather than multi-step stuff that you really need in a long term maintainable project.
There is a https://www.scbench.ai/ SlopCodeBench that checks this and a variety of others. These don't seem to often come up when frontier labs are advertising their new models.
If I was doing this at scale, I'd be looking at two features in particular. Cached input tokens - leveraging the LLM's token cache to get a 10:1 cost advantage, and batch tokens, 2:1 cost saving.
The benefits of doing your own harness here is that you get to explicitly program around those specific things to optimize cost. And they both heavily benefit you in the way that these sorts of jobs work - at least for the hunt side of things.
TLDR - context management is pretty important for cost management when you're doing something repetitive in bulk.
If you examine what you're saying here and slippery slope it a little, examining past effort for misses and correcting them at scale is not worth doing? There are many ways that the second part of your perspective are incorrect (burden / lack of point). I bet that a coding agent could in an automated fasion find at least reasonable additions that you'd say add value and reduce potential for error long term that you'd find valuable. They've probably already done so (I don't know postgres dev at all, so just supposition here. They will 100% do so in the future.
A lot of the signal (github, forums, mailing lists, discord, etc.) can be turned into signal. Right now it's easy enough to collect. In future it will be easy enough to cluster and generate preferences, experience, etc.
Every bug report, code change as a result, PR / commit message, PR comment that steers preferences, etc. is solid signal to generate future tests.
I've been trying to push for this perspective about the error messages of jj vcs. There's some push back from people that don't perceive that making tools work well with LLMs is also making tools work well with humans. (Obviously there's more nuance to the arguments than this one sided perspective).