One of my professors used an example like this to encourage you to pause if you ever find yourself doing something like "maybe if I just flip this < to > the tests will pass". The space of incorrect programs (even ones that pass your tests) is enormous compared to the space of correct programs.
If you like thinking about this and haven't seen this before, check out the busy beaver problem[1].
Then you run the risk of overfitting to whatever seed you picked. I think fixing a seed is probably a good idea most of the time but it’s worth considering
I've got a gitea instance and a systemd timer polling for issues assigned to my bot. The systemd timer clones the repo etc and spawns the agent in a restricted environment where it has a private localhost (enforced by systemd), and then I set HTTP_PROXY to an inner proxy that connects to an outer proxy over a unix socket. The outer proxy enforces an allowlist and injects credentials. The agent doesn't have access to any credentials inside its sandbox.
For the agent I was using `claude -p` with a pro subscription, but they've been treating their paying subscribers like they're on a free trial (they're subsidizing it so heavily it might as well be). So now I'm using an ollama pro subscription and a homebuilt agent with a bash tool and a str_replace tool. It gets on just fine with only those two
> If you wrote a function that takes a PRNG and generates a random object, you already have a function capable of enumerating all objects.
Something often forgotten here: if your PRNG only takes e.g. a 32-bit seed, you can generate at most 2^32 unique objects. Which you might chew through in seconds of fuzzing.
Edit: this is addressed later in the article/in a reference where they talk about using an exhaustive implementation of a PRNG interface. Neat!
I actually started a collection of annoying bugs I’ve seen in the wild. I give the llm the buggy implementation and ask it to write a test that catches it. So far not even a frontier model (Claude Sonnet) can do it, even though they can find and fix the bug itself.
> provided you understand the semantics of your target architecture
Unless you're writing inline assembly or intrinsics or something like that, the semantics of your target architecture are quite irrelevant. If you're reasoning about the target architecture semantics that's a pretty good indication that what you're writing is undefined behavior. Reasoning about performance characteristics of your target architecture is definitely ok though.
If you like thinking about this and haven't seen this before, check out the busy beaver problem[1].
[1]: https://en.wikipedia.org/wiki/Busy_beaver