The minimum fix would be to read it once per user turn, rather than once per tool call turn. As it is, if you tell an agent to update the file and it decides to do multiple edits on separate turns, you take a full cache miss per edit. If you're waiting 5-10 minutes on each miss (deep into a window, which is when you'd usually want to update notes) then you notice the difference.
The post is pretty explicit about it happening on every SSE turn, but I realise if I write something that is (to put it mildly) a rant, then I shouldn't expect people to read it charitably.
I also wouldn't mind being prompted before invalidating so I can decide whether to keep my cache, like most editors will prompt you if a file is edited externally.
> some things mention are outdated - particularly the tool call pruning feature. this has been disabled by default for a while specifically because of complaints like this
Thanks, I added a small update section near the top to call this out. I also saw that paths for bash redirects are now checked.
> and you need to hash chunks of tokens in order to look up in these, meaning you need to be able to slice up your tokens by token count.
The hash just has to uniquely identify the contents. I still don't see what stops you from walking the chunk tree by chunks of characters instead of chunks of tokens, then lazily finding the token boundary once you've found the longest common chunk prefix and also (in parallel) tokenized the input.
> It's all above board though, as you, the user, agreed to the terms & conditions for installing said app.
I have an LG TV, about 6 years old now. For a while it was connected to the internet. Every time I turned it on, after a ~20s delay, it would pop up asking me to install an update and I would press RIGHT + OK to select the "No" button because it worked fine and previous updates just made it slower, added ads etc.
Eventually, the remote dropped the button press for RIGHT, and just got the OK. Or maybe I just pressed the buttons too fast -- the interface is laggy and I'd developed muscle memory. This started the software update with no way to cancel from the UI. Once it updated, it made me accept a new EULA just to continue using the TV in the way I already used it (which was mostly as an HDMI display). There was no way to even get to the settings dialog to perform a factory reset without accepting the EULA. The device was held hostage until I said "Accept."
This is the level of consent implied by accepting the EULA on a TV: none. It's bullshit. Computers should never have been put in TVs, and a smart TV should never be connected to the internet.
> I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache
Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. That means tokenisation can proceed in parallel with your cache query, and since these caches are distributed in production systems I imagine the query itself could be slow.
I'm not trying to undermine the utility, and this is obviously excellent work. Being able to tokenise faster on the client also seems useful (precise token counts for context pruning heuristics, instead of `chars / 4`), and on a phone your work translates directly to energy savings. I'm just curious about the cache lookup point.
Having watched Qwen kill its own llama-server instance to free up a port, I think this is a bold presumption and you should test it at your earliest convenience.
Title is editorialised. Here is one editorialised in the opposite direction, for balance: "OpenAI model breached HF, meanwhile OpenAI model safeguards refused to help HF's defense."
Author here. Just wanted to say: thank you for posting this. You've given me something to think about, and I regret writing this post the way that I did.
It felt unreasonable when I was using it at midnight and had to wait 10 minutes to refill the KV cache on my local GPU :-)
A simple solution here would be to evaluate the date once per session, or once each time the `opencode` binary is launched (to avoid old, long-running sessions getting stuck in the past).
> You are confusing CPU-level security with filesystem-level security.
> I'm curious - how do you expect an LLM harness to build and test executables without being able to build and execute executables?
W^X applies to file systems as well as page tables, and this is called out in paragraph 4 of the Wikipedia article you didn't read. Respectfully, I'm leaving this conversation at this point because it doesn't seem productive.
This doesn't require cooperation from the shell. If writable paths are non-executable, and executable paths are non-writable, then executing `cp $(which git) my-new-favourite-binary` doesn't grant you permission to run it.
The part which does require cooperation (from outside of the shell, i.e. from the harness) is the ability to upgrade writable paths to executable, at the cost of writability and whatever other restrictions you place on the binaries generated by your agent.
This kind of W^X invariant is widely deployed in every desktop operating system, and the type of W-X upgrade hook I described is used by every JIT engine.
I mean, ideally enforce W ^ X, and enforce stricter permissions on executables where the agent has flipped the W to X. My broader point is we should be using operating system primitives instead of regex and wishful thinking.
Sure, the two main approaches are revoking execute permission from the executable itself, and write-protecting the things that it would modify (like $PWD/.git). Both of these are achievable with sandbox-exec or landlock_restrict_self(2).
This works for the `bash` example too. Spawning a new shell and running arbitrary commands is not an escalation because the child shell inherits the parent's restrictions.
Yeah, this is one of the weaker parts of the post because I didn't really make my position clear: that sandboxing should be a first-class integral part of every harness, not an afterthought that risks putting something you care about on the same side of the trust boundary as the LLM.
Some of my own frustrations with bad Docker deployments leaked through. I did snip some of them out way back when I posted this, but maybe not enough.
Prefix caching is inherent in how global softmax attention works. Modifying the prefix invalidates the suffix because the suffix's cached KV projections are a function of the hidden state after the model has evaluated the prefix.
There have been some attempts to approximately stitch KV blocks without full re-evaluation (see: CacheBlend on arxiv), but the results aren't promising.
You mentioned HiCache: as far as I can tell this is a prefix tree cache that efficiently shares prefixes across related histories. If you change the prefix, everything after the change still misses the cache and needs to be prefilled again.
Yeah, I don't think that line quite landed. My point was that LLMs are inherently adversarial (read, "relentlessly proactive") and sandboxing should be a first-class integral feature of your harness, not an afterthought. The problem with dev containers is you still tend to end up with something you care about on the same side of the trust boundary as the LLM.