They could always just extend the cache timeout beyond 5 minutes themselves. They don't do that because it is an expensive resource and there is a trade-off bewteen saving computation and reserving VRAM. Running a tool like this will force them to do one of the following:
Any person doing this will be a much more expensive customer than someone not doing this. Reserving the VRAM is one of the most expensive parts of inference.
1) To be clear, it costs you 10x more for uncached input tokens _for the next call_, which are still 5x cheaper than output tokens.
2) Now imagine Anthropic or OpenAI now charge your per minute of reserved VRAM time. It would be more fair if they did. Would you still want to run a tool like this?
The models themselves support up to 1M. You are just charged more for all context over 400k. The 272k limit is just a client limitation to ensure you can never go over 400k since the maximum output size of the model is 128k.
MTP can give you small batches, but it is still WAY smaller than the batches you can get with prefill, which is limited only by the number of input tokens you have (but has diminishing returns on performance).
But:
1) It still makes no sense to upload the weights to the GPU with MTP as you are still bottlenecked by the weight upload.
For prefill, you are compute bound, and it is trivial to batch multiple input tokens together. When using cpu offload, software like llama.cpp will batch weight uploads with tokens that need those weights and perform work on the GPU. It works very well. With a large batch size and pcie5 you can get prefill speeds close to having all weights on the GPU.
For decode, you are bandwidth bound, and it is difficult to batch multiple output tokens together. There is no benefit to sending your weights to the GPU because even if it internally has insane bandwidth, you are still bottlenecked by system RAM (and adding a pcie5 upload would bottleneck it further). This is the number people usually talk about when they say they are getting a certain tk/s.
Sindorov, who won the candidates, and will be playing for the world championship, has also had accounts banned on chess.com for (presumably) cheating as a teenager. Do you think it would be ok if a large part of the chess world conspired to keep him out of tournaments. Do you think it would be ok if the world made jokes that he cheated using a sex toy? Do you think it would be ok for the largest youtube chess channel (GothamChess) to publish a video titled "Chess cheating device" featuring a thumbnail showing his face next to a butt plug? And all this while he was 19?
What happened to Hans is FAR worse than anything Kramnik did. But maybe you think it's ok because you find him to be an asshole?
My experience of working in the industry is that many of my peers have had great animosity and little sympathy for blue collar workers who lost their livelihood to globalization.
Just to clarify, many third wave shops do serve medium and dark roasts. My point is that third wave coffee shops often call a roast dark when they are actually medium or light. Additionally, I think only light roasts really qualify as third wave.
Most of the coffees you listed look like actual dark roasts.
"Full City Roast" with "lemon zest" in the flavor notes raises some red flags for me. Indeed, from the reviews: "Yum! I don't like dark roasts, and this is not a dark roast which is why I liked it. Lots of chocolate. Very smooth, not bitterness, a hint of fruit as it cools. I really enjoyed drinking this."
"Imunika Black" might also be an exception: the "blackberry" tasting note makes me think it isn't very dark.
I see more with citrus or other fruity descriptions, and I doubt those are dark roasts either.
Here the first sentence of Third-wave coffee's wikipedia page:
> Third-wave coffee is a term primarily in the United States coffee industry emphasizing higher quality, single-origin farms and light roast to bring out distinctive flavors.
I also strongly suspect your definition of "decently dark" is still below a medium roast. If not, I would be interested if you could cite a "decently dark" roast you have tried from a third-wave coffee shop.
1) Start charging for VRAM reservations.
2) Charge _other_ customers more.
3) Eat the cost themselves.