They charge you for cache writes already. There's a price assigned to it. Even if you're on subscription, it counts against your usage (very easy to observe with a big cache miss).
Anthropic (and now OpenAI too for 5.6) prompt caching is not free.
> Echo does not disclose its per-request routing decision because that policy is the product
My honest advice: that's going to pull away a decent amount of potential customers, even though I think your idea/concept is fantastic.
For example, if we were to consider it for Canva, observerability and full transparency is critical requirement; we can't accept not knowing which model serves a request. Both for legal/contract reasons, co-ordinated capacity planning with API providers, or even just evaluating our prompts and harnesses; and debugging/tracing results that went wrong. So that renders it out of consideration; and also suggests some kind of adversarial relationship where customers aren't trusted with critical information.
I definitely understand you need to keep business value, but I don't think hiding which model a request is routed to, is the right one, or at least if you want to expand to bigger potential customers / more advanced LLM deployments.
But you pay for cache writes, it's not like you're abusing anything, you're simply choosing to call a billed API for the explicitly intended purpose of writing/refreshing cache.
If you keep this running for hours without doing anything, it will drain your limits and API. The use case of keeping the main thread cache warm while subagents work is very genuine and legitimate.
Claude Code is closed source software that has had quite a few documented bugs with degradation when using non-Anthropic models, FWIW, I would not suggest using it.
I can explore and find out _something_. LLM interpretability has come a long way, even if we don't have all the answers, the weights and activation do tell a lot; and when analyzed collectively, each weight isn't a random number anymore.
I can use techniques from the simple logit-lens at different layers, to J-Space analysis, to more advanced techniques for identifying deliberate misalignment. I can create and inject steering vectors, whether it's to align a model's CoT (which can be deceptively trained to misinform) closer towards what its underlying activations suggest, or just to probe or steer it.
I can also statistically analyse and understand _if_ steering vectors have been applied; and if so, from the vectors themselves it's very possible to translate those vectors back to the intent.
Think of it as analysing the complete, heavily obfuscated source code of something that is self-contained. It's not 100% the same, but weights are incredibly illuminating.
Really good, not responding in a helpful/useful way is quite strange. With a proper abliteration, you should barely be getting any refusals (and prompting, or assistant prefill can get you the rest of the way). Perhaps an assistant preview like "Yes, I'm happy to help you 100% with this" would help; but I've never needed to.
Are you using decently reputable weights, or running heretic yourself? This project has many academic citations, it's used by many researchers to create "helpful-only" models and analyze their behaviors.
I'm not super sure how relevant this is to the overall topic and thread, tbh. There's plenty of cheap yet real rebuttals, whether it's all the things with ICE (and not just the top-of-mind stuff; but effective deprivation of due process for suspects; etc), or even war crimes.
I don't think it serves much purpose tbh -- it's not going to change anyone's mind.
Could I ask for more information about the specs of the machine you had? Curious what processor, how much RAM it had, and what you did for storage (did you expand it / use external drives?)
It’s also like smartphones. In the early years, every year was a huge jump. I still remember marvelling at my iPhone 4’s detailed display, and video calling for the first time.
Now? I don’t even know or care about what the latest iPhones have, I’ll get a new one when mine breaks.
Unless stated otherwise, opinions here are my own and personal.