In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.
Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.
So even though it had better generated tok/s, because so many more were generated, the clock time was longer.
And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)
The huge spike of "lk-99" in science & frontier tech is amusing...
This is cool concept, would love a positive/negative sentiment computed for each comment that refers to a given word, so you can see trends of "cloudflare (positive)" vs "cloudflare (negative)" where first one counts comments only if sentiment confidence is greater than say 0.6 and the other one counts comments only if sentiment is less than 0.4 (assuming [0,1] sentiment score)
I do use it the same way as you're describing on personal projects at home, in a very crude manner (pasting code snippets in llama server web UI prompt. Next will attempt OpenCode)
At work I use it in similar manner with more mature tools, but the vast majority of token spend comes from a totally different workflow: "pretend the AI is a fleet of junior/intern engineer you're delegating work to", where the agent will on its own do the implementation, commit the changes etc.
It does indeed spend a lot of tokens wandering the codebase, talking to MCPs, loading skills etc.
> About the generation speed: ~100-150 t/s on the RTX 5090 and ~40 t/s on the Mac
Curious if you can share the prefill speed too?
I run locally on a crappy desktop (some AMD iGPU with Vulkan llama.cpp, 32 GB DDR4 RAM) for experimentation. I get 15 tok/s on generation for the qwen & gemma4 MoE models. I get around 150 tok/s prefill speed.
Reason I'm asking about the prefill is looking at my stats at work, I use between 20M to peaks of 300M input tokens daily. Some of those token are cached but in general, I seem to have roughly 500x more input tokens than output. So interested in prefill tok/s stats.
In ~2015 got an Xbox one, as a media center it was an awesome experience:
Kinect voice control to play/pause and other things way before Google home/Amazon echo ecosystem were mature.
Free OTA channels via TV tuner and well designed OneGuide (with ability to pause and rewind).
And of course all the Netflix and other apps, Plex server etc.
But strategically it seems Microsoft decided they wanted to look more like Playstation, focused on gaming (at that time paid Xbox live subscription vs free Playstation)
And as gaws points out, they seem to recently announce to double down on the gaming stuff.
So when they discontinued OneGuide. I picked Roku since they seem to be focused on the media experience primarily... but unsure how I feel about this acquisition news.
Since parent mentions "toxic byproduct": Say you're the company that invented Teflon pans. you made billions. You saved billions in time for all the users of the pans... A true entrepreneurial success.
But, by how many billions did you fuck up the environment, people's health etc with the spread of PFAS everywhere?
- media/tech shortened content: shorter tv shows, short video content, etc.
(Tiktok is the "state of the art" of those 2 trends pushed to the max)
Specifically, we're getting more & more addicted to things that increase the dopamine spikes frequency, making it increasingly difficult to go in deep focus work.
But isn't the prefill speed the bottleneck in some systems* ?
Sure it's order of magnitude faster (10x on Apple Metal?) but there's also order of magnitude more tokens to process, especially for tasks involving summarization of some sort.
But point taken that the parent numbers are probably decode
* Specifically, Mac metal, which is what parent numbers are about
Some might be tempted to brush aside that Server Linux threat model is very different from Desktop Linux (to snarkily reply "we'll it's powering a vast majority of GDP via all of AWS, Azure, etc.").
However comparing apples to apples, what makes you say this isn't ready for government usage, when it's ready for trillion dollar big tech companies' majority of their workforce? (Aside from Microsoft, Apple obviously). Large employers like IBM etc also must be using red hat or some other distro
> I don't know how to force this issue as a European. There are just too many levels of abstraction between me and Brussels.
> EU moves so much faster when it comes to regulations like forcing all of us in Denmark to use timesheets, annoying lids on our bottles, and invasive surveillance laws.
Rediscovering the principle of subsidiarity from first principles...
> I'll need to investigate further but it doesn't seem promising.
That's what I meant by "waiting a few days for updates" in my other comment. Qwen 3.5 release, I remember a lot of complaints about: "tool calling isn't working properly" etc.
That was fixed shortly after: there was some template parsing work in llama.cpp. and unsloth pulled out some models and brought back better one for improving something else I can't quite remember, better done Quantization or something...
Not anymore, OpenAI did sign it:
https://www.microsoft.com/en-us/corporate-responsibility/top...