I hear you on the insane amount of time vllm takes to launch (atlas is a move in the right direction in that regard).
But mostly I wanted to raise awareness to readers of your article that no, if you want to do inference, paying 15K for a single 96GB card almost certainly makes no sense. Buy 4 GX10s with the same money, and enjoy dramatically better models and user scalability.
Regardless - thanks for putting the effort to share your findings! I keep postponing doing the same... there's tons of things everyone is re-discovering on their own.
IMHO, the author could have done two things better:
- vllm instead of llama.cpp. With NVIDIA HW, there is huge difference in multi-user loads and caching with vllm; when he was complaining about what happens when more than one user uses the model, and about losing caching, I was "well, duh".
- The budget he used for a single card could have instead be put to far, far better use with SPARKs. I have access to a cluster of 2 x GX10 - total cost less than half what he paid, even today - and I am running vllm and Deepseek v4 Flash. The difference compared to any Qwen is tremendous - I've NEVER seen it loop, and in all my experiments so far, it's the most Sonnet-y model I've ever tried (antirez seems to agree, hence his ds4 fork).
Performance: 2K t/s prefill ( very useful for feeding tons of source code into its massive context window ) and around 50-60 tg/s in my coding sessions in the pi.dev harness. With the money the author paid, he could have bought 4 GX10s, and double both numbers ( vllm basically scales almost linearly with tensor parallelism ).
As soon as I met Forth, I felt I had to hack my own (https://github.com/ttsiodras/MiniForth/) - and did so over a single week, two years ago. I targeted the Blue Pill and the original Arduino, but developed via cross-compilers so that I could test my code in the host.
I became so obsessed with the project that I was looking forward to tinkering with it after coming back from work every day; so it was hacked in 5 evenings and a weekend.
It was that much fun, to build a Forth.
This is why I come to HN - brilliant ideas that I should have thought of before, but didn't!
I just changed my server, and uninstalled the - now truly useless - fail2ban. I use SSH keys of course, but without fail2ban my server's logs were constantly flooded with hacking attempts.
No longer - wireguard for the win. Thank you, chlorion!
> I wonder if using RLE encoding would not have been but vastly simpler and more compact...
No, mate - it wouldn't. Don't be confused by the "printf"-dump; the Python script processed it into pairs of (frequency,delay). That is, when you see...
Well, if the bird in question used an LZ77 followed by Huffman, he could compress all that wood/chuck stuff down to almost nothing. So he could chuck a lot :-D
I played the game with them, when they were very young. They loved every minute of it - with their uncle performing impromptu dramatization-translations :-)
I am seriously considering playing it again with them this summer :-D
A USB tester - it just reports the voltage (5V) and the current. As you can see in the video, the current hovers around 10mA, spiking to 20mA when we use the speaker.
If you are OK with external USB drives, I did just that with my Atomic PI [1] (Atom-based, 35$ SBC, 3.5W idle, running Debian on its internal EMMC, with two external USB drivers in ZFS-mirror configuration).
Instructions to reproduce, and benchmarks here: https://forums.developer.nvidia.com/t/deepseek-v4-flash-offi...