This is really cool! It's interesting that you can "cheat" by knowing the historical situation in advance. But I guess there's no way around that - if the situations involved were hypothetical, it would be unsatisfying when you guessed "wrong". It'd be neat to see really obscure examples drawn from history.
Hi, I wrote the post! Also not a ML researcher, just an interested engineer, so I'm sure I got some things wrong.
> MoE should be the better tradeoff for the local/single-user scenario since the downside of batching being harder / less efficient doesn't matter.
What I meant was that the single-user scenario is going to get dramatically worse throughput-per-GPU, because they're not able to reap the benefits of multi-user batching (unless they're somehow doing massively parallel inference requests, I suppose).
> Is it really that the matrixes being multiplied are larger? My mental model is that the purpose of batching isn't to get larger input matrices. It's to move the bottleneck from memory bandwidth to compute.
As I understand it, you want larger input matrices in order to move the bottleneck from memory to compute: if you do no batching at all, your multiplications will be smaller (the weights will be the same, of course, but the next-token data you're multiplying with the weights will be 1xdim instead of batch-size x dim), so your GPUs will be under-utilized and your inference will spend more time doing memory operations and less time multiplying.
> The post has no numbers on the time to first token for any of the three providers.
I probably should have hunted down specific numbers, but I think people who've played with DeepSeek and other models will notice that DeepSeek is noticeably more sluggish.
That’s how it does work, but unfortunately denoising the last paragraph requires computing attention scores for every token in that paragraph, which requires checking those tokens against every token in the sequence. So it’s still much less cacheable than the equivalent autoregressive model.
It’s partially because my articles often sit in draft for a while. Then while editing I have another thought that spins off into a separate article, which then gets posted in quick succession.
Yeah, I agree this post would have been better with a concrete example. It's hard to talk about a specific project though, since it comes down to describing in detail facts about a company's internal workings (often embarrassing facts). I couldn't figure out how to anonymize it sufficiently.
Nope! I threw this up in the time I had just to see what it'd look like. I think it'd be a lot of fun to try different formats or different prompts (e.g. eliciting CoT before making a decision).
This repo is sort of a companion piece to https://github.com/sgoedecke/fish-tank, which is a more physics-based 2d simulation. Interestingly, 4o-mini absolutely cleans up in the 2d space but doesn't win all the time in poker.
The specific example here - pushing an organization to use your bespoke Python framework, that you believe to be the best solution for all Python programming tasks - does not inspire confidence.
As for udp connections, WebRTC looks like the only mature solution (but it's not designed for games and is way more complicated than game devs want it to be). Alternatively, I know about https://github.com/networkprotocol/netcode.io but it's very much in the early stages - afaik you even need a Chrome extension to use it on the client. It's not easy to do udp on the web.
> Wrt mining/ads, I'm considering to default to mining if I detect ad-blocker, or you're on Chrome + Desktop PC (not laptops).
Do what you've gotta do to get paid for your work, but if I found out this was happening it would strongly discourage me from remaining on the website.
>Write game bots to test your load non-synthetically.
Can you expand a bit more on that? Cool tips, for sure. Personally I'd appreciate a flip-switch between ads and mining, as long as it defaulted to ads.