I understand this is important mainly for Nvidia GPUs.
Is there any benefit at all for this (vs the existing VA-API) on Intel and AMD graphics?
VA-API seems to work very well on both of these platforms, as far as I've tested.
Phones are already running models locally which can be used in the field for specific use cases. Maybe not for frontier coding just yet.
Also you don't need to be connected to the network to use a local AI in many instances. If all mobile apps were done with a local-first approach, then you could use a local AI to query your emails, lookup already visited pages, summarise recently received documents, and lots more. Lots of apps could use an inbox/outbox approach for receiving and sending updates instead of relying on the network at all times. And this pattern could be greatly leveraged by local agents.
This piece is a bit all over the place. I immediately toggled off the `AI enhancements` and read the draft instead. The internet is already full of AI slop, I find human text a lot more valuable, even if unpolished.
LLMs will not be centralised or restrained to any 'clergy', the rabbit is already out of the hat, and open-weights models exist and are widely used. Probably not as good as the latest Sol and Fable but 95% there.
Codex and Claude Code without a doubt have very good models behind them. But they also have really good harnesses built around them. An LLM is only a brain stuck in a cranium in the dark. It can generate endless code/prose, but it can't walk or see on its own, it needs additional tools. If you read any of the local LLM subreddits you will notice people mentioning again and again that the harness/tool-use/template-tweaking makes all the difference on how a model behaves/on how smart it is perceived.
Some folks are already using Qwen models for their daily work. Maybe it can't work in a hands-off/one-shot fashion like the frontier models, but they can help tremendously if you already have some domain knowledge.
People are excited about local LLMs and it's not going away any time soon.
The Strix Halo is a great dev machine and a mediocre AI machine. You can run Qwen 3.6 27B at a decent speed, or larger MoE models, and that's about it. For some that's more than enough though, myself included.
Glinet are doing a great job with their routers. I have the Beryl AX which is fully openwrt compatible. The new Beryl 7 is also fully compatible now. Mediatek chips might not be as high performance as Qualcomm but they make up in openness.
Edit:
They just announced Flint 4 with a Mediatek chip:
And this is exactly my point, the OEMs have more lobbying power and leverage. Anthropic might be valuated at whatever amount, but they're a new player and their only product is a piece of software - which others like Google, OpenAI, etc also have (not identical but similar enough).
Laws restricting the use of local AI/LLMs are not going to happen, no matter how much Anthropic might want it. All the major OEMs are now counting on local LLMs to take off. Just look at the OEM support for the upcoming Nvidia RTX Spark platform: Asus, Dell, HP, Lenovo, Microsoft, MSI. All the big names in the industry will have, by the end of this year, Nvidia-powered machines made specifically for local LLM use.
If you are from Europe, even if you're not living in the UK, the en-GB locale will feel a lot more familiar to you than the en-US one.
It uses the dd-mm-yyyy date format like the rest of Europe, the start of the week is on Monday (vs Sunday in the US), the default paper size is A4 (vs US letter), measurement defaults are metric (indeed UK roads use imperial, but the default is otherwise metric), the time format uses 24hrs (vs AM/PM in the US).
There's also `systemctl soft-reboot` which initiates a userspace-only reboot, which quickly restarts the system without going through the full hardware and kernel initialization process.
Thanks for the detailed response, I really appreciate it.
What I had in mind was an AMD Strix Halo machine, but it seems to have none of the advantages you mentioned. It's neither high bandwidth, nor does it have CUDA support, nor does it have support from the big OEMs. All the boards are from relatively obscure Chinese vendors.
It seems like all the major OEMs have rallied behind Nvidia, if you look at the upcoming RTX Spark laptops.
> The biggest thing to watch out for is not just RAM/VRAM but memory bandwidth. You can try to "future proof" yourself with lots of RAM, but if it's 400 GB/S you're still constrained to smaller models.
I'm thinking of getting a SoC machine with 128GB RAM but the bandwidth is limited to 256 GBps. Would you even consider such a machine a decent investment, or should I wait for the newer gen of chips? Thanks!