Have you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.
Have you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes
Xeon Scalable in general seems like a good idea due to 6-channel (relatively) inexpensive RDIMM memory, but I've been reading that NUMA kills inference performance. Anyone got experience with multi-socket systems? IIRC even within the socket these cpus are divided into sub-numa nodes.
Starlite Mk IV is my favourite "netbook" successor. 0.9 kg, 11.6 1080p IPS display and Linux-first, with coreboot. Unfortunately it is a couple years old by now and you can feel that (Intel N5030 and 8GB of RAM). Sadly the company changed the form factor of Mk V to a detachable but if you can live with that it's also an option.
I suppose that depends on how recent is what you consider recent. I'd have hoped with the coreboot upstreaming effort they would keep official releases coming a bit longer.
FWIW I have had a StarLite Mk IV for three years now and haven't run into a single issue with it (except maybe the speakers being quite poor).
Unfortunately the company stopped releasing firmware updates for it soon after they launched Mk V. I don't know if it can be still built from source for the older devices.