KV cache is, well, a cache that can fill up and trigger eviction. You require enough space to execute at least 1 fwd pass of 1 request at your context length. KV cache hits reduce TTFT by avoiding prefill. You don’t get to skip decode.
MoE is kinda related in terms of lower usage requirements vs a dense model of same total param size, but I think your mental model is a bit off.
“It depends”: what’s your prior experience, what kind of roles interest you, how big is the gap between what you have + a little ML knowledge/side projects?
I’d argue there’s a big need for people with solid fundamental CS, sysadmin, infra skills who can bridge the gap into ML practitioner/researcher understanding. Applications or inference generally are probably easiest to break into, especially if you already have service knowledge. If you want to work on distributed training or kernel/model optimization, you probably need to prove your chops there.
Neoclouds, startups in the AI space, maybe hw vendors are probably good places to look.
as someone in the space this ticks a lot of boxes: kubernetes-native, strong isolation, python sdk (ideal for ML scenarios). devmapper is a nice ootb approach.
Glancing at the readme, is your business model technical support? Or what's your plan with this?
Anything interesting to share around startup time for large artifacts, scaling, passing through persistent storage (or GPUs) to these sandboxes?
Curious what things like 'Multi-node cluster capabilities for distributed workloads' mean exactly? inter-VM networking?
Oh nice! I just had an excuse to try mojo via max inference, it was pretty impressive. Basically on par with vllm for some small benchmarks, bit of variance in ttft and tpot. Very cool!
Larger memory, weaker comms. You can optimize for this by doing things like increasing batch size/data parallelism vs sharding schemes with more comms.
At scale training won’t be able to avoid comms entirely, while many models can fit in a single MI300 for serving.
can't speak to GCP specifically but usually the issue is they are host-attached and can't be migrated, so need to be wiped on VM termination or migration -- that's when you lose data.
Reboots typically don't otherwise do anything special unless they also trigger a host migration. GCP live migration has some mention of support though
note that stop/terminate via cloud APIs usually releases host capacity for other customers and would trigger data wipe, a guest initiated reboot typically will not.
yeah, the system/application distinction feels somewhat superficial. The “multiple user space” inside a container thing sounds interesting (not sure what that means exactly), but maybe more similar to a Kubernetes pod, except maybe instead of different rootfs there’s another isolation mechanism?
The "first" link (after the home button) on bbchallenge is the header bar link to https://bbchallenge.org/story which cites Aaronson in the first sentence (double first!). I would not describe it like OP for someone trying to find the actual link ;)
"One Collatz Coincidence", the 2nd story on the blog, also mentions Aaronson
Supabase would be a good pick. Vercel and Cloudflare both do SQL DBs now too, curious how they are. Neon seemed interesting, haven’t heard much about them in a while.
None of those integrated of course, but easier than going big cloud or self hosted
Isn’t there an argument we’re simply better at brain statistics and modeling than current AI? Forget architectural limitations. What is the nature of the extrapolation? How do individuals balance their experiences and determine likely outcomes?
MoE is kinda related in terms of lower usage requirements vs a dense model of same total param size, but I think your mental model is a bit off.