While I think that local LLMs are the future, i think these setups are insane. You shouldn't be trying to push the SOTA, most people underestimate how much you can get out of small LLMs.
Why ask FABLE 5000 to "summarize this email thread" when a tiny model can do the job?
Sure Codex3000 can oneshot your backlog, but why not use a subsidized subscription to do it for now? We're clearly not at the peak of these model's capabilities yet.
I tried making one, but I couldn't make through the camera position tracking bit, software was super unintuitive. Very interested in gaming applications for this tech, but still waiting for it to be more approachable from a layman's point of view.
Yeah, but no. Proton is the best thing that happened to Linux gaming, but sometimes it's still not good enough to get full Windows-like experience.
HDR still doesn't work properly, there are no codecs (Atmos, DTS:X) for my surround system, latest nvidia driver broke Microsoft Flight Sim and other games on wayland. The list goes on.
And as for Copilot - I'm sure I'll be able to rip it out, just like the other crap MS forcefully pushes on people.
Yeah I guess the card companies will adapt somehow as well (in fact there's a follow-up video on how to discourage this technique [1]), but there's still a huge market for "vintage" cards and packs that will be affected.
They ship a k8s operator that deploys Synapse, will try that later. Supposedly it's free up to 200 users. But it seems like there's 0 documentation past the README.
A reminder of how centralized and dependent the whole industry has become on GH, which is ironic, considering that git itself is designed to be decentralized.
Good opportunity to think about mirroring your repos somewhere else like Gitea or Gitlab.
Because it wasn't the deal when they were consolidating the all online video content into themselves. And then once they've gotten "too big to fail", the greed rolled in.
They bait and switched us.
Youtube killed private video hosting.
Reddit killed small thematic php-bb forums.
If people knew that they'd eventually do that, none of these platforms would be as big as they are today.
Sure it looks different, but I'm still dragging windows on a virtual desktop.
You'd think there would be some innovation. For example I've always dreamed of a scrolling-timeline-like interaction with desktop, where your activities would scroll to the side when you switch to the new one and then you'd be able to "scrub back in time" to see what you were doing before.
Works on all platforms, but runs much better on Linux.
Running this in Docker on my 2080Ti, can barely fit 13B-4bit models into 11G of VRAM, but it works fine, produces around 10-15 tokens/second most of the time. It also has an API, that you can use with something like LangChain.
Supports multiple ways to run the models, purely with CUDA (I think AMD support is coming too) or on CPU with llama.cpp (also possible to offload part of the model to GPU VRAM, but the performance is still nowhere near CUDA).
Don't expect open-source models to perform as well as ChatGPT though, they're still pretty limited in comparison. Good place to get the models is TheBloke's page - https://huggingface.co/TheBloke. Tom converts popular LLM builds into multiple formats that you can use with textgen and he's a pillar of local LLM community.
I'm still learning how to fine-tune/train LoRAs, it's pretty finicky, but promising, I'd like to be able to feed personal data into the model and have it reliably answer questions.
In my opinion, these developments are way more exciting than whatever OpenAI is doing. No way I'm pushing my chatlogs into some corp datacenter, but running locally and storing checkpoints safely would achieve my end-goal of having it "impersonate" myself on the web.
Why ask FABLE 5000 to "summarize this email thread" when a tiny model can do the job?
Sure Codex3000 can oneshot your backlog, but why not use a subsidized subscription to do it for now? We're clearly not at the peak of these model's capabilities yet.