Reliable timing across TX/RX was by far the biggest hurdle. Raw USB access and overall throughput really hasn't been as big of an issue as I thought it would be.
Can you help me understand MetalRT a bit more? Based on the name, it sounds like something that's Apple-only (although, Apple basically co-opted the name Metal, which was traditionally more generic). Does or will MetalRT run on more platforms?
What about MetalRT's relationship to llama.cpp, onnx, MLX, transformers, etc? Is MetalRT a replacement for those? Designed to be compatible with a wide variety of model formats? Or are you just providing an abstraction on top of these?
Super interesting. Auth and Payments seem like things you don't want to or can't build yourself. (I'm waiting for the day when I can just use BTC or UDSC for everything.)
Have you thought about supporting additional auth providers? Or providing a way for other auth services to add support for their products?
I do really appreciate you taking the time to drop by and leave a comment. But I'm curious...why do you think building agents is so important vs. building more of the "AI infrastructure" (which is really what LlamaFarm is trying to do) that will enable the devs that are building integrated AI systems (including agents).
Appreciate the pep talk, but let’s not pretend infra and developer experience plays can’t scale. GitLab, HashiCorp, and Vercel all "just built better DX for open source" and somehow ended up billion-dollar companies.
Agents will come and go (and probably run into the same orchestration headaches), but someone still has to build the reliable, open foundation they’ll stand on.
Multiple areas of degradation. Typically, you don't ship a dataset to prod and then never change it. You want the system to continue to learn and improve as new data is available. This can create performance issues as the dataset grows in size. But also, your model's performance in terms of quality can degrade over time if you're not constantly evaluating its responses. This can occur because of new info within RAG, a model swap/upgrade, or changes to prompts. Keeping all of those knives in the air is tricky. We're hoping we can solve a bunch of pain points around this so that reliable AI systems are accessible to anyone.
Yes, our goal is to provide a stable, open source platform on top of the cutting-edge AI tools. We can systematically update dependencies as needed and ensure that outputs meet quality requirements.
We also have plans for eval features in the product so that users can measure the quality of changes over time, whether to their own project configs or actual LlamaFarm updates.
Yes, all that's a bit hand-wavy, I know. :-) But we do recognize the problem and have real ideas on solutions. But execution is everything. ;-)
Oh! Muna looks cool as well! I've just barely glanced at your docs page so far, but I'm definitely going to explore further. One of the biggest issues in the back of our minds is getting models running on a variety of hardware and platforms. Right now, we're just using Ollama with support for Lemonade coming soon. But both of these will likely require some manual setup before deploying LlamaFarm.
I think that enterprises and small businesses alike need stuff like this, regardless of whether they're software companies or some other vertical like healthcare or legal. I worked at IBM for over a decade and it was always preferable to start with an open source framework if it fit your problem space, especially for internal stuff. We shipped products with components built on Elastic, Drupal, Express, etc.
You could make the same argument for Kubernetes. If you have the cash and the team, why not build it yourself? Most don't have the expertise or the time to find/train the people who do.
People want AI that works out of the box on day one. Not day 100.
Right...there are lots of ways you could do that. Most of the ways we've seen enabling that sort of thing tend to be programmatic in nature. That's great for some people, but you have to deal with shifting dependencies, sorting out bugs, making sure everything connects properly, etc. Some people will want that for sure, because you do get control over every little piece.
LlamaFarm provides an abstraction over most (eventually all) of those pieces. Something that should work out of the box wherever you deploy it but with various knobs to customize as needed (we're working on an agent to help you with this as well).
In your example (alarm monitoring), I think right now you'd still need to write the agent, but you could use LlamaFarm to deploy an LLM that relied on increasingly accurate examples in RAG and very easily adjust your system prompt.
Cool idea. Thanks for sharing. I was really annoyed by the way Google nerfed the maps timeline stuff last year. Obviously this project is way more ambitious than that, but just goes to show you how little Google cares about the longevity of your data.
100%. We don't know what's going to happen in the future. Things are evolving so quickly. Hopefully pushing back on centralization now will keep the ecosystem healthier and give developers real options outside the big two/three cloud providers.
I love what they're doing in expanding the program and such, but the one problem that all "startup tiers" seem to have is that they only last one year.
Most startups are not profitable after just one year and paying cloud bills is a drag on growth.
A graduated drop-off would be better maybe 100% off for year one, then 75% year two, 50% year three, 25% year four, and full price year five. All subject to some sort of annual or total cap, of course.
That's a more realistic picture of what it takes to get off the ground.
Auth0/Okta uses jwt.io to promote themselves, so why shouldn't we do the same? ;-)
But this was also more about scratching an itch that we had internally where similar tools didn't support all of the features we wanted them to. So, we wrote our own.
It's definitely more than an afternoon of work, though I can understand how you might think that. Things are often harder and more complex than we realize at first. :-)
One of the biggest gaps with the current implementation is lack of support for JWKs. We wanted to support that. Also, dark mode. It's the little things.
For the record, no tokens are ever sent anywhere. Everything stays local to the browser. If you hit the "GPT" button, it just transmits the payload--nothing else. That part is definitely a fun experiment. It's more "useful" with more complex payloads. But it may not stick around long-term. We'll see.