I run 3 servers for 200 EUR, thanks to Hetzner, exactly for this reason (and I’m cheap and I never understood cloud/services like Vercel and Railway as serious alternatives ;-)).
I played a lot with LLMs over the last year and built a multitude of products with it and it’s always just the same bottom line outcome:
- be specific
- keep it small
- be precise when adding context
- don’t expect magic and shift deterministic requirements to deterministic code execution layers
All this is awfully painful to manage with current frameworks and SDKs, somehow a weird mix of over-engineered stuff while missing the actual point of making things traceable and easy changeable once it gets complex (my unpopular personal opinion, sorry). So I have built something out of my own need and started to offer it (quite successfully so far)to family & friends to get a handle on it: Have a look: https://llm-flow-designer.com
While working on a new startup I had exactly the same challenges and issues. As soon as it gets complex, data gets bigger, amount of tools increases it becomes tremendously difficult to control agents, network of agents or anything LLM related. Add then specific domains like legal or finance where most of the time there is just 0 or 1 and nothing in between (metaphorical speaking) it becomes a nightmare in code and side effects.
So I started to actually build something to solve most of my own problems reliably and pushing deterministic outputs with help of LLMs (e.g. imagine finding the right columns/sheets in massive spreadsheets to create tool execution flows and fine tuning finding a range of data sources).
My idea and solution which helped not only me but also quite a few business folks so far to fix and test agents is visualizing flows, connect and extract data visually, test and deploy changes in real time while keeping it very close to static types and predictable output (other than e.g. llama flow).
Just find a Hoster with low traffic egress cost, reverse proxy normal traffic to Cloudflare and reply with 2GB files for the bot, they annoy you/cost you money, make them pay.
would be good to have an indicator if it’s available with your distro by default or what package you’ll need to install it since all tools are only as useful as available they are…
+1 I dropped school relatively early (I was extremely bored and the way of education was certainly somewhere close from the Stone Age times). I did an apprenticeship as software engineer with some (extremely useless) school component. Most of the time in the late 90ties was trial and error, for me, the master and the master of masters. Playing around with Linux and make it ISDN routers with servers for websites built in HTML, Perl, PHP. This was devops (before it got hyped) and real engineering by figuring stuff out with almost no documentation, a lot of crazy creativity and push the boundaries of what’s possible. And it reminds me just a little like today’s world with AI and vibe coding just on a complete different level and with significant more pressure…fun times :-).
I input text and preferably I output JSON but doesn’t matter much as long as it’s somewhat structured.
Ultimately I’d like to extract information like date ranges, specific indications of tool usages (e.g. I have a bunch of data apis with their own individual data and semantic meaning which need to be picked and then a combination of tools to transform the data)
I switched as well and I actually use the AI assistant since then primarily. It’s awesome to connect search directly with AI, almost always get what I want immediately.
yeah, but it doesn’t need to be that way. It can be simple and makes it easier adoptable, why over engineering and reinventing the wheel of at least 2 decades experience and better practices.
OpenAI takes the backseat and wait until something stable/usable comes out of it which gains traction and takes it over then. Old classic playbook to let others make the mistakes and profit from it…
Couldn’t agree more, played the whole day today trying to get a HTTP MCP server with Claude running.
Absolutely terrible, no clear spec, absolute useless errors and/or just broken behaviour without telling what’s wrong. Reference implementations and frameworks are not working either, so only reverse engineering + trial & error until it runs, yaaay.
Feels like the early 2000 over and over again, trying to make something work.
This story is for me a real painful death I have experienced way too many times, absolutely nuts.
But then you find an open source solution which is in general better and can do everything you want (simply tested already with just a docker compose up) but for deployment you get hit hard by compliance who just checks the SOC2 certifications and wants a in-depth due diligence of the code since everyone in the world can theoretically change it.
Then your manager asks how it can be so good if it's for free and open source.
And of course, last but not least, your overloaded team in general not happy to support just another unpredictable piece of software...
So it's the question to rather burn money and nerves with an awful SaaS offering and their endless and useless sales cycles and terrible and super expensive vendor-lock-ins or burn some money and nerves by utilising and running open source inhouse...
So typically I prefer to chose for the open source option and especially if the SaaS option isn't allowing me easy and fast self-onboarding, meaningful testing periods and a predictable and transparent pricing.
And then, if it get's widely adopted, I allocate some budget to support the authors and/or get some support plan (for more complex open source software) in place even though you most likely never need it...