It isn't open... but drop me an email and I can send you it. Basically just tracks a list of known lmstudios on the network, queries their models every 15 seconds and routes to the ones who have the requested models loaded in a FIFO queue tracking the number of tokens/model (my servers are uniform... m4 max 128gb studios but could also track the server) and routes to the one that has just finished. I used to have it queue one just as it was expected to finish but was facing timeout issues due to an edgecase.
Exactly. Using lmstudio you can get structured output like with openai... I have everything in a zod schema then wrote an adapter to both openai and lmstudio instance.
I run lmstudio for ease of use on several mac studios that are fronted by a small token aware router that estimates resource usage on the mac studios.
Lots of optimization left there, but the systems are pinned most of the time so not focused on that at the moment as the gpus are the issue not the queuing.
I'm just now moving my main workflows off openai over to local models and I'm starting to find that these smaller models main failure mode is that they will accept edgecases with the goal of being helpful.
Especially in extraction tasks. This appears as inventing data or rationalizing around clear roadblocks.
My biggest hack so far is giving them an out named "edge_case" and telling them it is REALLY helpful if they identify edgecases. Simply renaming "fail_closed" or "dead_end" options to "edge_case" with helpful wording causes qwen models to adhere to their prompting more.
It feels like there are 100s of these small hacks that people have to have discovered... why isn't there a centralized place where people are recording these learnings?
Note: Running out the door, here is a brief summary that hasn't been deeply considered.
======
Example: I casually commented at the gym that I thought Trump would win about a week ago and went into a very long conversation/debate with several people in the gym.
I asked each of them to show me their filter bubbles... once I had explained it to them. 2 Spaniards, 2 Andorran, 1 French guy. The general consensus seemed to be that Trump was a major step back socially and would pollute the environment, not care about climate change, etc.
Their information came from traditional news, Instagram/Tiktok and one from Reddit/HN/X. The only one that was even open to hearing my views during the conversation was the guy who read Reddit/HN/X. Everyone else had their minds made up.
Today at the gym, I had the 2nd part of this conversation with 3 of the guys that were there this morning. The general take away that the guy who read Reddit/HN/X summed up is one of: "Europe has yet to reach "peak tolerance" where it appears the US is already there." This realization was came to due to the issues of immigrants in both Spain/France who don't assimilate which was a hot button issue for them.
I kinda agree with this sentiment. I think that Europe (at least the part I'm in) doesn't seem to have a great immune system for people who abuse the system and generally punishes tall poppies both socially and economically. The US is completely different, tall poppies are celebrated and if you fail or get sick you have no safety net.
For me the difference in world views is best summed up by what people are focusing on. Climate, equality, and tolerance are the key issues I see pushed heavily in Europe... I see this as stemming from the "tall poppy" syndrome that is prevalent in both Spain/France. In the US, people care about other things and are generally focused on things that directly impact them. That is what this election was about. Less focus on perceived injustices or injustices of the past and more focus on making the future better with something different.
Is Europe a great place? For sure... but it has wildly different problems and world views than the US does. I think it is hard to appreciate that until you've bene immersed in both cultures long enough.
In general, I'd just say there is more nuance to everything than our brains can handle. My little mission has been to try and bring back nuance into the conversation. Black and white thinking is lazy. Nuance exists, find it and challenge your filter bubble.
I'm excited to see how this casual gym conversation continues.
As an American living in Europe, it is fun to see the filter bubble Europeans live in with regards to how the US is framed in the media here and even in common coversation. I was talking to my French buddy about it and we were comparing our filter bubbles. Please realize that there is a filter bubble that all of us live in.
Nate - We discussed the issues extensively on discord and you even did a video call with me and my team. The entire core theming structure changed multiple times and worked with Eshan (which you represented as being a great developer and contributor to Tamagui) to solve the issues… just to have them break again on a minor version for you to release takeout. Having donated over $1,000 to the project and spent a ton of time in the discord and messaged with you more than a handful of times I feel like this comment isn’t genuine but is to save face on HN.
We were an early adopter and sponsor of Tamagui. Impressive tech but massively unstable even on minor version bumps. We even had a core team member working on the project and the breakage was every version. Lost more than 6 months of time relying on it. Ultimately we killed the project because of the decision to go with Tamagui. Just an FYI for those considering this. Abstractions on unstable abstractions is a good way to hate your project.
Kinda hijacking the thread but... My hypothesis is that we will look back and see that Streetview imagery is a goldmine for AI and will be a path to being able to answer HARD questions about the real world.
The insane thing is there are only like 7 companies that actually have meaningful datasets.
I spent 1.5 years studying the geospatial space and went so far as buying a Mosaic51 and scanning the entire country of Andorra as a test before looking at buying the camera manufacture.
Ultimately I walked away from buying the company after issues with the family office I was working with... but long story short I believe streetview imagery will be a gold mine in the future.
If anyone is working in the space. Feel free to ping me, happy to chat and even make intros to the space. If you are training an AI, ping me as well. Happy to open my images up to the right person to make something "country scale" (160k images... every 3 meters with RTK labeled gnss data).
* https://nicholasreese.com old blog.
Background in marketing, js, front end dev, and SEO.
Now working on a new side project and advising startups.
Email: nick[at][pickawebsiteabove].com