I’m Paul, the CEO of Mystic (YC W21). Today, we're launching our Turbo Registry, a new type of Docker registry built in Rust.
It's designed specifically for AI/ML inference deployments, which typically have larger container sizes. Traditional Docker registries suffer from slow download times, which can be a significant bottleneck in AI inference workflows when going from 0->1. Our new registry has high bandwidth parallel downloading when connected with our new containerd adapter. This adapter changes the way images are mounted and downloaded and interfaces with a new V3 API specification we have introduced in our registry. We will be open-sourcing this later this year but will keep it closed-source as we develop it further for the moment.
With Turbo Registry, we achieved:
• 5GB Docker images loading in 10.23 seconds (down from 82.21 seconds)
• 10GB Docker images loading in 14.75 seconds (down from 147 seconds)
• 20GB Docker images loading in 23.72 seconds (down from 270.47 seconds)
You can use the registry today behind our serverless endpoints.
Information about the roadmap and current limitations can be found in the documentation. Check out our documentation here to get started with Turbo Registry and experience the speed boost in your AI projects. We’re looking forward to your feedback and contributions!
Their new 7B model beats the Llama 2 7B in all benchmarks that they provided, and in many cases the 13B variation.
This is a basic demo for you to try it it, seeing the best results with the instruct variation, and JSON extraction seems good!
In the context of building a product it’s normally best to do some of that final 10% as early as possible. Fail fast, see what works fast.
I’ve had so many projects not get to 100% due to things I could have validated in the first 5%
I know people are working on it - but IMO it's not spoken about anywhere near as much as slapping more GPUs on a problem. I'm on about enterprise workload orchestration not just making a model faster
Activating type checking mode in VSCode was a game changer for me when running python, help catch loads of the edge cases for correctly for some production code
By far the biggest issue it utilisation of the GPUs, if people worked on that instead of throwing more power at problems this would be way less of a problem