Yeah having traits for this in the stdlib would be nice.
You might find Lunchbox [1] interesting. I needed an async virtual filesystem interface for a project a few years ago (and didn't find an existing library that fit my needs) so I built one:
> Lunchbox provides a common interface that can be used to interact with any filesystem (e.g. a local FS, in-memory FS, zip filesystem, etc). This interface closely matches `tokio::fs::` ...
It includes a few traits (`ReadableFileSystem`, `WritableFileSystem`) along with an implementation for local filesystems. I also used those traits to build libraries that enable things like read-only filesystems backed by zip files [2] and remote filesystems over a transport (e.g. TCP, UDS, etc) [3].
That's a good question! There's an FAQ entry on the homepage that touches on this, but let me know if I can improve it:
> ONNX converts models while Carton wraps them. Carton uses the underlying framework (e.g. PyTorch) to actually execute a model under the hood. This is important because it makes it easy to use custom ops, TensorRT, etc without changes. For some sophisticated models, "conversion" steps (e.g. to ONNX) can be problematic and require additional validation. By removing these conversion steps, Carton enables faster experimentation, deployment, and iteration.
> With that said, we plan to support ONNX models within Carton. This lets you use ONNX if you choose and it enables some interesting use cases (like running models in-browser with WASM).
More broadly, Carton can compose with other interesting technologies in ways ONNX isn't able to because ONNX is an inference engine while Carton is an abstraction layer.
In addition to the benefits mentioned in the sibling comment, zip files let you seek to and access individual files in the archive without extracting all files (vs tar files for example).
This lets us do things like fetch model metadata [1] for a large remote model, by only fetching a few tiny byte ranges instead of the whole model archive.
It also means you can include sample data (images, etc) with your model and they're only fetched when necessary (for example with stable diffusion: https://carton.pub/stabilityai/sdxl)
It uses the NVIDIA drivers on your system, but it should be possible to make the rest of CUDA somewhat portable. I have a few thoughts on how to do this, but haven't gotten around to it yet.
The current GPU enabled torch runners use a version of libtorch that's statically linked against the CUDA runtime libraries. So in theory, they just depend on your GPU drivers and not your CUDA installation. I haven't yet tested on a machine that has just the GPU drivers installed (i.e without CUDA), but if it doesn't already work, it should be very possible to make it work.
For exmaple, if your model contains arbitrary Python code, you'd pack it using [1] and then you could load it from another language using [2]. In this case, Carton transparently spins up an isolated Python interpreter under the hood to run your model (even if the rest of your application is in another language).
You can take it one step further if you're using certain DL frameworks. For example, you can create a TorchScript model in Python [3] and then use it from any programming language Carton supports without requiring python at runtime (i.e. your model runs completely in native code).
I'm working on the second part of my Nerf Dart Missile Defense system project [0].
Building a robot that can track nerf darts and shoot them out of the air has a lot of interesting technical challenges so it's a fun project :) I also get to learn a lot about the process of making videos.
The second part was almost ready a few months ago, but then I had to redo a lot of stuff and I lost steam for a bit.
Hopefully I'll have a second (more well-put-together) video out soon!
First off, thank you for taking the time to write this. I really appreciate the feedback.
I agree with some of your points and in fact I was originally going to post the video with “[part 1]” in the title, but I decided to leave it out and rename it when I post part 2. I figured this was fine since I say “this is the first part of a series” in the first 30 or 40 seconds of the video (and also included it in the HN post description).
I do agree with the idea of having a focused cohesive theme to a video, but I didn’t really find a natural spot to cut it that didn’t make the video boring or not provide the motivation/context of the end goal (eg “firing an electronic airsoft gun from a computer” isn’t conveying what I want to convey).
The goal of this series is to show the process in detail instead of just a high level overview. I’m hoping to post a high level overview video at the end that’s more appealing to people who don’t necessarily want to dig through all the details.
I think future videos will be a little more scoped because they don’t need to include an in depth project overview. Kinda a focused story within the context of a larger backdrop.
I do agree with the point about audience and I think that’s important in general. However, I think there are several things that seem simple and commonplace, but may actually be important to talk about briefly to make the content more accessible for people who don’t have context on a particular area I’m diving into.
Thanks again for the feedback and I’m glad you enjoyed the video!
Good luck starting a channel. I definitely learned a lot during the process of making this video. One thing I found helpful was to write out what I wanted to communicate and iterate on that before filming anything. Maybe that’ll help you get to explanations you’re happy with.
I’m building a robot that can track nerf darts and shoot them out of the air.
I’ve worked on other projects in robotics and object tracking (perception for self driving cars), but this has a whole different set of challenges.
Nerf rival rounds travel at over 100 feet per second and shooting them out of the air requires aiming systems that are precise to less than half the width of a human hair and timing precision of 600 times faster than the blink of an eye.
This is the first part of the series of me building the project (and my first video!) so I’d love to hear what you think!
Adding backends for TensorRT, ONNX, JAX, etc are on our TODO list (and we'd love to see PRs to add support for these and others)!
We actually do use TensorRT with several of our models, but our approach is generally to do all TRT related processing before the Neuropod export step. For example, we might do something like
TF model -> TF-TRT optimization -> Neuropod export
or
PyTorch model
-> (convert subset of model to a torchscript engine)
-> PyTorch model + custom op to run TRT engine
-> TorchScript model + custom op to run TRT engine
-> Neuropod export
Since Neuropod wraps the underlying model (including custom ops), this approach works well for us.
This is a good question. I want to write a more detailed post about this in the future, but here are a few points for now:
- Neuropod is an abstraction layer so it can do useful things on top of just running models locally. For example, we can transparently proxy model execution to remote machines. This can be super useful for running large scale jobs with compute intensive models. Including GPUs in all our cluster machines doesn’t make sense from a resource efficiency perspective so instead, if we proxy model execution to a smaller cluster of GPU-enabled servers, we can get higher GPU utilization while using fewer GPUs. The "Model serving" section of the blog post ([1]) goes into more detail on this. We can also do interesting things with model isolation (see the "Out-of-process execution" section of the post).
- ONNX converts models while Neuropod wraps them. We use TensorFlow, TorchScript, etc. under the hood to run a model. This is important because we have several models that use custom ops, TensorRT, etc. We can use the same custom ops that we use at training time during inference. One of the goals of Neuropod is to make experimentation, deployment, and iteration easier so not having to do additional "conversion" work is useful.
- When we started building Neuropod, ONNX could only do trace-based conversions of PyTorch models. We've generally had lots of trouble with correctness of trace-based conversions for non-trivial models (even with TorchScript). Removing intermediate conversion steps (and their corresponding verification steps) can save a lot of time and make the experimentation process more efficient.
- Being able to define a "problem" interface was important to us (e.g. "this is the interface of a model that does 2d object detection"). This lets us have multiple implementations that we can easily swap out because we concretely defined an interface. This capability is useful for comparing models across frameworks without doing a lot of work. The blog post ([1]) talks about this in more detail.
The blog post ([1]) goes into a lot more detail about our motivations and use cases so it's worth a read.
Neuropod was created at Uber ATG (not AI Labs) and powers hundreds of models across the company (ATG and the core business). It's been used in production for over a year and we're continuing to actively work on it.
The blog post I linked above goes into more detail, but here's a relevant quote about usage within Uber:
> Neuropod has been instrumental in quickly deploying new models at Uber since its internal release in early 2019. Over the last year, we have deployed hundreds of Neuropod models across Uber ATG, Uber AI, and the core Uber business. These include models for demand forecasting, estimated time of arrival (ETA) prediction for rides, menu transcription for Uber Eats, and object detection models for self-driving vehicles.
You might find Lunchbox [1] interesting. I needed an async virtual filesystem interface for a project a few years ago (and didn't find an existing library that fit my needs) so I built one:
> Lunchbox provides a common interface that can be used to interact with any filesystem (e.g. a local FS, in-memory FS, zip filesystem, etc). This interface closely matches `tokio::fs::` ...
It includes a few traits (`ReadableFileSystem`, `WritableFileSystem`) along with an implementation for local filesystems. I also used those traits to build libraries that enable things like read-only filesystems backed by zip files [2] and remote filesystems over a transport (e.g. TCP, UDS, etc) [3].
[1] https://crates.io/crates/lunchbox
[2] https://crates.io/crates/zipfs
[3] https://github.com/VivekPanyam/carton/tree/main/source/anywh...