My company was using Iroh for a production distributed ML training system & we LOVED it. The team was incredibly responsive even before we hooked up with an enterprise support contract, they're incredibly knowledgeable and the library itself worked amazingly. ++ to this lib. would use again over libp2p anytime.
This title is moderately clickbait-y and comes with a subtle implication that Rust might be getting removed from the kernel. IMO it should be changed to "Rust in the kernel is no longer experimental"
big ++ for iroh - using it for a project at work, it's evolving really nicely, has reliable holepunching, and the team is super responsive and happy to dig thru gigs of logs to find a bug :)
It's unlikely you can. They're generally only willing to set up corporate deals to sell data in massive bulk. You could buy a domain name, set up a decently real looking website with a corpo looking email, then go to any broker like https://www.acxiom.com/customer-data/ (not affiliated, first one i found on google) and do the whole corporate dance of signing a contract to get what you want.
you could absolutely use DisTrO for federated learning. The DeMo optimizer on its own doesn't solve the adverserial aspects of training on local-only data, nor does it solve tensor parallelism across devices, so you're still limited to only what fits on your local GPUs, but it does enable distributed data parallelism over the internet at a bandwidth orders of magnitude lower than before.
> Are you planning to eventually do a SETI@home style thing where anyone can join?
That's one of the goals of the stuff we're working on right now, I personally hope we can make it work!
> What was Durk Kingma's involvement in DisTrO?
As a co-author on the paper, we brought him in to bounce some ideas off of & have him validate DeMo's design and implementation to ensure we weren't hallucinating these results.
We're excited about the potential and want to find other folks also excited about it that are interested in working for/with us to build things on the foundations of DisTrO!
Plus also it's so cool and mind boggling to us that we wanted to share the hype a little bit, it was hard not being able to tell anyone we were working on it
We're pretty confident it's not a fluke, and paper + code are the next step, within a couple months.
It's not "synchronize every step", but it's "do something every step".
We double and triple and quadruple checked our results, to make sure that we are in fact getting results like this while only doing our thing every step, and it really keeps holding up.
Don't trust our word for it, though, you'll see when the paper comes out :)