We (I'm on the xet team at HF) are open sourcing our spec + protocol in the coming months. So, with the spec and protocol open-sourced, anyone can create xet clients and implement the protocol to build a xet backend.
The specific implementation of our xet backend is deeply integrated into HF backend so open-sourcing it directly wouldn't be very helpful. Once we get the spec + protocol released it should be easy to generate a compatible backend.
re: "modern development experience" - more and more technical projects can be seen as "just" software projects with huge data dependencies. For gaming, the data dependencies are binary texture files, sound files, and more. For biotech these are binary formats from equipment and for analysis. And for ML these are images, video, text (and binary formats for text like Parquet).
One of core motivations behind XetHub is to enable teams across industries to benefit from the workflow we've used in software for 15+ years. We've used this workflow for so long it is easy to overlook its benefits.
Software teams have a clear picture of who is working on what, what is in flight, what is in review, and what is remaining. Anyone on the team can easily pick up work in progress from someone else or start a new derivation of work without concern about interference. Teams can be distributed across timezones and yet everyone feels connected to the project and is able to contribute without disruption.
The power of a GitHub-style workflow for team collaboration comes from being able to experiment freely (branches or forks), review easily (pull requests), and observe (passively learn) best practices from the team (issues, code review feedback).
In my using this over the last couple days (I work with xdss and he asked me to look over the code a couple days ago) gitease works best if you find calling 'git add/commit/push' repeatedly to break your flow with git.
The main benefit is you just call 'ge save' and the tool takes care of calling 'git add/commit (uses LLM for summarizing the diff)'. If you call 'ge share' then also pushes the changes to remote.
Using gitease doesn't limit your usage of git in any way, I think of it as a utility to make it a little easier when first starting out on a project - when you just want to commit and keep going - without thinking too deeply about exactly what changed.
Since launching in December we’ve had a few teams use XetHub specifically as a drop-in replacement for Git LFS and their main reasons for switching are: easier to use (no .gitattributes file management), faster performance, and less storage.
Love to have you try XetHub and give us your feedback!
Great to see more people in this space! We are the authors of XetHub (posted in Dec ‘22, ShowHN: https://news.ycombinator.com/item?id=33969908) and also think a git-like workflow is perfect for ML dataset management, except that we actually integrate with git (like LFS). <A quick benchmark suggests we are 2x your published performance!>
XetHub Co-founder here. Yes, we use the same Git extension mechanism as Git LFS (clean/smudge filters) and we store pointer files in the git repository. Unlike Git LFS we do block-level deduplication (Git LFS does file-level deduplication) and this can result in a significant savings in storage and bandwidth.
As an example, a Unity game repo reduced in size by 41% using our block-level deduplication vs Git LFS. Raw repo was 48.9GB, Git LFS was 48.2GB, and with XetHub was 28.7GB.
Why do you think using a Git-based solution is a hack compared to p4? What part of the p4 workflow feels more natural to you?
XetHub Co-founder here. We are still trying to figure out pricing and would love to understand what sort of pricing tier would work for you.
In general, we are thinking about usage-based pricing (which would include bandwidth and storage) - what are your thoughts for that?
Also, where would you be mounting your repos from? We have local caching options that can greatly reduce the overall bandwidth needed to support data center workloads.
We (I'm on the xet team at HF) are open sourcing our spec + protocol in the coming months. So, with the spec and protocol open-sourced, anyone can create xet clients and implement the protocol to build a xet backend.
The specific implementation of our xet backend is deeply integrated into HF backend so open-sourcing it directly wouldn't be very helpful. Once we get the spec + protocol released it should be easy to generate a compatible backend.