This was written in golang. Specifically channels and io.Pipe objects made it pretty straight forward to "glue" different phases of download and extraction together into the final solution.
Oh nice! Yeah this is basically the same design/intuition that this project also started out with (I also just wanted some way to combine the benefits of aria2c with piping). Wish I found this earlier lol.
I only later decided to look into also parallelizing the tar extraction as well. This originally started with looking into making a change to the GNU tar source code itself. However upon cloning I realized there were a lot of design choices in tar that made the assumption of single threaded execution (global state, etc). At this point I decided it would just be easier to re-implement the tar extraction myself using golang's "archive/tar" package.
I did look into performing stuff like md5 inline while streaming the download, however it tended to slow down the extraction significantly. Other less rigorous checks like crc32 are still possible inline though with minimal performance reduction.
It should also be possible to split off and write the raw download to disk in the background with something like an io.TeeReader object [0]. You could then checksum it after the fact like you normally would.
1) Are there any ways to do this in a performant manner? Afaik, since the tarball is still sequential with no way to jump around, interacting with this filesystem would be fairly slow right?
2) Agreed, still need to look more into this, although it's more involved and entails changing more of our pipeline for packaging and distributing these images.
3) I need to look into how this would handle files that haven't been downloaded yet. From what I know about httpfs filesystems, I'm not sure how much would need to be done to let this block until file needed is downloaded vs the normal behavior of calling out to get the file being requested.
5) Could you go into more detail here? Not sure I understand how file boundaries, etc can help with latency/bandwidth.
Had not considered that. Reading about squashfs I see that it's readonly. Do you know if that's fine when mounting the directory for an LXC container? Also I wonder how much performance impact (if any) there might be on spark jobs due to the decompression at runtime.
To be completely honest this all started as a quick hackathon project just to speed up downloading before piping to GNU tar. Only later did I consider also re-implementing the tar extraction in a parallel manner. If considering changing the overall packaging method from tar to something else there's a lot more ways to consider going about this (including squashfs).
I'd say one of the nice things about this is that tar is fairly ubiquitous, not just in container images. For example, we also have tarball build artifacts in Jenkins jobs that could benefit from this tool.
Author here, it's true that GNU tar auto-detects compression when specifying a local tarball to extract. However when piping a file stream to tar to extract it actually doesn't auto-detect and fails if you don't explicitly specify a compression type. Should have made that more clear in the post.