I am happy to announce the availability of the first Zstandard compressor written in Go. This package will allow you to get very fast compression to Zstandard - an awesome compression format by Yann Collet.
The current compressor is focused mainly on speed and outperforms gzip/deflate typically by a factor of 3x for similar compression ratios. It is also within a reasonable margin, typically 0.75x the throughput of the reference library called with cgo.
Alongside the previously released decompressor, this will allow you to do very fast generic compression and decompression with unrivaled efficiency in pure Go.
Both a stream and byte blob compressor is made available and both operate without allocations if re-used, so it will also play nicely with the rest of your application.
As a bonus, there is a Snappy -> Zstandard stream converter. This was mainly a development tool, but could be useful for some.
Current status: The main goal for the initial release was to get something that would be a feasible replacement for both deflate/gzip and the cgo wrapper. The package in its current shape allows to replace both for scenarios where fast compression is required. The compressor has been (fuzz) tested, but may still contain subtle bugs - so if you are dealing with critical data, you might want to do your own testing first.
Feedback and experience reports are appreciated. This has taken most of my free time the last several months and has been a great learning experience. I plan to dive into the more efficient (and slower) compression modes next - but feel free to write if you desperately need something else.
Also, it seems like Go is the first language to get a Zstandard implementation that isn’t a wrapper around the reference library. Yay!
The current compressor is focused mainly on speed and outperforms gzip/deflate typically by a factor of 3x for similar compression ratios. It is also within a reasonable margin, typically 0.75x the throughput of the reference library called with cgo.
Alongside the previously released decompressor, this will allow you to do very fast generic compression and decompression with unrivaled efficiency in pure Go.
Both a stream and byte blob compressor is made available and both operate without allocations if re-used, so it will also play nicely with the rest of your application.
As a bonus, there is a Snappy -> Zstandard stream converter. This was mainly a development tool, but could be useful for some.
Current status: The main goal for the initial release was to get something that would be a feasible replacement for both deflate/gzip and the cgo wrapper. The package in its current shape allows to replace both for scenarios where fast compression is required. The compressor has been (fuzz) tested, but may still contain subtle bugs - so if you are dealing with critical data, you might want to do your own testing first.
Feedback and experience reports are appreciated. This has taken most of my free time the last several months and has been a great learning experience. I plan to dive into the more efficient (and slower) compression modes next - but feel free to write if you desperately need something else.
Also, it seems like Go is the first language to get a Zstandard implementation that isn’t a wrapper around the reference library. Yay!
Package & info: https://github.com/klauspost/compress/tree/master/zstd#zstd