zstd
 | |
| Original author(s) | Yann Collet |
|---|---|
| Developer(s) | Yann Collet, Przemysław Skibiński, Nick Terrell (and others) |
| Initial release | January 23, 2015 |
| Stable release | 1.5.2
/ January 20, 2022 |
| Written in | C |
| Operating system | Cross-platform |
| Platform | Portable |
| Type | Data compression |
| License | Dual: BSD License + GPLv2 |
zstd (or Zstandard) is a compression tool and algorithm offering both really fast compression at low compression ratios and decently fast compression speeds at high compression ratios. zstd decompression is very fast regardless of how high compression ratio was used.
zstd was developed by Yann Collet at Facebook.
Usage[edit]
zstd supports 19 basic compression levels plus additional faster and slower levels ranging from -5 to 22. Using compression levels above 19, which are very memory intensive, requires the --ultra parameter (to signal that you really want the Ultra experience). Similarly, super-fast modes enabled by negative values need the --fast parameter.
Compressing archives with zstd at the default compression level 3 using tar can be done with:
tar -c --zstd -f myarchive.tar.zst folder1/ folder2/
zstd will by default use one thread. It needs the -T for threads option to utilize more cores/threads. -T0 will make it auto-detect the number of real cores (not threads) and use them all.
Compressing using all cores and the highest (default is 3) compression is done with:
tar -c -I "zstd -19 -T0" -f myarchive.tar.zst folder1/ folder2/
Decompressing is similar:
tar -x --zstd -f myarchive.tar.zst
You can use multiple cores for decompression too, but it is gain is minimal.
The Linux kernel supports building zstd support into the kernel (CONFIG_ZSTD_COMPRESS and CONFIG_ZSTD_DECOMPRESS). That support can be used to compress the kernel itself, zswap, file system compression (for btrfs and squashfs) and other things the Linux kernel likes to compress.
Advanced Options[edit]
zstd supports dictionary compression. You can train it using sample files and create your own dictionary suitable for the files you tend to compress. The zstd manual page has in-depth details of the options you need to use this feature. You probably don't.
zstd has a --adapt option for "adaptive" compression. This mode makes zstd change the compression level depending on what it believes the machines I/O conditions are. You can optionally use --adapt[=min=#,max=#] to make it stick to a range of set compression levels. Adaptive compression will not work unless you also enable multiple threads with -T# (where # is the number of threads).
Comparison To Other Compression Algorithms[edit]
zstd can compress the Linux kernel (version 5.8.1) in 3 seconds OR 1 minute and 18 seconds depending on what compression level you use.
| Compressing Linux 5.8.1 | |||||
|---|---|---|---|---|---|
| algorithm | time | size | binary | parameters | info |
| gzip | 0m3.132s | 177M | pigz | ||
| pigz 2.4 | |||||
| xz | 1m33.441s | 110M | pxz | -c -I "pxz -9" -f | Parallel PXZ 4.999.9beta using its best possible compression. |
| zstd | 0m3.034s | 167M | zstd | -c --zstd -f | zstd using defaults (1 core, compression level 3)
|
| zstd | 1m18.238s | 117M | zstd | -c -I "zstd -19 -T0" -f | -19 gives the best possible compression, -T0 utilizes all cores.
|
zstd offers compression speeds comparable to both gzip and xz depending on what compression level is used.
Decompression is very fast regardless of what compression level was used.
| Decompressing Linux 5.8.1 | |||||
|---|---|---|---|---|---|
| algorithm | time | binary | command | info | |
| gzip2 | 0m4.232s | gzip2 | tar -xfz | ||
| gzip | 0m2.729s | pigz | tar -x -I pigz -f | ||
| xz | 0m7.419s | xz | tar -xfJ | ||
| zstd | 0m3.095s | zstd | tar -x --zstd -f | Archive compressed at default compression level 3 | |
| zstd | 0m2.556s | zstd | tar -x --zstd -f | Archive compressed at compression level 19 | |
The combination of both fast and slow compression with good speed/compression ratios and very fast decompression regardless of what compression level is used makes zstd a very good choice for a broad range of use-cases.
See Comparison of Compression Algorithms for comparisons to other compression algorithms.
Trivia[edit]
Fedora Linux has been using zstd for its packages since October 2019 (starting with Fedora 31) and Arch Linux has used zstd for its packages since January 2020.
Links[edit]
Facebook maintains a homepage for zstd at facebook.github.io/zstd/.
The code is publicly developed on Microsoft GitHub at github.com/facebook/zstd.
Enable comment auto-refresher