zstd

From LinuxReviews
Jump to navigationJump to search
Zstd
Zstd logo86.png
Original author(s)Yann Collet
Developer(s)Yann Collet, Przemysław Skibiński, Nick Terrell (and others)
Initial releaseJanuary 23, 2015; 5 years ago (2015-01-23)
Stable release
1.4.5 / May 22, 2020; 6 months ago (2020-05-22)
Written inC
Operating systemCross-platform
PlatformPortable
TypeData compression
LicenseDual: BSD License + GPLv2
Application-x-archive.png

zstd (or Zstandard) is a compression tool and algorithm offering both really fast compression at low compression ratios and decently fast compression speeds at high compression ratios. zstd decompression is very fast regardless of how high compression ratio was used.

zstd was developed by Yann Collet at Facebook.

Usage

zstd supports 19 basic compression levels plus additional faster and slower levels ranging from -5 to 22. Using compression levels above 19, which are very memory intensive, requires the --ultra parameter (to signal that you really want the Ultra experience). Similarly, super-fast modes enabled by negative values need the --fast parameter.

Compressing archives with zstd at the default compression level 3 using tar can be done with:

tar c --zstd -f myarchive.tar.zst folder1/ folder2/

zstd will by default use one thread. It needs the -T for threads option to utilize more cores/threads. -T0 will make it auto-detect the number of real cores (not threads) and use them all.

Compressing using all cores and the best (apart from Ultra) compression is done with:

tar c -I"zstd -19 -T0" -f myarchive.tar.zst folder1/ folder2/

Decompressing is similar:

tar x --std -f myarchive.tar.zst

You can use multiple cores for decompression too but it is typically pointless.

The Linux kernel supports building zstd support into the kernel (CONFIG_ZSTD_COMPRESS and CONFIG_ZSTD_DECOMPRESS). That support can be used to compress the kernel itself, zswap, file system compression (for btrfs and squashfs) and other things the Linux kernel likes to compress.

Advanced Options

zstd supports dictionary compression. You can train it using sample files and create your own dictionary suitable for the files you tend to compress. The zstd manual page has in-depth details of the options you need to use this feature. You probably don't.

zstd has a --adapt option for "adaptive" compression. This mode makes zstd change the compression level depending on what it believes the machines I/O conditions are. You can optionally use --adapt[=min=#,max=#] to make it stick to a range of set compression levels. Adaptive compression will not work unless you also enable multiple threads with -T# (where # is the number of threads).

Comparison To Other Compression Algorithms

zstd can compress the Linux kernel (version 5.8.1) in 3 seconds OR 1 minute and 18 seconds depending on what compression level you use.

Compressing Linux 5.8.1
algorithm time size binary parameters info
gzip 0m3.132s 177M pigz c -Ipigz -f pigz 2.4
xz 1m33.441s 110M pxz c -I"pxz -9" -f Parallel PXZ 4.999.9beta using its best possible compression.
zstd 0m3.034s 167M zstd c --zstd -f zstd using defaults (1 core, compression level 3)
zstd 1m18.238s 117M zstd c -I"zstd -19 -T0" -f -19 gives the best possible compression, -T0 utilizes all cores.

zstd offers compression speeds comparable to both gzip and xz depending on what compression level is used.

Decompression is very fast regardless of what compression level was used.

Decompressing Linux 5.8.1
algorithm time binary command info
gzip2 0m4.232s gzip2 tar xfz
gzip 0m2.729s pigz tar x -Ipigz -f
xz 0m7.419s xz tar xfJ
zstd 0m3.095s zstd tar x --zstd -f Archive compressed at default compression level 3
zstd 0m2.556s zstd tar x --zstd -f Archive compressed at compression level 19

The combination of both fast and slow compression with good speed/compression ratios and very fast decompression regardless of what compression level is used makes zstd a very good choice for a broad range of use-cases.

See Comparison of Compression Algorithms for comparisons to other compression algorithms.

Trivia

Fedora Linux has been using zstd for its packages since October 2019 (starting with Fedora 31) and Arch Linux has used zstd for its packages since January 2020.

Links

Facebook maintains a homepage for zstd at facebook.github.io/zstd/.

The code is publicly developed on Microsoft GitHub at github.com/facebook/zstd.

Add your comment
LinuxReviews welcomes all comments. If you do not want to be anonymous, register or log in. It is free.