Comparison of Compression Algorithms
GNU/Linux and *BSD has a wide range of compression algorithms available for file archiving purposes. There's gzip, bzip2, xz, lzip, lzma, lzop and less free tools like rar, zip, arc to choose from. Knowing which one to use can be such confusing. Here's an attempt to give you an idea how the various choices compare.
Most file archiving and compression on GNU/Linux and BSD is done with the
tar utility. It's name is short for tape archiver which is why every
tarcommand you will use ever has to include the
f flag to tell it that you are will be working on files not a ancient tape device. Creating a compressed file with tar is typically done by running tar
f and a compression algorithms flag followed by files and/or directories. The standard compression flags are:
|short option||long option||algorithm|
These are not your only options, there's more.
-I to invoke any third party compression utility.
||Parallel XZ (LZMA)|
|The above arguments will only work if you actually have |
So which should you use? It depends on the level of compression you want and speed you desire. You may have to pick just one of the two. Speed will depend widely on what binary you use for the compression algorithm you pick. As you will see below: There is a huge difference between using the standard bzip2 binary most (all?) distributions use by default and parallel pbzip2 which can into multi-core computing.
Compressing The Linux Kernel
|Note: These tests were done using a Ryzen 2600 with Samsung SSDs in RAID1. The differences between bzip2 and pbzip2 and xz and pxz will be much smaller on a dual-core. We could test on slower systems if anyone cares, but that seems unlikely given that only 3 people a month read this article and those three people use Windows, macOS and an Android phone respectively.|
The following results are what you can expect in terms of relative performance when using tar to compress the Linux kernel with
tar c --algo -f linux-5.8.1.tar.algo linux-5.8.1/ (or
tar cfX linux-5.8.1.tar.algo linux-5.8.1/ or
tar c -I"programname -options" -f linux-5.8.1.tar.algo linux-5.8.1/)
Ruling out cache impact was done by running
sync; echo 3 > /proc/sys/vm/drop_caches between runs.
The exact number will vary depending on your CPU, number of cores and SSD/HDD speed but the relative performance differences will be somewhat similar.
|none||0m0.934s||939M||just tar||cf||tar itself is an archiving tool, you do not need to compress the archives|
|gzip||0m3.132s||177M||pigz||c -Ipigz -f||pigz 2.4|
|bzip2||1m0.798s||134M||bzip2||cfj||Standard bzip2 will only use one core (at 100%)|
|bzip2||0m9.091s||135M||pbzip2||c -Ipbzip2 -f||Parallel bzip2. RAM usage topped out at about 900 MiB RAM.|
|lz4||0m3.914s||287M||lz4||c -I"lz4" -f||Really fast but the resulting archive is barely compressed. Worst compression king.|
|lz4||0m56.506s||207M||lz4||c -I"lz4 -12" -f||Supports levels |
Uses 1 core, does not appear to be any multi-threaded variant(?)
|lzip||4m42.017s||116M||lzip||c --lzip -f||v1.21. Standard lzip will only use one core (at 100%). Very slow.|
|lzip||0m42.542s||118M||plzip||c -Iplzip -f||plzip 1.8 (Parallel lzip), default level -6|
|lzip||1m39.697s||110M||plzip||c -I"plzip -9" -f||Parallel lzip at best compression -9. plzip process used 5.1 GiB RAM at its maximum.|
|xz||5m2.952s||114M||xz||cfJ||Standard xz will only use one core (at 100%). Unbearably slow.|
|xz||0m53.569s||115M||pxz||c -Ipxz -f||Parallel PXZ 4.999.9beta. Process used 1.4 GiB RAM at its maximum.|
|xz||1m33.441s||110M||pxz||c -I"pxz -9" -f||Parallel PXZ 4.999.9beta using its best possible compression. pxz process topped out at 3.5 GiB resident.|
|zstd||0m3.034s||167M||zstd||c --zstd -f||zstd uses 1 core if it is no told otherwise with |
|zstd||1m18.238s||117M||zstd||c -I"zstd -19 -T0" -f|
A few minor points should be apparent from above numbers:
- All the standard binaries GNU/Linux distributions give you as a default for all the commonly used compression algorithms are extremely slow compared to the parallel implementations that are available but not defaults.
- This is true for bzip, there is a huge difference between 10 seconds and one minute. And it is specially true for lzip and xz, the difference between one minute and five is significant.
- The difference between the
pigzparallel implementation of
gzipmay appear to be small since both are very fast. The difference between 3 and 23 seconds is huge in terms of percentage.
- lzip and xz offer the best compression. They are also the slowest alternatives. This is specially true if you do not use the parallel implementations.
- Both plzip (5.1 GiB) and pxz (3.5 GiB at -9) use a lot of memory. Expect much worse performance on memory-constrained machines.
- The difference between bzip2 and pbzip2 is huge. It may not appear that way since bzip is so much faster than xz and lzip but pbzip actually about ten times faster than regular bzip.
- pbzip2's default compression is apparently it's best at -9. A close-up inspection of the output files reveal that they are identical (130260727b) with and without -9.
pxz at compression level -9 comes out as a clear winner both when considering both compression rate and speed/compression. It does have two draw-backs: a) It is only fast if you have a lot of free RAM and b) pxz is not a drop-in replacement for xz. Simply making xz a symbolic link to pigz won't work, it has to be invoked with
-I"pxz -9" to be used as a tar compressor.
pxz is the best option if decompression speeds are a concern (see below).
plzip is the only pxz challenger with comparable speed and compression. Both xz and lzip use different Lempel-Ziv-Markov chain algorithm implementations which is why they perform somewhat similar. lzip requires more RAM so it may not be an option if RAM constraints are a concern.
pbzip2 wins hands-down when compression speed is a more important consideration than compression. Parallel gzip (pigz) is even faster than pbzip2 but the compression ratio is much worse. Do be aware that the bzip2 decompression speed is the worst, it is slower than xz, lzip and gzip.
zstd, the new and trendy kid in town, can easily replace gzip since it offers slightly better compression at speeds that are marginally faster than gzip with the default options. Compression is not at all great when the defaults are used but it does shine when
-19 -T0 is used and it is, when configured to use the best compression it offers and all CPU cores, comparable to lzip and xz.
Decompressing The Linux Kernel
Compression ratio is not the only concern one may want to consider, a well-compressed archive that takes forever to decompress will make end-users unhappy. Thus; it may be worth-while to look at the respective decompression speeds.
Keep in mind that most people will not use any parallel implementation to decompress their archives, it is much more likely that they will use whatever defaults the distributions provide. And those would be.. the single-threaded implementations.
We tested decompressing using a cheat:
tar xf<options> linux-5.8.1.tar.<algo> -C /tmp/ with
/tmp being a
tmpfs (=RAMdrive). The numbers will therefore absolutely not reflect real-world numbers. The reason we tested this way is to illustrate the difference between the pure decompression time without being bothered with disk I/O limitations.
|none||0m1.204s||none||tar xf||raw archive, no compression|
|gzip||0m2.729s||pigz||tar x -Ipigz -f||gzip is a clear winner if decompression speed is the only consideration.|
|bzip2||0m19.533s||pbzip2||tar x -Ipbzip2 -f||The difference between bzip2 and pbzip2 when decompressing is barely measurable|
|lzip||0m10.590s||lzip||tar x --lzip -f|
|lz4||0m1.873s||lz4||tar x -Ilz4 -f||Fastest of them all but not very impressive considering the compression it offers is horrible.|
|lzip||0m8.982s||plzip||x -Iplzip -f|
|xz||0m7.419s||xz||tar xfJ||xz offers the best decompression speeds of all the well-compressed algorithms.|
|xz||0m7.462s||pxz||tar x -Ipxz -f|
|zstd||0m3.095s||zstd||tar x --zstd -f||When compressed with no options (the default compression level 3)|
|zstd||0m2.556s||zstd||tar x --zstd -f||When compressed with |
- xz is the fastest decompressing well-compressed algorithm. gzip does offer much faster decompression but the compression ratio gzip offers is far worse. bzip2 offers much faster compression than xz but xz decompresses a lot faster than bzip2.
- zstd is also looking very good when the best compression level
19and multiple cores are used. Decompression is very fast and it is faster, not slower, when higher compression is used.
zram block drive compression
The Linux kernel allows you to create a compressed block device in RAM using the zram module. It is typically used to create a compressed RAM-backed RAM device but it does not have to be used for that purpose; you can use it like you would use any block device like a HDD or a NVMe drive. The Linux kernel supports several compression algorithms for zram devices:
$ cat /sys/block/zram0/comp_algorithm lzo lzo-rle lz4 lz4hc 842 [zstd]
Benchmarking how these in-kernel compression algorithms block devices work in a repeatable way is a bit tricky. Here's what happens if you extract Linux 5.9 rc4 to a uncompressed kernel tmpfs:
tar xf linux-5.9-rc4.tar.gz -C /tmp/
and then create and mount a compressed zram file system using the various compression algorithms:
# Make sure you use zramX not 0 if you already # have a zram device for swap. # # Create a new zram drive cat /sys/class/zram-control/hot_add # Select compression algorithm echo lzo > /sys/block/zram0/comp_algorithm # Make it big enough for the 1.1G kernel source tree echo 2G > /sys/block/zram0/disksize # Create a file system mkfs.ext4 /dev/zram0 # Mount it to /mnt/tmp mount -v /dev/zram0 /mnt/tmp
We repeated the above steps of each of the available compression algorithms (
lzo lzo-rle lz4 lz4hc 842 zstd) and did the same "benchmark":
time cp -r /tmp/linux-5.9-rc4/ /mnt/tmp/ sync;zramctl
We then used
zramctl to see the compressed and the total memory use by the zram device.
In case you want to try yourself, do this between each run:
umount /mnt/tmp echo 0 > /sys/class/zram-control/hot_remove
These are the results:
|Note: Time, in this case, is mostly irrelevant. There is a practical difference, and the numbers in the above table do vary. However, be aware that kernel write caching was not disabled. The numbers provide an indication, and they are what |
It would seem that the zstd compression algorithm is vastly superior when it comes to compressing the Linux kernel in memory. It is also notably slower than lzo-rle, not that the times listed above are very accurate, they should merely be taken as an indication.
We are not entirely clear on what compression level the kernel uses for zstd by default. For comparison,
tar c -I"zstd -19 -T0" -f linux-5.9-rc4.tar.zstd linux-5.9-rc4
produces a 117 MiB large
linux-5.9-rc4.tar.zstd file while
tar c -I"zstd -3 -T0" -f linux-5.9-rc4.tar.zstd linux-5.9-rc4
produces a 166 MiB file in 1.389 seconds. Going down to level 1 (
-1 increases the file size to 186M while the time is reduced to 0m1.064s. That's still one hundred megabyte less than what the Linux kernel version 5.9 rc4 uses to store itself on a zram block device. It is safe to say that the compression you can expect when you use the kernel-provided implementations of various compression algorithms differs from what you get when you create archives using tar.