Comparison of Compression Algorithms

From LinuxReviews
Jump to navigationJump to search
Application-x-archive.png

GNU/Linux and *BSD has a wide range of compression algorithms available for file archiving purposes. There's gzip, bzip2, xz, lzip, lzma, lzop and less free tools like rar, zip, arc to choose from. Knowing which one to use can be such confusing. Here's an attempt to give you an idea how the various choices compare.

Introduction

Most file archiving and compression on GNU/Linux and BSD is done with the tar utility. It's name is short for tape archiver which is why every tarcommand you will use ever has to include the f flag to tell it that you are will be working on files not a ancient tape device. Creating a compressed file with tar is typically done by running tar create f and a compression algorithms flag followed by files and/or directories. The standard compression flags are:

short option long option algorithm
z --gzip gzip
j --bzip2 bzip2
J --xz xz
z --compress compress
--lzip lzip
--lzma lzma
--zstd zstd

These are not your only options, there's more. tar accepts -I to invoke any third party compression utility.

long option algorithm
-Iplzip Parallel lzip
-Ipigz Parallel bzip
-Ipxz Parallel XZ (LZMA)
The above arguments will only work if you actually have plzip and pigz installed. Also note that you will have to have c or x before and -f after -I when you use -I. Example: tar c -I"pigz -9" -f archive.tar.gz folder/

So which should you use? It depends on the level of compression you want and speed you desire. You may have to pick just one of the two. Speed will depend widely on what binary you use for the compression algorithm you pick. As you will see below: There is a huge difference between using the standard bzip2 binary most (all?) distributions use by default and parallel pbzip2 which can into multi-core computing.

Compressing The Linux Kernel

Kemonomimi rabbit.svg
Note: These tests were done using a Ryzen 2600 with Samsung SSDs in RAID1. The differences between bzip2 and pbzip2 and xz and pxz will be much smaller on a dual-core. We could test on slower systems if anyone cares, but that seems unlikely given that only 3 people a month read this article and those three people use Windows, macOS and an Android phone respectively.

The following results are what you can expect in terms of relative performance when using tar to compress the Linux kernel with tar c --algo -f linux-5.8.1.tar.algo linux-5.8.1/ (or tar cfX linux-5.8.1.tar.algo linux-5.8.1/ or tar c -I"programname -options" -f linux-5.8.1.tar.algo linux-5.8.1/)

Ruling out cache impact was done by running sync; echo 3 > /proc/sys/vm/drop_caches between runs.

The exact number will vary depending on your CPU, number of cores and SSD/HDD speed but the relative performance differences will be somewhat similar.

algorithm time size binary parameters info
none 0m0.934s 939M just tar cf tar itself is an archiving tool, you do not need to compress the archives
gzip 0m23.502s 177M gzip cfz
gzip 0m3.132s 177M pigz c -Ipigz -f pigz 2.4
bzip2 1m0.798s 134M bzip2 cfj Standard bzip2 will only use one core (at 100%)
bzip2 0m9.091s 135M pbzip2 c -Ipbzip2 -f Parallel bzip2. RAM usage topped out at about 900 MiB RAM.
lz4 0m3.914s 287M lz4 c -I"lz4" -f Really fast but the resulting archive is barely compressed. Worst compression king.
lz4 0m56.506s 207M lz4 c -I"lz4 -12" -f Supports levels -[1-12].

Uses 1 core, does not appear to be any multi-threaded variant(?)

lzip 4m42.017s 116M lzip c --lzip -f v1.21. Standard lzip will only use one core (at 100%). Very slow.
lzip 0m42.542s 118M plzip c -Iplzip -f plzip 1.8 (Parallel lzip), default level -6
lzip 1m39.697s 110M plzip c -I"plzip -9" -f Parallel lzip at best compression -9. plzip process used 5.1 GiB RAM at its maximum.
xz 5m2.952s 114M xz cfJ Standard xz will only use one core (at 100%). Unbearably slow.
xz 0m53.569s 115M pxz c -Ipxz -f Parallel PXZ 4.999.9beta. Process used 1.4 GiB RAM at its maximum.
xz 1m33.441s 110M pxz c -I"pxz -9" -f Parallel PXZ 4.999.9beta using its best possible compression. pxz process topped out at 3.5 GiB resident.
zstd 0m3.034s 167M zstd c --zstd -f zstd uses 1 core if it is no told otherwise with -T# where # is a thread number and 0 means all cores (not all threads)
zstd 1m18.238s 117M zstd c -I"zstd -19 -T0" -f -19 gives the best possible compression, -T0 utilizes all cores.

Notable Takeaways

A few minor points should be apparent from above numbers:

  • All the standard binaries GNU/Linux distributions give you as a default for all the commonly used compression algorithms are extremely slow compared to the parallel implementations that are available but not defaults.
    • This is true for bzip, there is a huge difference between 10 seconds and one minute. And it is specially true for lzip and xz, the difference between one minute and five is significant.
    • The difference between the pigz parallel implementation of gzip and regular gzip may appear to be small since both are very fast. The difference between 3 and 23 seconds is huge in terms of percentage.
  • lzip and xz offer the best compression. They are also the slowest alternatives. This is specially true if you do not use the parallel implementations.
    • Both plzip (5.1 GiB) and pxz (3.5 GiB at -9) use a lot of memory. Expect much worse performance on memory-constrained machines.
  • The difference between bzip2 and pbzip2 is huge. It may not appear that way since bzip is so much faster than xz and lzip but pbzip actually about ten times faster than regular bzip.
  • pbzip2's default compression is apparently it's best at -9. A close-up inspection of the output files reveal that they are identical (130260727b) with and without -9.

pxz at compression level -9 comes out as a clear winner both when considering both compression rate and speed/compression. It does have two draw-backs: a) It is only fast if you have a lot of free RAM and b) pxz is not a drop-in replacement for xz. Simply making xz a symbolic link to pigz won't work, it has to be invoked with -Ipxz or -I"pxz -9" to be used as a tar compressor.

pxz is the best option if decompression speeds are a concern (see below).

plzip is the only pxz challenger with comparable speed and compression. Both xz and lzip use different Lempel-Ziv-Markov chain algorithm implementations which is why they perform somewhat similar. lzip requires more RAM so it may not be an option if RAM constraints are a concern.

pbzip2 wins hands-down when compression speed is a more important consideration than compression. Parallel gzip (pigz) is even faster than pbzip2 but the compression ratio is much worse. Do be aware that the bzip2 decompression speed is the worst, it is slower than xz, lzip and gzip.

zstd, the new and trendy kid in town, can easily replace gzip since it offers slightly better compression at speeds that are marginally faster than gzip with the default options. Compression is not at all great when the defaults are used but it does shine when -19 -T0 is used and it is, when configured to use the best compression it offers and all CPU cores, comparable to lzip and xz.

Decompressing The Linux Kernel

Compression ratio is not the only concern one may want to consider, a well-compressed archive that takes forever to decompress will make end-users unhappy. Thus; it may be worth-while to look at the respective decompression speeds.

Keep in mind that most people will not use any parallel implementation to decompress their archives, it is much more likely that they will use whatever defaults the distributions provide. And those would be.. the single-threaded implementations.

We tested decompressing using a cheat: tar xf<options> linux-5.8.1.tar.<algo> -C /tmp/ with /tmp being a tmpfs (=RAMdrive). The numbers will therefore absolutely not reflect real-world numbers. The reason we tested this way is to illustrate the difference between the pure decompression time without being bothered with disk I/O limitations.

algorithm time binary command info
none 0m1.204s none tar xf raw archive, no compression
gzip2 0m4.232s gzip2 tar xfz
gzip 0m2.729s pigz tar x -Ipigz -f gzip is a clear winner if decompression speed is the only consideration.
bzip2 0m20.181s bzip2 tar xfj
bzip2 0m19.533s pbzip2 tar x -Ipbzip2 -f The difference between bzip2 and pbzip2 when decompressing is barely measurable
lzip 0m10.590s lzip tar x --lzip -f
lz4 0m1.873s lz4 tar x -Ilz4 -f Fastest of them all but not very impressive considering the compression it offers is horrible.
lzip 0m8.982s plzip x -Iplzip -f
xz 0m7.419s xz tar xfJ xz offers the best decompression speeds of all the well-compressed algorithms.
xz 0m7.462s pxz tar x -Ipxz -f
zstd 0m3.095s zstd tar x --zstd -f When compressed with no options (the default compression level 3)
zstd 0m2.556s zstd tar x --zstd -f When compressed with tar c -I"zstd -19 -T0" (compression level 19)
Lovelyz Kei ProTip.jpg
TIP: tar is typically able to figure out what kind of archive you are trying to extract.
tar xf linux-5.8.1.tar.xz and tar xfJ linux-5.8.1.tar.xz will both work. You need to specify what kind of compression algorithm you want to use when you make an archive but you can omit algorithm-specific flags and let tar figure it out when you extract archives.
  • xz is the fastest decompressing well-compressed algorithm. gzip does offer much faster decompression but the compression ratio gzip offers is far worse. bzip2 offers much faster compression than xz but xz decompresses a lot faster than bzip2.
  • zstd is also looking very good when the best compression level 19 and multiple cores are used. Decompression is very fast and it is faster, not slower, when higher compression is used.

zram block drive compression

The Linux kernel allows you to create a compressed block device in RAM using the zram module. It is typically used to create a compressed RAM-backed RAM device but it does not have to be used for that purpose; you can use it like you would use any block device like a HDD or a NVMe drive. The Linux kernel supports several compression algorithms for zram devices:

$ cat /sys/block/zram0/comp_algorithm
lzo lzo-rle lz4 lz4hc 842 [zstd]

Benchmarking how these in-kernel compression algorithms block devices work in a repeatable way is a bit tricky. Here's what happens if you extract Linux 5.9 rc4 to a uncompressed kernel tmpfs:

tar xf linux-5.9-rc4.tar.gz -C /tmp/

and then create and mount a compressed zram file system using the various compression algorithms:

# Make sure you use zramX not 0 if you already
# have a zram device for swap.
#
# Create a new zram drive
cat /sys/class/zram-control/hot_add

# Select compression algorithm
echo lzo > /sys/block/zram0/comp_algorithm

# Make it big enough for the 1.1G kernel source tree
echo 2G > /sys/block/zram0/disksize

# Create a file system
mkfs.ext4  /dev/zram0

# Mount it to /mnt/tmp
mount -v /dev/zram0 /mnt/tmp

We repeated the above steps of each of the available compression algorithms (lzo lzo-rle lz4 lz4hc 842 zstd) and did the same "benchmark":

time cp -r /tmp/linux-5.9-rc4/ /mnt/tmp/
sync;zramctl

We then used zramctl to see the compressed and the total memory use by the zram device.

In case you want to try yourself, do this between each run:

umount /mnt/tmp
echo 0 > /sys/class/zram-control/hot_remove

These are the results:

Storing Linux 5.9 rc4
on a compressed zram block device
Algorithm cp time Data Compressed Total
lzo 4.571s 1.1G 387.8M 409.8M
lzo-rle 4.471s 1.1G 388M 410M
lz4 4.467s 1.1G 403.4M 426.4M
lz4hc 14.584s 1.1G 362.8M 383.2M
842 22.574s 1.1G 538.6M 570.5M
zstd 7.897s 1.1G 285.3M 298.8M
Kemonomimi rabbit.svg
Note: Time, in this case, is mostly irrelevant. There is a practical difference, and the numbers in the above table do vary. However, be aware that kernel write caching was not disabled. The numbers provide an indication, and they are what time returned. They just don't accurately reflect the total time before all data was actually "written" to the zram device.

It would seem that the zstd compression algorithm is vastly superior when it comes to compressing the Linux kernel in memory. It is also notably slower than lzo-rle, not that the times listed above are very accurate, they should merely be taken as an indication.

We are not entirely clear on what compression level the kernel uses for zstd by default. For comparison,

tar c -I"zstd -19 -T0" -f linux-5.9-rc4.tar.zstd linux-5.9-rc4

produces a 117 MiB large linux-5.9-rc4.tar.zstd file while

tar c -I"zstd -3 -T0" -f linux-5.9-rc4.tar.zstd linux-5.9-rc4

produces a 166 MiB file in 1.389 seconds. Going down to level 1 (-1 increases the file size to 186M while the time is reduced to 0m1.064s. That's still one hundred megabyte less than what the Linux kernel version 5.9 rc4 uses to store itself on a zram block device. It is safe to say that the compression you can expect when you use the kernel-provided implementations of various compression algorithms differs from what you get when you create archives using tar.

avatar

Rms

14 months ago
Score 0++
zstd is the new hot thing
avatar

Anonymous user #2

4 months ago
Score 0++

Here are size result in decreasing order. Zstandard is made for speed, but does not perform well with size

871833600 linux-5.1.11.tar 164204542 linux-5.1.11.tar.gz (option -9) 127157641 linux-5.1.11.tar.bz2 (option -9) 112615402 linux-5.1.11.tar.zstd (option -19) 106262492 linux-5.1.11.tar.xz 104914144 linux-5.1.11.tar.lz (option -9) 101275683 linux-5.1.11.tar.lz (options -m 273 -s 512Mi) 84033965 linux-5.1.11.tar.zpaq (option -m4) 76635908 linux-5.1.11.tar.zpaq (option -m5)

72992600 linux-5.1.11.tar.zpaq (option -m511 which uses 8Gio in RAM. Only 64bits computers can do this)
avatar

Anonymous user #1

12 months ago
Score 0++
Why are "J" and "j" used for the tar command? What does "J" stand for?
avatar

Chaekyung

12 months ago
Score 0++

j is a short-hand for --bzip2 and J is a short-hand for --xz

The J does not "stand for" anything and there's no logical reason why j means bzip2 and J means xz.
avatar

Anonymous user #3

3 months ago
Score 0++
pigz is parallel gz, not parallel xz. Typo?
avatar

Chaekyung

3 months ago
Score 0++
Thanks. Fixed that and re-did all the test with current versions (and the current kernel tree..)
avatar

Anonymous user #4

3 months ago
Score 0++
lz4 is missing.
avatar

Oyvinds

3 months ago
Score 0++
Thanks. lz4 is clearly fast but it is the absolute worst when it comes to compression ratio.
avatar

Anonymous user #5

one month ago
Score 0++
zstd has an "--ultra" switch (uses much more ram), then it has compression level up to 22. Would be nice to see how it compares to xz best compression.
avatar

Anonymous user #6

one month ago
Score 0++

Lucky you, Testing possible on dual core i3 and Celeron J1900 too

Using ssd and linux. Contact me telegram/@TERIIC
Add your comment
LinuxReviews welcomes all comments. If you do not want to be anonymous, register or log in. It is free.