GNU Parallel Ginsburg Is Released

From LinuxReviews
Jump to navigationJump to search
Gnu-head.jpg

The latest version of the most comprehensive tool for running terminal commands in parallel has new CPU detection code for GNU/Linux, bug fixes and a few changes to the very long and detailed manual page.

written by 윤채경 (Yoon Chae-kyung). published 2020-09-23last edited 2020-09-23

Supertuxkart-git-gnu.jpg
The current GNU Kart from SuperTuxKart git. GNU Parallel is a GNU project.

GNU Parallel is a hidden gem useful for doing all kinds of jobs in terminals where you would normally do something like:

for f in *.png; do 
  convert $f ${f%.*}.webp
done

convert is a single-threaded program, so if you have many cores and you want to use them all you can do:

parallel -j$(nproc)  convert {} {.}.webp ::: *.png

or, simply:

parallel convert {} {.}.webp ::: *.png

because GNU Parallel can detect how many CPU cores the machine has and -j$(nproc) is the default. This CPU detection has been improved in the latest GNU Parallel Ginsburg release.

GNU Parallel replaces {} with the input file and replaces {.} with input file names without the file extension. ::: is used to separate between the command to be executed and the files (or other arguments) each process should work on.

You can also pipe a list of files to it:

find recordings -type f -name '*.wav' | parallel ffmpeg -i {} -vn -sn -c:a flac -compression_level 6 {.}.flac

There are plenty of other examples in the very long and extremely detailed manual page. That manual page has seen some minor updates since the last release in May. A close-up inspection of the diff between the old and the new manual page reveal that a few examples have been altered and that's about it.

There is a paper-back book by GNU Parallel author Ole Tange from 2018 for sale at sa store called "Lulu" in case you find that the long detailed manual page isn't enough for you.

The source for the latest GNU Parallel version can be acquired from gnu.mirrors.hoobly.com/parallel/. It's a very small and quick compile with near-zero dependencies.

GNU Parallel really is quite handy if you need to process large amounts of files using some single-threaded program on a regular basis. It gives a bit more options than xargs -P (which can also be used to run jobs in parallel).

5.00
(2 votes)

Add your comment
LinuxReviews welcomes all comments. If you do not want to be anonymous, register or log in. It is free.