GNU Parallel Akihito released

From LinuxReviews
Jump to navigationJump to search
Gnu-head.jpg

The GNU project is proud to announce a new version of the command-line utility Parallel which is now nearly 10 years old. The new version supports grouping jobs and there is quite a few bug fixes.

Never heard of GNU Parallel? It is a tool for running multiple shell jobs (commands) in parallel. Consider this really meaningless example:

Shell command:
parallel echo 'Job for:' ::: *.png

This simple command, when ran in a folder with some png images, will produce something like:

Job for: Guix-installing-software-fs8.png
Job for: guix.png
Job for: KDE_Katie.png
Job for: kdenlive-wjsn.png
Job for: lives-2.10.2-fs8.png

You may or may not wonder what's the point of using parallel when you can do the exact same thing, echo a filename, by running

Shell command:
for f in *.png;do echo 'Job for: '$f;done
Gnu.jpg

and answer is simple: You can run 1000 echo jobs in an instant so it does not matter if it's single-threaded or multi-threaded but it does matter if you want to actually do something with a command like convert[1] and that something is single-threaded. Running a mostly single-threaded program in sequence or parallel can make a big difference if you have a 4 or 6 or 8-core system. There's is a very real and clear advantage in executing 8 or 10 single-threaded jobs instead of just one if you have 8 cores. This is what GNU parallel can do for you.

Shell command:
parallel pngquant ::: *.png

would obviously be faster than using for f in *.png;do on a larger collection of images since pngquant[2] takes up to half a second per image.

For more advanced use-cases the same syntax as xargs[3] applies.

The parallel manual page is rather long. You should want to read it, but if you don't want to there's a short video introduction series on YouTube. BTW, if you do look at the manual page and think "This is as long as a book" you'd be wrong. There is a paper-back book and it is 112 pages long.


Versions of Parallel are available in all distributions repositories. Just install it and you will get the new version when you update your system in the not too distant future.

You can read more about GNU Parallel at http://www.gnu.org/s/parallel/

footnotes

  1. ImageMagick's convert can convert and resize and manipulate images. See the manual page to learn how to use it
  2. pngquant is a tool for compressing PNG images, see the pngquant manual page to see what it does
  3. tool for running commands from standard input. See the manual if do not know how to use it


last edited 2019-05-23


avatar

Anonymous user #1

25 days ago
Score 0++
why use parallel when you can do the same thing with xargs -P?
avatar

Anonymous user #2

25 days ago
Score 0++
parallel is more flexible and has many more features. xargs -P is great if all you need is exactly what xargs -P does.
avatar

Anonymous user #2

25 days ago
Score 0++
what does parallel do that xargs doesn't?
avatar

Anonymous user #2

25 days ago
Score 0++
I believe another important benefit is limiting the amount of actively running jobs, like make does for recipes. You want 8 convert's on an 8 core system, not all 1000 at once that would all only sit waiting for CPU.
Add your comment
LinuxReviews welcomes all comments. If you do not want to be anonymous, register or log in. It is free.