Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

If you like xargs, but want jobs to execute in parallel, you may enjoy GNU parallel.

Your trivial example:

find images -name "*.jpg" | parallel convert -geometry 200x {} {.}_thumb.jpg



xargs --max-procs=#


However, unlike xargs, GNU Parallel gives you a guarantee on the order of the output. From the man page:

       GNU parallel makes sure output from the commands is the same output as
       you would get had you run the commands sequentially. This makes it
       possible to use output from GNU parallel as input for other programs.


Want to comment just to stress that this feature is EXTREMELY useful and has saved me from all sorts of tricks with file intermediate output with filenames that match process ids, etc.


The nice thing about parallel is that it can actually run some of the instances on remote machines using SSH, transparently (as long as they have the commands). You just need a couple of parameters and it takes care of uploading the files the command needs and then downloading the results. It's quite awesome.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: