Category Tech
Tags bash

The xargs utility allows to easily implement a process pool, where a constant pool of workers processes a list of jobs in parallel. We use the following example

echo "Called worker with args [email protected]"
sleep 2

The following call will run 26 times, always keeping 5 processes running in parallel.

xargs -L1 -P5 ./ <<EOF
a 1
b 2
c 3
--extra d 4
e 5
f 6
g 7
h 8
i 9
j 10
k 11
l 12
m 13
n 14
o 15
p 16
q 17
r 18
s 19
t 20
u 21
v 22
w 23
x 24
y 25
z 26

The -L1 argument ensures that each line below <<EOF is taken as one set of parameters for The -P5 argument results in 5 processes.

An alternative would be to place the lines between <<EOF and EOF in a file args.dat, and call <args.dat xargs -L1 -P5 ./