I happened across what I think will turn out to be a very useful open source application: Gnu parallel. Let me explain how I got to that point. I was uncompressing gzip’d files, of which I have lots. So many, in fact, it can take the better part of a day to go through them all. The server I was using is four years old but I don’t have access to anything better. In fact it seems processor speed has sort of plateaud. Moore’s Law only applies if you count the number of processors, which are multiplying like rabbits. So I start to think that maybe my single-threaded approach is wrong, see? I need to scale horizontally, not vetically.
It just happens that in my former life I spent quite some effort into parallelizing a compute-intensive program I wrote. I can’t believe that it’s still hanging around on the Internet! I haven’t looked for it for at least 17 years: my FERMISV Monte Carlo. So I thought, what if I could parallelize my zcat|grep? Writing such a thing from scratch would be too time-consuming, but after searching for parallelize zcat I quickly found Gnu Parallel. It’s one thing to identify such a program in a Google search, another to get it working. Believe me, I’ve put in my time with some of these WordPress plugins.
The upshot is that, yes, Gnu Parallel actually works and it’s not hard to learn to use. They have a nice Youtube video. For me I employed syntax like:
ls *gz|time parallel -k "zcat {}|grep 192.168.23.34" > /tmp/matches |
. This, of course, is to be compared with the normal operation I have been doing for eons:
time zcat *gz|grep 192.168.23.34 > /tmp/matches |
. The “time” is just for benchmarking and normally wouldn’t be present.
The results are that the matches file produced in the two different ways are identical. That’s good! The -k switch ensured the order is preserved in the output, and anyways, it didn’t cost me any time. Otherwise the order is not guaranteed. I tested on a server with four cores. Gnu Parallel reported that it used 375% of my CPU! It finished in less than half the time of my normal way of doing things. I need to put it through more real-world exercises, but it really looks promising. I never got onto the xargs bandwagon (perhaps I should be ashamed to admit it?), but Gnu Parallel can do a lot of that same type of thing, but in my opinion is more intuitive. Take a look!