Inter core i5, what a marvelous beast. 4 CPU cores in one tiny laptop. The problem is to use them properly. And when I had to compress a 700MB log file a few days ago, I realized that not all the tools on Linux are multi-core friendly.

Today, a fellow PLUG member pointed me to lbzip2, a multi-threaded implementation of bzip2. I just gave it a quick shot and the results are interesting:

Initial file:

$ ls -s jmeter-server-node1.log --block-size=1
689274880 jmeter-server-node1.log

=== with bzip2 ====

$ time bzip2 -z -9 jmeter-server-node1.log

real	8m33.220s
user	8m31.444s
sys	0m0.880s

$ ls -s jmeter-server-node1.log.bz2 --block-size=1
1589248 jmeter-server-node1.log.bz2

$ time bunzip2 jmeter-server-node1.log.bz2

real	0m35.801s
user	0m33.662s
sys	0m0.964s

=== with lbzip2 ====

$ time lbzip2 -n 4 -z -9 -S jmeter-server-node1.log 

real	5m37.425s
user	20m57.227s
sys	0m5.016s

$ ls -s jmeter-server-node1.log.bz2 --block-size=1
1601536 jmeter-server-node1.log.bz2

$ time lbzip2 -n 4 -d jmeter-server-node1.log.bz2 

real	0m20.370s
user	1m15.697s
sys	0m1.316s

Compression is of the same level, but I'm surprised to see that while lbzip2 is 65% faster, it also uses 250% more user time than bzip2. The efficiency per-core is a lot lower, but I'm happy to be using all my cores.