So you try to compress 8.5GB worth of source code and libraries to a 4.5GB tbz2 tarball, only to discover running:

tar -jcvpf dev20110411.tbz2 dev

takes about 30 minutes to complete, on a Mac Pro with 8 cores Xeon 2.8 GHz, 16GB RAM and a new 2TB 7200rpm WD Caviar Black HDD. The problem is explained below...

Look at the CPU utilisation. 13% over 8 cores implies one core is at its maximum. Indeed, tar is running at 100%. It is a single threaded application on the mac, and it tries to both archive files and compress them using bzip2 compression. This means 7/8 of the Mac Pro is sitting idle.

So I discovered this nifty enhancement to bzip2. It is a parallel implementation of bzip2, being very efficient with multiple cores. I downloaded it, compiled it using the gcc toolset that comes with XCode, and ran my compression again:

tar -cvp --use-compress-program=/usr/local/pbzip2/pbzip2-1.1.3/pbzip2 
   -f dev20110411.tbz2 dev

This time it took 5.5 minutes to complete. That is almost 6 times faster than the built in version. What is happening here can be explained by the following graph:

All 8 cores are working as efficiently as they can, keeping in mind smaller files have other overheads which my WD Caviar Black cannot sufficiently address (a SDD would have helped). But look at pbzip2 - it is running at 775% CPU, leaving some CPU for tar. It is amazingly efficient, and reduced the compression speed by a factor of 6 on the same hardware, by just being efficient.

Hope this tip helps you.