Wednesday, November 8, 2017

Parallel compression and decompression command

There is option for tar program:
option c for compression, x for extraction.
-I, --use-compress-program PROG
      filter through PROG (must accept -d)
You can use multithread version of archiver or compressor utility.
Most popular multithread archivers are pigz (instead of gzip) and pbzip2 (instead of bzip2). For instance:
$ tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 paths_to_archive
$ tar --use-compress-program=pigz -cf OUTPUT_FILE.tar.gz paths_to_archive
Archiver must accept -d. If your replacement utility hasn't this parameter and/or you need specify additional parameters, then use pipes (add parameters if necessary):
$ tar cf - paths_to_archive | pbzip2 > OUTPUT_FILE.tar.gz
$ tar cf - paths_to_archive | pigz > OUTPUT_FILE.tar.gz
Use lbzip2

$ time tar cf tmp.tar.bz2 tmp --use-compress-program=lbzip2
$ time tar xf tmp.tar.bz2 --use-compress-program=lbzip2

No comments:

Post a Comment