Archiving and compression - ArchWiki

Jump to content

# Archiving and compression

From ArchWiki

The traditional Unix archiving and compression tools are separated according to the Unix philosophy:

- A file archiver combines several files into one archive file, e.g. tar.
- A compression tool compresses and decompresses data, e.g. gzip.

These tools are often used in sequence by firstly creating an archive file and then compressing it.

Of course there are also tools that do both, which tend to additionally offer encryption, error detection and recovery.

## Archiving only

Name | Package | Manuals | Description
--- | --- | --- | ---
GNU tar | tar | tar(1), info | Core utility for manipulating the ubiquitous tar archives (tarballs).
libarchive | libarchive | bsdtar(1) bsdcpio(1) | Implementation of tar and cpio that also offers a library. Used by pacman and mkinitcpio.
ar | binutils | ar(1) | Legacy Unix archiver before tar. Today only used for creating static library files.
GNU cpio | cpio | cpio(1). info | File archiver via stdin/stdout, supports cpio and tar formats.

Tip Both GNU and BSD tar automatically do decompression delegation for bzip2, compress, gzip, lzip, lzma, lzop, zstd, and xz compressed archives. Only BSD tar supports lz4 natively (but GNU tar can do an equivalent with `--use-compress-program=lz4`/`-Ilz4`). When creating archives both support the `-a` switch to automatically filter the created archive through the right compression program based on the file extension. While BSD tar recognizes compression formats based on the format, GNU tar only guesses based on the file extension.

See also #Archiving only usage.

## Compression tools

### Compression only

These compression programs implement their own file format.

Name | Package | Manual | Ext | Tar ext | Description | Parallel implementations
--- | --- | --- | --- | --- | --- | ---
bzip2 | bzip2 | bzip2(1) | .bz2, .bz | .tbz2, .tbz | Uses the Burrows–Wheeler algorithm. | lbzip2, pbzip2
bzip3 | bzip3 | bzip3(1) | .bz3 | .tbz3 | Uses the Burrows–Wheeler algorithm. |
gzip | gzip | gzip(1) | .gz, .z | .tgz, .taz | GNU zip, based on DEFLATE algorithm. | pigz, bgzip(1) (part of htslibAUR), crabzAUR, python-rapidgzipAUR
lrzip | lrzip | lrzip(1) | .lrz |  | Improved version of rzip, uses multiple algorithms. | Is multithreaded by default
LZ4 | lz4 | lz4(1) | .lz4 |  | Written in C, focused on compression and decompression speed. | Is multithreaded by default. See https://lz4.org/ for alternatives.
lzip | lzip | lzip(1) | .lz |  | Uses LZMA. | plzipAUR
lzop | lzop | lzop(1) | .lzo | .tzo | Uses the LZO library (lzo). |
xz | xz | xz(1) | .xz, .lzma | .txz, .tlz | Uses LZMA2. Default for GNU coreutils and kernel archive files. | Is multithreaded by default. An alternative is pixz.
zstd | zstd | zstd(1) | .zst |  | Uses Zstandard algorithm. | is multithreaded

- Parallel implementations offer improved speeds by using multiple CPU cores.
- Tar extensions refers to compressed archives where `tar` and the compression tool is used, e.g. `.tzo` is `.tar.lzo`.
- See also #Compression only usage.

### Archiving and compression

Name | Packages | Manuals | Ext | Description
--- | --- | --- | --- | ---
7-Zip | 7zip | Official manual | .7z | A file archiver with a high compression ratio.
DAR | darAUR | dar(1) | .dar | Archiver to backup large live filesystems, takes care of hard links, extended attributes, sparse files and inode types.
GNU tar | tar | tar(1), info | .tar.compression-type | Supports passing the archive for compression to bzip2(1), xz(1), lzip(1), lzma(1), lzop(1), gzip(1), compress(1), zstd(1), as well as any other custom compression program. See tar(1) § Compression options.
libarchive | libarchive | bsdtar(1) | .tar.compression-type | Has built-in compression options for bzip2, gzip, lrzip, lzma, lzop, lz4, zstd and xz.
RAR | rarAUR, unrar | rar(1) | .rar | Both the format and the rar utility are proprietary.
t2sz | t2szAUR |  | .tar.zst .tzst | Tar archiving utility in C with member-aligned zstd-compression
tarlz | tarlzAUR | tarlz(1) | .tar.lz .tlz | Tar archiving utility in C++ with member-aligned lzip compression
ZIP | zip, unzip | zip(1), unzip(1) | .zip | Widely used outside of the Linux world.
Unarchiver | unarchiver | unar(1), lsar(1) | many | Command-line tool of a Mac application, supports over 40 archive formats.
ZPAQ | zpaqAUR | zpaq(1) | .zpaq | A high compression ratio archiver written in C++, uses several algorithms.
LHa | lhasa, lhaAUR | lha(1) | .lzh (on Amiga: .lha) | LZH/LHA archiver, supports the lh7-method.
WinAce | unace | unace(1) | .ace | Both the ACE file format and the archiving tool are proprietary.

See also #Archiving and compression usage.

### Feature charts

Some of the tools above are capable of handling multiple formats, allowing for fewer installed packages.

#### Decompress

Name | File | Archive
--- | --- | ---
gzip | bzip2 | LZMA | xz | zstd | ZIP | RAR | 7z | CAB
gzip | Yes | No | No | No | No | Partial1 | No | No | No
7zip | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes
unarchiver | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes
zstd | Yes | No | Yes | Yes | Yes | No | No | No | No

1. gunzip(1) can only decompress single member ZIP files.

## Usage comparison

### Archiving only usage

Name | Create archive | Extract archive | List content
--- | --- | --- | ---
tar(1) | `tar cfv archive.tar file1 file2` | `tar xfv archive.tar` | `tar -tvf archive.tar`
cpio(1) | `ls file1 file2 | cpio -o > archive.cpio` | `cpio -i -vd < archive.cpio` | `cpio -t < archive.cpio`

### Compression only usage

Name | Compress | Decompress | Decompress to stdout
--- | --- | --- | ---
bzip2(1) | `bzip2 file` | `bzip2 -d file.bz2` | `bzcat file.bz2`
gzip(1) | `gzip file` | `gzip -d file.gz` | `zcat file.gz`
lrzip(1) | `lrzip file` `lrztar folder` | `lrzip -d file.lrz` `lrztar -d folder.tar.lrz` | `lrzcat file.lrz`
lz4(1) | `lz4 file` | `lz4 file.lz4` | `lz4cat file.lz4`
xz(1) | `xz file` | `xz -d file.xz` | `xzcat file.xz`
zstd(1) | `zstd file` | `zstd -d file.zst` | `zstdcat file.zst`

### Archiving and compression usage

Name | Compress | Decompress | Decompress to stdout | List content
--- | --- | --- | --- | ---
7z | `7z a archive.7z file1 file2` | `7z x archive.7z` | `7z e -so archive.7z file1` | `7z l archive.7z`
dar(1) | `dar -c archive -g file1 -g file2` | `dar -x archive` |  | `dar -l archive`
tar(1) | `tar acvf archive.format file1 file2` | `tar xfv archive.format` | `tar xfvO archive.format` | `tar tvf archive.format`
rar(1) | `rar a archive.rar file1 file2` | `rar x archive.rar` | `rar p -inul archive.rar file1` | `rar l archive.rar`
zip(1), unzip(1) | `zip archive.zip file1 file2` | `unzip archive.zip` | `unzip -p archive.zip file1` | `unzip -l archive.zip`
lha(1) | `lha ao7 archive.lzh file1 file2` | `lha x archive.lzh` |  | minimal: `lha l archive.lzh` verbose: `lha v archive.lzh`

## Convenience tools

See also Bash/Functions#Extract.

- atool — Script for managing file archives of various types.

  https://www.nongnu.org/atool/ || atool

- dtrx — An intelligent archive extraction tool.

  https://github.com/brettcs/dtrx || dtrxAUR

- J7Z — GUI for Linux in java which attempts to simplify data compression and backup. It can create 7z, BZip2, Zip, GZip, Tar archives.

  https://sites.google.com/site/xavionsj7z || j7zAUR

- ouch — A command line utility for easily compressing and decompressing files and directories.

  https://github.com/ouch-org/ouch || ouch

- patool — Allows various archive types to be created, extracted, tested, listed, compared, searched and repacked.

  https://wummel.github.io/patool/ || patoolAUR

- python-unp — Command line tool that can unpack archives easily.

  https://github.com/mitsuhiko/unp || python-unpAUR

- unp — A script for unpacking a wide variety of archive formats.

  https://tracker.debian.org/pkg/unp || unp

- unpack — Wrapper script for handling multiple archive formats.

  https://github.com/githaff/unpack || unpack-gitAUR

## Determining archive format

To extract an archive, its file format needs to be determined. If the file is properly named you can deduce its format from the file extension.

Otherwise you can use the file tool, see file(1).

## Esoteric, rare or deprecated tools

Name | Packages | Ext | Description
--- | --- | --- | ---
ARC | arcAUR | .arc, .ark | Was very popular during the early days of the dial-up BBS. Superseded by ZIP.
ARJ | arj | .arj | An archiver used on DOS/Windows in mid-1990s. This is an open source clone.
Cabinet | cabextract, unshield | .cab, .exe | A variety of installation technologies in Windows use the CAB format.
compress | ncompress | .Z | The de facto standard UNIX compression utility to success the Huffman-based pack(1) before gzip become a thing.
Inno Setup | innoextract | .exe | Installers created by Inno Setup.
PAR2 | par2cmdline | .par2 | Parity archiver for increased data integrity. See also Parchive.
shar | sharutils | .shar | Creates self-extracting archives that are valid shell scripts.
Zoo | zooAUR | .zoo | Was mostly popular on the OpenVMS operating system before PKZIP became popular.

## File system compression

Some file systems support on-the-fly compression of file data:

- Btrfs can be configured to compress individual files, directories, or entire volumes by default.
- On ZFS, compression can be enabled on pools or file systems.

## Device mapper compression

The open-sourced VDO project was integrated into the Linux kernel project, which provides a deduplication and compression device mapper layer in the interest of increasing storage efficiency. A userspace tools for managing VDO volumes is available in the AUR: vdoAUR

## Compression libraries

- Brotli — Compression algorithm for data streams using the LZ77 algorithm, Huffman coding and 2nd order context modeling.

  https://github.com/google/brotli || brotli

- libzip — Provides creation and extraction of ZIP files. Used by KDE and Deepin in place of the zip/unzip tools.

  https://libzip.org || libzip

- zlib — Compression library implementing the deflate compression method found in gzip and PKZIP.

  https://www.zlib.net/ || zlib

- Zopfli — High compress ratio file compressor from Google, using a deflate-compatible algorithm called zopfli. Also available as compression level 11 in pigz.

  https://github.com/google/zopfli || zopfli-gitAUR

## Troubleshooting

### Garbled file names when extracted

See Character encoding#Troubleshooting.

## See also

- List of applications/Utilities#Archiving and compression tools
- List of applications/Multimedia#Image compression
- Wikipedia:Comparison of file archivers
- Wikipedia:List of archive formats
- Wikipedia:Comparison of archive formats

Categories:

- Data compression and archiving
- Command-line

Archiving and compression
Add topic