From 061321b668a8099bfafce7e86773969e2939b7f0 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Mon, 18 Dec 2017 05:44:36 +0100 Subject: [PATCH 1/3] update requirements and install docs about bundled lz4 and zstd --- docs/deployment/central-backup-server.rst | 2 -- docs/global.rst.inc | 1 + docs/installation.rst | 18 ++++++++++-------- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/deployment/central-backup-server.rst b/docs/deployment/central-backup-server.rst index 26e718985..f84a1e483 100644 --- a/docs/deployment/central-backup-server.rst +++ b/docs/deployment/central-backup-server.rst @@ -171,8 +171,6 @@ Salt running on a Debian system. - openssl - libacl1-dev - libacl1 - - liblz4-dev - - liblz4-1 - build-essential - libfuse-dev - fuse diff --git a/docs/global.rst.inc b/docs/global.rst.inc index 14a06bdf5..c4a12587c 100644 --- a/docs/global.rst.inc +++ b/docs/global.rst.inc @@ -14,6 +14,7 @@ .. _libacl: https://savannah.nongnu.org/projects/acl/ .. _libattr: https://savannah.nongnu.org/projects/attr/ .. _liblz4: https://github.com/Cyan4973/lz4 +.. _libzstd: https://github.com/facebook/zstd .. _libb2: https://github.com/BLAKE2/libb2 .. _ZeroMQ: http://zeromq.org/ .. _OpenSSL: https://www.openssl.org/ diff --git a/docs/installation.rst b/docs/installation.rst index 2061c1dd9..cd9f09c84 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -141,12 +141,17 @@ following dependencies first: optional install. * OpenSSL_ >= 1.0.0, plus development headers. * libacl_ (which depends on libattr_), both plus development headers. -* liblz4_ >= r129 (1.7.0), plus development headers. * ZeroMQ_ >= 4.0.0, plus development headers. +* We have bundled code of the following packages, but borg by default (see + setup.py if you want to change that) prefers a shared library if it can + be found on the system (lib + dev headers) at build time: + + - liblz4_ >= 1.7.0 (r129) + - libzstd_ >= 1.3.0 + - libb2_ * some Python dependencies, pip will automatically install them for you * optionally, the llfuse_ Python package is required if you wish to mount an archive as a FUSE filesystem. See setup.py about the version requirements. -* optionally libb2_. If it is not found a bundled implementation is used instead. If you have troubles finding the right package names, have a look at the distribution specific sections below or the Vagrantfile in the git repository, @@ -169,7 +174,6 @@ Install the dependencies with development headers:: sudo apt-get install python3 python3-dev python3-pip python-virtualenv \ libssl-dev openssl \ libacl1-dev libacl1 \ - liblz4-dev liblz4-1 \ libzmq3-dev libzmq3 \ build-essential sudo apt-get install libfuse-dev fuse pkg-config # optional, for FUSE support @@ -188,7 +192,6 @@ Install the dependencies with development headers:: sudo dnf install python3 python3-devel python3-pip python3-virtualenv sudo dnf install openssl-devel openssl sudo dnf install libacl-devel libacl - sudo dnf install lz4-devel sudo dnf install gcc gcc-c++ sudo dnf install redhat-rpm-config # not needed in Korora sudo dnf install fuse-devel fuse pkgconfig # optional, for FUSE support @@ -205,7 +208,7 @@ Install the dependencies automatically using zypper:: Alternatively, you can enumerate all build dependencies in the command line:: sudo zypper install python3 python3-devel \ - libacl-devel liblz4-devel openssl-devel \ + libacl-devel openssl-devel \ python3-Cython python3-Sphinx python3-msgpack-python \ python3-pytest python3-setuptools python3-setuptools_scm \ python3-sphinx_rtd_theme python3-llfuse gcc gcc-c++ @@ -218,7 +221,7 @@ Mac OS X Assuming you have installed homebrew_, the following steps will install all the dependencies:: - brew install python3 lz4 openssl + brew install python3 openssl brew install pkg-config # optional, for FUSE support pip3 install virtualenv @@ -238,7 +241,7 @@ and commands to make FUSE work for using the mount command. :: - pkg install -y python3 openssl liblz4 fusefs-libs pkgconf + pkg install -y python3 openssl fusefs-libs pkgconf pkg install -y git python3.4 -m ensurepip # to install pip for Python3 To use the mount command: @@ -270,7 +273,6 @@ Use the Cygwin installer to install the dependencies:: python3 python3-devel python3-setuptools binutils gcc-g++ libopenssl openssl-devel - liblz4_1 liblz4-devel git make openssh You can then install ``pip`` and ``virtualenv``:: From 2588e50f0b01181bc52699016e76e2647fbddd99 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Mon, 18 Dec 2017 22:53:24 +0100 Subject: [PATCH 2/3] readme / docs: mention zstd --- README.rst | 9 +++++++-- docs/faq.rst | 5 ++--- docs/internals/data-structures.rst | 9 +++++---- docs/quickstart.rst | 11 +++++++++++ 4 files changed, 25 insertions(+), 9 deletions(-) diff --git a/README.rst b/README.rst index 414901b24..21f0392a7 100644 --- a/README.rst +++ b/README.rst @@ -58,8 +58,13 @@ Main features authenticity is verified using HMAC-SHA256. Data is encrypted clientside. **Compression** - All data can be compressed by lz4 (super fast, low compression), zlib - (medium speed and compression) or lzma (low speed, high compression). + All data can be optionally compressed: + + * lz4 (super fast, low compression) + * zstd (wide range from high speed and low compression to high compression + and lower speed) + * zlib (medium speed and compression) + * lzma (low speed, high compression) **Off-site backups** Borg can store data on any remote host accessible over SSH. If Borg is diff --git a/docs/faq.rst b/docs/faq.rst index 7832d5da9..0bbb09571 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -788,7 +788,7 @@ Here's a (incomplete) list of some major changes: * more open, faster paced development (see `issue #1 `_) * less chunk management overhead (less memory and disk usage for chunks index) * faster remote cache resync (useful when backing up multiple machines into same repo) -* compression: no, lz4, zlib or lzma compression, adjustable compression levels +* compression: no, lz4, zstd, zlib or lzma compression, adjustable compression levels * repokey replaces problematic passphrase mode (you can't change the passphrase nor the pbkdf2 iteration count in "passphrase" mode) * simple sparse file support, great for virtual machine disk files * can read special files (e.g. block devices) or from stdin, write to stdout @@ -840,8 +840,7 @@ The default in Borg is lz4, which is fast enough to not use significant CPU time in most cases, but can only achieve modest compression. It still compresses easily compressed data fairly well. -zlib compression with all levels (1-9) as well as LZMA (1-6) are available -as well, for cases where they are worth it. +Borg also offers zstd, zlib and lzma compression, choose wisely. Which choice is the best option depends on a number of factors, like bandwidth to the repository, how well the data compresses, available CPU diff --git a/docs/internals/data-structures.rst b/docs/internals/data-structures.rst index d9d423020..7d7e06f03 100644 --- a/docs/internals/data-structures.rst +++ b/docs/internals/data-structures.rst @@ -913,15 +913,16 @@ Borg supports the following compression methods: - none (no compression, pass through data 1:1) - lz4 (low compression, but super fast) +- zstd (level 1-22 offering a wide range: level 1 is lower compression and high + speed, level 22 is higher compression and lower speed) - since borg 1.1.4 - zlib (level 0-9, level 0 is no compression [but still adding zlib overhead], level 1 is low, level 9 is high compression) - lzma (level 0-9, level 0 is low, level 9 is high compression). -Speed: none > lz4 > zlib > lzma -Compression: lzma > zlib > lz4 > none +Speed: none > lz4 > zlib > lzma, lz4 > zstd +Compression: lzma > zlib > lz4 > none, zstd > lz4 -Be careful, higher zlib and especially lzma compression levels might take a -lot of resources (CPU and memory). +Be careful, higher compression levels might use a lot of resources (CPU/memory). The overall speed of course also depends on the speed of your target storage. If that is slow, using a higher compression level might yield better overall diff --git a/docs/quickstart.rst b/docs/quickstart.rst index 3c197aee3..cdc976ea2 100644 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -186,6 +186,17 @@ Backup compression The default is lz4 (very fast, but low compression ratio), but other methods are supported for different situations. +You can use zstd for a wide range from high speed (and relatively low +compression) using N=1 to high compression (and lower speed) using N=22. + +zstd is a modern compression algorithm and might be preferable over zlib and +lzma, except if you need compatibility to older borg versions (< 1.1.4) that +did not yet offer zstd. + + $ borg create --compression zstd,N /path/to/repo::arch ~ + +Other options are: + If you have a fast repo storage and you want minimum CPU usage, no compression:: $ borg create --compression none /path/to/repo::arch ~ From 3d32407e9af6c4ea099a68b3469bdb3426657d0c Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Mon, 18 Dec 2017 23:21:34 +0100 Subject: [PATCH 3/3] add zstd to compression help --- src/borg/archiver.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/borg/archiver.py b/src/borg/archiver.py index f6df9c069..bf4be41a7 100644 --- a/src/borg/archiver.py +++ b/src/borg/archiver.py @@ -1968,7 +1968,13 @@ class Archiver: Do not compress. lz4 - Use lz4 compression. High speed, low compression. (default) + Use lz4 compression. Very high speed, very low compression. (default) + + zstd[,L] + Use zstd ("zstandard") compression, a modern wide-range algorithm. + If you do not explicitely give the compression level L (ranging from 1 + to 22), it will use level 3. + Archives compressed with zstd are not compatible with borg < 1.1.4. zlib[,L] Use zlib ("gz") compression. Medium speed, medium compression. @@ -1995,6 +2001,8 @@ class Archiver: Examples:: borg create --compression lz4 REPO::ARCHIVE data + borg create --compression zstd REPO::ARCHIVE data + borg create --compression zstd,10 REPO::ARCHIVE data borg create --compression zlib REPO::ARCHIVE data borg create --compression zlib,1 REPO::ARCHIVE data borg create --compression auto,lzma,6 REPO::ARCHIVE data