mirror of
https://github.com/borgbackup/borg.git
synced 2026-07-06 16:57:30 -04:00
0e3876d5e4
Normalized chunking switches between a stricter and a looser cut mask around the target chunk size. This greatly tightens the chunk-size distribution (coefficient of variation ~0.9 -> ~0.3 in tests) and removes the dedup-hostile max-size-clamped chunks, with unchanged deduplication. chunker-params for buzhash64 gains a required 6th field, nc_level: buzhash64,chunk_min,chunk_max,chunk_mask,window_size,nc_level Use nc_level=2 for the new default, nc_level=0 to disable (then behavior is byte-identical to the previous single-mask chunker). buzhash (32bit) is untouched and stays bit-compatible with borg 1.x. The mask transition point (normal_size) defaults to a principled formula (target minus the expected loose-phase tail) so the mean stays near the target; it can be tuned via the normal_size constructor arg. scripts/chunker_bench.py: evidence harness used to measure chunk-size distribution, dedup ratio, throughput and shift-resilience. Measurements (before = nc_level 0, after = nc_level 2; both at the default params buzhash64,19,23,21,4095; measured with scripts/chunker_bench.py): 5 GiB of incompressible data (~2000-2700 chunks, statistically stable): before: CV 0.739, 49 max-size-clamped (8 MiB) chunks, 953 MB/s after: CV 0.311, 0 max-size-clamped chunks, 1024 MB/s Re-backup of a 2.5 GiB file after a few scattered single-byte edits (deduplication ratio; 0.5 = v2 fully deduplicated against v1, lower is better): 64 edits: before 0.5424 -> after 0.5235 320 edits: before 0.6791 -> after 0.6142 Normalized chunking deduplicates better after edits: removing the max-size-clamped chunks means a single-byte change invalidates much less data (about 36% less dedup overhead at 320 edits). Throughput was also consistently higher with nc_level=2 at this scale. Also: fix bug when computing the mask, one needs to use 1ULL instead of 1, so the shifting computation is done in a uint64, not in a 32bit int. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
106 lines
4.4 KiB
ReStructuredText
106 lines
4.4 KiB
ReStructuredText
.. include:: transfer.rst.inc
|
|
|
|
Examples
|
|
~~~~~~~~
|
|
|
|
To keep the following examples short and readable, we export the repository
|
|
locations and passphrases first:
|
|
|
|
::
|
|
|
|
export BORG_REPO=ssh://borg2@borgbackup/./tests/b20
|
|
export BORG_PASSPHRASE='your-borg2-repo-passphrase'
|
|
export BORG_OTHER_REPO=ssh://borg2@borgbackup/./tests/b1x
|
|
export BORG_OTHER_PASSPHRASE='your-borg1-repo-passphrase'
|
|
|
|
::
|
|
|
|
# Borg 1.x repository -> Borg 2.0 repository (hmac-sha256 -> hmac-sha256, keeping the same chunk ID algorithm)
|
|
|
|
# 0. Have Borg 2.0 installed on the client AND server; have a Borg 1.x repository copy for testing.
|
|
|
|
# 1. Create a new "related" repository:
|
|
# Here, the existing Borg 1.x repository used repokey (and AES-CTR mode),
|
|
# thus we use aes256-ocb for the new Borg 2.0 repository.
|
|
# Staying with the same chunk ID algorithm (hmac-sha256) and with the same
|
|
# key material (via BORG_OTHER_REPO) will make deduplication work
|
|
# between old archives (copied with borg transfer) and future ones.
|
|
# The AEAD cipher does not matter (everything must be re-encrypted and
|
|
# re-authenticated anyway); you could also choose chacha20-poly1305.
|
|
$ borg repo-create -e aes256-ocb
|
|
|
|
# 2. Check what and how much it would transfer:
|
|
$ borg transfer --from-borg1 --dry-run
|
|
|
|
# 3. Transfer (copy) archives from the old repository into the new repository (takes time and space!):
|
|
$ borg transfer --from-borg1
|
|
|
|
# 4. Check whether we have everything (same as step 2):
|
|
$ borg transfer --from-borg1 --dry-run
|
|
|
|
::
|
|
|
|
# Borg 1.x repository -> Borg 2.0 repository (blake2 -> blake3, changing the chunk ID algorithm)
|
|
|
|
# 0. Have Borg 2.0 installed on the client AND server; have a Borg 1.x repository copy for testing.
|
|
|
|
# 1. Create a new "related" repository:
|
|
# Here, the existing Borg 1.x repository used repokey-blake2 (and AES-CTR mode),
|
|
# thus we use aes256-ocb with --id-hash blake3 for the new Borg 2.0 repository.
|
|
# We need to change from blake2 to blake3, because blake2 is not supported
|
|
# for borg2 repos (blake3 is much faster). Because we change how chunk IDs are
|
|
# computed, we need to re-chunk everything while doing the transfer.
|
|
# The chunker parameters you provide here should be the same as you will
|
|
# use for all future Borg 2.0 archives.
|
|
# The AEAD cipher does not matter (everything must be re-encrypted and
|
|
# re-authenticated anyway); you could also choose -e chacha20-poly1305 -i blake3.
|
|
$ borg repo-create -e aes256-ocb -i blake3
|
|
$ export CHUNKER_PARAMS="buzhash64,19,23,21,4095,2"
|
|
|
|
# 2. Check what and how much it would transfer:
|
|
$ borg transfer --from-borg1 --chunker-params=$CHUNKER_PARAMS --dry-run
|
|
|
|
# 3. Transfer (copy) archives from the old repository into the new repository (takes time and space!):
|
|
$ borg transfer --from-borg1 --chunker-params=$CHUNKER_PARAMS
|
|
|
|
# 4. Check whether we have everything (same as step 2):
|
|
$ borg transfer --from-borg1 --chunker-params=$CHUNKER_PARAMS --dry-run
|
|
|
|
Keyfile considerations when upgrading from Borg 1.x
|
|
++++++++++++++++++++++++++++++++++++++++++++++++++++
|
|
|
|
If you are using a ``keyfile`` encryption mode (not ``repokey``), Borg 2
|
|
may not automatically find your Borg 1.x key file, because the default
|
|
key file directory has changed on some platforms due to the switch to
|
|
the `platformdirs <https://pypi.org/project/platformdirs/>`_ library.
|
|
|
|
On **Linux**, there is typically no change -- both Borg 1.x and Borg 2
|
|
use ``~/.config/borg/keys/``.
|
|
|
|
On **macOS**, Borg 1.x stored key files in ``~/.config/borg/keys/``,
|
|
but Borg 2 defaults to ``~/Library/Application Support/borg/keys/``.
|
|
|
|
On **Windows**, Borg 1.x used XDG-style paths (e.g. ``~/.config/borg/keys/``),
|
|
while Borg 2 defaults to ``C:\Users\<user>\AppData\Roaming\borg\keys\``.
|
|
|
|
If Borg 2 cannot find your key file, you have several options:
|
|
|
|
1. **Copy the key file** from the old location to the new one.
|
|
2. **Set BORG_KEYS_DIR** to point to the old key file directory::
|
|
|
|
export BORG_KEYS_DIR=~/.config/borg/keys
|
|
|
|
3. **Set BORG_KEY_FILE** to point directly to the specific key file::
|
|
|
|
export BORG_KEY_FILE=~/.config/borg/keys/your_key_file
|
|
|
|
4. **Set BORG_BASE_DIR** to force Borg 2 to use the same base directory
|
|
as Borg 1.x::
|
|
|
|
export BORG_BASE_DIR=$HOME
|
|
|
|
This makes Borg 2 use ``$HOME/.config/borg``, ``$HOME/.cache/borg``,
|
|
etc., matching Borg 1.x behavior on all platforms.
|
|
|
|
See :ref:`env_vars` for more details on directory environment variables.
|
|
|