Add a new "fastcdc" content-defined chunker selectable via --chunker-params.
It uses the FastCDC Gear rolling hash (fp = (fp << 1) + Gear[byte]), which is
window-less and cheaper per byte than buzhash's cyclic-polynomial update, so it
chunks noticeably faster (see "borg benchmark cpu" output), while producing
the same chunk-size distribution and deduplication.
The Gear table is keyed: it is derived from the repo id key via CSPRNG (own
"fastcdc" domain), exactly like the buzhash64 table, so chunk cut points stay
unpredictable without the key (anti-fingerprinting). It implements the same
FastCDC techniques as buzhash64 (sub-minimum skipping, normalized chunking with
a required nc_level, min/max clamping); the mask uses the high bits of the hash
(Gear accumulates entropy there).
chunker-params: "fastcdc,chunk_min,chunk_max,chunk_mask,nc_level" - there is no
window field, because Gear is window-less. e.g. fastcdc,19,23,21,2
Also: borg benchmark cpu now measures the fastcdc chunker; tests in
borg.testsuite.chunkers (golden vector, size distribution, keyed gear table,
param parsing, slow fuzz); docs and changelog.
Benchmarks (scripts/chunker_bench.py, buzhash64 vs fastcdc, both nc_level=2,
incompressible data unless noted):
5 GiB, 2 MiB target (default params):
buzhash64: CV 0.294, 1011 MB/s
fastcdc: CV 0.295, 1313 MB/s (+30%)
64 MiB, 64 KiB target:
buzhash64: CV 0.374, shift-resilience 0.9928, 963 MB/s
fastcdc: CV 0.359, shift-resilience 0.9929, 1331 MB/s (+38%)
Re-backup of a 2.5 GiB file after scattered single-byte edits (dedup ratio,
0.5 = v2 fully deduplicated, lower is better):
64 edits: buzhash64 0.5237, fastcdc 0.5236
320 edits: buzhash64 0.6133, fastcdc 0.6161
borg benchmark cpu, 1 GB: fastcdc 3.80s, buzhash 4.36s, buzhash64 8.13s,
fixed 0.56s.
Chunk-size distribution, deduplication and shift-resilience match buzhash64
within noise; fastcdc is consistently faster.
Also: fix bug when computing the mask, one needs to use 1ULL instead of
1, so the shifting computation is done in a uint64, not in a 32bit int.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sphinxcontrib.jquery is already registered in the extensions list, so
the explicit app.setup_extension("sphinxcontrib.jquery") in setup() is
redundant.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Normalized chunking switches between a stricter and a looser cut mask
around the target chunk size. This greatly tightens the chunk-size
distribution (coefficient of variation ~0.9 -> ~0.3 in tests) and removes
the dedup-hostile max-size-clamped chunks, with unchanged deduplication.
chunker-params for buzhash64 gains a required 6th field, nc_level:
buzhash64,chunk_min,chunk_max,chunk_mask,window_size,nc_level
Use nc_level=2 for the new default, nc_level=0 to disable (then behavior
is byte-identical to the previous single-mask chunker).
buzhash (32bit) is untouched and stays bit-compatible with borg 1.x.
The mask transition point (normal_size) defaults to a principled formula
(target minus the expected loose-phase tail) so the mean stays near the
target; it can be tuned via the normal_size constructor arg.
scripts/chunker_bench.py: evidence harness used to measure chunk-size
distribution, dedup ratio, throughput and shift-resilience.
Measurements (before = nc_level 0, after = nc_level 2; both at the default
params buzhash64,19,23,21,4095; measured with scripts/chunker_bench.py):
5 GiB of incompressible data (~2000-2700 chunks, statistically stable):
before: CV 0.739, 49 max-size-clamped (8 MiB) chunks, 953 MB/s
after: CV 0.311, 0 max-size-clamped chunks, 1024 MB/s
Re-backup of a 2.5 GiB file after a few scattered single-byte edits
(deduplication ratio; 0.5 = v2 fully deduplicated against v1, lower is
better):
64 edits: before 0.5424 -> after 0.5235
320 edits: before 0.6791 -> after 0.6142
Normalized chunking deduplicates better after edits: removing the
max-size-clamped chunks means a single-byte change invalidates much less
data (about 36% less dedup overhead at 320 edits). Throughput was also
consistently higher with nc_level=2 at this scale.
Also: fix bug when computing the mask, one needs to use 1ULL instead of
1, so the shifting computation is done in a uint64, not in a 32bit int.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The --hostname/--username options on borg create (added in 2.0.0b21, #9402)
are removed in favor of the BORG_HOSTNAME / BORG_USERNAME environment variables.
When set, these override the hostname/username stored in newly created archives
and used by the {hostname}/{user} placeholders (so they also apply to archive
names, prune --glob-archives, check, etc.). Useful to run borg on host A but
impersonate host B.
The env vars are read at the point of use; fqdn/hostid (see BORG_HOST_ID) and
the auto-detection are intentionally left untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Makes specific notes on week numbers, clears some ambiguity in the
yearly candidate wording, and more.
The matching test was previously missing assertions on the weekly
archives, now fixed along with better explanations.
Implement the read-only repository check by hashing pack and index
objects and comparing against the stored hashes, without writing to the
repository.
Report check progress separately for the index and for the packs, each
ending at 100%.
Port of borgbackup/borg#9790 to master.
Note: the PR's second commit (Archive.delete: don't reuse msgpack Unpacker
after an unpacking failure) does not apply here - on master Archive.delete
no longer unpacks item metadata (it just removes the archive and lets
"borg compact" reclaim space), so the reuse-after-failure code path does
not exist. The streaming RobustUnpacker already creates a fresh Unpacker
on resync.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the dead BORG_TESTONLY_SHA256_PACK_ID tox env and CI job, fix the packs.rst pack-id docs, and reword comments/tests to describe sha256 pack naming instead of the removed shortcut.
The repo-create.rst examples were not updated after --encryption was
split into --encryption + --id-hash. Use the real mode name (aes256-ocb)
and show --id-hash as an orthogonal option.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
NAME was only read by "borg repo-info" to show the crypto suite. Remove it from
all (current and legacy) key classes and let repo-info assemble the display from
the two real dimensions instead: "<mode>, <ENC_NAME>, <IDHASH_NAME>", e.g.
"Encrypted: Yes (repokey, aes256-ocb, sha256)" or
"Encrypted: No (repokey, authenticated, blake3)".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stop using key.ARG_NAME (the combined crypto-suite name) for the JSON output.
The "encryption" object now mirrors the split CLI options: the "mode" field is
replaced by "encryption" (cipher / AE algorithm, key.ENC_NAME) and "id_hash"
(id hash function, key.IDHASH_NAME).
This is a breaking change to the documented JSON API.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The combined --encryption value packed two orthogonal dimensions (cipher / AE
algorithm and id hash function) into a single string, causing a combinatorial
explosion of mode names. Key location was already split out into --key-location.
Now:
- --encryption selects only the cipher / AE algorithm:
none, authenticated, aes256-ocb, chacha20-poly1305
- --id-hash selects the id hash function: sha256 (default) or blake3
- --key-location (unchanged) selects key storage: repokey (default) or keyfile
The old combined names were removed (clean break): select a BLAKE3 suite via
--encryption ... --id-hash blake3 instead of blake3-*. aes-ocb was renamed to
aes256-ocb (key NAME shown by repo-info and ARG_NAME in JSON updated to match).
"none" has no key, so it only supports the sha256 id hash.
No on-disk format, key-type byte, or crypto behavior changes: the existing key
classes form a clean cross-product of {cipher} x {id-hash}, selected via the new
ENC_NAME / IDHASH_NAME class attributes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The multi-key feature (#9743) added the "borg key add", "borg key list"
and "borg key remove" subcommands but never wired up their docs:
- scripts/make.py: map key_add/key_list/key_remove to the "key" usage
group, so build_man can locate their examples (it previously aborted
with FileNotFoundError: docs/usage/key_add.rst).
- docs/usage/key.rst: include the three new generated snippets so they
show up on the HTML key page.
- regenerate the affected docs: new key_{add,list,remove}.rst.inc usage
snippets, new borg-key-{add,list,remove}.1 man pages, and borg-key.1
(SYNOPSIS + SEE ALSO now reference the new subcommands).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The authenticated and authenticated-blake3 modes do not encrypt data, but
they still have a real key (id/auth key material) stored as a key blob.
That blob can live as a keyfile or as a repokey just like the encrypted
modes, so make it configurable instead of always forcing repokey storage.
- AuthenticatedKeyBase: set LOCATION_CONFIGURABLE = True so --key-location
(at repo-create) and "borg key change-location" apply.
- key change-location: only copy sessionid/cipher when present (those are
AEAD-only; authenticated keys do not have them).
- repo-info: report the key storage location for authenticated keys too,
and handle the authenticated-blake3 variant (was only "authenticated").
- repo-create help: stop claiming authenticated* has no keyfile/repokey
storage; only "none" truly has no key.
- add change-location round-trip tests for authenticated mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Borg used to read the manifest's key-type byte and then look for the key in
exactly one place (keyfile or repokey) depending on the key class that byte
selected. As a result every crypto suite was duplicated into a keyfile class
and a repokey class that differed only in TYPE, NAME, ARG_NAME and STORAGE.
Now key *location* is independent of the type byte: detection tries keyfiles
first and repokeys afterwards until a passphrase unlocks a key. The type byte
still selects the crypto suite (id hash, MAC, cipher) to instantiate. Where a
key is stored (keyfile vs repokey) is therefore a per-key property
(self.storage), not a separate class, so a repository may even hold a mix of
keyfile- and repo-stored borg keys.
With storage decoupled from class identity, the keyfile/repokey class pairs
collapse into one class per crypto suite:
- modern AEAD: AESOCBKey, CHPOKey, Blake3AESOCBKey, Blake3CHPOKey
- legacy borg 1.x (read-only): AESCTRKey, Blake2AESCTRKey
There is now exactly one type byte per modern crypto suite (the old separate
repokey type bytes 0x11/0x21/0x31/0x41 were removed; borg2 is beta and only
needs to read repos it created). identify_key() matches on TYPES_ACCEPTABLE.
CLI: --encryption selects only the crypto suite (aes-ocb, chacha20-poly1305,
blake3-aes-ocb, blake3-chacha20-poly1305, authenticated*, none); the storage
location is chosen with the new --key-location=repokey|keyfile (default
repokey). The old combined modes (repokey-aes-ocb etc.) were removed.
borg key import also gained --key-location. borg key change-location no longer
swaps key classes or rewrites the manifest; it just re-saves the unlocked key
at the new location.
Keyfile removal (key remove, change-location) now overwrites the keyfile with
random data via secure_erase() before unlinking, consistent with save().
borg 1.x legacy read compatibility is preserved (the legacy class merge is a
behavior-preserving rename; the legacy type bytes incl. PASSPHRASE stay in
TYPES_ACCEPTABLE).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The unix-socket transport (socket:// repositories, the --socket option and
"borg serve" over a socket) was never part of a stable borg 2 release and the
old RPC protocol it relied on is gone, so the remaining code was dead:
- legacy remote: drop the unreachable proto == "socket" connection branch and
the now-unused self.sock handling, "import socket" and get_socket_filename
import (LegacyRemoteRepository is only built for proto == "ssh")
- helpers: remove get_socket_filename() and its export
- parseformat: drop "socket" from local_path_re - socket:// is now treated like
any other unknown scheme (a local path) rather than being special-cased
- tests: drop test_socket and the self.sock check in the legacy reopen helper
- docs: drop the stale --socket entry from the manually maintained
common-options.rst.inc (the auto-generated usage/man docs are left untouched
here and will be rebuilt in a separate commit)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
For sftp/http(s)/s3/b2/rclone repositories, borg only detects the scheme now
and hands the raw URL to borgstore, which parses and validates it - removing
the duplicate parsing borg used to do. Precise field extraction (user/host/
port/path) is kept only for the protocols borg itself reads: file, rest and
legacy ssh.
- drop http_re, s3_re, rclone_re and the sftp arm of the old ssh_or_sftp_re
- add a single scheme-detection pass-through against BORGSTORE_SCHEMES; reject
unknown schemes (e.g. socket://) as before
- canonical_path() returns the processed URL for the delegated protocols, with
embedded credentials stripped so secrets never reach the security state file
or logs
- source local_path_re's scheme exclusions from BORGSTORE_SCHEMES
- create: use proto == "file" instead of "not location.host" for the local
repo-dir inode skip
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
If `borg compact` was interrupted after deleting repository objects but
before writing the updated chunk index, the still-existing cache/chunks.*
kept claiming the deleted objects were present. A later `borg create` would
trust that stale index, skip re-uploading the affected chunks and silently
create an archive with dangling object references that extracts to zero bytes.
Invalidate all cached chunk indexes via delete_chunkindex_cache() before the
first object is deleted, so an interruption is conservative: the next client
rebuilds the index from actual repository contents and re-uploads any deleted
data. The post-deletion save_chunk_index() still writes a fresh, valid index.
Add a regression test covering both compact paths (default and --stats) that
interrupts compaction right before save_chunk_index() and verifies no cached
chunk index survives and a later create+extract reproduces the original bytes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FilesCacheMixin initialized _newest_cmtime to 0, but _write_files_cache()
only treats None as "no file was chunked this run" (falling back to a
max_time_ns cutoff that keeps all current entries).
When a backup reuses all files from the cache without chunking anything,
_newest_cmtime stayed at 0, so the race-protection cutoff became the unix
epoch and every current (age == 0) entry was discarded. The next backup
then had to re-read, chunk and hash all files again.
Initialize _newest_cmtime to None to match the documented contract in
_write_files_cache(), and make the comparisons in _build_files_cache()
None-safe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enable the pdf output format on Read the Docs (the LaTeX build config
already existed in docs/conf.py) and add a "Downloads" line to the left
sidebar that links the offline formats (PDF, HTML zip, ePub). The links
are populated from the Read the Docs addons data, reusing the same
mechanism as the version selector, so they are version-correct and hidden
when unavailable. The line is left-aligned with the boxes above and the
table of contents below, with separators above and below it.
Also drop the stale 'resources' entry from latex_appendices (the page was
removed in #2088); it broke the now-enabled PDF build with a doctree
KeyError.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make `borg serve` able to be the server-side component of a rest:// repository,
selected with a new --rest option. Plain `borg serve` (no option) keeps serving
legacy borg-1.x repos and stays command-line compatible with borg 1.x.
- serve_cmd.py: add --rest and --backend. With --rest, serve the given
--backend FILE:<path> on stdio via borgstore.server.rest.serve(); honor
--restrict-to-path/--restrict-to-repository (validated against the FILE path)
and --permissions (mapped via borg_permissions). Without --rest, run the legacy
RepositoryServer as before.
- repository.py: for rest:// locations, build the borgstore REST backend with a
command that runs `borg serve --rest --backend FILE:<path>` (locally via
sys.executable, or over ssh reusing borgstore's ssh_cmd / BORG_REMOTE_PATH),
instead of borgstore's hardcoded `borgstore-server-rest`. So a remote only needs
borg installed. Extracted the permissions string->dict mapping into the reusable
borg_permissions().
- tests: unit tests for the rest serve command builder. The existing
remote_archiver (rest:///) suite now runs against `borg serve --rest`.
- docs: changelog + quickstart updated.
Legacy serve and the legacy ssh client are unchanged (client still spawns plain
`borg serve`).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The modern client/server transport (RemoteRepository served by `borg serve`
over an msgpack RPC protocol) is now redundant for current (borg 2) repos:
its functionality is replaced by rest:// (which can tunnel over ssh to a
remote borgstore REST server).
Remove the modern RemoteRepository (both ssh:// and socket://) entirely.
Legacy v1 (borg 1.x) repos remain reachable over ssh:// via the separate
LegacyRemoteRepository client, and `borg serve` / RepositoryServer is kept,
trimmed to the legacy-only path, so a remote borg2 can still serve a v1 repo
for `borg transfer --from-borg1`.
Details:
- remote.py: delete RemoteRepository, SleepingBandwidthLimiter and the `api`
decorator; trim RepositoryServer to legacy-only (drop modern _rpc_methods,
socket serving, non-legacy open() branch); keep cache_if_remote /
RepositoryCache / RepositoryNoCache (used by all repos).
- get_repository(): non-legacy ssh:// now raises a clear "use rest://" error;
socket:// route and the global --socket option removed.
- parseformat: drop the socket:// scheme (now an invalid location).
- borg serve: keep the command (serves legacy v1 ssh only); update epilog.
- borg version: drop modern remote query; keep legacy ssh path.
- update isinstance/import sites (cache, archive, fuse/hlfuse, analyze/compact,
archiver __init__ -> LegacyRemoteRepository.RPCError).
- tests/docs updated; obsolete socket serve test removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BLAKE3 is generally faster and provides a more modern construction for
keyed hashing (using its internal keyed mode instead of the construction
used for BLAKE2b).
Key types changed:
- authenticated-blake2 -> authenticated-blake3
- {keyfile,repokey}-blake2-aes-ocb -> {keyfile,repokey}-blake3-aes-ocb
- {keyfile,repokey}-blake2-chacha20-poly1305 -> {keyfile,repokey}-blake3-chacha20-poly1305
This also fixes the slightly unusual way how we used blake2b,
it is only supported for importing borg 1.x repos.
New repos either use HMAC-SHA256 or BLAKE3.
