The multi-key feature (#9743) added the "borg key add", "borg key list"
and "borg key remove" subcommands but never wired up their docs:
- scripts/make.py: map key_add/key_list/key_remove to the "key" usage
group, so build_man can locate their examples (it previously aborted
with FileNotFoundError: docs/usage/key_add.rst).
- docs/usage/key.rst: include the three new generated snippets so they
show up on the HTML key page.
- regenerate the affected docs: new key_{add,list,remove}.rst.inc usage
snippets, new borg-key-{add,list,remove}.1 man pages, and borg-key.1
(SYNOPSIS + SEE ALSO now reference the new subcommands).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The authenticated and authenticated-blake3 modes do not encrypt data, but
they still have a real key (id/auth key material) stored as a key blob.
That blob can live as a keyfile or as a repokey just like the encrypted
modes, so make it configurable instead of always forcing repokey storage.
- AuthenticatedKeyBase: set LOCATION_CONFIGURABLE = True so --key-location
(at repo-create) and "borg key change-location" apply.
- key change-location: only copy sessionid/cipher when present (those are
AEAD-only; authenticated keys do not have them).
- repo-info: report the key storage location for authenticated keys too,
and handle the authenticated-blake3 variant (was only "authenticated").
- repo-create help: stop claiming authenticated* has no keyfile/repokey
storage; only "none" truly has no key.
- add change-location round-trip tests for authenticated mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PackWriter requires a real/explicit index (no silent private one); PackLocationUnknown is now a loud ErrorWithTraceback and get() raises it unconditionally; close() persists via is_chunk_index_loaded/self.chunks; delete_chunkindex_cache calls invalidate_chunk_index() directly; drop per-access fragment consolidation (compact reclaims instead); capture BORG_TESTONLY_SHA256_PACK_ID at import; remove dead _current_offset; update tests.
Borg used to read the manifest's key-type byte and then look for the key in
exactly one place (keyfile or repokey) depending on the key class that byte
selected. As a result every crypto suite was duplicated into a keyfile class
and a repokey class that differed only in TYPE, NAME, ARG_NAME and STORAGE.
Now key *location* is independent of the type byte: detection tries keyfiles
first and repokeys afterwards until a passphrase unlocks a key. The type byte
still selects the crypto suite (id hash, MAC, cipher) to instantiate. Where a
key is stored (keyfile vs repokey) is therefore a per-key property
(self.storage), not a separate class, so a repository may even hold a mix of
keyfile- and repo-stored borg keys.
With storage decoupled from class identity, the keyfile/repokey class pairs
collapse into one class per crypto suite:
- modern AEAD: AESOCBKey, CHPOKey, Blake3AESOCBKey, Blake3CHPOKey
- legacy borg 1.x (read-only): AESCTRKey, Blake2AESCTRKey
There is now exactly one type byte per modern crypto suite (the old separate
repokey type bytes 0x11/0x21/0x31/0x41 were removed; borg2 is beta and only
needs to read repos it created). identify_key() matches on TYPES_ACCEPTABLE.
CLI: --encryption selects only the crypto suite (aes-ocb, chacha20-poly1305,
blake3-aes-ocb, blake3-chacha20-poly1305, authenticated*, none); the storage
location is chosen with the new --key-location=repokey|keyfile (default
repokey). The old combined modes (repokey-aes-ocb etc.) were removed.
borg key import also gained --key-location. borg key change-location no longer
swaps key classes or rewrites the manifest; it just re-saves the unlocked key
at the new location.
Keyfile removal (key remove, change-location) now overwrites the keyfile with
random data via secure_erase() before unlinking, consistent with save().
borg 1.x legacy read compatibility is preserved (the legacy class merge is a
behavior-preserving rename; the legacy type bytes incl. PASSPHRASE stay in
TYPES_ACCEPTABLE).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A repository can now have multiple borg keys protected by individual
(usually different) passphrases, each borg key is stored separately.
Any of the keys can be used to decrypt the repository, all contain
the same secret key material.
Keys now have a LABEL (first key always has "admin" label) for easier
identification. The "admin" key is protected and can't be removed.
Repository now stores N borg keys under keys/ instead of one.
LegacyRepository stays single-key.
To find the right key, borg tries a passphrase against every borg key.
CLI:
- borg key list
- borg key add --label
- borg key remove --label|--key|--passphrase
- borg key export --label|--key
Reading borg 1.x keyfiles and repokey (borg transfer --from-borg1)
keeps working: the legacy key classes share FlexiKey, so enumeration
falls back to load_key() and headerless 1.x repokey blobs still parse.
The keyblob on-disk format is unchanged; multiplicity lives only in the
storage/enumeration layer.
tests: adapt test_repo_key_detect_does_not_raise_integrity_error
Updated to the multiple-borg-keys API: store_key.call_args for the saved blob and
load_keys.return_value to feed it back to detect().
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a test run is aborted (Ctrl-C, a timeout, or a crashed xdist worker),
the fuse_mount context manager's `finally: umount` does not run, leaving a
live FUSE mount under the pytest temp dir. A later run then fails to rm_rf
the old temp tree (Resource busy / Device not configured / Read-only file
system), which surfaces as noisy pytest warnings.
Add pytest_sessionstart/sessionfinish hooks (run once on the xdist
controller) that walk the pytest temp tree for leftover "mountpoint"
directories and unmount them via borg's own cross-platform umount helper,
making runs self-healing across previously aborted sessions.
Discovery uses only the stdlib (os.walk + os.path.ismount, with an
os.listdir fallback for dead/stale mounts) and the mount is named by the
well-known on-disk layout, so there is no OS-specific mount-table parsing or
force-unmount handling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PackWriter and the cache now share repository.chunks instead of keeping their own copies; set_chunk_index is gone (use the writable .chunks property). The cache consolidates fragmented cache/chunks.* files so they no longer grow by one per backup until the next check or compact.
delete_chunkindex_cache() now also drops the repository's in-memory index, so
close() cannot persist a stale copy and callers no longer clear it by hand.
get() now raises a distinct PackLocationUnknown when a chunk is indexed but not
yet flushed, rather than a plain ObjectNotFound.
finish() deletes cache/chunks.* and now also invalidates the live index, so close() does not write it back. The next repository access rebuilds it from actual repository contents.
close() now writes the in-memory index back so the next session can resolve chunk locations instead of failing with ObjectNotFound. Empty incremental writes are skipped so a deleted index is not recreated as a stale, empty one.
Replace _pack_info (session-scoped dict) with a borrowed ChunkIndex reference.
Cache passes its index via set_chunk_index(); get() routes correctly for all sessions.
Ports #9755 (1.4-maint) to master.
macOS flags files whose content lives in cloud storage (e.g. iCloud Drive)
and is not present locally with SF_DATALESS. Reading such a file triggers
downloading its content.
This adds --exclude-dataless to borg create: the flags are checked right
after stat() and before any open(), so excluded files/dirs never get
materialized. Skipped paths are reported with the usual 'x' status,
analogous to --exclude-nodump.
Notes:
- stat.SF_DATALESS only exists from Python 3.13 on, so there is a fallback
to the value from macOS' sys/stat.h (0x40000000).
- The check covers directories too (they can also be dataless), and we skip
before opening/recursing into them.
- Test fakes get_flags via monkeypatch since SF_DATALESS cannot be set from
userspace.
- Fish shell completion updated; bash/zsh completions don't exist in master.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The unix-socket transport (socket:// repositories, the --socket option and
"borg serve" over a socket) was never part of a stable borg 2 release and the
old RPC protocol it relied on is gone, so the remaining code was dead:
- legacy remote: drop the unreachable proto == "socket" connection branch and
the now-unused self.sock handling, "import socket" and get_socket_filename
import (LegacyRemoteRepository is only built for proto == "ssh")
- helpers: remove get_socket_filename() and its export
- parseformat: drop "socket" from local_path_re - socket:// is now treated like
any other unknown scheme (a local path) rather than being special-cased
- tests: drop test_socket and the self.sock check in the legacy reopen helper
- docs: drop the stale --socket entry from the manually maintained
common-options.rst.inc (the auto-generated usage/man docs are left untouched
here and will be rebuilt in a separate commit)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
For sftp/http(s)/s3/b2/rclone repositories, borg only detects the scheme now
and hands the raw URL to borgstore, which parses and validates it - removing
the duplicate parsing borg used to do. Precise field extraction (user/host/
port/path) is kept only for the protocols borg itself reads: file, rest and
legacy ssh.
- drop http_re, s3_re, rclone_re and the sftp arm of the old ssh_or_sftp_re
- add a single scheme-detection pass-through against BORGSTORE_SCHEMES; reject
unknown schemes (e.g. socket://) as before
- canonical_path() returns the processed URL for the delegated protocols, with
embedded credentials stripped so secrets never reach the security state file
or logs
- source local_path_re's scheme exclusions from BORGSTORE_SCHEMES
- create: use proto == "file" instead of "not location.host" for the local
repo-dir inode skip
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
If `borg compact` was interrupted after deleting repository objects but
before writing the updated chunk index, the still-existing cache/chunks.*
kept claiming the deleted objects were present. A later `borg create` would
trust that stale index, skip re-uploading the affected chunks and silently
create an archive with dangling object references that extracts to zero bytes.
Invalidate all cached chunk indexes via delete_chunkindex_cache() before the
first object is deleted, so an interruption is conservative: the next client
rebuilds the index from actual repository contents and re-uploads any deleted
data. The post-deletion save_chunk_index() still writes a fresh, valid index.
Add a regression test covering both compact paths (default and --stats) that
interrupts compaction right before save_chunk_index() and verifies no cached
chunk index survives and a later create+extract reproduces the original bytes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FilesCacheMixin initialized _newest_cmtime to 0, but _write_files_cache()
only treats None as "no file was chunked this run" (falling back to a
max_time_ns cutoff that keeps all current entries).
When a backup reuses all files from the cache without chunking anything,
_newest_cmtime stayed at 0, so the race-protection cutoff became the unix
epoch and every current (age == 0) entry was discarded. The next backup
then had to re-read, chunk and hash all files again.
Initialize _newest_cmtime to None to match the documented contract in
_write_files_cache(), and make the comparisons in _build_files_cache()
None-safe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In modern borg these were just a pass-through repository wrapper (there
is no RepositoryCache), with one variant doing inline decryption and
returning (csize, plaintext) tuples. Drop both and make all consumers
use the raw repository directly:
- fuse.py: ItemCache / FuseOperations / FuseBackend now take the raw
repository + repo_objs and decrypt via repo_objs.parse(ROBJ_DONTCARE),
matching hlfuse.py. The csize value was discarded at both call sites.
- mount_cmds.py: drop the cache_if_remote wrapper around FuseOperations.
- archive.py (rebuild_archives / check): drop the pass-through wrapper;
robust_iterator now uses self.repository directly.
- repository.py: delete the RepositoryNoCache class and cache_if_remote.
- repository_test.py: remove TestCacheIfRemote and orphaned imports.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Preloading was used only by extract and export-tar, and the modern
borgstore-based Repository.preload() was already a no-op. Remove the
preload calls from both commands and the now-dead supporting code:
- DownloadPipeline.preload_item_chunks / Archive.preload_item_chunks
- hlids_preloaded tracking
- is_preloaded parameter from DownloadPipeline.fetch_many and
Repository.get_many
- the no-op Repository.preload()
Also: remove preload support from borg.legacy
With no remaining callers, drop the legacy-side preload machinery:
LegacyRemoteRepository:
- preload_ids / chunkid_to_msgids state and the pop_preload_msgid helper
- is_preloaded parameter and handling in call_many() (get requests now
always go through the normal send path; MAX_INFLIGHT pipelining of
regular calls is unchanged)
- is_preloaded from get_many() and the preload() method
LegacyRepository:
- is_preloaded from get_many() and the no-op preload() stub
Both legacy repo classes now match the modern Repository interface.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On omniOS the test data now lives on ZFS (via TMPDIR=/var/tmp), and ZFS
rejects the year-2261 os.utime() with EOVERFLOW. Treat that as an
unsupported-filesystem condition and skip, rather than failing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With TMPDIR=/var/tmp/borg-ci on omniOS, platformdirs' user_runtime_dir
fallback yields /var/tmp/borg-ci/runtime-<uid>/borg. Add it to the
accepted values, like the existing NetBSD CI entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On omniOS /tmp is swap-backed tmpfs (small, RAM-bound), so the pip/cargo
build temps and the pytest temp tree quickly exhaust it ("no space left on
device"). Point TMPDIR at disk-backed /var/tmp instead, mirroring what the
NetBSD job already does.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix PackWriter.flush() to use max_count == 1 (not len == 1) for the pack_id hack,
so final partial packs under max_count > 1 correctly use SHA256. Add covering test.
Move sha256 import to module level in repository_test.
PackWriter buffers (chunk_id, cdata) pairs and flushes as pack files via borgstore.
At N=1 pack_id == chunk_id; UNKNOWN_INT32 (0xFFFFFFFF) placeholders in the index
are replaced by real pack location fields after flush() via update_pack_info().
Update test_chunkindex_add to expect UNKNOWN_INT32 sentinels from add().
Enable the pdf output format on Read the Docs (the LaTeX build config
already existed in docs/conf.py) and add a "Downloads" line to the left
sidebar that links the offline formats (PDF, HTML zip, ePub). The links
are populated from the Read the Docs addons data, reusing the same
mechanism as the version selector, so they are version-correct and hidden
when unavailable. The line is left-aligned with the boxes above and the
table of contents below, with separators above and below it.
Also drop the stale 'resources' entry from latex_appendices (the page was
removed in #2088); it broke the now-enabled PDF build with a doctree
KeyError.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RepositoryServer stored self.permissions but never read it: open() builds
a LegacyRepository without any permissions, and legacy (borg 1.x / v1)
repositories have no permission system at all. Remove the dead __init__
parameter and stop forwarding args.permissions from do_serve.
The --permissions CLI option stays - it applies to the non-legacy
"borg serve --rest" path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remote access is restricted via an SSH forced command in authorized_keys
that hardcodes the restriction, e.g.
command="borg serve --rest --restrict-to-path=/srv/repos"
get_args() merges the forced command with the client's intended command
(SSH_ORIGINAL_COMMAND), copying only allowlisted options from the client.
For legacy serve the repo path travels inside the RPC protocol, so the
server enforces restrictions against it. But a rest:// repo passes the
repo as "--backend FILE:<path>" on the command line, and "backend" was in
neither allow- nor denylist, so under a forced command the client's
--backend was dropped: args.backend ended up None and do_serve_rest failed
with "requires --backend" - restrictions for rest were effectively broken.
Add "backend" to the allowlist so the client chooses which repo while the
forced command pins the restriction and the rest mode; do_serve_rest then
validates the client backend against restrict_to_paths/repositories via
check_rest_restrictions. The --rest mode flag stays out of the allowlist
so a forced legacy serve cannot be flipped to rest by the client.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
