Convert if/elif chains that dispatch on a single value to match
statements: repository.borg_permissions (permission preset),
CompressionSpec.__init__/compressor/__str__ (compression name),
calculate_relative_offset (relative ts unit) and the cockpit widget
file-status counter. Membership checks like 'in ("none", "lz4")'
become alternative patterns ('none' | 'lz4').
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the remaining typing.Optional[...] annotations with the
X | None syntax (PEP 604, Python 3.10) and drop the now-unused
Optional/List imports; List[str] -> list[str] in cockpit/runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the hand-rolled changedir() context manager with stdlib
contextlib.chdir (kept under the changedir name for importers), and
use it in the archiver fixture so the cwd is restored even if the
test body raises.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Manifest.Operation now derives from enum.StrEnum, so its members are
real str instances; drop the .value indirection in the feature-flag
lookups.
Replace timezone.utc with the datetime.UTC alias (3.11) across the
non-test modules and drop the now-unused timezone imports.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
flush() drops the entries add() pre-marked when store.store() raises, so a never-stored chunk is not left indexed for dedup. adds rollback tests, bumps large-meta test to 5KB, drops security wording from the read_data=False clamp comments.
Several check_cmd tests corrupted a repo object by overwriting a byte at
a fixed position with a fixed value, e.g.:
manifest[:250] + b"x" + manifest[251:]
Manifests/chunks are stored as AEAD-encrypted repo objects, so their
bytes are ~random. When the target byte already happened to hold the
overwrite value (~1/256), the "corruption" was a no-op: the object
stayed valid, "check" returned 0 instead of 1, and the test failed
intermittently (observed in test_spoofed_archive).
Introduce a corrupt(data, position) helper that flips the byte (XOR
0xFF), so the result is guaranteed to differ, and use it in all the
byte-overwrite corruption sites: test_corrupted_manifest,
test_spoofed_manifest, test_manifest_rebuild_corrupted_chunk,
test_spoofed_archive, test_verify_data and test_corrupted_file_chunk.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Python 3.15-dev is not compatible with some of our requirements yet,
so use 3.14 (py314-mfusepy) for the canary matrix entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The multi-key feature (#9743) added the "borg key add", "borg key list"
and "borg key remove" subcommands but never wired up their docs:
- scripts/make.py: map key_add/key_list/key_remove to the "key" usage
group, so build_man can locate their examples (it previously aborted
with FileNotFoundError: docs/usage/key_add.rst).
- docs/usage/key.rst: include the three new generated snippets so they
show up on the HTML key page.
- regenerate the affected docs: new key_{add,list,remove}.rst.inc usage
snippets, new borg-key-{add,list,remove}.1 man pages, and borg-key.1
(SYNOPSIS + SEE ALSO now reference the new subcommands).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The authenticated and authenticated-blake3 modes do not encrypt data, but
they still have a real key (id/auth key material) stored as a key blob.
That blob can live as a keyfile or as a repokey just like the encrypted
modes, so make it configurable instead of always forcing repokey storage.
- AuthenticatedKeyBase: set LOCATION_CONFIGURABLE = True so --key-location
(at repo-create) and "borg key change-location" apply.
- key change-location: only copy sessionid/cipher when present (those are
AEAD-only; authenticated keys do not have them).
- repo-info: report the key storage location for authenticated keys too,
and handle the authenticated-blake3 variant (was only "authenticated").
- repo-create help: stop claiming authenticated* has no keyfile/repokey
storage; only "none" truly has no key.
- add change-location round-trip tests for authenticated mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PackWriter requires a real/explicit index (no silent private one); PackLocationUnknown is now a loud ErrorWithTraceback and get() raises it unconditionally; close() persists via is_chunk_index_loaded/self.chunks; delete_chunkindex_cache calls invalidate_chunk_index() directly; drop per-access fragment consolidation (compact reclaims instead); capture BORG_TESTONLY_SHA256_PACK_ID at import; remove dead _current_offset; update tests.
Borg used to read the manifest's key-type byte and then look for the key in
exactly one place (keyfile or repokey) depending on the key class that byte
selected. As a result every crypto suite was duplicated into a keyfile class
and a repokey class that differed only in TYPE, NAME, ARG_NAME and STORAGE.
Now key *location* is independent of the type byte: detection tries keyfiles
first and repokeys afterwards until a passphrase unlocks a key. The type byte
still selects the crypto suite (id hash, MAC, cipher) to instantiate. Where a
key is stored (keyfile vs repokey) is therefore a per-key property
(self.storage), not a separate class, so a repository may even hold a mix of
keyfile- and repo-stored borg keys.
With storage decoupled from class identity, the keyfile/repokey class pairs
collapse into one class per crypto suite:
- modern AEAD: AESOCBKey, CHPOKey, Blake3AESOCBKey, Blake3CHPOKey
- legacy borg 1.x (read-only): AESCTRKey, Blake2AESCTRKey
There is now exactly one type byte per modern crypto suite (the old separate
repokey type bytes 0x11/0x21/0x31/0x41 were removed; borg2 is beta and only
needs to read repos it created). identify_key() matches on TYPES_ACCEPTABLE.
CLI: --encryption selects only the crypto suite (aes-ocb, chacha20-poly1305,
blake3-aes-ocb, blake3-chacha20-poly1305, authenticated*, none); the storage
location is chosen with the new --key-location=repokey|keyfile (default
repokey). The old combined modes (repokey-aes-ocb etc.) were removed.
borg key import also gained --key-location. borg key change-location no longer
swaps key classes or rewrites the manifest; it just re-saves the unlocked key
at the new location.
Keyfile removal (key remove, change-location) now overwrites the keyfile with
random data via secure_erase() before unlinking, consistent with save().
borg 1.x legacy read compatibility is preserved (the legacy class merge is a
behavior-preserving rename; the legacy type bytes incl. PASSPHRASE stay in
TYPES_ACCEPTABLE).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A repository can now have multiple borg keys protected by individual
(usually different) passphrases, each borg key is stored separately.
Any of the keys can be used to decrypt the repository, all contain
the same secret key material.
Keys now have a LABEL (first key always has "admin" label) for easier
identification. The "admin" key is protected and can't be removed.
Repository now stores N borg keys under keys/ instead of one.
LegacyRepository stays single-key.
To find the right key, borg tries a passphrase against every borg key.
CLI:
- borg key list
- borg key add --label
- borg key remove --label|--key|--passphrase
- borg key export --label|--key
Reading borg 1.x keyfiles and repokey (borg transfer --from-borg1)
keeps working: the legacy key classes share FlexiKey, so enumeration
falls back to load_key() and headerless 1.x repokey blobs still parse.
The keyblob on-disk format is unchanged; multiplicity lives only in the
storage/enumeration layer.
tests: adapt test_repo_key_detect_does_not_raise_integrity_error
Updated to the multiple-borg-keys API: store_key.call_args for the saved blob and
load_keys.return_value to feed it back to detect().
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a test run is aborted (Ctrl-C, a timeout, or a crashed xdist worker),
the fuse_mount context manager's `finally: umount` does not run, leaving a
live FUSE mount under the pytest temp dir. A later run then fails to rm_rf
the old temp tree (Resource busy / Device not configured / Read-only file
system), which surfaces as noisy pytest warnings.
Add pytest_sessionstart/sessionfinish hooks (run once on the xdist
controller) that walk the pytest temp tree for leftover "mountpoint"
directories and unmount them via borg's own cross-platform umount helper,
making runs self-healing across previously aborted sessions.
Discovery uses only the stdlib (os.walk + os.path.ismount, with an
os.listdir fallback for dead/stale mounts) and the mount is named by the
well-known on-disk layout, so there is no OS-specific mount-table parsing or
force-unmount handling.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PackWriter and the cache now share repository.chunks instead of keeping their own copies; set_chunk_index is gone (use the writable .chunks property). The cache consolidates fragmented cache/chunks.* files so they no longer grow by one per backup until the next check or compact.
delete_chunkindex_cache() now also drops the repository's in-memory index, so
close() cannot persist a stale copy and callers no longer clear it by hand.
get() now raises a distinct PackLocationUnknown when a chunk is indexed but not
yet flushed, rather than a plain ObjectNotFound.
finish() deletes cache/chunks.* and now also invalidates the live index, so close() does not write it back. The next repository access rebuilds it from actual repository contents.
close() now writes the in-memory index back so the next session can resolve chunk locations instead of failing with ObjectNotFound. Empty incremental writes are skipped so a deleted index is not recreated as a stale, empty one.
Replace _pack_info (session-scoped dict) with a borrowed ChunkIndex reference.
Cache passes its index via set_chunk_index(); get() routes correctly for all sessions.
Ports #9755 (1.4-maint) to master.
macOS flags files whose content lives in cloud storage (e.g. iCloud Drive)
and is not present locally with SF_DATALESS. Reading such a file triggers
downloading its content.
This adds --exclude-dataless to borg create: the flags are checked right
after stat() and before any open(), so excluded files/dirs never get
materialized. Skipped paths are reported with the usual 'x' status,
analogous to --exclude-nodump.
Notes:
- stat.SF_DATALESS only exists from Python 3.13 on, so there is a fallback
to the value from macOS' sys/stat.h (0x40000000).
- The check covers directories too (they can also be dataless), and we skip
before opening/recursing into them.
- Test fakes get_flags via monkeypatch since SF_DATALESS cannot be set from
userspace.
- Fish shell completion updated; bash/zsh completions don't exist in master.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The unix-socket transport (socket:// repositories, the --socket option and
"borg serve" over a socket) was never part of a stable borg 2 release and the
old RPC protocol it relied on is gone, so the remaining code was dead:
- legacy remote: drop the unreachable proto == "socket" connection branch and
the now-unused self.sock handling, "import socket" and get_socket_filename
import (LegacyRemoteRepository is only built for proto == "ssh")
- helpers: remove get_socket_filename() and its export
- parseformat: drop "socket" from local_path_re - socket:// is now treated like
any other unknown scheme (a local path) rather than being special-cased
- tests: drop test_socket and the self.sock check in the legacy reopen helper
- docs: drop the stale --socket entry from the manually maintained
common-options.rst.inc (the auto-generated usage/man docs are left untouched
here and will be rebuilt in a separate commit)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
For sftp/http(s)/s3/b2/rclone repositories, borg only detects the scheme now
and hands the raw URL to borgstore, which parses and validates it - removing
the duplicate parsing borg used to do. Precise field extraction (user/host/
port/path) is kept only for the protocols borg itself reads: file, rest and
legacy ssh.
- drop http_re, s3_re, rclone_re and the sftp arm of the old ssh_or_sftp_re
- add a single scheme-detection pass-through against BORGSTORE_SCHEMES; reject
unknown schemes (e.g. socket://) as before
- canonical_path() returns the processed URL for the delegated protocols, with
embedded credentials stripped so secrets never reach the security state file
or logs
- source local_path_re's scheme exclusions from BORGSTORE_SCHEMES
- create: use proto == "file" instead of "not location.host" for the local
repo-dir inode skip
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
If `borg compact` was interrupted after deleting repository objects but
before writing the updated chunk index, the still-existing cache/chunks.*
kept claiming the deleted objects were present. A later `borg create` would
trust that stale index, skip re-uploading the affected chunks and silently
create an archive with dangling object references that extracts to zero bytes.
Invalidate all cached chunk indexes via delete_chunkindex_cache() before the
first object is deleted, so an interruption is conservative: the next client
rebuilds the index from actual repository contents and re-uploads any deleted
data. The post-deletion save_chunk_index() still writes a fresh, valid index.
Add a regression test covering both compact paths (default and --stats) that
interrupts compaction right before save_chunk_index() and verifies no cached
chunk index survives and a later create+extract reproduces the original bytes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
FilesCacheMixin initialized _newest_cmtime to 0, but _write_files_cache()
only treats None as "no file was chunked this run" (falling back to a
max_time_ns cutoff that keeps all current entries).
When a backup reuses all files from the cache without chunking anything,
_newest_cmtime stayed at 0, so the race-protection cutoff became the unix
epoch and every current (age == 0) entry was discarded. The next backup
then had to re-read, chunk and hash all files again.
Initialize _newest_cmtime to None to match the documented contract in
_write_files_cache(), and make the comparisons in _build_files_cache()
None-safe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In modern borg these were just a pass-through repository wrapper (there
is no RepositoryCache), with one variant doing inline decryption and
returning (csize, plaintext) tuples. Drop both and make all consumers
use the raw repository directly:
- fuse.py: ItemCache / FuseOperations / FuseBackend now take the raw
repository + repo_objs and decrypt via repo_objs.parse(ROBJ_DONTCARE),
matching hlfuse.py. The csize value was discarded at both call sites.
- mount_cmds.py: drop the cache_if_remote wrapper around FuseOperations.
- archive.py (rebuild_archives / check): drop the pass-through wrapper;
robust_iterator now uses self.repository directly.
- repository.py: delete the RepositoryNoCache class and cache_if_remote.
- repository_test.py: remove TestCacheIfRemote and orphaned imports.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
