test_create_changed_file_retry_rolls_back_chunks and
test_create_erroneous_file_read_retry_rolls_back_chunks open a local
Cache/Repository to inspect chunk refcounts. In RemoteArchiverTestCase the
repo was created via a remote (__testsuite__:) location, so opening it
locally triggers the "repository was previously located at ssh://..."
relocation prompt, which reads stdin and fails under pytest.
These checks are inherently local (they poke at cache internals), so skip
them in the remote variant, like test_debug_put_get_delete_obj already does.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The changed-file retry path already rolls back the chunks it added before
re-reading. Do the same when a read error (BackupOSError) interrupts reading
a file: the chunks added since the last checkpoint (item.chunks[from_chunk:])
are not referenced by any committed part item, so without rolling them back
they leak as an inflated chunk refcount (or, if content also shifted, an
orphan chunk).
The rollback lives in process_file_chunks, where from_chunk is known: chunks
before from_chunk are referenced by part files we already wrote and must be
kept; only the uncommitted tail is decref'd.
Adds test_create_erroneous_file_read_retry_rolls_back_chunks: the re-read
chunks dedup to the same ids, so the leak shows up as a too-high refcount
(not an orphan id) - the test checks refcounts directly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Backport of #7351 to 1.4-maint. Retry reading a backup source file when
it raises a BackupError / BackupOSError (e.g. transient I/O error, or the
file changed while we read it):
- retry the same file after some sleep time (1ms, growing exponentially up to 10s, 10 tries)
- BackupOSError: if retries do not help, skip the file, log it with "E" status
- BackupError (changed while reading): last try backs it up, logged with "C" status
Works for borg create's builtin fs recursion, --paths-from-command and --paths-from-stdin.
1.4-specific: do NOT retry a file once checkpoint part files (.borg_part_N)
have been written for it. Re-reading it from the start would create duplicate /
inconsistent part files (concatenating all part files would then no longer yield
the complete file). ChunksProcessor.last_part_number tracks this; such a file is
kept as-is ("C") or skipped ("E") instead of being retried.
Permission errors (EPERM/EACCES) are not retried (they won't get better),
so we don't waste ~15s of backoff on every unreadable file.
Adds a ChunkerFailing test chunker (chunker-params=fail,block_size,map) to drive
the error handling, and tests:
- test_create_erroneous_file: file is read on a later retry
- test_create_no_permission_file: permission error is not retried, file skipped
- test_create_erroneous_file_with_part_files: file with part files is not retried
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
process_file() ran process_file_chunks() inside `with backup_io('read')`.
That block is meant to guard reading the *source* file, but the source reads
are already guarded individually by backup_io_iter(). The outer wrapper also
caught add_chunk()'s (and maybe_checkpoint()'s) *repository* writes, so a
repository IO failure -- e.g. the repo running out of space -- was wrapped
into a per-file BackupOSError tagged "read". Borg then emitted misleading
"<path>: read: [Errno 28] No space left on device" warnings, pointlessly
retried the file, and only treated a critical repository error as a
non-critical per-file one, contrary to the BackupOSError docstring ("Any
unwrapped IO error is critical and aborts execution (for example repository
IO failure)").
In 1.4 the transactional repository still rolls the partial transaction back
on the eventual failure, so this is not data loss here (unlike borg2, where
it silently commits a corrupt archive). But the misclassification -- wrong
warning text and needless read-retries of a repository-full condition -- is
wrong regardless.
Drop the outer backup_io('read') wrapper. Source reads stay per-file
warnings (backup_io_iter is unchanged); repository OSErrors are now left
unwrapped and critical, aborting promptly with the correct error.
Verified on a space-limited macOS ramdisk: before, create emitted many
"read: [Errno 28]" retry warnings then rolled back; after, it aborts
immediately ("No space left on device, cleaning up partial transaction"),
commits no archive, and the repo stays consistent. Normal backups and
unreadable-source-file handling (per-file warning) are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When set, these override the hostname/username that is stored in newly
created archives and that is used for the {hostname}/{user} placeholders
(e.g. in archive names, prune --glob-archives, check). Useful to run borg
on host A but impersonate host B.
fqdn/hostid and the auto-detection are intentionally left untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
msgpack does not support reusing an Unpacker instance after an unpacking
failure - a new instance must be created instead.
In the forced-delete path, item metadata was unpacked via "for item in
unpacker" with an "except (TypeError, ValueError)" handler. msgpack's
FormatError/StackError are ValueError subclasses, so a real unpacking
failure was caught there and the (now broken) unpacker kept being reused
for the following chunks, making them all fail too - leaking their chunk
references.
Restructure the loop to use next(unpacker) so each failure mode gets a
tight handler:
- StopIteration: buffer drained, feed next chunk
- msgpack.UnpackException: corruption; in forced mode replace the unpacker
with a fresh instance and skip the rest of the chunk
- TypeError/ValueError: only wraps Item processing (bad types yielded)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ProgressIndicatorBase installs a handler on the process-global
'borg.output.progress' logger and only removes it in __del__. In-process
(fork=False) command execution leaves this logger at level INFO /
propagate=False with lingering handlers, and that state leaks across tests.
When a later progress test runs, the "if not self.logger.handlers" guard
skips the stderr handler setup, so progress output goes to the logging
system instead of the stderr captured by capfd. Under xdist this surfaces
as order/distribution-dependent failures of the progress tests
(test_progress_percentage_*, test_extract_progress, test_progress_on,
test_check_usage, ...).
Add an autouse fixture that removes handlers and resets level/propagate on
the progress logger after each test. This also makes the intra-test
workaround in test_check_usage unnecessary (the progress logger level is
set per command run by _setup_implied_logging), so remove it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
macOS flags files whose content is stored in cloud storage (e.g. iCloud
Drive) and not present locally with SF_DATALESS. Reading such a file
triggers downloading its content.
With --exclude-dataless, borg create checks the flags (cheap, after
stat, before open) and skips such files/dirs, so backups do not force
a download of all cloud-stored data.
stat.SF_DATALESS is only available from Python 3.13 on, so we fall
back to the value from macOS' sys/stat.h on older Pythons.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Style the sidebar downloads line to align with the boxes above and the
table of contents below (padding 22px), with a horizontal separator both
between the search box and the downloads line and between the downloads
line and the table of contents.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the multi-line "Download docs" block with a compact single line
"Downloads: PDF | HTML | ePub" placed right below the search box.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The resources doc page was removed long ago (#2088), but it was still
referenced in latex_appendices. This went unnoticed because PDF builds
were not enabled; enabling pdf surfaced it as a doctree KeyError.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enable the pdf format on Read the Docs (the LaTeX build config already
existed in conf.py) and add a "Download docs" section to the sidebar that
links the offline formats (PDF, HTML zip, ...). The links are populated
from the Read the Docs addons data, reusing the same mechanism as the
version selector, so they are version-correct and hidden when unavailable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- update to 'borg key change-passphrase' in env help
- remove outdated borg-change-passphrase man page from 2017
- remove outdated change-passphrase.rst.inc
