process_file() ran process_file_chunks() inside `with backup_io("read")`.
That block was meant to guard reading the *source* file, but the source
reads are already guarded individually by backup_io_iter(). The outer
wrapper additionally caught add_chunk()'s *repository* writes, so a critical
repository IO failure -- e.g. the repo running out of space during a pack
flush -- was wrapped into a per-file BackupOSError. Borg then only warned,
skipped the file, and continued, and create_inner() still committed the
archive via archive.save().
The result: `borg create` on an out-of-space repo printed a normal success
summary ("Error files: 0"), exited 0, and committed an archive that
references chunks which were never durably stored. `borg check` afterwards
reports "Missing file chunk detected" and `borg compact` reports "Repository
has N missing objects!" -- silent, unrestorable-backup data loss.
Drop the outer backup_io("read") wrapper. Source reads stay per-file
warnings (backup_io_iter is unchanged); repository OSErrors are now left
unwrapped and therefore critical, aborting create before archive.save()
runs, exactly as the BackupOSError docstring prescribes.
Reproduced on a space-limited macOS ramdisk (source > free space): before,
create exited 0 with a corrupt archive; after, create fails and commits
nothing, and the repository stays consistent across a create/delete/compact
churn matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
compact_pack() updated repository.chunks, but compact computes keep/drop from
and persists its own index, which kept pointing at the deleted packs. Also:
tolerate deleting already-gone packs, start rebuilt index entries as unused.
max_count=3 was a testing value. The default is now max_count=None,
max_size=50_000_000; the testsuite sets BORG_PACK_MAX_COUNT=3 instead,
so small test data still produces multiple multi-object packs.
close() persists the files cache but never drops its reference to
self._files, unlike self._chunks which is cleared right after being
saved. For a backup of many files this dict can hold hundreds of MB and
stays alive for as long as the closed cache object does.
Set self._files = None after writing it, mirroring the _chunks handling.
Accessing .files after close() was never supported anyway (close() also
tears down cache_config, which _read_files_cache needs), so this only
frees memory earlier.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AdHocWithFilesCache._chunks is the very same ChunkIndex object the
Repository holds (bound via the .chunks property). On close() we persist
it and clear the table in place (_maybe_write_chunks_cache(clear=True)),
then set self._chunks = None. But that only drops the *cache's* reference:
the Repository still points at the now-empty table and
is_chunk_index_loaded stays True.
A later access to repository.chunks in the same process (e.g.
delete_object doing self.chunks.get(id)) would then see a valid-looking
but empty index and conclude every chunk is gone, instead of rebuilding
from the repository. delete_chunkindex_from_repo() already guards its
analogous case with invalidate_chunk_index(); do the same here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
get_config/cache/data/runtime_dir now accept *subdirs and ensure_dir the
full path. get_keys_dir and get_security_dir use this instead of
hand-joining "keys"/"security", and the store cache in repository.py uses
get_cache_dir("storecache") instead of its own makedirs.
On filesystems with large allocation units like tmpfs with hugepages,
SEEK_DATA can report an offset before the actual first non-zero byte.
The test already verifies file contents and checks st_blocks to confirm
the file is stored sparsely, so only require that the first data region
is not past the expected content.
Signed-off-by: Zhou Qiankang <wszqkzqk@qq.com>
fill() is declared 'except 0' and always returns 1: errors propagate as
exceptions, so the 'if not self.fill(): return None' branches were
unreachable (and process() returning None would have crashed __next__
anyway). Call fill() plainly instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
