10311 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Mrityunjay Raj	bccbfde960	CI: run one linux job per event with BORG_STORE_CACHE enabled	2026-07-02 20:29:28 +05:30
Mrityunjay Raj	7cc2701cc0	repository: rename BORG_PACK_CACHE to BORG_STORE_CACHE and document the shared pack cache	2026-07-02 17:32:33 +05:30
Mrityunjay Raj	260e21a3c5	repository: add env-gated writethrough cache for the packs/ namespace ... BORG_PACK_CACHE enables it; BORG_PACK_MAX_COUNT/SIZE override pack sizing.	2026-07-01 13:28:16 +05:30
TW	8addc58416	Merge pull request #9823 from ThomasWaldmann/buzhash64-normalized-chunking ... buzhash64: add FastCDC-style normalized chunking	2026-06-30 11:58:42 +02:00
TW	52fdfc7596	Merge pull request #9828 from mr-raj12/chunkindex-pending-flag ... hashindex: track unresolved pack location with F_PENDING flag	2026-06-30 10:03:52 +02:00
TW	2ae04200a3	Merge pull request #9833 from borgbackup/dependabot/github_actions/actions-6d1c06d137 ... build(deps): bump actions/cache from 5 to 6 in the actions group	2026-06-30 10:01:20 +02:00
TW	68455a34b2	Merge pull request #9831 from mr-raj12/legacy-keys-dir-macos-master ... legacy: use borg 1.x keys dir for v1 repos	2026-06-30 10:00:45 +02:00
TW	0e2f3b8c0d	Merge pull request #9832 from borgbackup/dependabot/pip/requirements.d/pip-dependencies-43c8ae297d ... build(deps-dev): bump the pip-dependencies group in /requirements.d with 2 updates	2026-06-30 09:59:34 +02:00
dependabot[bot]	e3b5a81d93	build(deps): bump actions/cache from 5 to 6 in the actions group ... Bumps the actions group with 1 update: [actions/cache](https://github.com/actions/cache). Updates `actions/cache` from 5 to 6 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/cache dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com>	2026-06-30 05:43:29 +00:00
dependabot[bot]	4dbfb2d652	build(deps-dev): bump the pip-dependencies group ... Bumps the pip-dependencies group in /requirements.d with 2 updates: [virtualenv](https://github.com/pypa/virtualenv) and [tox](https://github.com/tox-dev/tox). Updates `virtualenv` from 21.3.3 to 21.4.1 - [Release notes](https://github.com/pypa/virtualenv/releases) - [Changelog](https://github.com/pypa/virtualenv/blob/main/docs/changelog.rst) - [Commits](https://github.com/pypa/virtualenv/compare/21.3.3...21.4.1) Updates `tox` from 4.54.0 to 4.55.0 - [Release notes](https://github.com/tox-dev/tox/releases) - [Changelog](https://github.com/tox-dev/tox/blob/main/docs/changelog.rst) - [Commits](https://github.com/tox-dev/tox/compare/4.54.0...4.55.0) --- updated-dependencies: - dependency-name: virtualenv dependency-version: 21.4.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: pip-dependencies - dependency-name: tox dependency-version: 4.55.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: pip-dependencies ... Signed-off-by: dependabot[bot] <support@github.com>	2026-06-30 05:43:09 +00:00
Mrityunjay Raj	1206808449	hashindex: make F_PENDING a user flag, drop redundant pending check ... F_PENDING moves to the user flag range so it is set/cleared via normal assignment; flush()'s store-failure path no longer re-checks is_pending.	2026-06-30 08:44:02 +05:30
Mrityunjay Raj	46642da292	legacy: use borg 1.x keys dir for v1 repos ... keyfile lookup always used the borg2 keys dir; on macOS that path differs from borg 1.x, so v1 repo keyfiles were not found.	2026-06-30 08:23:11 +05:30
TW	b08dc6334f	Merge pull request #9822 from mr-raj12/legacy-fs-extraction ... legacy: split borg 1.x directory resolution into borg.legacy.fs	2026-06-29 20:21:37 +02:00
Mrityunjay Raj	f50fd4fa52	hashindex: add tests for F_PENDING branches in repository list/get/flush	2026-06-29 22:45:55 +05:30
TW	467d37e856	Merge pull request #9826 from mr-raj12/check-tests-corrupt-at-store-level ... tests: corrupt chunks at store level in check tests	2026-06-29 18:21:54 +02:00
Mrityunjay Raj	3b3a647b27	tests: corrupt chunks at store level in check tests ... test_manifest_rebuild_corrupted_chunk now corrupts the pack at the store level and is skipped (TODO #8572) until repair rewrites store-corrupted packs.	2026-06-29 07:15:27 +05:30
Mrityunjay Raj	061530d199	hashindex: track unresolved pack location with F_PENDING flag ... Replace the UNKNOWN_BYTES32 pack_id sentinel (in-band signalling) with a ChunkIndex system flag, queried via is_pending(). Follow-up to #9821.	2026-06-29 06:57:56 +05:30
TW	fb257de35f	Merge pull request #9827 from mr-raj12/fix-manifest-rebuild-corrupt-offset ... tests: corrupt manifest off-centre to fix flaky check --repair rebuild test on Windows	2026-06-28 23:21:16 +02:00
Mrityunjay Raj	26d3a5d441	tests: corrupt manifest off-centre to fix flaky check --repair rebuild test on Windows	2026-06-29 00:39:00 +05:30
Thomas Waldmann	0e3876d5e4	buzhash64: add FastCDC-style normalized chunking ... Normalized chunking switches between a stricter and a looser cut mask around the target chunk size. This greatly tightens the chunk-size distribution (coefficient of variation ~0.9 -> ~0.3 in tests) and removes the dedup-hostile max-size-clamped chunks, with unchanged deduplication. chunker-params for buzhash64 gains a required 6th field, nc_level: buzhash64,chunk_min,chunk_max,chunk_mask,window_size,nc_level Use nc_level=2 for the new default, nc_level=0 to disable (then behavior is byte-identical to the previous single-mask chunker). buzhash (32bit) is untouched and stays bit-compatible with borg 1.x. The mask transition point (normal_size) defaults to a principled formula (target minus the expected loose-phase tail) so the mean stays near the target; it can be tuned via the normal_size constructor arg. scripts/chunker_bench.py: evidence harness used to measure chunk-size distribution, dedup ratio, throughput and shift-resilience. Measurements (before = nc_level 0, after = nc_level 2; both at the default params buzhash64,19,23,21,4095; measured with scripts/chunker_bench.py): 5 GiB of incompressible data (~2000-2700 chunks, statistically stable): before: CV 0.739, 49 max-size-clamped (8 MiB) chunks, 953 MB/s after: CV 0.311, 0 max-size-clamped chunks, 1024 MB/s Re-backup of a 2.5 GiB file after a few scattered single-byte edits (deduplication ratio; 0.5 = v2 fully deduplicated against v1, lower is better): 64 edits: before 0.5424 -> after 0.5235 320 edits: before 0.6791 -> after 0.6142 Normalized chunking deduplicates better after edits: removing the max-size-clamped chunks means a single-byte change invalidates much less data (about 36% less dedup overhead at 320 edits). Throughput was also consistently higher with nc_level=2 at this scale. Also: fix bug when computing the mask, one needs to use 1ULL instead of 1, so the shifting computation is done in a uint64, not in a 32bit int. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-28 12:00:09 +02:00
TW	8e6d1788da	Merge pull request #9821 from mr-raj12/pack-files-max-size ... repository: add max_size byte limit to PackWriter, OR-ed with max_count	2026-06-28 01:17:51 +02:00
TW	c4c7c84e02	Merge pull request #9820 from mr-raj12/fix-repository-delete-edge-cases ... repository: fix Repository.delete edge cases (#9815)	2026-06-28 01:17:16 +02:00
Mrityunjay Raj	2dcb50f30e	repository: assert compact_pack complete coverage reaches pack end	2026-06-28 03:46:35 +05:30
Mrityunjay Raj	0e2025885c	repository: update PackWriter comments	2026-06-28 03:34:45 +05:30
Mrityunjay Raj	2a4671c1ab	tests: extract corrupt_chunk_on_disk helper into repository_test.py	2026-06-28 01:43:01 +05:30
Mrityunjay Raj	e06ce2c64b	repository: allow max_count=None and tighten PackWriter comments per review	2026-06-28 00:45:31 +05:30
Mrityunjay Raj	f295b3e77a	repository: simplify delete/compact_pack docstrings and comments ... Rename delete's persist kwarg to update_index; corrupt pack bytes on disk in test_verify_data instead of via delete/put.	2026-06-28 00:07:38 +05:30
Mrityunjay Raj	1ebdf865fe	legacy: move borg 1.x directory resolution into borg.legacy.fs ... helpers.fs keeps only the borg2 layout; the v1 home/config/cache/keys/security dir logic now lives in legacy.fs. Part of #9556.	2026-06-27 11:53:34 +05:30
Mrityunjay Raj	4fe7e0ea16	repository: fix Repository.delete edge cases (#9815) ... Tolerate a stale earlier object in the pack via a compact_pack complete flag, skip the per-delete index write in check --repair via a delete persist flag, and make verify_data report the files whose chunks it removes. Add a regression test for the stale-earlier-object case.	2026-06-27 10:54:57 +05:30
Mrityunjay Raj	2206c3468f	repository: add max_size byte limit to PackWriter, OR-ed with max_count ... max_size=None (default) keeps count-only behaviour. Also bump default max_count 1->3 and flush test_list's last partial pack explicitly.	2026-06-27 10:02:02 +05:30
TW	f3933988c5	Merge pull request #9813 from mr-raj12/pack-files-cleanup ... pack-files: never drop a whole pack to delete a single object	2026-06-26 20:58:01 +02:00
Mrityunjay Raj	106dfbadae	tests: drop the delete_chunk wrapper, call Repository.delete directly ... Inline repository.delete at the check/extract/mount call sites. Drop the borgstore HEAD note from compact_pack now that it is fixed upstream.	2026-06-26 23:24:57 +05:30
Mrityunjay Raj	e74d7ef9a0	repository: make Repository.delete a real single-object delete ... Rewrite the object's pack via compact_pack and persist a full chunk index. Wire debug delete-obj and the delete_chunk test helper to it, and drop the store.info HEAD coverage check from compact_pack.	2026-06-26 09:51:57 +05:30
Mrityunjay Raj	5747a19cf8	tests: delete_chunk drops a single object via compact_pack, not the whole pack ... Rewrite the damage helper to rewrite the pack without the target chunk (neighbours survive) and persist the index, then revive three check tests that were skipped because the old whole-pack drop left an inconsistent index.	2026-06-26 04:05:32 +05:30
Mrityunjay Raj	82c589f3da	debug delete-obj: make single-object delete a no-op instead of dropping the whole pack ... Add explicit flush() in two repository tests so they no longer rely on the put count filling whole packs before close()/get().	2026-06-26 03:40:51 +05:30
Mrityunjay Raj	9fff040d58	comments: drop N=1/N>1 pack-files scaffolding, describe the general case ... These comments were phrased around development milestones (N=1, N>1, pack_id == chunk_id); with multi-object packs in place that framing is stale. Reword for the general case. Comment-only, no behavior change.	2026-06-26 01:46:32 +05:30
Mrityunjay Raj	62f2e8b05f	repository: store more than one object per pack (#9808) ... * repository: store more than one object per pack Raise PackWriter max_count to 2 and flush buffered packs before the archives pointer is written; also flush after debug put-obj and in the affected tests. * archive: flush buffered packs in Archives.create() so no caller writes the pointer over buffered objects	2026-06-25 21:33:09 +02:00
TW	e57e22d174	Merge pull request #9801 from mr-raj12/pack-files-compact ... compact: free space per pack with --threshold	2026-06-24 18:04:01 +02:00
Mrityunjay Raj	4a28995f04	compact: --stats reports sizes from the cached index, drop the redundant full scan and fix the epilog	2026-06-24 20:15:46 +05:30
Mrityunjay Raj	0586cc7b7b	compact: report progress after each unit so it reaches 100%	2026-06-24 18:14:05 +05:30
Mrityunjay Raj	78b8bbab50	compact: pack-level compaction with --threshold and --dry-run ... Rewrite a pack only when its unused bytes reach --threshold percent; --dry-run reports the space compact would free without changing the repository (#9379).	2026-06-24 11:10:42 +05:30
Mrityunjay Raj	58cd86c3d4	compact: free space per pack, honor --threshold ... Group objects by their pack and act per pack: drop fully-unused packs, and rewrite mixed packs whose unused bytes reach --threshold (default 40%) by copying the survivors forward with compact_pack. Two index scans keep the memory use bounded. refs #8572 #8514	2026-06-24 11:10:42 +05:30
TW	13fa6237ee	Merge pull request #9804 from ThomasWaldmann/interals-docs-update ... docs: update / fix "internals" section	2026-06-23 11:36:08 +02:00
Thomas Waldmann	66d010c2ac	docs: update / fix "internals" section	2026-06-23 11:19:38 +02:00
TW	dd67bf371f	Merge pull request #9800 from ThomasWaldmann/requirements-fixes-master ... Requirements fixes (master)	2026-06-23 10:45:37 +02:00
Thomas Waldmann	5237f9a002	setuptools-scm: tag.strict = false	2026-06-23 09:46:25 +02:00
Thomas Waldmann	84281c3670	requirements.d/development.*: remove requirements that are in pyproject.toml build-system	2026-06-23 09:46:22 +02:00
Thomas Waldmann	4fdf38ebcf	setuptools-scm: be consistent with dash and toml requirement	2026-06-23 09:46:20 +02:00
Thomas Waldmann	aa9a4c1416	blake3: exclude 1.0.9 release ... 1.0.9 does not build on windows/msys2/mingw.	2026-06-23 09:46:14 +02:00
TW	cf6d70fd61	Merge pull request #9803 from borgbackup/dependabot/github_actions/actions-d9db5a10fc ... build(deps): bump the actions group with 2 updates	2026-06-23 09:44:04 +02:00