mirror of
https://github.com/borgbackup/borg.git
synced 2026-07-06 16:57:30 -04:00
0e3876d5e4
Normalized chunking switches between a stricter and a looser cut mask around the target chunk size. This greatly tightens the chunk-size distribution (coefficient of variation ~0.9 -> ~0.3 in tests) and removes the dedup-hostile max-size-clamped chunks, with unchanged deduplication. chunker-params for buzhash64 gains a required 6th field, nc_level: buzhash64,chunk_min,chunk_max,chunk_mask,window_size,nc_level Use nc_level=2 for the new default, nc_level=0 to disable (then behavior is byte-identical to the previous single-mask chunker). buzhash (32bit) is untouched and stays bit-compatible with borg 1.x. The mask transition point (normal_size) defaults to a principled formula (target minus the expected loose-phase tail) so the mean stays near the target; it can be tuned via the normal_size constructor arg. scripts/chunker_bench.py: evidence harness used to measure chunk-size distribution, dedup ratio, throughput and shift-resilience. Measurements (before = nc_level 0, after = nc_level 2; both at the default params buzhash64,19,23,21,4095; measured with scripts/chunker_bench.py): 5 GiB of incompressible data (~2000-2700 chunks, statistically stable): before: CV 0.739, 49 max-size-clamped (8 MiB) chunks, 953 MB/s after: CV 0.311, 0 max-size-clamped chunks, 1024 MB/s Re-backup of a 2.5 GiB file after a few scattered single-byte edits (deduplication ratio; 0.5 = v2 fully deduplicated against v1, lower is better): 64 edits: before 0.5424 -> after 0.5235 320 edits: before 0.6791 -> after 0.6142 Normalized chunking deduplicates better after edits: removing the max-size-clamped chunks means a single-byte change invalidates much less data (about 36% less dedup overhead at 320 edits). Throughput was also consistently higher with nc_level=2 at this scale. Also: fix bug when computing the mask, one needs to use 1ULL instead of 1, so the shifting computation is done in a uint64, not in a 32bit int. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| general | ||
| analyze.rst | ||
| analyze.rst.inc | ||
| benchmark.rst | ||
| benchmark_cpu.rst.inc | ||
| benchmark_crud.rst.inc | ||
| borgfs.rst | ||
| borgfs.rst.inc | ||
| break-lock.rst.inc | ||
| check.rst | ||
| check.rst.inc | ||
| common-options.rst.inc | ||
| compact.rst | ||
| compact.rst.inc | ||
| completion.rst | ||
| completion.rst.inc | ||
| create.rst | ||
| create.rst.inc | ||
| debug.rst | ||
| delete.rst | ||
| delete.rst.inc | ||
| diff.rst | ||
| diff.rst.inc | ||
| export-tar.rst.inc | ||
| extract.rst | ||
| extract.rst.inc | ||
| general.rst | ||
| help.rst | ||
| help.rst.inc | ||
| import-tar.rst.inc | ||
| info.rst | ||
| info.rst.inc | ||
| key.rst | ||
| key_add.rst.inc | ||
| key_change-location.rst.inc | ||
| key_change-passphrase.rst.inc | ||
| key_export.rst.inc | ||
| key_import.rst.inc | ||
| key_list.rst.inc | ||
| key_remove.rst.inc | ||
| list.rst | ||
| list.rst.inc | ||
| lock.rst | ||
| mount.rst | ||
| mount.rst.inc | ||
| notes.rst | ||
| prune.rst | ||
| prune.rst.inc | ||
| recreate.rst | ||
| recreate.rst.inc | ||
| rename.rst | ||
| rename.rst.inc | ||
| repo-create.rst | ||
| repo-create.rst.inc | ||
| repo-delete.rst | ||
| repo-delete.rst.inc | ||
| repo-info.rst | ||
| repo-info.rst.inc | ||
| repo-list.rst | ||
| repo-list.rst.inc | ||
| repo-space.rst | ||
| repo-space.rst.inc | ||
| serve.rst | ||
| serve.rst.inc | ||
| tag.rst | ||
| tag.rst.inc | ||
| tar.rst | ||
| transfer.rst | ||
| transfer.rst.inc | ||
| umount.rst.inc | ||
| undelete.rst | ||
| undelete.rst.inc | ||
| usage_general.rst.inc | ||
| version.rst | ||
| version.rst.inc | ||
| with-lock.rst.inc | ||