mirror of
https://github.com/borgbackup/borg.git
synced 2026-07-08 17:51:06 -04:00
0e3876d5e4
Normalized chunking switches between a stricter and a looser cut mask around the target chunk size. This greatly tightens the chunk-size distribution (coefficient of variation ~0.9 -> ~0.3 in tests) and removes the dedup-hostile max-size-clamped chunks, with unchanged deduplication. chunker-params for buzhash64 gains a required 6th field, nc_level: buzhash64,chunk_min,chunk_max,chunk_mask,window_size,nc_level Use nc_level=2 for the new default, nc_level=0 to disable (then behavior is byte-identical to the previous single-mask chunker). buzhash (32bit) is untouched and stays bit-compatible with borg 1.x. The mask transition point (normal_size) defaults to a principled formula (target minus the expected loose-phase tail) so the mean stays near the target; it can be tuned via the normal_size constructor arg. scripts/chunker_bench.py: evidence harness used to measure chunk-size distribution, dedup ratio, throughput and shift-resilience. Measurements (before = nc_level 0, after = nc_level 2; both at the default params buzhash64,19,23,21,4095; measured with scripts/chunker_bench.py): 5 GiB of incompressible data (~2000-2700 chunks, statistically stable): before: CV 0.739, 49 max-size-clamped (8 MiB) chunks, 953 MB/s after: CV 0.311, 0 max-size-clamped chunks, 1024 MB/s Re-backup of a 2.5 GiB file after a few scattered single-byte edits (deduplication ratio; 0.5 = v2 fully deduplicated against v1, lower is better): 64 edits: before 0.5424 -> after 0.5235 320 edits: before 0.6791 -> after 0.6142 Normalized chunking deduplicates better after edits: removing the max-size-clamped chunks means a single-byte change invalidates much less data (about 36% less dedup overhead at 320 edits). Throughput was also consistently higher with nc_level=2 at this scale. Also: fix bug when computing the mask, one needs to use 1ULL instead of 1, so the shifting computation is done in a uint64, not in a 32bit int. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| 3rd_party | ||
| _static | ||
| _templates | ||
| binaries | ||
| borg_theme/css | ||
| deployment | ||
| internals | ||
| man | ||
| misc | ||
| usage | ||
| authors.rst | ||
| book.rst | ||
| changes.rst | ||
| changes_0.x.rst | ||
| changes_1.x.rst | ||
| conf.py | ||
| deployment.rst | ||
| development.rst | ||
| faq.rst | ||
| global.rst.inc | ||
| index.rst | ||
| installation.rst | ||
| internals.rst | ||
| introduction.rst | ||
| Makefile | ||
| man_intro.rst | ||
| quickstart.rst | ||
| quickstart_example.rst.inc | ||
| support.rst | ||
| usage.rst | ||
| usage_general.rst.inc | ||