Add a new "fastcdc" content-defined chunker selectable via --chunker-params.
It uses the FastCDC Gear rolling hash (fp = (fp << 1) + Gear[byte]), which is
window-less and cheaper per byte than buzhash's cyclic-polynomial update, so it
chunks noticeably faster (see "borg benchmark cpu" output), while producing
the same chunk-size distribution and deduplication.
The Gear table is keyed: it is derived from the repo id key via CSPRNG (own
"fastcdc" domain), exactly like the buzhash64 table, so chunk cut points stay
unpredictable without the key (anti-fingerprinting). It implements the same
FastCDC techniques as buzhash64 (sub-minimum skipping, normalized chunking with
a required nc_level, min/max clamping); the mask uses the high bits of the hash
(Gear accumulates entropy there).
chunker-params: "fastcdc,chunk_min,chunk_max,chunk_mask,nc_level" - there is no
window field, because Gear is window-less. e.g. fastcdc,19,23,21,2
Also: borg benchmark cpu now measures the fastcdc chunker; tests in
borg.testsuite.chunkers (golden vector, size distribution, keyed gear table,
param parsing, slow fuzz); docs and changelog.
Benchmarks (scripts/chunker_bench.py, buzhash64 vs fastcdc, both nc_level=2,
incompressible data unless noted):
5 GiB, 2 MiB target (default params):
buzhash64: CV 0.294, 1011 MB/s
fastcdc: CV 0.295, 1313 MB/s (+30%)
64 MiB, 64 KiB target:
buzhash64: CV 0.374, shift-resilience 0.9928, 963 MB/s
fastcdc: CV 0.359, shift-resilience 0.9929, 1331 MB/s (+38%)
Re-backup of a 2.5 GiB file after scattered single-byte edits (dedup ratio,
0.5 = v2 fully deduplicated, lower is better):
64 edits: buzhash64 0.5237, fastcdc 0.5236
320 edits: buzhash64 0.6133, fastcdc 0.6161
borg benchmark cpu, 1 GB: fastcdc 3.80s, buzhash 4.36s, buzhash64 8.13s,
fixed 0.56s.
Chunk-size distribution, deduplication and shift-resilience match buzhash64
within noise; fastcdc is consistently faster.
Also: fix bug when computing the mask, one needs to use 1ULL instead of
1, so the shifting computation is done in a uint64, not in a 32bit int.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Moved the `ChunkerFixed` implementation from `chunker` to a new `fixed` module for better modularity. Updated imports and type hints.
Removed now empty chunkers.chunker module.
Moved `buzhash` implementation from `chunker` to a new `buzhash` module for better separation of concerns. Updated imports, adjusted `setup.py` and build configuration accordingly. Removed deprecated `Chunker` definitions from `chunker.pyi`.
Extracted the `reader` logic from `chunker` into a dedicated `reader` module to improve modularity and maintainability. Updated imports, references, and build configurations accordingly.
The `-Wno-unreachable-code-fallthrough` compiler flag suppresses warnings about fallthrough annotations in unreachable code.
In C switch statements, "fallthrough" occurs when execution continues from one case to the next without a break statement. This is often a source of bugs, so modern compilers warn about it. To indicate intentional fallthrough, developers use annotations like `__attribute__((fallthrough))`.
In Cython-generated C code, the `CYTHON_FALLTHROUGH` macro is defined to expand to the appropriate fallthrough annotation for the compiler being used. For example, in `compress.c`:
```c
#define CYTHON_FALLTHROUGH __attribute__((fallthrough))
```
The issue occurs because Cython generates code with conditional branches that may be unreachable on certain platforms or configurations. When these branches contain switch statements with fallthrough annotations, compilers like Clang issue warnings like:
```
warning: fallthrough annotation in unreachable code [-Wunreachable-code-fallthrough]
```
These warnings appear in the generated C code, not in the original Cython source. They're harmless but noisy, cluttering the build output with warnings about code we don't control.
By adding `-Wno-unreachable-code-fallthrough` to the compiler flags in `setup.py`, we specifically tell the compiler to ignore these particular warnings, resulting in a cleaner build output without affecting the actual functionality of the code.
This is a common practice when working with generated code - suppress specific warnings that are unavoidable due to the code generation process while keeping other useful warnings enabled.
`setup.py` hardcoded crypto library paths for OpenBSD, causing build
issue when OpenBSD drops specific OpenSSL version. Solution is to make
paths configurable.
Addresses #8553.
That "failed to map segment from shared object" error msg is not
very helpful. Add a hint that the filesystem needs to be +exec
(== not noexec mounted, like it might be the case for /tmp on
some systems).
Looks like borg's setup.py has hidden the real cause of a cythonize ImportError.
There are basically 2 cases:
- either there is no Cython installed, then the import fails because the module can not be found, or
- there is some issue within Cython and the import fails due to that.
It's important not to hide the real cause, especially if we run into case 2.
case 1 is kind of expected and frequent, case 2 is rare.
this was recently set to a relatively high minimum version when
locating it via pkgconfig was added. this broke the binary builds
on buster and bullseye.
i don't think borg requires a specific libacl version as long as
the api is compatible, so i now set this to 2.2.47 (from 2008).
borg2's new repo format does not need computing crc32 over big amounts of
(content) data any more (we now use xxh64 for that).
thus, having a quick crc32 implementation via libdeflate is not important
enough any more to rectify having libdeflate as a requirement.
