Commit graph

9181 commits

Author SHA1 Message Date
Thomas Waldmann
bc86bd4bb0
Move chunker module to chunkers package
Refactor by relocating the `chunker` module under a new `chunkers` package, adjusting imports and file references accordingly.
2025-06-04 22:12:45 +02:00
TW
ae80be7f43
Merge pull request #8901 from ThomasWaldmann/chunker-refactor3
Small Chunker refactor
2025-06-04 21:55:39 +02:00
Thomas Waldmann
f58d26671d
ChunkerFixed: do not assert on short header read
ChunkerFixed can be configured to support files with a specific header size.

But we do not want to get an AssertionError if we encounter a 0-byte file
or a file that is shorter than the header size.
2025-06-04 13:40:16 +02:00
Thomas Waldmann
66f10712b4
chunker: use safe_fadvise
also: refactor safe_advise a bit, use has_posix_fadvise.
2025-06-04 13:28:32 +02:00
TW
2295c83c7e
Merge pull request #8900 from ThomasWaldmann/gitignore-update
.gitignore: update entries
2025-06-04 11:51:13 +02:00
Thomas Waldmann
b26f222ca4
.gitignore: update entries 2025-06-04 11:40:55 +02:00
TW
6e2b550a0e
Merge pull request #8897 from ThomasWaldmann/update-changes
update CHANGES
2025-06-04 01:01:46 +02:00
Thomas Waldmann
5ca22e9c10
update CHANGES 2025-06-03 20:26:10 +02:00
TW
7ec02dce46
Merge pull request #8895 from ThomasWaldmann/exclude-based-on-xattrs
Exclude based on xattrs or NODUMP flag
2025-06-03 19:54:00 +02:00
Thomas Waldmann
2129d81fdb
create: auto-exclude items based on xattrs or NODUMP, see #4972
no options yet, just hardcoded macOS and Linux xattrs.
removed the --exclude-nodump option, it is also done automagically now.

also: create: call stat_ext_attrs early

this reads bsdflags, xattrs and ACLs from the
filesystem, except if the user chose to disable that.

notable:
- borg always reads these, even for unchanged files
- if we read them early, borg can now behave differently
  based e.g. on a xattr value (and e.g. exclude the file)
2025-06-03 12:56:30 +02:00
TW
1c07087e00
Merge pull request #8883 from ThomasWaldmann/chunker-refactor2
Chunker refactor
2025-06-03 12:51:30 +02:00
TW
0fa548beac
Merge pull request #8894 from ThomasWaldmann/bandit
use bandit
2025-06-03 09:07:35 +02:00
Thomas Waldmann
3b94130b1c
suppress some bandit warnings 2025-06-01 18:00:37 +02:00
Thomas Waldmann
338f9b1db9
add bandit, a security-oriented static analysis tool 2025-06-01 18:00:35 +02:00
TW
dda9c445e1
Merge pull request #8893 from ThomasWaldmann/borg-serve-permissions
serve: add --permissions option as an alternative to BORG_REPO_PERMISSIONS env var
2025-05-30 13:08:29 +02:00
Thomas Waldmann
bdc36d451f
serve: add --permissions option as an alternative to BORG_REPO_PERMISSIONS env var 2025-05-30 11:42:56 +02:00
TW
8c235d760b
Merge pull request #8891 from ThomasWaldmann/refactor-chunker-slow-master
test_chunkpoints_unchanged: do not use blake2b_256
2025-05-29 19:36:24 +02:00
Thomas Waldmann
6841897692
test_chunkpoints_unchanged: do not use blake2b_256
we want to get rid of legacy stuff(*) one day and sha256 is as
good for this purpose (and might be even hw accelerated).

(*) considered legacy due to the way it gives the key to the
blake2b function (just padding and prepending it to the data,
instead of using the key parameter, see #8867 ).
2025-05-29 16:38:53 +02:00
Thomas Waldmann
9f73862d51
sparse test: test for transformation of all-zero blocks into CH_ALLOC chunks 2025-05-28 22:33:27 +02:00
Thomas Waldmann
2ef669945f
skip test_create_read_special_symlink for now
it tends to hang, not only on cygwin.
2025-05-28 22:12:27 +02:00
Thomas Waldmann
439c193662
add tests for archiving big all-zero and all-random files 2025-05-28 13:05:13 +02:00
Thomas Waldmann
29cd100e3b
Cython: do optimisations only where really needed 2025-05-28 11:29:23 +02:00
Thomas Waldmann
a235cff0c5
Chunker: fix infinite loop
Could happen at EOF when remaining data is insufficient and no new data can be added.
2025-05-28 11:07:15 +02:00
Thomas Waldmann
5445a52994
Chunker: add sparse/fmap support 2025-05-27 23:58:12 +02:00
Thomas Waldmann
ad6d0ef2e8
Chunker: integrate FileReader for unified read logic
Replaced inline file reading logic with `FileReader` to standardize handling across chunkers. Improved buffer updates and allocation handling for sparse files and optimized read operations.
2025-05-27 23:58:08 +02:00
Thomas Waldmann
1c0d3eaa9f
FileFMAPReader: add extensive test coverage
Includes unit tests for basic functionality, handling of custom file maps, allocation types, sparse file support, and `_build_fmap` method.
2025-05-27 23:58:06 +02:00
Thomas Waldmann
e65755e114
FileReader: add tests for read and chunk handling
Includes cases for simple reads, multiple reads, and mock chunk scenarios to verify behavior with mixed allocation types.

Also: change Chunk type for empty read result for better consistency.
2025-05-27 23:58:02 +02:00
Thomas Waldmann
4db522dc0d
remove unneeded assertion, use 1MiB read size 2025-05-27 23:58:00 +02:00
Thomas Waldmann
43635a2edc
FileReader: refactor read method
Simplified and improved handling of mixed types of chunks during reading. The allocation type of resulting chunks is now determined based on contributing chunks.
2025-05-27 23:57:58 +02:00
Thomas Waldmann
3dac8f180c
Refactor FileReader.read to always return Chunk objects 2025-05-27 23:57:57 +02:00
Thomas Waldmann
1f03a776d5
Remove support for header_size in file readers.
The `header_size` parameter and related logic have been removed from file readers, simplifying their implementation. This change eliminates unnecessary complexity while maintaining all functional capabilities via `read_size` and `fmap`.
2025-05-27 23:57:55 +02:00
Thomas Waldmann
f036152789
Chunker: split logic into FileFMAPReader and FileReader
`FileFMAPReader` deals with sparse files (data vs holes) or fmap and yields blocks of some specific read_size using a generator.

`FileReader` uses the `FileFMAPReader` to fill an internal buffer and lets users use its `read` method to read arbitrary sized chunks from the buffer.

For both classes, instances now only deal with a single file.
2025-05-27 23:57:53 +02:00
Thomas Waldmann
2818a0c26e
Refactor ChunkerFixed: move file reading into FileReader class
Replaced `ChunkerFixed`'s block-reading functionality with a new `FileReader` class to streamline code and improve separation of concerns. Adjusted `ChunkerFixed` to delegate file reading to `FileReader` while focusing on chunk assembly.

`FileReader` is intended to be useful for other chunkers also, so they can easily implement sparse file reading / fmap support.
2025-05-27 23:57:52 +02:00
Thomas Waldmann
6a17829c25
refactor ChunkerFixed: move filemap building to _build_fmap method 2025-05-27 23:57:48 +02:00
TW
04000f7fed
Merge pull request #8880 from Atharva-Varpe/master
tox: use native pyproject.toml configuration
2025-05-27 17:12:46 +02:00
TW
a7bc9771d9
Merge pull request #8882 from ThomasWaldmann/buzhash-chunker-cy-translation
Buzhash chunker: C to Cython translation
2025-05-26 23:18:35 +02:00
Thomas Waldmann
8d3767efe7
inline BARREL_SHIFT code 2025-05-26 21:52:41 +02:00
Thomas Waldmann
522eaf6899
Convert _chunker.c into Cython code within Chunker class
This was mostly done by Junie AI, but needed some manual fixes:

fix exception handling
======================

-    cdef object process(self):
+    cdef object process(self) except *:

fix construction of memoryviews
===============================

placement of the [:size] is important.

-                return memoryview(<char*>(self.data + self.position))[:self.remaining]
+                return memoryview((self.data + self.position)[:self.remaining])

-        return memoryview(<char*>(self.data + old_last))[:n]
+        return memoryview((self.data + old_last)[:n])
2025-05-26 21:32:14 +02:00
Atharva Varpe
7b8137c646 removing tox.ini 2025-05-24 06:34:20 +00:00
Atharva Varpe
a4633ec144 added native tox support to pyproject.toml 2025-05-24 05:38:44 +00:00
TW
ee1f0fdb9c
Merge pull request #8876 from ThomasWaldmann/docs
docs: fix mistyped CVE number
2025-05-23 09:48:07 +02:00
Thomas Waldmann
456e319bb6
docs: fix mistyped CVE number 2025-05-23 09:26:26 +02:00
TW
2b655fccf7
Merge pull request #8872 from ThomasWaldmann/chunker-params-reject-even-window-size
ChunkerParams: reject even window size for buzhash, fixes #8868
2025-05-23 07:36:23 +02:00
TW
6fc0cd5ef3
Merge pull request #8873 from ThomasWaldmann/disable-windows-ci
CI: disable windows for now
2025-05-23 07:35:17 +02:00
Thomas Waldmann
7072f6263d
ChunkerParams: reject even window size for buzhash, fixes #8868 2025-05-23 06:49:52 +02:00
Thomas Waldmann
5940bad06e
CI: disable windows for now
The file:// repo URLs are still broken on windows.
2025-05-23 06:47:59 +02:00
TW
5707a37702
Merge pull request #8871 from ThomasWaldmann/rel200b17
Release 2.0.0b17
2025-05-23 06:37:05 +02:00
Thomas Waldmann
6b87fec1f7
update CHANGES 2025-05-22 14:22:10 +02:00
Thomas Waldmann
4514b957e4
require borgstore 0.3.x 2025-05-22 13:23:29 +02:00
Thomas Waldmann
fa01a2d46a
build_usage build_man 2025-05-22 13:21:21 +02:00