A) the compaction code needs the shadow index only for this case:
segment A: PUT x, segment B: DEL x, with A < B (DEL shadows the PUT).
B) for the following case, we have no shadowing DEL (or rather: it does not matter,
because there is a PUT right after the DEL) and x is in the repo index,
thus the shadow_index is not needed for the special case in the compaction code:
segment A: PUT x, segment B: DEL x PUT x
see also PR #5636.
reverts f079a83fed
and clarifies the code by more comments.
we keep the code deduplication of 5f32b5666a
and just add a update_shadow_index param to make it not look like there was
something accidentally forgotten, which was the whole reason for the reverted
"fix".
The shadow_index should be in same state after both of these sequences
(let's assume that A is not in repo yet for simplicity, but it does not matter):
a) explicit delete: put(A), delete(A), put(A), resulting in: PUT A, DEL A, PUT A repo contents
b) implicit delete: put(A), put(A), resulting in: PUT A, DEL A, PUT A repo contents
add remote upload buffer (--remote-buffer)
- added new option --remote-buffer
- allow to_send to grow to selected size
- don't grow if wait is specified
- fill pipe on any command (including 'async_response')
- add new option to docs
- create EfficientBytesQueue to prevent recreation of buffer each time we send something
- add tests for EfficientBytesQueue
a file map can be:
- created internally inside chunkify by calling sparsemap, which uses
SEEK_DATA / SEEK_HOLE to determine data and hole ranges inside a
seekable sparse file.
Usage: borg create --sparse --chunker-params=fixed,BLOCKSIZE ...
BLOCKSIZE is the chunker blocksize here, not the filesystem blocksize!
- made by some other means and given to the chunkify function.
this is not used yet, but in future this could be used to only read
the changed parts and seek over the (known) unchanged parts of a file.
sparsemap: the generate range sizes are multiples of the fs block size.
the tests assume 4kiB fs block size.
emit a warning message giving the path, xattr key and error message.
also: continue trying to restore other xattrs and bsdflags afterwards
(it did not continue with this before this fix).
/vagrant/borg/borg/.tox/py36-none/lib/python3.6/site-packages/borg/testsuite/remote.py:73:
PytestDeprecationWarning: @pytest.yield_fixture is deprecated.
Use @pytest.fixture instead; they are the same.
Docs: https://docs.pytest.org/en/stable/warnings.html
