Commit graph

1597 commits

Author SHA1 Message Date
Elmar Hoffmann
938e7f295c add progress indicator for archive check
Depending on the number of archives in a repository, the archive check part
of the check operation can take some time, so it should have a progress
indicator as well.
2021-05-15 23:15:31 +02:00
Thomas Waldmann
b04b7a77fd fix --iec option when given after "borg"
now both behave the same:
borg --iec info REPO
borg info --iec REPO
2021-05-03 18:28:58 +02:00
TW
c114e060ec
Merge pull request #5788 from RonnyPfannschmidt/move-chunks-equals
move chunk_equals to module level and modernize tox.ini
2021-05-02 21:14:59 +02:00
Ronny Pfannschmidt
603023bbd5 transform _chunk_content_equal into a global function to ease later benchmarking 2021-05-02 17:29:37 +02:00
Thomas Waldmann
65ea375968 get rid of distutils, use packaging 2021-05-01 20:40:48 +02:00
Thomas Waldmann
76dfd64aba create/recreate: print preliminary file status early, fixes #5417
if we back up stdin / pipes / regular files (or devices with --read-special),
that may take longer, depending on the amount of content data (could be many GiBs).

usually borg announces file status AFTER backing up the file,
when the final status of the file is known.

with this change, borg announces a preliminary file status before
the content data is processed. if the file status changes afterwards,
e.g. due to an error, it will also announce that as final file status.
2021-04-30 20:34:13 +02:00
Romain Vimont
9ddcfaf4f7 info / create --stats: add --iec option
If --iec is passed, then sizes are expressed in powers of 1024
instead of 1000.
2021-04-28 15:17:40 +02:00
Thomas Waldmann
dec1664a7e missing / healed chunks: always tell chunk ID, fixes #5704 2021-04-19 23:46:21 +02:00
Thomas Waldmann
02a9db50d2 do not load files cache for commands not using it, fixes #5673 2021-04-19 22:40:21 +02:00
Thomas Waldmann
adc89c488c fix repeated cache tag file writing bug
this bug was introduced by PR #5485 and also affects backport PR #5774.
2021-04-19 21:07:29 +02:00
Thomas Waldmann
7290333051 help string fixup 2021-04-16 19:09:52 +02:00
Thomas Waldmann
6b838febe8 s/remote_(buffer|ratelimit)/upload_(buffer/ratelimit)/g 2021-04-16 15:48:10 +02:00
Thomas Waldmann
40e4aa085f add --upload-(ratelimit|buffer), deprecate --remote-*, fixes #5611 2021-04-16 15:45:10 +02:00
Thomas Waldmann
6f9b9e5a53 s/numeric_owner/numeric_ids/g 2021-04-16 15:02:16 +02:00
Thomas Waldmann
251ce11fd8 add --numeric-ids, deprecate --numeric-owner option, fixes #5724 2021-04-16 14:52:12 +02:00
adrian5
6f371d5522 docs: Add commas to pattern help section 2021-03-19 09:34:51 +01:00
Robert Blenis
b2dea4422e add --json-lines option to diff command 2021-03-13 11:50:55 -05:00
Thomas Waldmann
bbccdbd81c mount: implement --numeric-owner (default: False!), fixes #2377
this is different default behaviour than in borg < 1.2:

default (numeric_owner=False) is to use the user/group name from the archive,
look up the local uid / gid and then use that for the FUSE fs.

when --numeric-owner is given (numeric_owner=True), then the uid/gid
from the archive is directly used (as it was the default behaviour in
borg < 1.2).

this was implemented like this (changing the default behaviour) to make
borg mount and borg extract behave more similar considering usage of
user/group numeric archived ids or archived names mapped to corresponding
numeric local system ids.

also, both now use the same function to get the uid/gid from the item.

fuse:
- add user and group name entries to default_dir
- also: set internal_dict(!) of new Item with data from Item.as_dict()
2021-03-07 18:16:23 +01:00
TW
086c0f2a6e
Merge pull request #5713 from ThomasWaldmann/files-cache-debug-logging-master
verbose files cache logging via --debug-topic=files_cache, fixes #5659
2021-02-28 23:14:28 +01:00
Thomas Waldmann
2211b840a3 verbose files cache logging via --debug-topic=files_cache, fixes #5659 2021-02-28 22:39:44 +01:00
Thomas Waldmann
d44836a865 config: accept non-int value for max_segment_size
borg config REPO max_segment_size 500M

note: when setting a non-int value for this in a repo config, using the repo will require borg >= 1.1.16.
2021-02-28 22:28:58 +01:00
Thomas Waldmann
99aa15b850 config: accept non-int value for storage_quota
borg config REPO storage_quota 100G

note: when setting a non-int value for this in a repo config, using the repo will require borg >= 1.1.16.
2021-02-28 22:27:48 +01:00
Thomas Waldmann
d4971e2819 some micro-opts in stat_ext_attrs 2021-02-16 23:24:05 +01:00
Thomas Waldmann
1b65db990d create/extract: add --noxattrs option, #3955
when given with borg create, borg will not get xattrs from input files (and thus, it will not archive xattrs).

when given with borg extract, borg will not read xattrs from archive and it will not set xattrs on extracted files.
2021-02-16 23:20:28 +01:00
Thomas Waldmann
9412a8430e create/extract: add --noacls option, #3955
when given with borg create, borg will not get ACLs from input files (and thus, it will not archive ACLs).

when given with borg extract, borg will not read ACLs from archive and it will not set ACLs on extracted files.
2021-02-16 22:43:08 +01:00
Manu
a84ead8e7c Pass args.log_json to FilesystemObjectProcessors/Statistics instance 2021-02-07 10:42:46 +08:00
Thomas Waldmann
10368f9c2c minor fix: fix order issue in test 2021-02-06 01:24:14 +01:00
Thomas Waldmann
3d0c61a184 revert incorrect fix for put updating shadow_index, fixes #5661
A) the compaction code needs the shadow index only for this case:

segment A: PUT x, segment B: DEL x, with A < B  (DEL shadows the PUT).

B) for the following case, we have no shadowing DEL (or rather: it does not matter,
because there is a PUT right after the DEL) and x is in the repo index,
thus the shadow_index is not needed for the special case in the compaction code:

segment A: PUT x, segment B: DEL x PUT x

see also PR #5636.

reverts f079a83fed
and clarifies the code by more comments.

we keep the code deduplication of 5f32b5666a
and just add a update_shadow_index param to make it not look like there was
something accidentally forgotten, which was the whole reason for the reverted
"fix".
2021-02-04 02:29:43 +01:00
Thomas Waldmann
a77db94b01 fix bad default: manifest.archives.list(consider_checkpoints=False), fixes #5668
also, add a comment about it, to avoid future similar mistakes.
2021-02-03 14:13:27 +01:00
Thomas Waldmann
649603f247 only print stats if not Ctrl-C'ed
if the user interrupts by ctrl-c, we do not save the archive,
thus we can not show stats because archive.id will not be in
the chunks index.
2021-02-03 01:56:52 +01:00
Thomas Waldmann
12d9110882 also accept msgpack up to 1.0.2
exclude 1.0.1 though, which had some issues (not sure whether they affect borg).
2021-01-30 21:31:33 +01:00
Thomas Waldmann
f079a83fed fix updating shadow_index also in put
The shadow_index should be in same state after both of these sequences
(let's assume that A is not in repo yet for simplicity, but it does not matter):

a) explicit delete: put(A), delete(A), put(A), resulting in: PUT A, DEL A, PUT A repo contents

b) implicit delete: put(A), put(A), resulting in: PUT A, DEL A, PUT A repo contents
2021-01-29 17:05:01 +01:00
Thomas Waldmann
5f32b5666a deduplicate code of put and delete, no functional change 2021-01-29 17:05:01 +01:00
Thomas Waldmann
6f00b025d8 remove empty shadowed_segments lists, fixes #5275
also:
- add test for removed empty shadowed_segments list
- add some comments
- add repo_dump test debug tool
2021-01-29 15:44:49 +01:00
Thomas Waldmann
a83fdd7de9 implement borg debug dump-hints 2021-01-29 14:11:32 +01:00
Thomas Waldmann
c8e9131158 remove bundled blake2 code, usage of libb2
we just use it via python3 now.
2021-01-28 18:00:00 +01:00
Thomas Waldmann
1dbe86a14e use blake2b from hashlib 2021-01-28 18:00:00 +01:00
Thomas Waldmann
6fa5bb4630 add tests for blake2b_128 2021-01-28 17:59:46 +01:00
TW
699256edbd
Merge pull request #5620 from ThomasWaldmann/sparse-file-integr2
Sparse file support (integration)
2021-01-17 17:45:53 +01:00
Thomas Waldmann
6dc334422e fixup: improve comment about assumptions in the item metadata stream chunker 2021-01-15 21:51:15 +01:00
Thomas Waldmann
2391d160a8 add all-zero detection to buzhash chunk data processing 2021-01-15 21:27:29 +01:00
Thomas Waldmann
2d76365214 cosmetic: directly set allocation instead going via is_zero 2021-01-15 21:10:07 +01:00
Thomas Waldmann
8162e2e67b cached_hash is only used in archive, move it there 2021-01-14 20:50:12 +01:00
Thomas Waldmann
e41dc6e96f use zeros for benchmarks 2021-01-14 20:19:10 +01:00
Thomas Waldmann
be257728ca move zeros to constants module 2021-01-14 20:02:18 +01:00
Thomas Waldmann
3b9798cffc remove max_chunk_size (unused) 2021-01-14 19:56:39 +01:00
TW
4041bdf169
Merge pull request #5606 from ThomasWaldmann/fix-5603-master
do not recurse into duplicate roots, fixes #5603 (master)
2021-01-11 16:51:49 +01:00
Thomas Waldmann
4e3be1db5e reuse zeros also in fixed-size chunker for all-zero chunk detection
also: zeros.startswith() is faster
2021-01-08 23:39:53 +01:00
Thomas Waldmann
ef19d937ed use cached_hash also to generate all-zero replacement chunks
at least for major amounts of fixed-size replacement hashes,
this will be much faster. also less memory management overhead.
2021-01-08 23:39:53 +01:00
Thomas Waldmann
f3088a9893 rename chunk_to_id_data to cached_hash 2021-01-08 23:39:53 +01:00