Commit graph

1721 commits

Author SHA1 Message Date
Thomas Waldmann
8edb1dc085 manual formatting fixes (cosmetic / pep8) 2022-02-27 21:13:04 +01:00
Thomas Waldmann
c4ed840c7b pyupgrade --py38-plus ./**/*.py 2022-02-27 21:12:44 +01:00
TW
97de7e1fe2
Merge pull request #6380 from ThomasWaldmann/fix-zstd-include-1.2
fix zstd.h include for bundled zstd, fixes #6369
2022-02-27 14:39:09 +01:00
Thomas Waldmann
9390d2712b fix zstd.h include for bundled zstd, fixes #6369 2022-02-27 14:35:19 +01:00
James Buren
1618c98975 src/borg/cache_sync/unpack.h: fix compiler warnings
The key argument being sent to hashindex_get and hashindex_set by
multiple functions is a different signedness from what the functions
expect. This resolves the issue by changing the key type in the
unpack_user struct to unsigned char.
2022-02-26 16:13:07 -06:00
James Buren
4427c16d31 src/borg/_hashindex.c: fix compiler warnings
The value argument of hashindex_set is causing harmless pointer type
mismatches. This resolves the issue by changing the type to void*
which silences these types of warnings.
2022-02-26 16:13:02 -06:00
James Buren
8e640ae0f3 src/borg/compress.pyx: fix compiler warning, closes #6365
This resolves a compiler warning from the generated code that
resulted from a comparison of two local variables of different
signedness. The issue is resolved by changing the type of both
to int since this seems like the safest choice available.
2022-02-26 13:20:39 -06:00
James Buren
067dd10027 src/borg/crypto/low_level.pyx: fix compiler warning
The generated source code was producing a compiler warning due to
the pointers differing in constness. The called function expects
a non-const pointer while the generated code produces a const pointer
via a cast. This changes the cast to drop 'const' to make the compiler
happy.
2022-02-25 17:36:48 -06:00
TW
8a409ec1fb
Merge pull request #6306 from ThomasWaldmann/fix-savefile-races-master
SaveFile: fix race conditions
2022-02-17 16:51:43 +01:00
Laurent
6015ded828
docs: explain the difference between a path that ends with or without a slash (#6297)
docs: explain the difference between a path that ends with or without a slash
2022-02-16 23:53:11 +01:00
Thomas Waldmann
14b5c005d8 SaveFile: fix race conditions
Thanks to Andrey Bienkowski (@hexagonrecursion) for reporting this and writing reproducer code.

Changes:
- use different, randomly (but recognizably) named temp files while writing (securely made by os.mkstemp())
- make sure temp files are cleaned up in normal and error conditions
- SyncFile can now get corresponding pair of path + open os-level fd
- cleaned up: fd now means os-level fd, f means python-file-like object
- fixed a caller of SaveFile
2022-02-16 23:23:47 +01:00
Thomas Waldmann
b292e158a6 rename truncate_and_unlink to safe_unlink
it usually does not truncate any more,
only under "disk full" circumstances and only if there is only one hardlink.
2022-02-15 21:08:34 +01:00
Thomas Waldmann
4a2ab496e0 safer truncate_and_unlink implementation
the previous implementation caused collateral damage on hardlink-copies of a repository,
see: https://github.com/borgbackup/borg/discussions/6286
2022-02-15 12:58:37 +01:00
TW
4896fe1560
Merge pull request #6296 from ThomasWaldmann/cache-pre12-archive-meta
info: use a pre12-meta cache to accelerate stats for borg < 1.2 archives
2022-02-14 18:29:17 +01:00
Thomas Waldmann
a2fb9cde4e calc_stats progress display: add archive name 2022-02-14 18:00:02 +01:00
Thomas Waldmann
25e27a1539 info: use a pre12-meta cache to accelerate stats for borg < 1.2 archives
first time borg info is invoked on a borg 1.1 repo, it can take
a rather long time computing and caching some stats values for
1.1 archives, which borg 1.2 archives have in their archive
metadata structure. be patient, esp. if you have lots of old
archives.

following invocations are much faster.
2022-02-14 18:00:02 +01:00
Tomás Andrighetti
a2ae36bb54 Exclude directories in is_hardlink_master 2022-02-13 19:23:40 -03:00
TW
0cebe62264
Merge pull request #6290 from ThomasWaldmann/compact-free-space-master
compact segments: improve freeable / freed space log output
2022-02-13 18:25:01 +01:00
Thomas Waldmann
603b58f6a1 implement more standard hashindex.setdefault behaviour
the .get() like behaviour (== returning the value) was missing.

it's still not 100% like dict.setdefault, because there is no
default value None. but None doesn't make sense here, because we
usually need a N-tuple matching the hash table's value format.

note: this "bug" (or unusual implementation) was without consequences,
      because hashindex.setdefault is not used anywhere in borg, so
      it was also not used in a wrong way anywhere.

https://docs.python.org/3/library/stdtypes.html#dict.setdefault
2022-02-13 03:47:44 +01:00
Thomas Waldmann
17e8aef394 compact: not "freeable", but "maybe freeable"
e.g. if there is a ton of DELs in a segment, they all are maybe freeable,
but only if we also got rid of the respective PUTs (see also #6289).
2022-02-12 20:37:28 +01:00
Thomas Waldmann
e80b5c2272 compact: derive freed space from quota use before/after, fixes #5679
due to the way quota accounting is done, this is likely not
100% precise, but much better than selling the hints as the truth.
2022-02-12 20:37:18 +01:00
TW
c56b15f271
Merge branch 'master' into docs-links-master 2022-02-08 21:57:58 +01:00
Andrey Bienkowski
24e8b38f1a
Doc: borg init: explain the encryption modes better (#6184)
docs:borg init: explain the encryption modes better

The documentation for borg init was not structured logically:
1. The topic is switched from the general discussion of `borg init`
   to the discussion of encryption modes without a title.
2. Obscure technical details (chunking, id generation etc) were
   above the high-level overview and other key information.
2022-02-08 21:55:51 +01:00
Thomas Waldmann
3428b9f2b4 docs: link to python 3.8 docs
because py38 is our minimum requirement.
2022-02-07 23:23:15 +01:00
Thomas Waldmann
510ceb86f8 docs: link to borg_placeholders 2022-02-07 22:36:24 +01:00
Thomas Waldmann
d79b325494 docs: link to borg_patterns 2022-02-07 22:33:28 +01:00
Thomas Waldmann
42fb25d4a7 docs: use same phrasing in misc. help texts 2022-02-07 21:40:15 +01:00
bobthebadguy
45df345d29
Clarify usage of patternfile roots (#6242) 2022-02-07 20:53:25 +01:00
Thomas Waldmann
f0809cfeba docs: rst markup: remove accidental indentation in patterns help 2022-02-06 00:47:33 +01:00
Thomas Waldmann
3f02d3cd3d docs: rst markup: remove accidental indentation, fixes #6248 2022-02-06 00:44:51 +01:00
Andrey Bienkowski
e21343d2a8 Fix ArchiverTestCaseBinary.test_init_refuse_to_overwrite_keyfile 2022-02-05 22:32:09 +03:00
Andrey Bienkowski
29201a0eb5 Test: borg init refuses to overwrite keyfile
https://github.com/borgbackup/borg/pull/6046 neglected to
add a regression test
2022-02-05 22:32:09 +03:00
TW
4fa9f1faa4
Merge pull request #6188 from hexagonrecursion/nonce
docs: impact of deleting path/to/repo/nonce
2022-02-05 13:31:07 +01:00
Andrey Bienkowski
4931eadd91 Refactor: extract get_security_dir() 2022-02-05 09:06:53 +03:00
Andrey Bienkowski
e663c9aa10 Strengthen the test: we can read data w/o nonces 2022-02-05 09:06:25 +03:00
Andrey Bienkowski
37dde58154 Add tests for path/to/repo/nonce deletion
I am about to add documentation for this feature. Per the "If you liked
it, you should have put a CI test on it" rule I am adding tests to
detect if the feature regresses (causing a discrepancy between the docs
and the real behavior).
2022-02-05 09:06:17 +03:00
TW
2197e94a31
Merge pull request #6241 from ThomasWaldmann/argparse-highlander-master
argument parsing: accept some options only once, fixes #6026
2022-02-04 22:33:16 +01:00
Thomas Waldmann
c70788cef9 argument parsing: accept some options only once, fixes #6026 2022-02-04 22:08:09 +01:00
Thomas Waldmann
7edba854b2 use create=True when creating new keys, fixes #6036 2022-02-04 19:33:45 +01:00
Thomas Waldmann
d299b8bc9c KeyfileKeyBase: add create=False param to save method
If we create a new repo (and a new keyfile key, create=True),
there must not already exist a keyfile at the path/filename
where we want to write the new one.

In other use cases (e.g. if we overwrite a keyfile due
to the user changing their passphrase, create=False),
of course overwriting at the same path/fname is desired.
2022-02-04 19:33:45 +01:00
Andrey Bienkowski
9f311abd79
Doc: borg key export: add examples (#6227)
docs: borg key export: add examples

see: https://github.com/borgbackup/borg/issues/6204#issuecomment-1027150308
2022-02-04 18:37:33 +01:00
Thomas Waldmann
925daf30b7 fix intermediate commits, shall be at end of segment
compact_segments produced separate 17b files for intermediate commits, although they were intended to be end-of-segment-file commits.

this is because when the intermediate commit is triggered, we are already at an offset beyond the limit.
 thus needed to add the no_new flag to indicate that we do not want a new segment file just for the commit IF it is an intermediate commit.
2022-02-01 19:45:29 +01:00
Thomas Waldmann
79964d1a2b if ensure_dir() fails, give more informative error message, fixes #5952
previously, it just said PermissionDenied with giving the filename/path.
2022-02-01 04:02:09 +01:00
Thomas Waldmann
c4116b26c8 Location: fix bad naming: rename .orig -> .processed
- .raw is the unprocessed location (as given by user / env).
- .processed is the processed location (after placeholder replacement).
2022-01-31 21:55:12 +01:00
Thomas Waldmann
ad122e554a repo::archive location placeholder expansion fixes, fixes #5826, fixes #5998
- use expanded location for log output
- support placeholder expansion for BORG_REPO env var
- use Location.raw for the unprocessed, not expanded location string
2022-01-31 21:33:41 +01:00
Thomas Waldmann
5d4dda75d3 docs: --pattern* not experimental any more, fixes #6134 2022-01-31 20:55:08 +01:00
Thomas Waldmann
d917c3a43b delete --force: do not ask when deleting a repo, fixes #5941 2022-01-30 23:43:59 +01:00
Thomas Waldmann
20c5c6afbd delete: don't commit if nothing was deleted, avoid cache sync, fixes #6060 2022-01-30 23:15:05 +01:00
TW
a65f298477
Merge pull request #6157 from hexagonrecursion/ununit
Refactor: remove assert_true (master)
2022-01-29 23:19:11 +01:00
Andrey Bienkowski
6a5d24650f Refactor: improve a test name
https://github.com/borgbackup/borg/pull/6186#issuecomment-1023504905
> could you rename the strip test a little, so that there is
> ..._remote_repo in the test name. if one does not fully read the test
> (like me), one would assume that a strip-related test also applies to
> local repos, but in this case, the condition checked only applies to
> remote repos.
2022-01-27 23:03:15 +03:00