Commit graph

967 commits

Author SHA1 Message Date
Thomas Waldmann
d8cfd91b15 fix borg create never showing M status
the problem was that the upper layer code did not have enough information
about the file, whether it is known or not - and thus, could not decide
correctly whether status should be M)odified or A)dded.

now, file_known_and_unchanged method returns an additional "known"
boolean to fix this.

also: add comment about files cache loading in cache_mode='r'
2018-03-02 15:44:41 +00:00
TW
b602612a40
Merge pull request #3555 from leo-b/1.1-maint-archive-metadata
add chunker_params to borg archive info (--json)
2018-01-29 17:52:26 +01:00
Thomas Waldmann
21742faf26 fixup: tests
(cherry picked from commit 835fede85f)
2018-01-20 17:13:45 +01:00
Thomas Waldmann
423ec4ba1e files cache: improve exception handling, fixes #3553
now deals with:
- corrupted files cache (truncated or modified not by borg)
- inaccessible/unreadable files cache
- missing files cache

The latter fix is not sufficient, the cache transaction processing
would still stumble over expected, but missing files in the cache.
2018-01-20 06:36:54 +01:00
Alexander 'Leo' Bergolth
b5fbeed6ed remove unneeded sort from scandir_generic() 2018-01-19 17:38:15 +01:00
Alexander 'Leo' Bergolth
9163e064fd ignore exceptions in scandir_inorder() caused by an implicit stat() 2018-01-18 16:25:50 +01:00
Alexander 'Leo' Bergolth
7e75cc3080 add chunker_params to archive info (at least to json output) 2018-01-18 11:32:40 +01:00
Abogical
46692c139a Revert copyright year change in shellpattern 2018-01-02 19:44:53 +02:00
Josh Holland
5138db20e2 Correct some confusing error messages from borg init (#3485)
init: more clear exception messages for borg create, fixes #3465

also: refactor

(cherry picked from commit 9f400633f2)
2017-12-29 03:18:02 +01:00
Thomas Waldmann
d11c9e9b15 use patched version of socket.getfqdn(), fixes #3471
(cherry picked from commit 9b0d0f3127)
2017-12-25 06:05:53 +01:00
Thomas Waldmann
f00e9de492 refactor/move hostname/fqdn related funcs, see #3471
- move stuff to platform.base (should be platform independent according
  to the docs).
- bump platform API version
- parseformat: import fqdn from platform instead of recomputing it

This is not yet fixing #3471, just a preparation for it.

(cherry picked from commit 5e4df7782b)
2017-12-25 06:05:52 +01:00
TW
128ed2da47
Merge pull request #3482 from ThomasWaldmann/fix-borg-init-quota-1.1
borg init: use given --storage-quota for local repo, fixes #3470 (1.1)
2017-12-23 21:54:08 +01:00
TW
b500ab7509
Merge pull request #3480 from ThomasWaldmann/borg-base-dir-1.1
added BORG_BASE_DIR, s/get_home_dir/get_base_dir/g, docs, fixes #3338
2017-12-23 19:53:16 +01:00
Thomas Waldmann
b101717818 borg init: use given --storage-quota for local repo, fixes #3470
(cherry picked from commit d1cdc1451f)
2017-12-23 19:26:52 +01:00
Thomas Waldmann
28baa78595 added BORG_BASE_DIR, s/get_home_dir/get_base_dir/g, docs, fixes #3338
also: sorted env var docs by level
(cherry picked from commit c5a339384e)
2017-12-23 18:09:16 +01:00
Josh Holland
b263888a4f Properly quote repo path in borg init output
Instead of printing this invalid command:

    borg upgrade --disable-tam '/path/test'repo'

print this valid one:

    borg upgrade --disable-tam '/path/test'"'"'repo'

(cherry picked from commit e780660830)
2017-12-23 17:49:18 +01:00
Thomas Waldmann
7d2f7be414 add zstd to compression help
(cherry picked from commit 3d32407e9a)
2017-12-20 03:25:18 +01:00
Thomas Waldmann
1bdaffc62c crypto.low_level extension: build blake2 like zstd, see #3415
(cherry picked from commit 4a2fec32b6)
2017-12-18 04:29:33 +01:00
Thomas Waldmann
0118b939d9 update blake2 reference impl file (cosmetic)
only whitespace changes.

(cherry picked from commit b56f6cdbc0)
2017-12-18 04:26:15 +01:00
Thomas Waldmann
ba38d5199b move blake2 files to subdir ref/ (as seen in orig. repo)
(cherry picked from commit 8924d78d44)
2017-12-18 04:25:57 +01:00
TW
b8a7238622
Merge pull request #3456 from ThomasWaldmann/lz4-ext-building-1.1
compress ext building (lz4 part, 1.1)
2017-12-18 04:14:25 +01:00
Thomas Waldmann
21ba3ff63c fix lz4 deprecation warning, require lz4 >= 1.7.0 (r129)
as we bundle lz4 1.8.0 now, platforms not having a recent liblz4 can
now just use the bundled code.

(cherry picked from commit fe48caf853)
2017-12-18 02:47:44 +01:00
Thomas Waldmann
7d8ec62b50 compress extension: build lz4 like zstd, see #3415
(cherry picked from commit 73a70082c2)
2017-12-18 02:41:30 +01:00
Thomas Waldmann
49f76a1e32 bundle lz4 1.8.0 src code, license
(cherry picked from commit c8b2820225)
2017-12-18 02:38:52 +01:00
Thomas Waldmann
1d64e5c227 also delete security dir when deleting a repo, fixes #3427
(cherry picked from commit 8f772437f2)
2017-12-18 01:55:11 +01:00
TW
696f2a2d25
Merge pull request #3445 from ThomasWaldmann/check-hangs-fix
fix for borg check --repair malfunction, #3444 (1.1-maint)
2017-12-16 21:59:44 +01:00
Thomas Waldmann
e09892caec check --repair: fix malfunctioning validator, fixes #3444
the major problem was the ('path' in item) expression.
the dict has bytes-typed keys there, so it never succeeded as it
looked for a str key. this is a 1.1 regression, 1.0 was fine.

the dict -> StableDict change is just for being more specific,
the check triggered correctly as StableDict subclasses dict,
it was just a bit too general.
2017-12-16 19:44:25 +01:00
Thomas Waldmann
a68d28bfa4 modify borg check unit test so it "hangs", see #3444
it doesn't infinitely hang, but slows down considerably.
2017-12-16 19:44:24 +01:00
Thomas Waldmann
c7383589cc bundle zstd 1.3.2 source code
only .c and .h files + license

(cherry picked from commit dc883f62ae)
2017-12-15 03:14:33 +01:00
Thomas Waldmann
0271ae6f95 support code to build bundled zstd code into the compress extension
setup_zstd.py modified so it is just amending the Extension() kwargs,
but the Extension is initialized by the caller.

this way, amending can happend multiple times (e.g. for multiple
compression algorithms).

also:
- move include/library dirs processing for system-library case
- move system zstd prefix detection to setup_zstd module
- cosmetic: setup.py whitespace fixes
- prefer system zstd option, document zstd min. requirement

(cherry picked from commit 34b92ffdaa)
2017-12-15 03:13:54 +01:00
Thomas Waldmann
01078328e2 zstd: use own Cython-based binding, remove python-zstandard dep
currently requires an externally available libzstd >= 1.3.0,
no bundled zstd yet.

(cherry picked from commit aec36f64a2)
2017-12-15 03:08:28 +01:00
Thomas Waldmann
6bad2395dd add zstd compression
based on willyvmm's work in PR #3116, but some changes:

- removed any mulithreading changes
- add zstandard in setup.py install_requires
- tests
- fix: minimum compression level is 1 (not 0)
- use 3 for the default compression level
- use ID 03 00 for zstd
- only convert to bytes if we don't have bytes yet
- move zstd code so that code blocks are ordered by ID
- other cosmetic fixes

(cherry picked from commit 11b2311e6e)
2017-12-15 02:13:15 +01:00
Nils Steinger
00865ae89e List help topics when invalid topic is requested
(cherry picked from commit 9e81a76172)
2017-12-15 00:58:29 +01:00
Gregor Kleen
96675e06c8 Refactor: call getfqdn() once per call of replace_placeholders()
(cherry picked from commit 294f06b565)
2017-12-15 00:56:43 +01:00
Gregor Kleen
dbf8d582f9 Add placeholder for fqdn in reverse notation
(cherry picked from commit 85fb38e2f3)
2017-12-15 00:56:31 +01:00
Thomas Waldmann
981a936f47 add parens for C preprocessor macro argument usages
this is needed for correctness because the preprocessor is just
doing text replacement.

This is the correct way:

#define MUL(x, y) ((x) * (y))

MUL(1+2, 3-4) -> ((1+2) * (3-4))    # not: (1+2 * 3-4)

I didn't put parens around all arg usages for readability.

Some stuff (like index) is not expected to be an expression.

Also, when a arg is only used in another macro or function call,
no parens are needed either.

I reviewed the code: no harm was done (yet) due to this fault.

Thanks to @rciorba who found this.

(cherry picked from commit a3cecf599f)
2017-12-15 00:50:06 +01:00
Thomas Waldmann
dc8de36109 borg mount: fix hardlink processing, fixes #3388
when the result list after stripping was empty,
os.path.join(*emptylist) fails as it want 1+ args.
2017-11-27 04:10:17 +01:00
Thomas Waldmann
23a1d62b25 crc32-slice-by-8.c: fix for C90 compilers
crc32_slice_by_8.c:344:3: error: ISO C90 forbids mixed declarations
and code [-Werror=declaration-after-statement]
2017-11-27 01:01:34 +01:00
Thomas Waldmann
831a06a07d move 3rd party docs/license file from package dir to docs/3rd_party
setup.py excludes all .c .h .pyx files from installation, but such
docs / license files would still be copied to the target directory
if they are inside the python "borg" package dir.
2017-11-27 00:02:37 +01:00
Thomas Waldmann
ea0203bb0d security: fix enforcement of --restrict-to-path in args processing
Fixes CVE-2017-15914 (affects releases 1.1.0, 1.1.1, 1.1.2,
but not 1.0.x).

Thanks to Florian Apolloner for discovering/reporting this!

Also: added tests for this.
2017-11-27 00:02:36 +01:00
Milkey Mouse
248c408b5b
Fix build_man issues (fixes #3364) 2017-11-25 19:38:23 -08:00
Thomas Waldmann
ef39dc6554 docs: move bsdflags tuning comments to notes docs section
the atime comments are already there.

(cherry picked from commit 91ff2f81a6)
2017-11-26 04:04:20 +01:00
TW
26cfaf3681
Merge pull request #3359 from milkey-mouse/borg-config-cmd-bp1.1
Add borg config command (1.1 backport)
2017-11-26 03:56:01 +01:00
Milkey Mouse
5331b378f7
fixup! Add borg config command (fixes #3304)
Don't use list unpacking for function calls in order to support py3.4

Reword basic borg config help

It doesn't edit the repo config, but any borg-related config.
2017-11-25 15:58:08 -08:00
Thomas Waldmann
3bdf1193ad docs: point out tuning options for borg create, fixes #3239
(cherry picked from commit 520a6a2ef1)
2017-11-25 15:25:15 +01:00
Milkey Mouse
15d9c94981
Fix borg config flake8 failures
Suppressed E731 so lambdas can be assigned to vars
2017-11-24 16:39:27 -08:00
Milkey Mouse
2ac0bf4980
Add borg config command (fixes #3304)
This command works similarly to "git config" - it parses repo and
cache configs to get, set, and delete values. It only works on local
repos so a malicious client can't e.g. override their storage quota
or reset the append_only flag.

Add tests for borg config

Add documentation for borg config

Change manual config edits -> borg config

There were a couple places in the documentation where it was advised
to edit the repository or cache config file, a process that is stream-
lined by borg config.
2017-11-24 16:38:56 -08:00
Thomas Waldmann
47d16e0f62 borg mount: support --consider-part-files correctly, fixes #3347
(cherry picked from commit caece370b8)
2017-11-24 01:45:42 +01:00
Thomas Waldmann
0e07647576 borg mount: support hardlinks correctly, add tests
previous commit did not yet support hardlinks correctly, if the
hardlink master was excluded somehow.

added some tests for this, also refactored related tests slightly.

(cherry picked from commit e97deafb16)
2017-11-24 00:30:28 +01:00
Thomas Waldmann
52410b6976 borg mount: support exclusion group options and paths, fixes #2138
borg mount [options] repo_or_archive mountpoint path [paths...]

paths: you can just give some "root paths" (like for borg extract) to
only partially populate the FUSE filesystem.

Similar for these exclusion group options:
--exclude
--exclude-from
--pattern
--patterns-from
--strip-components

(cherry picked from commit 77df1cfe8c)
2017-11-24 00:09:08 +01:00