diff --git a/MANIFEST.in b/MANIFEST.in index d6f230182..0e677767e 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -4,4 +4,6 @@ exclude .coafile .editorconfig .gitattributes .gitignore .mailmap .travis.yml Vagrantfile prune .travis prune .github -include src/borg/platform/darwin.c src/borg/platform/freebsd.c src/borg/platform/linux.c src/borg/platform/posix.c src/borg/platform/windows.c +include src/borg/platform/darwin.c src/borg/platform/freebsd.c src/borg/platform/linux.c src/borg/platform/posix.c +include src/borg/platform/syncfilerange.c +include src/borg/platform/windows.c diff --git a/docs/changes.rst b/docs/changes.rst index 166cef555..6c22a8fc1 100644 --- a/docs/changes.rst +++ b/docs/changes.rst @@ -217,8 +217,8 @@ The best check that everything is ok is to run a dry-run extraction:: Changelog ========= -Version 1.2.0a8 (not released yet) ----------------------------------- +Version 1.2.0a8 (2020-04-22) +---------------------------- Please note: @@ -304,6 +304,7 @@ Other changes: - upgrade bundled lz4 to 1.9.2 - upgrade xxhash to 0.7.3 - require recent enough llfuse for birthtime support, #5064 +- only store compressed data if the result actually is smaller, #4516 - check: improve error output for matching index size, see #4829 - ignore --stats when given with --dry-run, but continue, fixes #4373 - replaced usage of os.statvfs with shutil.disk_usage (better cross-platform support). @@ -337,6 +338,7 @@ Other changes: - fixed "doc downplays severity of Nonce reuse issue", #4883 - borg repo restore instructions needed, #3428 - new FAQ: A repo is corrupt and must be replaced with an older repo. + - clarify borg init's encryption modes - native windows port: - update README_WINDOWS.rst diff --git a/docs/man/borg-benchmark-crud.1 b/docs/man/borg-benchmark-crud.1 index 0cdcf0377..5b611b1be 100644 --- a/docs/man/borg-benchmark-crud.1 +++ b/docs/man/borg-benchmark-crud.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-BENCHMARK-CRUD 1 "2020-04-12" "" "borg backup tool" +.TH BORG-BENCHMARK-CRUD 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-benchmark-crud \- Benchmark Create, Read, Update, Delete for archives. . diff --git a/docs/man/borg-benchmark.1 b/docs/man/borg-benchmark.1 index 56a2be2e6..3238ae2c0 100644 --- a/docs/man/borg-benchmark.1 +++ b/docs/man/borg-benchmark.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-BENCHMARK 1 "2020-04-12" "" "borg backup tool" +.TH BORG-BENCHMARK 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-benchmark \- benchmark command . diff --git a/docs/man/borg-break-lock.1 b/docs/man/borg-break-lock.1 index 54bcabb61..d9a3907de 100644 --- a/docs/man/borg-break-lock.1 +++ b/docs/man/borg-break-lock.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-BREAK-LOCK 1 "2020-04-12" "" "borg backup tool" +.TH BORG-BREAK-LOCK 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-break-lock \- Break the repository lock (e.g. in case it was left by a dead borg. . diff --git a/docs/man/borg-check.1 b/docs/man/borg-check.1 index a8ce9a793..6bf8cb8f9 100644 --- a/docs/man/borg-check.1 +++ b/docs/man/borg-check.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-CHECK 1 "2020-04-12" "" "borg backup tool" +.TH BORG-CHECK 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-check \- Check repository consistency . diff --git a/docs/man/borg-common.1 b/docs/man/borg-common.1 index 02163a150..86b883ad2 100644 --- a/docs/man/borg-common.1 +++ b/docs/man/borg-common.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-COMMON 1 "2020-04-12" "" "borg backup tool" +.TH BORG-COMMON 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-common \- Common options of Borg commands . diff --git a/docs/man/borg-compact.1 b/docs/man/borg-compact.1 index d5a667e07..6da8ab847 100644 --- a/docs/man/borg-compact.1 +++ b/docs/man/borg-compact.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-COMPACT 1 "2020-04-12" "" "borg backup tool" +.TH BORG-COMPACT 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-compact \- compact segment files in the repository . diff --git a/docs/man/borg-compression.1 b/docs/man/borg-compression.1 index a3fec6f6a..d7a8dbbf0 100644 --- a/docs/man/borg-compression.1 +++ b/docs/man/borg-compression.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-COMPRESSION 1 "2020-04-12" "" "borg backup tool" +.TH BORG-COMPRESSION 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-compression \- Details regarding compression . diff --git a/docs/man/borg-config.1 b/docs/man/borg-config.1 index 66c1c6de8..246db7156 100644 --- a/docs/man/borg-config.1 +++ b/docs/man/borg-config.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-CONFIG 1 "2020-04-12" "" "borg backup tool" +.TH BORG-CONFIG 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-config \- get, set, and delete values in a repository or cache config file . diff --git a/docs/man/borg-create.1 b/docs/man/borg-create.1 index 30c39aedc..6e1662034 100644 --- a/docs/man/borg-create.1 +++ b/docs/man/borg-create.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-CREATE 1 "2020-04-12" "" "borg backup tool" +.TH BORG-CREATE 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-create \- Create new archive . diff --git a/docs/man/borg-delete.1 b/docs/man/borg-delete.1 index d1b2c98e5..d073a22ad 100644 --- a/docs/man/borg-delete.1 +++ b/docs/man/borg-delete.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-DELETE 1 "2020-04-12" "" "borg backup tool" +.TH BORG-DELETE 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-delete \- Delete an existing repository or archives . diff --git a/docs/man/borg-diff.1 b/docs/man/borg-diff.1 index af50c2a9e..150b38ba9 100644 --- a/docs/man/borg-diff.1 +++ b/docs/man/borg-diff.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-DIFF 1 "2020-04-12" "" "borg backup tool" +.TH BORG-DIFF 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-diff \- Diff contents of two archives . diff --git a/docs/man/borg-export-tar.1 b/docs/man/borg-export-tar.1 index 961c10956..6880f82ae 100644 --- a/docs/man/borg-export-tar.1 +++ b/docs/man/borg-export-tar.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-EXPORT-TAR 1 "2020-04-12" "" "borg backup tool" +.TH BORG-EXPORT-TAR 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-export-tar \- Export archive contents as a tarball . diff --git a/docs/man/borg-extract.1 b/docs/man/borg-extract.1 index 193d3f7df..0124adcf2 100644 --- a/docs/man/borg-extract.1 +++ b/docs/man/borg-extract.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-EXTRACT 1 "2020-04-12" "" "borg backup tool" +.TH BORG-EXTRACT 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-extract \- Extract archive contents . diff --git a/docs/man/borg-info.1 b/docs/man/borg-info.1 index 8e4dc3fb3..2e77b8511 100644 --- a/docs/man/borg-info.1 +++ b/docs/man/borg-info.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-INFO 1 "2020-04-12" "" "borg backup tool" +.TH BORG-INFO 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-info \- Show archive details such as disk space used . diff --git a/docs/man/borg-init.1 b/docs/man/borg-init.1 index 85303ba2a..a62843f6a 100644 --- a/docs/man/borg-init.1 +++ b/docs/man/borg-init.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-INIT 1 "2020-04-12" "" "borg backup tool" +.TH BORG-INIT 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-init \- Initialize an empty repository . @@ -43,7 +43,11 @@ Encryption can be enabled at repository init time. It cannot be changed later. It is not recommended to work without encryption. Repository encryption protects you e.g. against the case that an attacker has access to your backup repository. .sp -But be careful with the key / the passphrase: +Borg relies on randomly generated key material and uses that for chunking, id +generation, encryption and authentication. The key material is encrypted using +the passphrase you give before it is stored on\-disk. +.sp +You need to be careful with the key / the passphrase: .sp If you want "passphrase\-only" security, use one of the repokey modes. The key will be stored inside the repository (in its "config" file). In above @@ -81,6 +85,12 @@ a different keyboard layout. You can change your passphrase for existing repos at any time, it won\(aqt affect the encryption/decryption key or other secrets. .SS Encryption modes +.sp +You can choose from the encryption modes seen in the table below on a per\-repo +basis. The mode determines encryption algorithm, hash/MAC algorithm and also the +key storage location. +.sp +Example: \fIborg init \-\-encryption repokey ...\fP .\" nanorst: inline-fill . .TS @@ -126,7 +136,8 @@ _ .\" nanorst: inline-replace . .sp -\fIMarked modes\fP are new in Borg 1.1 and are not backwards\-compatible with Borg 1.0.x. +Modes \fImarked like this\fP in the above table are new in Borg 1.1 and are not +backwards\-compatible with Borg 1.0.x. .sp On modern Intel/AMD CPUs (except very cheap ones), AES is usually hardware\-accelerated. @@ -159,8 +170,8 @@ from the other blake2 modes. This mode is new and \fInot\fP compatible with Borg 1.0.x. .sp \fInone\fP mode uses no encryption and no authentication. It uses SHA256 as chunk -ID hash. Not recommended, rather consider using an authenticated or -authenticated/encrypted mode. This mode has possible denial\-of\-service issues +ID hash. This mode is not recommended, you should rather consider using an authenticated +or authenticated/encrypted mode. This mode has possible denial\-of\-service issues when running \fBborg create\fP on contents controlled by an attacker. Use it only for new repositories where no encryption is wanted \fBand\fP when compatibility with 1.0.x is important. If compatibility with 1.0.x is not important, use @@ -203,9 +214,11 @@ $ borg init \-\-encryption=repokey\-blake2 /path/to/repo $ borg init \-\-encryption=none /path/to/repo # Remote repository (accesses a remote borg via ssh) +# repokey: stores the (encrypted) key into /config $ borg init \-\-encryption=repokey\-blake2 user@hostname:backup -# Remote repository (store the key your home dir) +# Remote repository (accesses a remote borg via ssh) +# keyfile: stores the (encrypted) key into ~/.config/borg/keys/ $ borg init \-\-encryption=keyfile user@hostname:backup .ft P .fi diff --git a/docs/man/borg-key-change-passphrase.1 b/docs/man/borg-key-change-passphrase.1 index 23f0d822f..eadc0bbfe 100644 --- a/docs/man/borg-key-change-passphrase.1 +++ b/docs/man/borg-key-change-passphrase.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-KEY-CHANGE-PASSPHRASE 1 "2020-04-12" "" "borg backup tool" +.TH BORG-KEY-CHANGE-PASSPHRASE 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-key-change-passphrase \- Change repository key file passphrase . diff --git a/docs/man/borg-key-export.1 b/docs/man/borg-key-export.1 index 488cbc344..df933de63 100644 --- a/docs/man/borg-key-export.1 +++ b/docs/man/borg-key-export.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-KEY-EXPORT 1 "2020-04-12" "" "borg backup tool" +.TH BORG-KEY-EXPORT 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-key-export \- Export the repository key for backup . diff --git a/docs/man/borg-key-import.1 b/docs/man/borg-key-import.1 index 76e8efc55..49b13f369 100644 --- a/docs/man/borg-key-import.1 +++ b/docs/man/borg-key-import.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-KEY-IMPORT 1 "2020-04-12" "" "borg backup tool" +.TH BORG-KEY-IMPORT 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-key-import \- Import the repository key from backup . diff --git a/docs/man/borg-key-migrate-to-repokey.1 b/docs/man/borg-key-migrate-to-repokey.1 index 3d7fe21bb..0879dc321 100644 --- a/docs/man/borg-key-migrate-to-repokey.1 +++ b/docs/man/borg-key-migrate-to-repokey.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-KEY-MIGRATE-TO-REPOKEY 1 "2020-04-12" "" "borg backup tool" +.TH BORG-KEY-MIGRATE-TO-REPOKEY 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-key-migrate-to-repokey \- Migrate passphrase -> repokey . diff --git a/docs/man/borg-key.1 b/docs/man/borg-key.1 index 843fef71a..cde8b81c3 100644 --- a/docs/man/borg-key.1 +++ b/docs/man/borg-key.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-KEY 1 "2020-04-12" "" "borg backup tool" +.TH BORG-KEY 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-key \- Manage a keyfile or repokey of a repository . diff --git a/docs/man/borg-list.1 b/docs/man/borg-list.1 index eb65a0565..f24120ee3 100644 --- a/docs/man/borg-list.1 +++ b/docs/man/borg-list.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-LIST 1 "2020-04-12" "" "borg backup tool" +.TH BORG-LIST 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-list \- List archive or repository contents . diff --git a/docs/man/borg-mount.1 b/docs/man/borg-mount.1 index 299adc9ab..932aa548c 100644 --- a/docs/man/borg-mount.1 +++ b/docs/man/borg-mount.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-MOUNT 1 "2020-04-12" "" "borg backup tool" +.TH BORG-MOUNT 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-mount \- Mount archive or an entire repository as a FUSE filesystem . diff --git a/docs/man/borg-patterns.1 b/docs/man/borg-patterns.1 index 08355ed74..e77e45c7f 100644 --- a/docs/man/borg-patterns.1 +++ b/docs/man/borg-patterns.1 @@ -1,6 +1,6 @@ .\" Man page generated from reStructuredText. . -.TH BORG-PATTERNS 1 "2020-04-12" "" "borg backup tool" +.TH BORG-PATTERNS 1 "2020-04-21" "" "borg backup tool" .SH NAME borg-patterns \- Details regarding patterns . @@ -37,17 +37,27 @@ currently active recursion root. You usually give the recursion root(s) when invoking borg and these can be either relative or absolute paths. .sp So, when you give \fIrelative/\fP as root, the paths going into the matcher -will look like \fIrelative/.../file.ext\fP\&. When you give \fI/absolute/\fP as root, -they will look like \fI/absolute/.../file.ext\fP\&. This is meant when we talk -about "full path" below. +will look like \fIrelative/.../file.ext\fP\&. When you give \fI/absolute/\fP as +root, they will look like \fI/absolute/.../file.ext\fP\&. +.sp +File paths in Borg archives are always stored normalized and relative. +This means that e.g. \fBborg create /path/to/repo ../some/path\fP will +store all files as \fIsome/path/.../file.ext\fP and \fBborg create +/path/to/repo /home/user\fP will store all files as +\fIhome/user/.../file.ext\fP\&. .sp File patterns support these styles: fnmatch, shell, regular expressions, path prefixes and path full\-matches. By default, fnmatch is used for -\fB\-\-exclude\fP patterns and shell\-style is used for the experimental \fB\-\-pattern\fP -option. +\fB\-\-exclude\fP patterns and shell\-style is used for the experimental +\fB\-\-pattern\fP option. .sp -If followed by a colon (\(aq:\(aq) the first two characters of a pattern are used as a -style selector. Explicit style selection is necessary when a +Starting with Borg 1.2, for all but regular expression pattern matching +styles, all paths are treated as relative, meaning that a leading path +separator is removed after normalizing and before matching. This allows +you to use absolute or relative patterns arbitrarily. +.sp +If followed by a colon (\(aq:\(aq) the first two characters of a pattern are +used as a style selector. Explicit style selection is necessary when a non\-default style is desired or when the desired pattern starts with two alphanumeric characters followed by a colon (i.e. \fIaa:something/*\fP). .INDENT 0.0 @@ -65,14 +75,15 @@ to match a pattern, the full path must match, or it must match from the start of the full path to just before a path separator. Except for the root path, paths will never end in the path separator when matching is attempted. Thus, if a given pattern ends in a path -separator, a \(aq*\(aq is appended before matching is attempted. +separator, a \(aq*\(aq is appended before matching is attempted. A leading +path separator is always removed. .TP .B Shell\-style patterns, selector \fIsh:\fP This is the default style for \fB\-\-pattern\fP and \fB\-\-patterns\-from\fP\&. Like fnmatch patterns these are similar to shell patterns. The difference is that the pattern may include \fI**/\fP for matching zero or more directory levels, \fI*\fP for matching zero or more arbitrary characters with the -exception of any path separator. +exception of any path separator. A leading path separator is always removed. .TP .B Regular expressions, selector \fIre:\fP Regular expressions similar to those found in Perl are supported. Unlike @@ -86,13 +97,14 @@ the re module\fP\&. .TP .B Path prefix, selector \fIpp:\fP This pattern style is useful to match whole sub\-directories. The pattern -\fIpp:root/somedir\fP matches \fIroot/somedir\fP and everything therein. +\fIpp:root/somedir\fP matches \fIroot/somedir\fP and everything therein. A leading +path separator is always removed. .TP .B Path full\-match, selector \fIpf:\fP This pattern style is (only) useful to match full paths. This is kind of a pseudo pattern as it can not have any variable or -unspecified parts \- the full path must be given. -\fIpf:root/file.ext\fP matches \fIroot/file.txt\fP only. +unspecified parts \- the full path must be given. \fIpf:root/file.ext\fP matches +\fIroot/file.txt\fP only. A leading path separator is always removed. .sp Implementation note: this is implemented via very time\-efficient O(1) hashtable lookups (this means you can have huge amounts of such patterns @@ -144,7 +156,7 @@ $ borg create \-e \(aq*.o\(aq backup / $ borg create \-e \(aq/home/*/junk\(aq backup / # Exclude the contents of \(aq/home/user/cache\(aq but not the directory itself: -$ borg create \-e /home/user/cache/ backup / +$ borg create \-e home/user/cache/ backup / # The file \(aq/home/user/cache/important\(aq is *not* backed up: $ borg create \-e /home/user/cache/ backup / /home/user/cache/important @@ -159,8 +171,8 @@ $ cat >exclude.txt <