Compare commits

...

1269 commits

Author SHA1 Message Date
Thomas Waldmann
083eaa0418
update CHANGES 2026-03-18 23:56:15 +01:00
Thomas Waldmann
8ddf789ec3
add .DS_Store to .gitignore 2026-03-18 23:56:12 +01:00
Thomas Waldmann
ff5be71f73
build_usage build_man 2026-03-18 23:56:10 +01:00
TW
b3e695cc49
Merge pull request #9510 from ThomasWaldmann/new-version-selector-location-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / asan_ubsan (push) Blocked by required conditions
CI / native_tests (push) Blocked by required conditions
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Blocked by required conditions
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Blocked by required conditions
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Blocked by required conditions
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
docs: move RTD version selector to sidebar top-left, fixes #8204
2026-03-18 23:39:00 +01:00
Thomas Waldmann
c046a154f2
docs: move RTD version selector to sidebar top-left
That way, right below the docs version number that is currently
being displayed, it is easier to find for users.

Also: hide the default readthedocs-flyout (bottom right)

Co-authored-by: Junie <junie@jetbrains.com>
2026-03-18 23:34:38 +01:00
TW
519cc99f30
Merge pull request #9505 from ThomasWaldmann/fix/borgfs-fstab-home-dir-3395
helpers: get_base_dir: avoid using incorrect HOME, fixes #3395
2026-03-18 23:12:08 +01:00
TW
65422547a1
Merge pull request #9507 from ThomasWaldmann/hashindex-fixes-1.4
hashindex: check values in read HashHeader, fixes #9485
2026-03-18 22:07:28 +01:00
TW
21353a34c5
Merge pull request #9506 from ThomasWaldmann/improve-serve-append-only-docs-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / asan_ubsan (push) Blocked by required conditions
CI / native_tests (push) Blocked by required conditions
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Blocked by required conditions
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Blocked by required conditions
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Blocked by required conditions
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
clarify append-only != write-only, fixes #9304
2026-03-18 19:52:31 +01:00
Thomas Waldmann
9917dd8541
hashindex: check values in read HashHeader 2026-03-18 19:33:40 +01:00
Thomas Waldmann
4efaaca751
clarify append-only != write-only, fixes #9304 2026-03-18 19:11:35 +01:00
TW
f4dacad528
Merge pull request #9498 from ThomasWaldmann/hashindex-memleak-1.4
hashindex: fix memory leak, fixes #9497
2026-03-18 18:55:01 +01:00
TW
7f6226ca09
Merge pull request #9504 from ThomasWaldmann/warn-replaying-segments-1.4
warn when replaying segments, fixes #9233
2026-03-18 18:53:32 +01:00
Thomas Waldmann
77db9c114f
helpers: get_base_dir: avoid using HOME when it incorrectly points to root's home for non-root users (fstab borgfs), fixes #3395 2026-03-18 18:42:43 +01:00
Thomas Waldmann
d2bc38acaa
warn when replaying segments, fixes #9233
If the repo index is not uptodate, any borg operation might trigger replay_segments and thus even operations that are usually quick can take rather long, leaving the user wondering about what's going on.

While there is already a progress indicator in replay_segments, it is logged at INFO level and thus not visible if borg runs at the default WARNING level.
2026-03-18 18:08:01 +01:00
Thomas Waldmann
5df07b51ef
hashindex: fix memory leak, fixes #9497
Thanks to @rolandrc for finding and fixing this!

Also: set index->buckets = NULL to indicate "we are finished with this" and avoid use-after-free.
2026-03-18 17:45:54 +01:00
TW
1fbbbe3da2
Merge pull request #9496 from ThomasWaldmann/update-changes-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / asan_ubsan (push) Blocked by required conditions
CI / native_tests (push) Blocked by required conditions
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Blocked by required conditions
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Blocked by required conditions
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Blocked by required conditions
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
update CHANGES (1.4)
2026-03-17 15:43:06 +01:00
Thomas Waldmann
d4a6542977
update CHANGES 2026-03-17 15:35:53 +01:00
TW
773bfef9a9
Merge pull request #9494 from ThomasWaldmann/fix-6204-1.4
docs: consolidate key backup info in borg key export, fixes #6204
2026-03-17 15:34:29 +01:00
Thomas Waldmann
d53ff5f175
docs: consolidate key backup info in borg key export, fixes #6204
Consolidate key backup documentation into `borg key export` and reference
it from Quickstart and FAQ to avoid duplication and inconsistency.

Clarify that while `repokey` or `authenticated` mode stores the key in the
repo, a separate backup is still recommended to protect against repository
corruption or data loss.
2026-03-17 14:33:19 +01:00
TW
3186cc4114
Merge pull request #9490 from ThomasWaldmann/fix-with-lock-test-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / asan_ubsan (push) Blocked by required conditions
CI / native_tests (push) Blocked by required conditions
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Blocked by required conditions
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Blocked by required conditions
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Blocked by required conditions
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
fix race condition in test_with_lock, fixes #8810
2026-03-17 01:23:55 +01:00
Thomas Waldmann
c1e31b8865
fix race condition in test_with_lock, fixes #8810 2026-03-17 00:52:10 +01:00
TW
974340c58a
Merge pull request #9491 from ThomasWaldmann/trailing-sparse-windows-1.4
fix spurious sparse test fail on win32, fixes #7616
2026-03-17 00:30:19 +01:00
Thomas Waldmann
7565b410d3
fix spurious sparse test fail on win32, fixes #7616
Python's `os.truncate()` on Windows relies on `SetEndOfFile()`, which does
not initialize the extended disk space with zeroes. This means that
trailing sparse holes simply leave uninitialized garbage data at the end
of the file.

During sparse file extraction, when the very last chunk is a sparse hole,
the VDL (Valid Data Length) is not properly advanced by `os.truncate()`.
As a result, reading from the end of the file fetches random disk garbage
instead of zeroes, causing spurious test failures at boundaries (like
2MB or 8MB) depending on what was in the uninitialized disk sectors.

Fix this by tracking trailing holes and manually writing a single `b"\0"`
byte at the end of the file before truncating on Windows. Writing explicit
data forces NTFS to officially advance the VDL and securely zero-fill the
preceding hole space.

Re-enable `test_sparse_file` on Windows.
2026-03-16 22:44:42 +01:00
TW
b7659140e6
Merge pull request #9489 from ThomasWaldmann/windows-fixes-1.4
cygwin: skip ~root base dir test
2026-03-16 22:36:44 +01:00
TW
f452ad7fca
Merge pull request #9488 from ThomasWaldmann/archive-cwd-1.4
info: show cwd at the time of backup creation, fixes #6191
2026-03-16 21:58:54 +01:00
Thomas Waldmann
b1b9e781af
cygwin: skip ~root base dir test 2026-03-16 21:57:22 +01:00
TW
c5ee7fc7f1
Merge pull request #9487 from ThomasWaldmann/filter-xattrs-1.4
no_selinux -> filter_xattrs
2026-03-16 21:21:06 +01:00
Thomas Waldmann
80e60b4f6f
info: show cwd at the time of backup creation, fixes #6191
Useful if one archives relative paths. Together with the cwd
one can know the full path of the files.
2026-03-16 21:18:55 +01:00
Thomas Waldmann
de92a3dace
no_selinux -> filter_xattrs
Originally, we only wanted to get rid of selinux xattrs,
but macOS also uses some xattr keys that disturb our test results,
so we filter them also.
2026-03-16 20:35:55 +01:00
TW
289364b7d7
Merge pull request #9452 from borgbackup/backport-9449-to-1.4-maint
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
[Backport 1.4-maint] tests: fix coverage collection for daemonized borg mount (fuse/hlfuse)
2026-03-07 23:27:01 +01:00
Divyesh
5c42d34a1d tests: collect coverage from daemonized borg mount processes
Fixes #9448.

borg mount forks into a background daemon, so coverage was missing the process that actually handles the FUSE requests, leaving fuse.py and hlfuse.py at 0%.

Enable coverage patches:

    patch = ["subprocess", "_exit"]

This lets coverage follow spawned subprocesses and still record data for paths that terminate via os._exit().

(cherry picked from commit 6df608ab6b)
2026-03-07 22:24:56 +00:00
Divyesh
c7fa503565 tests: make coverage patch config work on Windows CI
(cherry picked from commit 00ab5b0f25)
2026-03-07 22:24:56 +00:00
Divyesh
164594fbe1 tests: collect coverage from daemonized borg mount processes
(cherry picked from commit 99be942448)
2026-03-07 22:24:56 +00:00
TW
0a0eed485b
Merge pull request #9426 from ThomasWaldmann/hashindex_size-64bit
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
hashindex_size: return int64_t, fixes #9423
2026-03-01 19:17:03 +01:00
Thomas Waldmann
f67bfc4b8b
hashindex_size: return int64_t, fixes #9423 2026-03-01 18:25:29 +01:00
TW
86f6b8e7ee
Merge pull request #9370 from ThomasWaldmann/fix-hashindex-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
Fix hashindex .iteritems (1.4-maint)
2026-02-19 00:24:15 +01:00
Thomas Waldmann
c5a7a3cda6
hashindex: moved the tests from hashindex_stress to hashindex_pytest 2026-02-18 23:04:17 +01:00
Thomas Waldmann
eb80e53fc2
hashindex: always check return value of hashindex_get in the same way 2026-02-18 22:52:01 +01:00
Thomas Waldmann
2e6af653f6
hashindex: fixed iteritems segfaulting with non-existent marker, fixes #9368
Never happened in borg, because borg always gives existing markers to iteritems.

Also: added test for this.
2026-02-18 22:40:22 +01:00
TW
b1aa1a4eb0
Merge pull request #9364 from mr-raj12/port-pr-template-1.4-maint
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
github: enhance pull request template (port from master)
2026-02-17 18:29:50 +01:00
Mrityunjay Raj
99ee41e404 github: enhance pull request template (port from master) 2026-02-17 22:34:10 +05:30
TW
0e13fbdd7f
Merge pull request #9326 from ThomasWaldmann/misc-fixes-1.4
Some checks failed
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
CI / lint (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
Misc. fixes (1.4-maint)
2026-02-17 15:15:31 +01:00
Thomas Waldmann
bc370eefcc
update CHANGES 2026-02-14 17:21:07 +01:00
Thomas Waldmann
4277a6ae16
rename Dockerfile to match script name 2026-02-14 15:57:57 +01:00
Thomas Waldmann
5f6c794cb0
CI: build linux binaries with pyfuse3 2026-02-14 15:04:45 +01:00
Thomas Waldmann
a3dbdda264
pyinstaller binary: do not exclude ssl, needed for pyfuse3/trio, fixes #9196 2026-02-14 15:02:24 +01:00
Thomas Waldmann
f98257d4cb
tox: use pytest -n auto by default to speed up tests 2026-02-14 12:57:31 +01:00
TW
a9abc4217e
Merge pull request #9323 from borgbackup/backport-9322-to-1.4-maint
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
[Backport 1.4-maint] linux-run: run commands (e.g. tox) in a podman linux container
2026-02-14 12:30:06 +01:00
Thomas Waldmann
61609f60a7 linux-run: run commands (e.g. tox) in a podman linux container
this is useful to run the linux tests on a mac, e.g. to test fuse stuff.

(cherry picked from commit 40a2bb7032)
2026-02-14 11:26:13 +00:00
TW
7752addbee
Merge pull request #9297 from mr-raj12/fix/codespell-typos-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
fix typos found by codespell (1.4-maint)
2026-02-12 20:50:48 +01:00
Mrityunjay Raj
22297f4680 fix typos found by codespell (1.4-maint)
- backupped → backed-up (docs/faq.rst)
- abount → about (docs/faq.rst)
- unflexible → inflexible (docs/faq.rst)
- shrinked → shrunk (docs/internals/data-structures.rst)
- backupped → backed up (src/borg/archive.py)
- splitted → expected (src/borg/testsuite/xattr.py)

Closes #9295
2026-02-13 00:40:54 +05:30
TW
97c570957a
Merge pull request #9291 from MultisampledNight/backport/fix-padme-ui-to-1.4-maint
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
Backport "Fix padme UI" to 1.4-maint
2026-02-10 09:31:26 +01:00
MultisampledNight
261705c4c1
test(compress): roundtrip through UI parsing 2026-02-09 23:42:46 +01:00
MultisampledNight
9b84a2c235
compress: expose Padmé size obfuscation via UI
Fixes #9286.
Tested manually via:

 = export BORG_REPO=repo
 = alias b='python -m borg'
 = echo meow > file
 = b create ::test ./file -C obfuscate,250,none
 = b info
Archive name: ::test
Archive fingerprint: 2c67ba6c04faa6d911b5dd02b99593ea2cf6ebe7c38e2e6fca90afad47a7095e
[...]
Command line: /home/user/c/d0/software/backup/borg/src/borg/__main__.py create ::test ./file -C obfuscate,250,none
[...]
Number of files: 1
Original size: 40 B
 = rm file
 = b list ::test
-rw-r--r-- user   users         5 Mon, 2026-02-09 22:41:54 +0100 file
 = b extract ::test file
 = cat file
meow
 = # Yup, all good. Let's go for something nontrivial to make sure it's stripped accordingly.
 = dd if=/dev/urandom of=file bs=1M count=4
 = b create ::test ./file -C obfuscate,250,zstd
 = mv file original
 = b info
Archive name: ::test
Archive fingerprint: 2c67ba6c04faa6d911b5dd02b99593ea2cf6ebe7c38e2e6fca90afad47a7095e
[...]
Command line: /home/user/c/d0/software/backup/borg/src/borg/__main__.py create ::test ./file -C obfuscate,250,none
Working Directory: /home/user/l/bug/borg
Number of files: 1
Original size: 40 B

Archive name: ::test
Archive fingerprint: 01014345dd1187f014f19ac59a2313a4e6d644532b287d806fe87882ca47ea0d
[...]
Command line: /home/user/c/d0/software/backup/borg/src/borg/__main__.py create ::test ./file -C obfuscate,250,zstd
Working Directory: /home/user/l/bug/borg
Number of files: 1
Original size: 4.19 MB
 = b extract aid:01014345dd1187f014f19ac59a2313a4e6d644532b287d806fe87882ca47ea0d 
 = sha256sum file original
d74be01406764ef11f5aa3ee27033bdced2c9573c0b3ed980fcfb2116c7d84de  file
d74be01406764ef11f5aa3ee27033bdced2c9573c0b3ed980fcfb2116c7d84de  original
 =
2026-02-09 23:31:52 +01:00
Konmous04
260500f3a2
prune: added -v / --info output, fixes #9262
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
2026-01-31 20:48:28 +01:00
TW
ea5a4a1a92
Merge pull request #9267 from ThomasWaldmann/FileType-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
replace deprecated `argparse.FileType`, fixes #9266
2026-01-22 17:34:41 +01:00
Thomas Waldmann
3630eaafdf
replace deprecated argparse.FileType, fixes #9266
also: fix trailing whitespace in archiver.py.
2026-01-22 17:10:17 +01:00
TW
8770990fe7
Merge pull request #9265 from ThomasWaldmann/borg-version-1.4
Version: do not access private attributes, fixes #9263
2026-01-22 17:01:38 +01:00
Thomas Waldmann
000a5818ef
Version: do not access private attributes, fixes #9263
DeprecationWarning: Version._version is private and will be removed soon
2026-01-22 15:57:55 +01:00
MartinKurtz
f379420e7c
mount: warn about symlinks pointing outside of the mountpoint, see #9254
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
2026-01-12 23:09:34 +01:00
TW
70c87b637f
Merge pull request #9246 from ThomasWaldmann/fuse-fixes-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
mount: fuse fs performance fix, docstring
2026-01-04 04:38:55 +01:00
Thomas Waldmann
4eed3ecd5b
fuse: add thread/async safety warning 2026-01-03 19:47:15 +01:00
Thomas Waldmann
646f279062
mount: performance improvement
### Performance Comparison: `bytearray.extend()` vs. Concatenation

The efficiency difference between `meta.extend(bytes(N))` and `meta = meta + bytes(N)` stems from how Python manages memory and objects during these operations.

#### 1. In-Place Modification vs. Object Creation
- **`bytearray.extend()`**: This is an **in-place** operation. If the current memory block allocated for the `bytearray` has enough extra capacity (pre-allocated space), Python simply writes the new bytes into that space and updates the length. If it needs more space, it uses `realloc()`, which can often expand the existing memory block without moving the entire data set to a new location.
- **Concatenation (`+`)**: This creates a **completely new** `bytearray` object. It allocates a new memory block large enough to hold the sum of both parts, copies the contents of `meta`, copies the contents of `bytes(N)`, and then reassigns the variable `meta` to this new object.

#### 2. Computational Complexity
- **`bytearray.extend()`**: In the best case (when capacity exists), it is **O(K)**, where K is the number of bytes being added. In the worst case (reallocation), it is **O(N + K)**, but Python uses an over-allocation strategy (growth factor) that amortizes this cost, making it significantly faster on average.
- **Concatenation (`+`)**: It is always **O(N + K)** because it must copy the existing `N` bytes every single time. As the `bytearray` grows larger (e.g., millions of items in a backup), this leads to **O(N²)** total time complexity across multiple additions, as you are repeatedly copying an ever-growing buffer.

#### 3. Memory Pressure and Garbage Collection
- Concatenation briefly requires memory for **both** the old buffer and the new buffer simultaneously before the old one is garbage collected. This increases the peak memory usage of the process.
- `extend()` is more memory-efficient as it minimizes the need for multiple large allocations and relies on the underlying memory manager's ability to resize buffers efficiently.

In the context of `borg mount`, where `meta` can grow to be many megabytes or even gigabytes for very large repositories, using concatenation causes a noticeable slowdown as the number of archives or files increases, whereas `extend()` remains performant.
2026-01-03 19:31:54 +01:00
TW
c935dbe3b0
Merge pull request #9228 from ThomasWaldmann/build-on-macos15-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
CI: use macos-15 to build the binaries
2025-12-16 21:18:04 +01:00
Thomas Waldmann
af232a539c
CI: use macos-15 to build the binaries
Github EOLed the macOS 13 Intel runner and only
supports Intel on macOS 15 from now on.

Let's also use macOS 15 for the Apple Silicon build.
2025-12-16 20:22:22 +01:00
TW
400be5fcb5
Merge pull request #9207 from ThomasWaldmann/misc-updates-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
misc. updates (1.4-maint)
2025-12-02 15:31:52 +01:00
Thomas Waldmann
1298840de0
CI: increase timeout 2025-12-02 11:04:56 +01:00
Thomas Waldmann
9f61c1c693
freebsd: fix upload name for artifact zip 2025-12-02 08:59:33 +01:00
Thomas Waldmann
04164ecf1f
update binary readme 2025-12-02 08:51:56 +01:00
TW
6cfdd72433
Merge pull request #9204 from ThomasWaldmann/rel143
Some checks are pending
CI / lint (push) Waiting to run
CI / asan_ubsan (push) Blocked by required conditions
CI / native_tests (push) Blocked by required conditions
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Blocked by required conditions
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Blocked by required conditions
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Blocked by required conditions
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
Release 1.4.3
2025-12-02 08:47:08 +01:00
Thomas Waldmann
509c5695e0
build_usage build_man 2025-12-02 00:20:09 +01:00
Thomas Waldmann
8334b41f82
diff help: remove trailing blank 2025-12-02 00:19:23 +01:00
Thomas Waldmann
6a65d23e89
update CHANGES 2025-12-02 00:09:36 +01:00
TW
ad05ba8f37
Merge pull request #9203 from ThomasWaldmann/fix-compact-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / asan_ubsan (push) Blocked by required conditions
CI / native_tests (push) Blocked by required conditions
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Blocked by required conditions
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Blocked by required conditions
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Blocked by required conditions
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
compact: fix dealing with mismatching hints (1.4-maint)
2025-12-01 23:56:12 +01:00
Thomas Waldmann
32946a9f81
compact: also fix segments hints data for lost segment files
The code used to remove the missing segment only from "compact" hints,
but we need to also remove it from "segments" hints.
2025-12-01 23:12:32 +01:00
Thomas Waldmann
3a78a3cf49
compact_segments: replace AssertionError by warning, fixes #8535
emit only a warning, but let compaction complete.
after that, borg check --repair can fix the hints successfully.

This is the result of a longer discussion with Antigravity AI and me:

Detailed Explanation: Why Converting AssertionError to Warning is Correct
=========================================================================

PROBLEM OVERVIEW
----------------
The assertion `assert segments[segment] == 0` in compact_segments() was causing
borg compact to crash when segment reference counts in the hints file didn't
match the actual repository state. This typically occurred after index corruption
or repository recovery scenarios.

ROOT CAUSE ANALYSIS
-------------------
The crash happens due to a fundamental mismatch between two data structures:

1. self.segments (loaded from hints file)
   - Contains reference counts for each segment
   - Persisted to disk in the hints file
   - Represents the "last known state"

2. self.index (loaded from index file)
   - Contains mappings of object IDs to (segment, offset) pairs
   - Can be corrupted or lost
   - When corrupted, triggers auto-recovery

The Problem Scenario:
1. Repository has valid data with consistent hints.N and index.N
2. Index file gets corrupted (crash, disk error, etc.)
3. Borg detects corruption and auto-recovers:
   - Loads hints.N (with old reference counts)
   - Rebuilds index by replaying segments
   - Commits the rebuilt index
4. State is now inconsistent IF segments were deleted/lost:
   - self.segments[X] = 10 (from old hints, assumes segment X exists)
   - Segment X was actually deleted/lost
   - self.index has 0 entries for segment X (rebuilt from remaining segments)
5. During compact_segments():
   - Tries to iterate objects in segment X
   - Segment X doesn't exist (was deleted/lost)
   - OR: segment X exists but objects aren't in index (superseded)
   - segments[X] is never decremented
   - segments[X] remains 10 instead of becoming 0
   - Assertion fails!

WHY THE FIX IS CORRECT
----------------------

1. Hints are Advisory, Not Authoritative
   The hints file is an optimization to avoid scanning all segments. It's
   explicitly designed to be rebuildable from scratch by scanning segments.
   Therefore, incorrect hints should not cause a fatal error.

2. Self-Healing Behavior
   By converting the assertion to a warning and allowing compaction to proceed:
   - Compaction completes successfully
   - New hints are written with correct reference counts
   - Repository is automatically healed
   - No manual intervention required

3. Data Safety is Preserved
   The fix does NOT compromise data integrity because:
   - Compaction first copies all live data from segments to new segments
   - Only after all live data is safely copied are segments marked for deletion
   - The index determines what's "live" (authoritative source of truth)
   - Segments are deleted only when they contain no live data (per index)
   - The refcount warning indicates stale hints, not actual data loss risk
   - After compaction, new hints are written with correct counts

4. Consistent with Design Philosophy
   Borg already handles many corruption scenarios gracefully:
   - Missing hints → regenerated from segments
   - Corrupted index → rebuilt from segments
   - Missing segments → detected and handled
   This fix extends that philosophy to hint/index mismatches.

5. Alternative Solutions are Worse
   Other approaches considered:
   a) Crash and require manual intervention
      - Current behavior, user-hostile
      - Requires expert knowledge to fix
   b) Automatically run check --repair
      - Too aggressive, may hide real problems
      - User should decide when to repair
   c) Refuse to compact
      - Leaves repository in degraded state
      - Prevents normal operations

VERIFICATION
------------
The fix has been verified with test cases that reproduce both scenarios:

1. test_missing_segment_in_hints
   - Simulates missing segment files
   - Verifies compact succeeds and updates hints correctly

2. test_index_corruption_with_old_hints
   - Simulates the root cause: corrupted index with old hints
   - Verifies compact succeeds despite reference count mismatch

3. test_subtly_corrupted_hints_without_integrity
   - Existing test updated to expect warning instead of crash
   - Verifies repository remains consistent after compaction

OPERATIONAL IMPACT
------------------
After this fix:
1. Users experiencing this crash can now run `borg compact` successfully
2. The warning message alerts them to the inconsistency
3. They can optionally run `borg check --repair` for peace of mind
4. Repository continues to function normally

The warning message provides enough information for debugging while not
blocking normal operations.

CONCLUSION
----------
Converting the assertion to a warning is the correct fix because:
- It aligns with Borg's design philosophy of graceful degradation
- It enables self-healing behavior
- It preserves data safety
- It improves user experience
- It's consistent with how other corruption scenarios are handled

The assertion was overly strict for a data structure (hints) that is
explicitly designed to be advisory and rebuildable.
2025-12-01 23:11:28 +01:00
Thomas Waldmann
3226b5b084
add test for issue #8535 2025-12-01 20:27:57 +01:00
TW
3d32703336
Merge pull request #9198 from ThomasWaldmann/update-changes2-1.4
Some checks failed
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
update CHANGES
2025-11-30 17:16:07 +01:00
Thomas Waldmann
8fdae12c2d
update CHANGES
grammar / typo fixes
2025-11-30 14:17:19 +01:00
TW
dc3b938d0b
Merge pull request #9197 from ThomasWaldmann/update-changes-1.4
Some checks failed
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
CI / lint (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / native_tests (push) Has been cancelled
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Has been cancelled
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Has been cancelled
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Has been cancelled
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Has been cancelled
update CHANGES
2025-11-30 12:17:35 +01:00
Thomas Waldmann
691a5a3640
update CHANGES 2025-11-30 12:00:45 +01:00
TW
402dd30388
Merge pull request #9187 from ThomasWaldmann/fix-binary-build-gh-1.4
CI: install fuse bindings for binary build, fixes #9182
2025-11-30 11:25:30 +01:00
Thomas Waldmann
c845565c4a
CI: FUSE related fixes/improvements (1.4-maint)
fixes #9182

- install OS fuse support packages as indicated by the tox env
  on the macOS runners, we do not have any fuse support.
  on the linux runners, we may have fuse2 or fuse3.
  on FreeBSD, we have fuse2.
- install fuse python library for binary build
- first build/upload binaries, then run tests (including binary tests).
  early uploading makes inspection of a malfunctioning binary possible.
- for now, use llfuse, as there is an issue with pyinstaller and pyfuse3.

Also:
- remove || true - this just hides errors, not what we want.
2025-11-30 10:54:46 +01:00
TW
968c454b20
Merge pull request #9192 from ThomasWaldmann/backport-haiku-bsd-ci-1.4-maint
Some checks are pending
CI / lint (push) Waiting to run
CI / asan_ubsan (push) Blocked by required conditions
CI / native_tests (push) Blocked by required conditions
CI / vm_tests (Haiku, false, haiku, r1beta5) (push) Blocked by required conditions
CI / vm_tests (NetBSD, false, netbsd, 10.1) (push) Blocked by required conditions
CI / vm_tests (OpenBSD, false, openbsd, 7.7) (push) Blocked by required conditions
CI / vm_tests (borg-freebsd-14-x86_64-gh, FreeBSD, true, freebsd, 14.3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
CI: backport vm_tests (FreeBSD/NetBSD/OpenBSD/Haiku) from master branch
2025-11-29 19:30:02 +01:00
Thomas Waldmann
c43a78accf
freebsd/netbsd: set and display full hostname 2025-11-29 18:37:24 +01:00
Thomas Waldmann
4525e8dfe6
rename posix_tests -> native_tests 2025-11-21 23:53:23 +01:00
Thomas Waldmann
af707b802e
freebsd/netbsd: skip some tests we have github issues for
skip test_hard_link_deletion_and_replacement, #9147, #9153

The test fails on these platforms.

I could not find the root cause of this issue, but it is likely a minor
problem with ctime and doesn't affect borg usage much.

So I rather like to have CI on freebsd/netbsd not failing because of this.

tests(diff): on NetBSD only expect mtime for touched file in JSON diff
(treat like Windows); completes backport of #9161 to 1.4-maint layout

tests(diff): also skip DiffArchiverTestCase.test_multiple_link_exclusion
when hardlinks unsupported (include are_hardlinks_supported in skip)
2025-11-21 23:46:27 +01:00
Thomas Waldmann
0a553ddc9f
testsuite: improve haiku compatibility
haiku does not have os.mknod.
and it looks like it does not have hardlinks either.
2025-11-21 22:21:54 +01:00
Thomas Waldmann
a022d5623c
CI: backport vm_tests (FreeBSD/NetBSD/OpenBSD/Haiku) from master branch 2025-11-21 21:55:29 +01:00
TW
b386060f1a
Merge pull request #9191 from ThomasWaldmann/backport-9139-to-1.4-maint
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / asan_ubsan (push) Has been cancelled
CI / posix_tests (push) Has been cancelled
[1.4-maint] Backport: CI dynamic code analysis (#9139)
2025-11-21 20:38:33 +01:00
Thomas Waldmann
ea6450bf6f
tests: use context manager when opening files in patterns_test 2025-11-21 19:13:56 +01:00
Thomas Waldmann
5b92b86e13
CI: do dynamic code analysis, #6819
Use:
- AddressSanitizer ASan
- UndefinedBehaviorSanitizer UBSan

(cherry picked from commit add19da678)
2025-11-21 19:13:54 +01:00
TW
2a263ece85
Merge pull request #9190 from ThomasWaldmann/backport/pr-9151-to-1.4-maint
Backport: add granularity_sleep, fixes #9150 (from #9151) to 1.4-maint
2025-11-21 18:45:43 +01:00
Thomas Waldmann
e06935f275
add granularity_sleep, fixes #9150 2025-11-21 18:28:50 +01:00
TW
2b1fa7ad97
Merge pull request #9189 from ThomasWaldmann/backport-9188-to-1.4-maint
[1.4-maint] Backport PR #9188: docs: some fixes/update to the FAQ
2025-11-21 17:51:24 +01:00
Thomas Waldmann
3e7ecd6133
docs: some fixes/update to the FAQ
(cherry picked from commit 3642c7903b)
2025-11-21 17:45:42 +01:00
TW
9ff4f5dc98
Merge pull request #9137 from ThomasWaldmann/attestation-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / posix_tests (push) Has been cancelled
ci: add tag-based workflows and provenance attestation for GitHub-built binaries
2025-11-02 22:22:07 +01:00
Thomas Waldmann
a25856a33d
ci: add tag-based workflows and provenance attestation for GitHub-built binaries
backport of #9135 and #9136.
2025-11-02 22:06:08 +01:00
TW
7acb023d31
Merge pull request #9122 from ThomasWaldmann/drop-py39-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / pytest (push) Has been cancelled
drop Python 3.9 (EOL'ed by python.org) (1.4-maint)
2025-11-01 14:34:00 +01:00
Thomas Waldmann
cb0157ca55
drop Python 3.9 (EOL'ed by python.org) 2025-11-01 13:09:28 +01:00
TW
48a2887f9b
Merge pull request #9121 from ThomasWaldmann/release-process-updates-1.4
Some checks are pending
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
Release process updates (1.4-maint)
2025-11-01 12:39:54 +01:00
Thomas Waldmann
8388b074f8
update binary readme 2025-11-01 11:35:54 +01:00
Thomas Waldmann
6493743ffc
docs: release binaries are built on GitHub now
bad:
- no *BSD testing and FreeBSD binary building on gh
- binaries not signed by me, because they are built on gh

good:
- for linux intel/amd64 and arm64, built on ubuntu
- for macOS intel and arm64, built on a relatively recent macOS
- I can get rid of that ancient macOS VM I used for building.
- the source code distribution (sdist) is still made locally on
  my machine and thus signed with my signature (*.asc).
2025-11-01 11:35:51 +01:00
Thomas Waldmann
0036fe3a92
upload-pypi: use borg-specific repository names
so it finds the right api token in .pypirc
2025-10-31 19:46:03 +01:00
TW
4c5eb36b86
Merge pull request #9119 from ThomasWaldmann/rel142
Some checks are pending
CI / lint (push) Waiting to run
CI / pytest (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
Release 1.4.2
2025-10-31 19:32:47 +01:00
Thomas Waldmann
8e1a4248c0
build_usage build_man 2025-10-31 19:12:42 +01:00
Thomas Waldmann
92486d82e9
update CHANGES 2025-10-31 19:02:42 +01:00
Thomas Waldmann
6e19159385
run test_extract_restores_append_flag also on freebsd 2025-10-31 18:49:10 +01:00
TW
90457eba26
Merge pull request #9117 from ThomasWaldmann/vagrant-updates-1.4
Vagrant updates (1.4-maint)
2025-10-31 17:59:06 +01:00
Thomas Waldmann
7ba091b8a4
vagrant: use python 3.11.14 for the binary build 2025-10-31 16:34:17 +01:00
Thomas Waldmann
93cc4faa1a
coverage: disable no-ctracer warning
On a BSD vagrant box, it spams us with these warnings...

It also works without the c-tracer, just slower.
2025-10-31 16:34:16 +01:00
Thomas Waldmann
05b63a41fe
vagrant: fix openindiana box
- working nameserver
- run pkg update and reboot, otherwise it does not let us
  install gcc-13 and git.
2025-10-31 16:33:33 +01:00
Thomas Waldmann
a706fbdde2
vagrant: add a OpenBSD 7.7 box
The base box has an invalid hostname and no comp package installed,
so we need to fix that.
2025-10-31 15:35:30 +01:00
TW
91a360496f
Merge pull request #9116 from ThomasWaldmann/fix/freebsd-flags-9090
Some checks are pending
CI / lint (push) Waiting to run
CI / pytest (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
extract: fs flags: use get/set to influence only specific flags, #9090, FreeBSD only
2025-10-31 01:57:45 +01:00
Thomas Waldmann
8a895f55ae
extract: fs flags: use get/set to influence only specific flags, #9090, FreeBSD only. 2025-10-31 00:41:52 +01:00
TW
631046c63c
Merge pull request #9115 from ThomasWaldmann/fix/macos-flags-9090
extract: fs flags: use get/set to influence only specific flags, #9039, macOS only
2025-10-30 23:30:49 +01:00
Thomas Waldmann
83571aa00d
extract: fs flags: use get/set to influence only specific flags, #9039, macOS only.
preserve UF_COMPRESSED and SF_DATALESS when restoring flags,
get-modify-set in macOS set_flags, keeping system-managed read-only flags.
2025-10-30 23:09:33 +01:00
TW
3082c59e9a
Merge pull request #9114 from ThomasWaldmann/doc/compact-threshold-max-9112
Some checks are pending
CI / lint (push) Waiting to run
CI / pytest (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
doc: compact threshold — document how to get maximum compaction (#9112)
2025-10-30 11:16:00 +01:00
Thomas Waldmann
177f7d5b9a
docs(compact): explain how to get maximum compaction with --threshold 0 and trade-offs (#9112, discussion #8716) 2025-10-30 11:08:17 +01:00
TW
83d9d28fa4
Merge pull request #9103 from PhrozenByte/docs-repokey-change-passphrase
Some checks are pending
CI / lint (push) Waiting to run
CI / pytest (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
Docs: Rewrite `borg init --encryption` docs
2025-10-29 19:24:45 +01:00
Daniel Rudolf
3edf5b3475
Docs: Improve borg init encryption table formatting 2025-10-29 17:48:14 +01:00
Thomas Waldmann
9226d9c6a9
reformat init_epilog to 80chars wide 2025-10-29 13:26:31 +01:00
Thomas Waldmann
b22cc77453
better clarity, generalisation, precision, less typos 2025-10-29 13:10:45 +01:00
Thomas Waldmann
3c2daef474
fix typos and grammar
Junie AI + TW
2025-10-29 11:19:49 +01:00
Thomas Waldmann
4eb6773c2d
fix rst_plain_text_references 2025-10-29 10:52:56 +01:00
TW
eb63682df6
Merge pull request #9111 from borgbackup/msgpack-check-env-var-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / pytest (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
BORG_MSGPACK_VERSION_CHECK=no to disable the version check, fixes #9109
2025-10-29 01:36:09 +01:00
Thomas Waldmann
2d63dc9a4f
BORG_MSGPACK_VERSION_CHECK=no to disable the version check, fixes #9109 2025-10-28 23:20:53 +01:00
TW
b712d55de9
Merge pull request #9096 from ThomasWaldmann/fix-9095-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / pytest (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Windows CI / msys2-ucrt64 (push) Waiting to run
json: include archive keys in JSON lines when requested via --format, fixes #9095
2025-10-27 22:19:58 +01:00
Thomas Waldmann
0c27bd3047
json: include archive keys in JSON lines when requested via --format, fixes #9095
Add a test to ensure correct behavior.
2025-10-27 22:04:01 +01:00
TW
0f85500314
Merge pull request #9106 from ThomasWaldmann/update-changes-1.4
update CHANGES (1.4-maint)
2025-10-27 22:03:02 +01:00
Thomas Waldmann
88ec307895
update CHANGES
some new entries.
also fixed typos/grammar.
2025-10-27 21:54:10 +01:00
TW
7a69499c0b
Merge pull request #9003 from ThomasWaldmann/fix-8898
fix inconsistencies in original size computation, fixes #8898 (1.4-maint)
2025-10-27 21:50:49 +01:00
Thomas Waldmann
ce27418791
create/info: fix discrepancies in archive stats, fixes #8898, #9003
do not account archive metadata, only file contents.
2025-10-27 21:36:37 +01:00
Thomas Waldmann
ff35e07997
tests: add tests for stats consistency, #8898 #9003
add tests ensuring:
- borg info and create report same "This archive" deduplicated size
- borg info and create report same "All archives" deduplicated size
- before/after borg recreate it reports same "This archive" deduplicated size

note that some stats differences are expected, because the repo-level
deduplication ("All archives") is computed in a different way than "This archive".
2025-10-27 21:35:08 +01:00
Daniel Rudolf
d42453fde4
Docs: Rewrite borg init docs 2025-10-26 23:13:59 +01:00
TW
cec8d87a5e
Merge pull request #9089 from ThomasWaldmann/fix-set_flags-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / pytest (push) Has been cancelled
extract: misc. fixes for restoring linux fs flags
2025-10-25 03:34:35 +02:00
Thomas Waldmann
56dda84162
set_flags: remove compression flag
This flag needs to be set BEFORE writing to the file.
But "borg extract" sets the flags last (to support IMMUTABLE),
thus the compression flag would not work as expected.
2025-10-17 02:41:53 +02:00
Thomas Waldmann
9c600a9571
set_flags: better give up than corrupt
Thanks to Earnestly for the feedback on IRC.
2025-10-17 00:27:56 +02:00
Thomas Waldmann
9214197a2c
set_flags: use get/set to only influence specific flags, fixes #9039
Linux platform only.
2025-10-16 21:59:28 +02:00
TW
4cbc893f31
Merge pull request #9078 from ThomasWaldmann/ci-windows-yaml-fix-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
Windows CI / msys2-ucrt64 (push) Has been cancelled
CI / pytest (push) Has been cancelled
CI: fix indentation in windows.yml
2025-10-15 18:24:19 +02:00
Thomas Waldmann
687c34e89b
CI: fix indentation in windows.yml 2025-10-15 17:42:12 +02:00
TW
b18fe879ed
Merge pull request #9077 from ThomasWaldmann/brewfile-1.4
Brewfile: use openssl@3
2025-10-15 17:34:03 +02:00
Thomas Waldmann
5a5f596367
Brewfile: use openssl@3
@3.0 is old, @3 installs something recent (or
even that is installed already).
2025-10-15 17:20:30 +02:00
TW
6715dedda8
Merge pull request #9076 from ThomasWaldmann/fix-grammar-typos-changelog-1.4
CHANGES: fix typos / grammar (1.4-maint)
2025-10-15 17:16:41 +02:00
Thomas Waldmann
a88d1eb5d5
CHANGES: fix typos / grammar 2025-10-15 17:10:33 +02:00
TW
37c3d0d7d6
Merge pull request #9075 from ThomasWaldmann/update-changes-1.4
update CHANGES (1.4-maint)
2025-10-15 16:28:06 +02:00
Thomas Waldmann
a2f0cda69e
update CHANGES 2025-10-15 16:22:37 +02:00
TW
bd4c9ea35c
Merge pull request #9074 from ThomasWaldmann/drop-macos-vm-1.4
vagrant: drop macOS 10.12 VM
2025-10-15 16:17:32 +02:00
Thomas Waldmann
d3d1d97f87
vagrant: drop macOS 10.12 VM
The VM was used for local macOS testing and
also for building a macOS intel fat binary.

We also do macOS CI testing on GitHub and I
recently added binary building on GitHub for
Apple Silicon and Intel.

The macOS 10 VM was very outdated, super slow
and a pain to use. I didn't succeed in building
a recent macOS vagrant VM, so we'll just use
GitHub from now on...
2025-10-15 16:11:56 +02:00
TW
94db900579
Merge pull request #9071 from ThomasWaldmann/build-binaries-on-gh-fixes5-1.4
Some checks are pending
CodeQL / Analyze (push) Waiting to run
go back to macOS 13 for Intel
2025-10-14 22:51:51 +02:00
Thomas Waldmann
4d6ad22353
go back to macOS 13 for Intel
there are no other free macOS Intel gh
action runners for opensource projects. :-(

the macOS 13 runner is deprecated, so guess
it will be game-over for macOS Intel 2025-12.
2025-10-14 22:50:07 +02:00
TW
0e26390ff2
Merge pull request #9070 from ThomasWaldmann/build-binaries-on-gh-fixes4-1.4
fix macos-14 label
2025-10-14 22:35:00 +02:00
Thomas Waldmann
d595f04398
fix macos-14 label
macos-14-intel does not exist (yet?).
2025-10-14 22:33:38 +02:00
TW
77753318a9
Merge pull request #9069 from ThomasWaldmann/build-binaries-on-gh-fixes3-1.4
use macos-14 for the intel binary
2025-10-14 22:24:52 +02:00
Thomas Waldmann
1a75569590
use macos-14 for the intel binary
the macOS 13 runner is already deprecated
and going away in 2 months already.
2025-10-14 22:16:08 +02:00
TW
ef65705a71
Merge pull request #9068 from ThomasWaldmann/build-binaries-on-gh-fixes2-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / pytest (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Build binaries on gh: fixes2 (1.4-maint)
2025-10-14 19:35:43 +02:00
Thomas Waldmann
1759ff295a
ci: check out exact tag during tagged commit builds 2025-10-14 19:30:28 +02:00
Thomas Waldmann
815aa560ac
ci: improved tagged commit detection 2025-10-14 19:27:28 +02:00
TW
f3d2aea67d
Merge pull request #9066 from ThomasWaldmann/build-binaries-on-gh-fixes-1.4
binary building on gh: fixes
2025-10-14 19:04:40 +02:00
Thomas Waldmann
5a3a0c4808
ci: build and test binaries only for tagged commits 2025-10-14 18:32:48 +02:00
Thomas Waldmann
a768027de6
binaries: don't use .exe extension 2025-10-14 15:36:39 +02:00
Thomas Waldmann
424ac2e1c0
update README for the binaries 2025-10-14 15:36:37 +02:00
Thomas Waldmann
14fa8be25c
update fetch-binaries script 2025-10-14 15:29:22 +02:00
Thomas Waldmann
43a39f5f08
binaries: directly give the binary name 2025-10-14 15:23:56 +02:00
Thomas Waldmann
8a4782d1aa
ci: speed up pull requests
only run py39 fuse2 ubuntu 22.04 and py314 fuse3 ubuntu 24.04.
there are no binary builds in these matrix entries.

the macOS runners are usually super slow.
testing all pythons and building binaries consumes quite some
resources.
thus, only do that on "push" (e.g. when the PR is merged).
2025-10-14 15:23:54 +02:00
Thomas Waldmann
33ab2dc688
binary builds for macOS and Ubuntu 2025-10-14 15:23:53 +02:00
TW
71db823d92
Merge pull request #9064 from ThomasWaldmann/drop-buster-vm-1.4
vagrant: drop debian buster VM / borg-linux-glibc228
2025-10-14 14:37:24 +02:00
TW
b2d92d3091
Merge pull request #9065 from ThomasWaldmann/build-binaries-on-gh-1.4
Some checks failed
CI / lint (push) Waiting to run
CI / pytest (macos-14, 3.11, py311-none) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.10, py310-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.11, py311-fuse2) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.9, py39-fuse2) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.12, py312-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.13, py313-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.14, py314-fuse3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
Build binaries / Build (macOS macos-12 / ) (push) Has been cancelled
Build binaries / Build (macOS macos-14 / ) (push) Has been cancelled
ci: add GitHub Actions workflow for building binaries on macOS
2025-10-14 00:16:14 +02:00
Thomas Waldmann
116c2eee23
ci: add GitHub Actions workflow for building binaries on macOS 2025-10-14 00:12:49 +02:00
Thomas Waldmann
32754d0657
vagrant: drop debian buster VM / borg-linux-glibc228
vagrant fails when trying to install rsync.

buster is EOL, even in LTS.

you can still use the binary from borg <= 1.4.1.
2025-10-13 21:54:37 +02:00
TW
f9b23ca5ab
Merge pull request #9061 from ThomasWaldmann/document-extract-wildcard-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / pytest (macos-14, 3.11, py311-none) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.10, py310-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.11, py311-fuse2) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.9, py39-fuse2) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.12, py312-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.13, py313-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.14, py314-fuse3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
extract: document how to use wildcards in PATHs
2025-10-13 19:59:18 +02:00
Atemu
908c506e3f
extract: document how to use wildcards in PATHs
Fixes https://github.com/borgbackup/borg/issues/8589
2025-10-13 19:54:49 +02:00
TW
924d94be75
Merge pull request #9054 from ThomasWaldmann/fix-7144-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / pytest (macos-14, 3.11, py311-none) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.10, py310-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.11, py311-fuse2) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.9, py39-fuse2) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.12, py312-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.13, py313-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.14, py314-fuse3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
docs: improve borg help patterns, fixes #7144
2025-10-12 15:24:42 +02:00
TW
81b3b767be
Merge pull request #9057 from ThomasWaldmann/mount-show-rc-1.4
mount --show-rc: display main process rc, fixes #8308
2025-10-12 15:19:11 +02:00
TW
cb19d62057
Merge pull request #9055 from ThomasWaldmann/non-tty-progress-1.4
improved tty-less progress reporting (1.4-maint)
2025-10-12 15:18:09 +02:00
Thomas Waldmann
41bd6dd35b
docs: improve borg help patterns, fixes #7144 2025-10-12 14:21:37 +02:00
TW
6cd3b162a8
Merge pull request #9058 from ThomasWaldmann/patterns-style-1.4 2025-10-12 14:02:29 +02:00
Thomas Waldmann
f4a6ea370b
patterns: clarify scope of default pattern style, fixes #9004 2025-10-11 20:07:26 +02:00
Thomas Waldmann
702de47197
mount --show-rc: display main process rc, fixes #8308
when borg mount is used without -f/--foreground (so that the FUSE
borg process was started daemonized in the background), it did not
display the rc of the main process, even when --show-rc was used.

now it does display the rc of the main process.

note that this is rather a consistency fix than being super useful,
because the main "action" happens in the background daemon process,
not in the main process.
2025-10-11 18:55:54 +02:00
Thomas Waldmann
3e2f560a61
improved tty-less progress reporting (1.4-maint)
Previously when running borg in a systemd service (and similar when piping to
a file and co.), these problems occurred:

- The carriage return both made it so that journald interpreted the output as
  binary, therefore not printing the text, while also not buffering
  correctly, so that log output was only available every once in a while
  in the form [40k blob data]. This can partially be worked around by
  using `journalctl -a` to view the logs, which at least prints the text,
  though only sporadically.

- The path was getting truncated to a short length, since the default
  get_terminal_size returns a column width of 80, which isn't relevant
  when printing to e.g. journald.

This commit fixes this by introducing a new code path for when stream is
not a tty, which always prints the full paths and ends lines with a linefeed.

This is based on unfinished PR #8939 by @infinisil, thanks for your suggestion!
2025-10-11 14:17:02 +02:00
TW
98b831ba8d
Merge pull request #9049 from ThomasWaldmann/keyfile-same-repo-path-1.4
Some checks failed
CI / lint (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
CI / pytest (macos-14, 3.11, py311-none) (push) Has been cancelled
CI / pytest (ubuntu-22.04, 3.10, py310-fuse3) (push) Has been cancelled
CI / pytest (ubuntu-22.04, 3.11, py311-fuse2) (push) Has been cancelled
CI / pytest (ubuntu-22.04, 3.9, py39-fuse2) (push) Has been cancelled
CI / pytest (ubuntu-24.04, 3.12, py312-fuse3) (push) Has been cancelled
CI / pytest (ubuntu-24.04, 3.13, py313-fuse3) (push) Has been cancelled
CI / pytest (ubuntu-24.04, 3.14, py314-fuse3) (push) Has been cancelled
docs: What happens when a new keyfile repo is created at the same path?
2025-10-10 20:25:10 +02:00
TW
5a1f21fee2
Merge pull request #9052 from ThomasWaldmann/borg-mount-debugging-1.4
docs: how to debug borg mount, fixes #5461
2025-10-10 20:19:16 +02:00
Thomas Waldmann
92418883dc
docs: how to debug borg mount, fixes #5461 2025-10-10 14:57:25 +02:00
Thomas Waldmann
80818c1588
docs: What happens when a new keyfile repo is created at the same path?, fixes #6230
Also add a test.
2025-10-10 14:12:48 +02:00
TW
b5ced17534
Merge pull request #9048 from ThomasWaldmann/fix-3318-1.4
Some checks are pending
CodeQL / Analyze (push) Waiting to run
docs: borg serve: recommend using a simple shell, fixes #8318
2025-10-10 13:54:36 +02:00
Thomas Waldmann
5bce73301d
docs: borg serve: recommend using a simple shell, fixes #8318 2025-10-10 13:48:26 +02:00
TW
bc42d80fcc
Merge pull request #9047 from ThomasWaldmann/update-changes-1.4
Some checks are pending
CI / lint (push) Waiting to run
CI / pytest (macos-14, 3.11, py311-none) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.10, py310-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.11, py311-fuse2) (push) Blocked by required conditions
CI / pytest (ubuntu-22.04, 3.9, py39-fuse2) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.12, py312-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.13, py313-fuse3) (push) Blocked by required conditions
CI / pytest (ubuntu-24.04, 3.14, py314-fuse3) (push) Blocked by required conditions
CodeQL / Analyze (push) Waiting to run
update CHANGES
2025-10-10 03:31:41 +02:00
Thomas Waldmann
67e846568b
update CHANGES 2025-10-09 20:42:27 +02:00
TW
dfd3b0e6ee
Merge pull request #9046 from ThomasWaldmann/backport-9042-setuptools-min-1.4
backport(1.4): build: require setuptools>=78.1.1
2025-10-09 20:05:38 +02:00
TW
e7d8fba1df
Merge pull request #9044 from ThomasWaldmann/py314-1.4
python 3.14 release, use recent cython, allow latest msgpack
2025-10-09 20:05:14 +02:00
Thomas Waldmann
0069ce1f2d
backport(1.4): build: require setuptools>=78.1.1 to address GHSA-5rjg-fvgr-3xxf (backport of #9045) 2025-10-09 17:56:53 +02:00
Thomas Waldmann
bbaac19c94
Cython 3.1.4 2025-10-09 17:39:12 +02:00
Thomas Waldmann
d35ec93968
allow msgpack 1.1.2
1.1.2 provides wheels for py 3.14.
2025-10-09 17:38:10 +02:00
Thomas Waldmann
4915c26795
test on Python 3.14 (w/o -dev) 2025-10-09 17:36:03 +02:00
TW
edb758e8f7
Merge pull request #9023 from ThomasWaldmann/fix-readonly-tests-1.4
read_only CM: skip test if cmd_immutable is unsuccessful, fixes #9021
2025-09-22 15:12:29 +02:00
Thomas Waldmann
4edd0e6e9d
read_only CM: skip test if cmd_immutable is unsuccessful, fixes #9021 2025-09-22 11:17:16 +02:00
TW
aa39d63718
Merge pull request #9020 from PhrozenByte/docs-build-src-version-1.4
Update install docs to include `SETUPTOOLS_SCM_PRETEND_VERSION` (v1.4 backport)
2025-09-21 17:26:40 +02:00
Daniel Rudolf
465937f5a3
Unify master and 1.4-maint install docs 2025-09-21 15:54:48 +02:00
Daniel Rudolf
d9dae4220f
Update install docs to include SETUPTOOLS_SCM_PRETEND_VERSION 2025-09-21 15:53:04 +02:00
TW
9053596cb1
Merge pull request #9019 from PhrozenByte/docs-arch-dev-deps-1.4
Add Arch Linux to installing from source docs (v1.4 backport)
2025-09-21 15:52:51 +02:00
Daniel Rudolf
1066154106
Add Arch Linux to installing from source docs
Note that Arch Linux usually has no separate `-dev` packages, but ships development headers with the normal package.
2025-09-21 15:39:41 +02:00
TW
07a4f2ec5a
Merge pull request #9005 from ThomasWaldmann/diff-sort-field-1.4
diff: --sort-by=field[,field,...], fixes #8998
2025-09-19 14:29:38 +02:00
Thomas Waldmann
7f9dd37536
diff --sort-by: enhanced sorting, fixes #8998
use borg diff --sort-by=spec1,spec2,spec2 for enhanced sorting.

keep legacy --sort behaviour (sort by path) for compatibility,
but deprecate it.

Co-authored-by: Daniel Rudolf <github.com@daniel-rudolf.de>
2025-09-18 23:32:23 +02:00
Thomas Waldmann
f837e6a27a
preprocess_args: fix option name matching
previously, it also matched on --sort-by when it only should have matched on --sort.
2025-09-18 23:32:10 +02:00
TW
b91fb89466
Merge pull request #9002 from ThomasWaldmann/chunker-fuzzing-test-1.4
fuzzing test for default chunker
2025-09-03 01:33:40 +02:00
Thomas Waldmann
9485564fcf
fuzzing test for default chunker 2025-09-02 21:10:27 +02:00
TW
f8d9f8c371
Merge pull request #8990 from ThomasWaldmann/typos-grammar-1.4
Typos / grammar (1.4-maint)
2025-08-17 18:32:53 +02:00
Thomas Waldmann
9973296c09
update frontend.rst
there were some fixes to error messages in the code.
2025-08-17 17:24:01 +02:00
TW
7f22f6ce95
Merge pull request #8985 from ThomasWaldmann/vagrant-debian-testing-1.4
Vagrant: debian testing/trixie box (1.4-maint)
2025-08-17 16:50:28 +02:00
Thomas Waldmann
3f9ad883cb
manual fixes
sometimes the AI got it wrong.
also some other fixes.
2025-08-17 16:05:51 +02:00
Thomas Waldmann
2a25764273
binary readme: fix typos and grammar 2025-08-12 17:04:46 +02:00
Thomas Waldmann
87af3a4b47
paperkey.html: fix typos, grammar, html structure
The original markup included a paragraph element wrapping a block-level pre element, which is invalid per HTML’s content model (a p can only contain phrasing content; pre is flow content).

The fix separated text and pre blocks into valid sibling elements, ensuring no pre is nested inside a p.
2025-08-12 16:56:00 +02:00
Thomas Waldmann
48ad1a8fa8
README: fix typos and grammar 2025-08-12 11:21:50 +02:00
Thomas Waldmann
1bb32057d2
github workflows: fix typos and grammar 2025-08-12 11:15:11 +02:00
Thomas Waldmann
1b68676b8e
github templates: fix typos and grammar 2025-08-11 22:21:49 +02:00
Thomas Waldmann
eab8be76a0
_*.c: fix typos and grammar 2025-08-11 21:15:14 +02:00
Thomas Waldmann
c956e1f50e
changes.rst: fix typos and grammar 2025-08-11 18:01:24 +02:00
Thomas Waldmann
71888265f9
epilogs: fix typos and grammar 2025-08-11 17:51:26 +02:00
Thomas Waldmann
a4b2bb3d76
.pyx files: fix typos and grammar 2025-08-11 16:56:11 +02:00
TW
3f824ac2dc
Merge pull request #8991 from dataprolet/backport/add-faq
Backport/add-faq
2025-08-11 15:50:19 +02:00
dataprolet
ad44548b04 Adds systemd-inhibit and examples (#8989) 2025-08-11 15:35:28 +02:00
Thomas Waldmann
6602cf1679
docs: fix typos and grammar 2025-08-11 15:26:24 +02:00
Thomas Waldmann
404bb1ca20
scripts: fix typos and grammar 2025-08-10 22:06:55 +02:00
Thomas Waldmann
a7f130f146
toplevel files: fix typos and grammar 2025-08-10 20:40:03 +02:00
Thomas Waldmann
477366f4a5
tests: fix typos and grammar 2025-08-10 17:40:03 +02:00
Thomas Waldmann
3e6a9d677f
code: fix typos and grammar 2025-08-10 15:51:29 +02:00
TW
90e745212f
Merge pull request #8986 from ThomasWaldmann/fuse-testing-config-1.4
FUSE testing config (1.4-maint)
2025-08-09 16:53:25 +02:00
Thomas Waldmann
890605ce5f
pyproject.toml: correctly define test envs for fuse testing
Looks like the toml config format of tox does not yet support
an easier way for generative test environments.
2025-08-09 15:59:44 +02:00
Thomas Waldmann
0991d38e07
pyinstaller spec: avoid pkg_resources warning 2025-08-09 00:29:49 +02:00
Thomas Waldmann
69e009ec82
vagrant: add debian trixie box
currently still known as "testing",
but soon to be released.
2025-08-08 23:13:17 +02:00
TW
d756b936bf
Merge pull request #8984 from ThomasWaldmann/requirements-1.4
update requirements (1.4-maint)
2025-08-08 23:11:54 +02:00
Thomas Waldmann
27ae6971fd
vagrant: use Python 3.11.13 2025-08-08 21:29:39 +02:00
Thomas Waldmann
461f9d7a3e
update requirements.lock.txt to current versions 2025-08-08 21:27:58 +02:00
TW
cf8303217c
Merge pull request #8983 from ThomasWaldmann/license-spdx-1.4
pyproject.toml: SPDX expression for license, add license-files, see #8771
2025-08-08 20:57:06 +02:00
Thomas Waldmann
0d2b866df8
pyproject.toml: SPDX expression for license, add license-files, see #8771
Also: raise setuptools version requirement appropriately.

In case the setuptools requirement turns out to be a problem when
packaging borg for some OS distribution and an older setuptools must
be used, just do a reverse patch when packaging borg - there should
be no problem as long as you use some older setuptools.
2025-08-08 17:20:12 +02:00
TW
28bcefded8
Merge pull request #8978 from ThomasWaldmann/chunkerparams-validation-1.4
fix ChunkerParams validation
2025-08-05 16:14:15 +02:00
Thomas Waldmann
a361a5d02e
fix ChunkerParams validation
logging a warning there does not work, because the logging
system is not initialized yet, thus we always raise an error.

also: fix tests, so they don't use invalid window size.
2025-08-05 03:30:19 +02:00
TW
ebfe007cba
Merge pull request #8959 from ThomasWaldmann/files-changed-option
create --files-changed=MODE option
2025-08-03 15:49:23 +02:00
TW
3f9ea06b20
Merge pull request #8974 from ThomasWaldmann/update-changes
update CHANGES
2025-08-03 06:39:11 +02:00
Thomas Waldmann
798350b633
update CHANGES 2025-08-03 06:24:04 +02:00
TW
cdf884be60
Merge pull request #8962 from ThomasWaldmann/py314
support Python 3.14
2025-07-26 11:14:42 +02:00
Thomas Waldmann
779a94bb20
support Python 3.14 2025-07-25 13:36:31 +02:00
TW
89b627206e
Merge pull request #8961 from FelixSchwarz/1.4-maint
tests: fix ZeroDivisionError msg test for py314 (for borg 1.4)
2025-07-24 23:56:35 +02:00
Felix Schwarz
e934ed01dc tests: fix ZeroDivisionError msg test for py314 (for borg 1.4) 2025-07-24 22:11:55 +02:00
Thomas Waldmann
e5d04710ad
create --files-changed=MODE option
control how borg detects whether a file has changed while it was backed up, valid modes are ctime, mtime or disabled.

ctime is the safest mode and the default.

mtime can be useful if ctime does not work correctly for some reason
(e.g. OneDrive files change their ctime without the user changing the file).

disabled (= disabling change detection) is not recommended as it could lead to
inconsistent backups. Only use if you know what you are doing.
2025-07-19 00:39:40 +02:00
TW
a557251ec4
Merge pull request #8953 from ThomasWaldmann/test-save-temp-space-1.4
tests: save temp space (1.4-maint)
2025-07-06 17:21:29 +02:00
TW
33929210a8
Merge pull request #8952 from ThomasWaldmann/coverage-pyproject-1.4
coverage: use pyproject.toml (1.4-maint)
2025-07-06 17:20:55 +02:00
Thomas Waldmann
5dab679ee8
tests: test_list_chunk_counts, save space on TMPDIR 2025-07-06 16:37:15 +02:00
Thomas Waldmann
a4cb2d7ff3
tests: test_max_data_size, save space on TMPDIR 2025-07-06 16:34:52 +02:00
Thomas Waldmann
43610a1048
tests: test_sparse_file, save space on TMPDIR 2025-07-06 16:30:31 +02:00
Thomas Waldmann
75e5db0857
coverage: use pyproject.toml 2025-07-06 15:59:56 +02:00
TW
16feb6d42c
Merge pull request #8945 from ThomasWaldmann/tar-use-normalized-path-1.4
import-tar: when printing the path, use the already normalized item.path.
2025-06-28 14:10:36 +02:00
Thomas Waldmann
f69561aeda
import-tar: when printing the path, use the already normalized item.path. 2025-06-28 12:13:16 +02:00
TW
f5301e20e2
Merge pull request #8941 from ThomasWaldmann/import-tar-dotslash-1.4
import-tar dotslash issue (1.4-maint)
2025-06-27 11:39:31 +02:00
Thomas Waldmann
15d6f031e1
import-tar: fix the dotslash issue, add test 2025-06-27 10:01:23 +02:00
TW
65ed665f41
Merge pull request #8909 from ThomasWaldmann/backports-1.4
Backports to 1.4-maint
2025-06-09 18:39:06 +02:00
Thomas Waldmann
188e96c60b
msgpack version check: ignore "rc" or other version elements
Only compare the main version number, e.g. 1.1.1 (first 3 elements
of the version tuple).

Without this change, it would not accept 1.1.1rc1 because that is
not "<= (1, 1, 1)" in that simplistic version comparison.
2025-06-09 14:52:13 +02:00
Thomas Waldmann
f6724bfef2
msgpack: allow 1.1.1
1.1.1rc1 looked good in testing, so hopefully 1.1.1 will also be ok.
2025-06-09 14:50:37 +02:00
Thomas Waldmann
850734be77
reformat text 2025-06-09 14:45:57 +02:00
Thomas Waldmann
144a52f4ee
fix grammar 2025-06-09 14:43:47 +02:00
Max-Julian Pogner
c77f005307
docs: borg-serve: simplify example of env in authorized_keys
see #8318

so long as it can be assumed that the user has configured a POSIX
compliant login shell, using a simple command [1] looks cleaner, as
no ``export`` or ``;`` are used.

[1] Section "2.9.1 Simple Commands" in volume "Shell & Utilities" of POSIX.1-2024
2025-06-09 14:43:11 +02:00
TW
391248da6c
Merge pull request #8889 from Atharva-Varpe/1.4-maint
tox: use native pyproject.toml configuration
2025-05-30 11:33:14 +02:00
Atharva Varpe
330497e230
Update pyproject.toml 2025-05-30 12:05:14 +05:30
TW
6aba879f12
Merge pull request #8890 from ThomasWaldmann/refactor-chunker-slow-1.4
test_chunkpoints_unchanged: do not use blake2b_256
2025-05-29 19:36:49 +02:00
Thomas Waldmann
217cb79cb4
test_chunkpoints_unchanged: do not use blake2b_256
we want to get rid of legacy stuff(*) one day and sha256 is as
good for this purpose (and might be even hw accelerated).

(*) considered legacy due to the way it gives the key to the
blake2b function (just padding and prepending it to the data,
instead of using the key parameter, see #8867 ).
2025-05-29 16:32:49 +02:00
Atharva Varpe
1b77ddf5f9 tox: use native pyproject.toml configuration 2025-05-29 08:19:15 +00:00
TW
0736bf1239
Merge pull request #8877 from ThomasWaldmann/chunker-params-warn-even-window-size
Chunker params: warn for even window size (1.4)
2025-05-23 17:47:53 +02:00
Thomas Waldmann
1de41731f9
ChunkerParams: reject even window size for buzhash, fixes #8868 2025-05-23 11:00:56 +02:00
TW
6799b8756b
Merge pull request #8875 from ThomasWaldmann/backports-1.4
Backports to 1.4-maint
2025-05-23 10:11:15 +02:00
Thomas Waldmann
fd0b1ace59
cosmetic: move padme obfuscation method closer to the other o. methods 2025-05-23 08:34:37 +02:00
Thomas Waldmann
ade92e9e2c
docs: minor formatting changes in development docs 2025-05-23 08:30:19 +02:00
Thomas Waldmann
775733e585
docs: announce releases on Mastodon, bsky, X 2025-05-23 08:28:11 +02:00
Thomas Waldmann
9773a8ea19
docs: refer to more current 1.4-maint 2025-05-23 08:27:52 +02:00
Thomas Waldmann
77eb5992fe
tests: fix diff cmd test on macOS HFS+, fixes #8860
If we touch too quickly after file creation, it would not noticably update the timestamp.
2025-05-23 08:20:53 +02:00
Thomas Waldmann
830c5c8fea
suppress compiler warning about CYTHON_FALLTHROUGH 2025-05-23 08:14:43 +02:00
Thomas Waldmann
4c45087698
minor cython code fixes 2025-05-23 08:13:19 +02:00
Thomas Waldmann
8e08273128
minor tox.ini improvements
- consistent indentation with spaces
- use pytest, not py.test
- consistent empty lines between sections
- remove empty changedir lines
2025-05-23 08:10:05 +02:00
TW
66a5e95e74
Merge pull request #8770 from ThomasWaldmann/simplify-padme-1.4
Remove unnecessary check that Padmé overhead is at most 12%
2025-04-19 18:55:54 +02:00
ash
1011268437
Remove unnecessary check that Padmé overhead is at most 12%
It's easy enough to verify exhaustively for any plausible chunker params
that Padmé always produces at most a 12% overhead. Checking that again
at runtime is pointless.
2025-04-19 11:54:10 +02:00
TW
a40bbdbee2
Merge pull request #8766 from ThomasWaldmann/rel141
Release 1.4.1
2025-04-18 22:01:10 +02:00
Thomas Waldmann
ecf2dfa170
build_usage / build_man 2025-04-18 18:26:48 +02:00
Thomas Waldmann
1bed2155d8
update CHANGES 2025-04-18 18:24:59 +02:00
TW
51434dd038
Merge pull request #8765 from ThomasWaldmann/padme-obfuscate-1.4
implement padme chunk size obfuscation (SPEC 250), fixes #8705
2025-04-18 17:57:57 +02:00
Divyansh Agrawal
3880c7e127
implement padme chunk size obfuscation (SPEC 250), fixes #8705
(backport from master)
2025-04-18 17:25:03 +02:00
TW
a6813cea24
Merge pull request #8763 from ThomasWaldmann/ignore-macos-provenance-xattr-1.4
tests: ignore 'com.apple.provenance' xattr (macOS specific)
2025-04-18 15:10:57 +02:00
Divyansh Agrawal
490a841717
tests: ignore 'com.apple.provenance' xattr (macOS specific) 2025-04-17 20:45:40 +02:00
TW
972750402f
Merge pull request #8762 from ThomasWaldmann/update-changes-1.4
update CHANGES
2025-04-17 20:32:08 +02:00
Thomas Waldmann
49fa6c476a
update CHANGES 2025-04-17 20:26:44 +02:00
TW
611708a1f2
Merge pull request #8759 from ThomasWaldmann/fix-nfs4-acl-freebsd-1.4
freebsd: fix nfs4 acl processing, fixes #8756 (1.4-maint)
2025-04-17 17:17:14 +02:00
TW
dd98328c17
Merge pull request #8755 from ThomasWaldmann/fix-remote-modern-exit-codes-1.4
fix remote repository exception handling / modern exit codes, fixes #8631
2025-04-17 17:01:58 +02:00
Thomas Waldmann
aba21ee185
freebsd: fix nfs4 acl processing, fixes #8756
This only happened when:
- using borg extract --numeric-ids
- processing NFS4 ACLs

It didn't affect POSIX ACL processing.

This is rather old code, so it looks like nobody used that
code or the bug was not reported.

The bug was discovered by PyCharm's "Junie" AI. \o/
2025-04-17 15:44:27 +02:00
Thomas Waldmann
f3a61ae350
CI: test on ubuntu 22.04 or 24.04
20.04 was removed.
2025-04-17 15:17:57 +02:00
Thomas Waldmann
dddb6b1556
fix remote repository exception handling / exit codes, fixes #8631
also: reduce code duplication.
2025-04-17 15:13:19 +02:00
TW
1659be23f8
Merge pull request #8751 from songxiaocheng/fix/macos-birthtime-1.4
fix: retrieve `birthtime` in nanosecond precision via system call (1.4-maint)
2025-04-14 14:01:52 +02:00
Xiaocheng Song
a6be4c4f6c macOS: retrieve birthtime in nanosecond precision via system call, fixes #8724 2025-04-14 11:41:04 +08:00
TW
0fe038192c
Merge pull request #8748 from destructor64/key_import_error_message_fix-1.4
Fix paper key import error message
2025-04-13 22:54:29 +02:00
Peter Zangerl
0b6150c741 Fix paper key import error message 2025-04-13 14:42:02 +02:00
TW
6276435126
Merge pull request #8744 from ThomasWaldmann/vagrant-updates-1.4
Vagrant updates (1.4-maint)
2025-04-11 22:58:25 +02:00
Thomas Waldmann
28e69e5ad4
update development.lock.txt
use same versions as we have in master right now.
2025-04-11 21:45:56 +02:00
Thomas Waldmann
8c92d69def
vagrant: use Python 3.11.12 2025-04-11 21:45:43 +02:00
TW
94194f3bd4
Merge pull request #8742 from ThomasWaldmann/update-rtd-1.4
docs: theme fixes (1.4-maint)
2025-04-10 21:50:50 +02:00
Thomas Waldmann
11ac0cdc65
docs: theme fixes
- register theme as extension
- add sphinxcontrib.jquery

The html generated uses jquery ($(...)),
but sphinx does not include it anymore since a while.
2025-04-10 21:47:48 +02:00
Thomas Waldmann
46c44b9d9d
docs: fix template to support search result preview 2025-04-10 21:38:54 +02:00
Thomas Waldmann
ecd7976114
docs: add layout.html template from guzzle_sphinx_theme 0.7.11 2025-04-10 21:38:41 +02:00
TW
e0c5c151b8
Merge pull request #8733 from ThomasWaldmann/backport-8625-1.4
work around sudden failure of sphinx ini lexer
2025-04-09 16:51:50 +02:00
TW
8b856cb753
Merge pull request #8731 from ThomasWaldmann/update-changes-1.4
update CHANGES (1.4-maint)
2025-04-09 15:59:41 +02:00
TW
ed0c88ac29
Merge pull request #8732 from ThomasWaldmann/backport-8693-1.4
Clearer language
2025-04-09 15:59:20 +02:00
Thomas Waldmann
31784547bc
work around sudden failure of sphinx ini lexer
/Users/tw/w/borg/docs/internals/data-structures.rst:971:
WARNING: Lexing literal_block
'
[cache]
version = 1
repository = 3c4...e59
manifest = 10e...21c
timestamp = 2017-06-01T21:31:39.699514
key_type = 2
previous_location = /path/to/repo

[integrity]
manifest = 10e...21c
files = {"algorithm": "XXH64", "digests": {"HashHeader": "eab...39e3", "final": "e2a...b24"}}
'
as "ini" resulted in an error at token: '}'.
Retrying in relaxed mode. [misc.highlighting_failure]

Note: this part of the docs didn't change for a long time, so I guess
the sudden warning comes from a change in sphinx' lexers.

Main problem is that rc != 0 will abort our CI pipeline.
2025-04-09 15:56:33 +02:00
Coenraad Loubser
8b1840640d
Clearer language 2025-04-09 15:53:44 +02:00
Thomas Waldmann
a21f08ac6b
update CHANGES 2025-04-09 15:47:40 +02:00
TW
7cd75e11df
Merge pull request #8688 from 120EE0980/local-tz-prune-test
Fix hourly prune test failure due to local timezone
2025-03-12 11:39:29 +01:00
TW
95591c1b6c
Merge pull request #8687 from 120EE0980/revert-8686-prune-test-hourly
Revert "Fix hourly prune test failure due to non-full-hour timezone "
2025-03-12 11:37:50 +01:00
Nish_
8ffd26300e set archive timestamps in local timezone
Signed-off-by: Nish_ <120EE0980@nitrkl.ac.in>
2025-03-12 07:53:24 +05:30
Nish_
3a932c5878
Revert "Fix hourly prune test failure due to non-full-hour timezone " 2025-03-12 07:24:01 +05:30
TW
3d3e8ecb9d
Merge pull request #8686 from 120EE0980/prune-test-hourly
Fix hourly prune test failure due to non-full-hour timezone
2025-03-11 18:03:11 +01:00
Nish_
570c97eb12 run prune tests in specific full hour timezone
Signed-off-by: Nish_ <120EE0980@nitrkl.ac.in>
2025-03-11 18:57:57 +05:30
TW
d892bca8b9
Merge pull request #8685 from 120EE0980/replace-assert
quickfix :  borg exits when assertions are disabled with Python optimizations
2025-03-10 22:55:28 +01:00
Nish_
7cf8316099 quickfix to disallow running borg with assertions switched off
Signed-off-by: Nish_ <120EE0980@nitrkl.ac.in>
2025-03-10 20:05:38 +05:30
Abhijeet
8e6dc52a8a
Fix #8522: Add {unixtime} placeholders (1.4-maint) (#8675) 2025-03-01 11:17:13 +01:00
TW
839434891c
Merge pull request #8643 from httpdev/fix-docs-automated-local-sytemd-wants-1.4
docs: modify docs for automated backup to append to SYSTEMD_WANTS
2025-01-22 22:44:05 +01:00
TW
86fa210f69
Merge pull request #8645 from infectormp/patch-5
FAQ: Why are backups slow on a Linux server that is a member of a windows domain?
2025-01-19 15:41:00 +01:00
infectormp
a1a028cef9
FAQ: Why are backups slow on a Linux server that is a member of a windows domain?
backport to 1.4-maint
2025-01-19 17:32:38 +03:00
TW
0910efad6b
Merge pull request #8644 from httpdev/fix-docs-automated-local-1.4
docs: fix udev rule priority in automated-local.rst
2025-01-19 15:28:11 +01:00
jan
fc0c799dbb docs: modify docs for automated backup to append to SYSTEMD_WANTS rather than overwrite
fixes #8641
In the example, setting SYSTEMD_WANTS instead of appending may prevent
other autostart services attached by earlier udev rules from launching.
This commit changes = to += to fix this behavior.
2025-01-19 14:54:28 +01:00
jan
5634e49fe6 docs: fix udev rule priority in automated-local.rst
fixes #8639
The priority of 40 for the udev rules as stated in to documentation
applies the rule too early on some systems, which prevents the rule from
matching. This commit changes the priority to 80.
2025-01-19 14:53:36 +01:00
TW
4344e64436
Merge pull request #8633 from goebbe/1.4-maint
docs: clarify requirements when using command line options with special characters within a shell #8628
2025-01-13 10:48:22 +01:00
goebbe
15ec434b50
docs: clarify requirements when using command line options with special characters within a shell #8628
docs: clarify requirements when using command line options with special characters within a shell #8628 

port to the 1.4-maint branch
2025-01-13 08:19:53 +01:00
TW
e8c618f272
Merge pull request #8622 from ThomasWaldmann/misc-changes-1.4
Misc. changes (1.4)
2025-01-06 20:40:27 +01:00
Thomas Waldmann
54e526cfc2
fix security policy: 1.4.x is security supported 2025-01-01 17:59:43 +01:00
Thomas Waldmann
21c3b8688b
it's 2025 2025-01-01 17:58:18 +01:00
Thomas Waldmann
301aacecd7
docs: remove 3rd_party licenses
we do not bundle this sw anymore (neither in source
code nor as binary), thus we don't need to have the
licenses here anymore.
2025-01-01 17:53:43 +01:00
TW
31cdc080cf
Merge pull request #8615 from ThomasWaldmann/docs-slow-msgpack-binary-1.4
Docs about slow msgpack when using the binary (1.4)
2024-12-31 09:09:48 +01:00
Thomas Waldmann
cda01c7ba5
docs: add FAQ entry about pure-python msgpack warning, fixes #8323 2024-12-30 22:59:01 +01:00
Thomas Waldmann
16e4b30870
binary readme: first check faq and existing issues 2024-12-30 22:40:15 +01:00
TW
f28693b1f2
Merge pull request #8611 from ThomasWaldmann/update2-changes-1.4
update CHANGES (1.4)
2024-12-29 17:17:42 +01:00
Thomas Waldmann
d3b23b0e69
update CHANGES 2024-12-29 17:08:17 +01:00
TW
f10842f39b
Merge pull request #8610 from ThomasWaldmann/vagrant-updates2-1.4
Vagrant updates2 (1.4)
2024-12-29 01:53:40 +01:00
Thomas Waldmann
094a4a4d48
vagrant: pyenv: only use Python 3.11.11
Installing 3.9/10/12 involves downloading and compiling these,
which takes a lot of time and has some potential breaking the
boxes by installing misc. openssl versions.

We test misc. python versions on github actions CI all the time,
so we only need the one here which we want to use for the binary
builds.
2024-12-29 01:04:45 +01:00
Thomas Waldmann
2dfb5d9a06
use Cython 3.0.11 2024-12-29 00:53:50 +01:00
Björn Ketelaars
fa2aa5f5bd
Filter test output with LibreSSL related warnings on OpenBSD
We do not want that urllib spoils test output with LibreSSL related
warnings on OpenBSD.

`NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently
the 'ssl' module is compiled with 'LibreSSL 3.8.2'`.
2024-12-28 23:53:21 +01:00
Björn Ketelaars
1003ba9762
Fix OpenBSD Vagrant
This should address #8506. Unfortunately I'm unable to test.

#8506 is likely caused by the Vagrant box having a mirror in its
`etc/installurl`, which does not offer 7.4 packages. There are other
mirrors out there who do, e.g., https://ftp.eu.openbsd.org/pub/OpenBSD/.

Proposed 'fix' is to replace the mirror in `/etc/installurl`.
2024-12-28 23:52:53 +01:00
TW
2a044f9f9d
Merge pull request #8609 from ThomasWaldmann/vagrant-updates-1.4
Vagrant updates (1.4)
2024-12-28 23:08:17 +01:00
TW
8015d92896
Merge pull request #8608 from ThomasWaldmann/mount-docs-1.4
mount docs: apply jdchristensen's suggestion, better phrasing.
2024-12-28 23:05:55 +01:00
TW
f8564cc860
Merge pull request #8607 from ThomasWaldmann/faq-slow-fat-1.4
FAQ: Why is backing up an unmodified FAT filesystem slow on Linux?
2024-12-28 23:04:16 +01:00
Thomas Waldmann
8d4f7a00e5
vagrant: use Python 3.11.11 for binary build 2024-12-28 22:57:05 +01:00
Thomas Waldmann
3be57ca1f8
vagrant: use bento/ubuntu-24.04 for now 2024-12-28 22:53:18 +01:00
Thomas Waldmann
58f8554b6d
vagrant: macos: give more memory
sometimes the VM just got stuck (usually when building/testing openssl 3),
maybe more RAM helps?
2024-12-28 22:51:57 +01:00
Thomas Waldmann
136e9cbdc4
vagrant: install rust on *BSD 2024-12-28 22:48:52 +01:00
Thomas Waldmann
7d9028a221
mount docs: apply jdchristensen's suggestion, better phrasing. 2024-12-28 22:42:18 +01:00
Helmut Grohne
f0bd01b98d
FAQ: Why is backing up an unmodified FAT filesystem slow on Linux? 2024-12-28 22:33:56 +01:00
TW
f859100f84
Merge pull request #8606 from ThomasWaldmann/update-changes-1.4
update CHANGES (1.4)
2024-12-28 22:21:36 +01:00
Thomas Waldmann
88ce37b734
update CHANGES 2024-12-28 19:52:20 +01:00
TW
6ccffe414a
Merge pull request #8601 from ThomasWaldmann/check-clarify-messages-1.4
check: more consistent messaging considering --repair, fixes #8533
2024-12-28 18:18:57 +01:00
Thomas Waldmann
a6dc8c52d0
check: more consistent messaging considering --repair, fixes #8533 2024-12-26 18:38:46 +01:00
TW
f9d91f173a
Merge pull request #8600 from ThomasWaldmann/py313
support and test on Python 3.13
2024-12-25 20:55:40 +01:00
Thomas Waldmann
69151c7a1a
support and test on Python 3.13 2024-12-25 13:49:01 +01:00
TW
7bf22bdaf1
Merge pull request #8597 from ThomasWaldmann/yesno-ignore-unicode-error-1.4
yes: deal with UnicodeDecodeError in input(), fixes #6984
2024-12-25 00:54:16 +01:00
TW
491babfa37
Merge pull request #8598 from ThomasWaldmann/additional-free-space-sanity-1.4
config: reject additional_free_space < 10M, fixes #6066
2024-12-25 00:52:50 +01:00
Thomas Waldmann
4f794bbeea
config: reject additional_free_space < 10M, fixes #6066
Doesn't mean that 10M is very helpful in out of space conditions,
but avoids wrong unit assumptions as in #6066.
2024-12-24 22:41:39 +01:00
Thomas Waldmann
f0564036e9
yes: deal with UnicodeDecodeError in input(), fixes #6984
if retry is True, it will just retry to get a valid answer.
if retry is False, it will return the default.

the code can be tested by entering "error" (without the quotes).
2024-12-24 21:44:28 +01:00
TW
a0fbdd978c
Merge pull request #8596 from ThomasWaldmann/compact-dry-run-1.4
compact: support --dry-run (do nothing), fixes #8300
2024-12-24 21:16:09 +01:00
TW
6bbbf3c6ec
Merge pull request #8595 from ThomasWaldmann/storage-quota-zero-1.4
config: fix acceptance of storage_quota 0, fixes #8499
2024-12-24 21:14:43 +01:00
Thomas Waldmann
09745d22fd
compact: support --dry-run (do nothing), fixes #8300
Sole purpose of this is to simplify scripting.
2024-12-24 19:34:21 +01:00
TW
2dfb091d69
Merge pull request #8593 from ThomasWaldmann/docs-bundled-libs-1.4
docs: update install docs, nothing bundled anymore! fixes #8342.
2024-12-24 19:15:38 +01:00
Thomas Waldmann
e74c912856
config: fix acceptance of storage_quota 0, fixes #8499
We don't want to accept very small quota limits < 10M,
but 0 (quota disabled) is acceptable.
2024-12-24 19:13:05 +01:00
Thomas Waldmann
095b43ba3e
docs: update install docs, nothing bundled anymore! fixes #8342.
borg 1.2.x used to bundle some source code for lz4, zstd and xxhash,
but borg 1.4.x got rid of that - all that is meanwhile available as
a library in all OSes we support.

also, the python package pkgconfig does not need to be "preinstalled"
anymore, because our pyproject.toml cares for that. otoh, the cli tool
pkg-config must be preinstalled so that libs and headers can be found
automagically.

Also be a bit more clear about the FUSE stuff.
2024-12-24 17:30:19 +01:00
TW
576cd4ad35
Merge pull request #8563 from goebbe/goebbe-flags-borg1.4
Clarify  excluded and included flags for dry-run in borg the docs
2024-11-24 14:25:33 +01:00
goebbe
66e92636d2
Clarify excluded and included flags for dry-run in borg the docs
Rewording of the "flags" description in the docs of borg 1.4.
https://github.com/borgbackup/borg/issues/8556
2024-11-24 13:44:39 +01:00
TW
6b4e952dcc
Merge pull request #8542 from mrichtarsky/1.4-maint
Backport doc changes
2024-11-15 11:09:23 +01:00
Martin Richtarsky
8fe9c5ed6c docs: Small changes regarding compression
- Mention zstd as the best general choice when not using lz4
  (as often acknowledged by public benchmarks)
- Mention 'auto' more prominently as a good heuristic to improve
  speed while retaining good compression
- Link to compression options

(cherry picked from commit f656f6b1f2)
2024-11-15 07:40:42 +00:00
Martin Richtarsky
05837fbeb7 docs/FAQ: Clean up entries regarding SSH settings
Link to recommended ones

(cherry picked from commit b913d9dd1a)
2024-11-15 07:40:21 +00:00
TW
d94b954da2
Merge pull request #8344 from cr1901/relax_hardlink_assert
Do not assert on diff if hard link sources are not found due to exclusions
2024-10-06 14:00:01 +02:00
TW
47a85429d8
Merge pull request #8462 from ThomasWaldmann/doc-updates-1.4
Doc updates (1.4-maint)
2024-10-05 20:24:59 +02:00
Thomas Waldmann
2230932c02
docs: borg/borgfs detects internally under which name it was invoked, fixes #8207 2024-10-05 19:42:30 +02:00
Thomas Waldmann
1b878ab546
binary: using the directory build is faster, fixes #8008 2024-10-05 19:41:45 +02:00
Thomas Waldmann
278c266e7c
docs: add readme of the binaries
"as is" as of 1.4.0
2024-10-05 19:23:58 +02:00
Thomas Waldmann
cbbfc35db0
docs: mount: document on-demand loading, perf tips, fixes #7173 2024-10-05 19:21:17 +02:00
Thomas Waldmann
cf472302dc
docs: better link modern return codes, fixes #8370 2024-10-05 19:14:51 +02:00
William D. Jones
2720331213 Do not assert on diff if hard link sources are not found due to exclusions. 2024-10-02 00:05:20 -04:00
TW
c1bb7ec515
Merge pull request #8401 from ThomasWaldmann/fix-authenticated-no-key-1.4
fix authenticated_no_key (1.4-maint)
2024-09-20 22:25:07 +02:00
Thomas Waldmann
2c18620075
fix WORKAROUNDS=authenticated_no_key support for archive TAM authentication, fixes #8400 2024-09-20 20:19:30 +02:00
TW
30cd3b8be3
Merge pull request #8371 from AfroThundr3007730/AfroThundr3007730/issue8361
Update repository URLs in docs to use new syntax
2024-09-11 11:02:33 +02:00
Eddie Carswell
4f110f8bde
Update repository URLs in docs to use new syntax
Fixes #8361

Signed-off-by: Eddie Carswell <eddiecarswell13@yahoo.com>
2024-09-10 18:08:18 -05:00
William D. Jones
81cb1cdc56
Add quarterly pruning strategy. (#8337)
Add 13weekly and 3monthly quarterly pruning strategies.
2024-08-21 22:40:32 +02:00
TW
0ebb80c05d
Merge pull request #8334 from cr1901/json-no-changes
Suppress modified changes for files which weren't actually modified in JSON output.
2024-08-21 18:45:52 +02:00
William D. Jones
938e144f8b Ensure that 0B changes are hidden from text diffs too. 2024-08-19 11:25:06 -04:00
William D. Jones
f7673e3f06 Add test to make sure 0-added,0-removed modified changes aren't in JSON output. 2024-08-15 22:26:03 -04:00
William D. Jones
6e9857b3e0 Remove 0-added,0-removed modified entries from JSON output. 2024-08-15 20:42:40 -04:00
TW
ad5028484f
Merge pull request #8321 from ruben-rodriguez/1.4-maint
docs: align /etc/backups path references in automated backups deployment guide (1.4-maint)
2024-08-04 21:04:18 +02:00
Ruben Rodriguez
54c902474f docs: align /etc/backups path in all references 2024-08-04 19:46:34 +02:00
TW
94b067b00d
Merge pull request #8316 from ThomasWaldmann/cleanups-1.4
remove outdated comment, fixes #4240
2024-08-02 21:30:18 +02:00
Thomas Waldmann
adf2d8c047
remove outdated comment, fixes #4240 2024-08-02 17:26:10 +02:00
TW
4184cade37
Merge pull request #8303 from ThomasWaldmann/improve-healing-test-1.4
improve borg check --repair healing tests, see #8302 (1.4-maint)
2024-07-23 20:25:06 +02:00
Thomas Waldmann
85d7bdd75f
improve borg check --repair healing tests, see #8302
test the healing more thoroughly:
- preservation of correct chunks list in .chunks_healthy
- check that .chunks_healthy is removed after healing
- check that doing another borg check --repair run does not find
  something to heal, again.

also did a datatype consistency fix for item.chunks_healthy list
members: they are now post processed in the same way as item.chunks,
so they have type ChunkListEntry rather than simple tuple.
2024-07-23 18:12:11 +02:00
TW
5ef115b99f
Merge pull request #8283 from Aztorius/fix-borg-use-chunks-archive-1.4
Fix backwards compatibility of old chunks.archive.d file behavior
2024-07-15 00:17:00 +02:00
William Bonnaventure
924952fcdd Fix backwards compatibility of old chunks.archive.d file behavior
In case a file named chunks.archive.d, cache would not create a folder
with that name to store cache chunks archive. We need to support both
the new environment variable and the old "hacky" way of creating a file.
2024-07-14 18:55:19 +02:00
TW
2c364fca40
Merge pull request #8282 from Aztorius/borg_use_chunks_archive_1.4
Backport BORG_USE_CHUNKS_ARCHIVE env var to 1.4-maint
2024-07-14 14:52:17 +02:00
William Bonnaventure
f3c1444a41 Update docs for BORG_USE_CHUNKS_ARCHIVE 2024-07-14 07:24:37 +02:00
William Bonnaventure
0575f87580 Add BORG_USE_CHUNKS_ARCHIVE option 2024-07-14 07:23:06 +02:00
TW
83eca0d8b0
Merge pull request #8278 from ThomasWaldmann/ci-windows-fix-1.4
github CI: windows msys2 build: broken, disable it for now, see #8264
2024-07-07 16:18:08 +02:00
TW
2f85725031
Merge pull request #8277 from Aztorius/automatic-rebuild-deleted-chunks-file-1.4
Automatic rebuild cache on exception, fixes #5213 (#8257) - Backport to 1.4-maint
2024-07-07 14:56:38 +02:00
Thomas Waldmann
de913cc251
github CI: windows msys2 build: broken, disable it for now, see #8264 2024-07-07 14:42:02 +02:00
William Bonnaventure
0bd41ba65a Automatic rebuild cache on exception, fixes #5213 (#8257)
Try to rebuild cache if an exception is raised, fixes #5213

For now, we catch FileNotFoundError and FileIntegrityError.

Write cache config without manifest to prevent override of manifest_id.
This is needed in order to have an empty manifest_id.
This empty id triggers the re-syncing of the chunks cache by calling sync() inside LocalCache.__init__()

Adapt and extend test_cache_chunks to new behaviour:

- a cache wipe is expected now.
- borg detects the corrupt cache and wipes/rebuilds the cache.
- check if the in-memory and on-disk cache is as expected (a rebuilt chunks cache).
2024-07-07 10:05:39 +02:00
TW
add9cafce6
Merge pull request #8275 from wzyboy/fix/typo-1.4-maint
docs: fix typo (backport #8272)
2024-07-06 20:37:29 +02:00
Zhuoyun Wei
b379202beb
docs: fix typo (backport #8272) 2024-07-06 09:30:50 -07:00
TW
9a40344ec4
Merge pull request #8267 from ThomasWaldmann/vagrant-machines-1.4
Vagrant machines (1.4-maint)
2024-07-03 14:20:28 +02:00
Thomas Waldmann
91452e3079
vagrant: netbsd/openbsd: add version number 2024-07-03 14:04:57 +02:00
Thomas Waldmann
fe8ef751ee
vagrant: macos: add version number to call param 2024-07-03 14:03:26 +02:00
Thomas Waldmann
e36eaabb1c
vagrant: darwin -> macos
Also: add macOS version number.
2024-07-03 13:57:56 +02:00
Thomas Waldmann
81faea9288
vagrant: adapt fetch-binaries script 2024-07-03 13:52:42 +02:00
Thomas Waldmann
e63cd2c467
vagrant: get rid of the "64" suffix for VM names.
We only have 64bit machines anyway. And even in the unlikely
case we would add 32bit machines, we could suffix them by "32".
2024-07-03 13:50:06 +02:00
Thomas Waldmann
9cee77525c
vagrant: use freebsd(13|14)_64 as VM name 2024-07-03 13:44:27 +02:00
Thomas Waldmann
b6a5baa92c
add freebsd 13 box, fixes #8266
users of TrueNAS do not have fbsd 14 yet.
2024-07-03 13:40:13 +02:00
TW
88fa2e6557
Merge pull request #8263 from ThomasWaldmann/rel140
release 1.4.0
2024-07-03 12:20:09 +02:00
Thomas Waldmann
8b7cce3b7c
update CHANGES 2024-07-02 21:21:30 +02:00
Thomas Waldmann
dbac621671
build_man / build_usage 2024-07-02 21:20:41 +02:00
Thomas Waldmann
30b28ba8c4
docs: update standalone binary section 2024-07-02 21:20:39 +02:00
TW
de91c79f64
Merge pull request #8259 from ThomasWaldmann/borg-mount-docs-1.4
docs: add examples using :: positional argument, fixes #8255
2024-06-30 17:46:04 +02:00
Thomas Waldmann
7b8acc925f
docs: add examples using :: positional argument, fixes #8255
Simply omitting the first positional argument (REPO) does
not work because borg then gets confused.
2024-06-30 17:31:22 +02:00
TW
68ca4a8ffb
Merge pull request #8246 from tree-wall/1.4-maint
Installation: update Arch Linux repo name
2024-06-04 13:04:17 +02:00
tree-wall
82bb14ccab
Installation: update Arch Linux repo name
The "community" repository was merged into the "extra" repo last year.

https://archlinux.org/news/git-migration-completed/
2024-06-04 10:38:58 +01:00
TW
9b5d6d789c
Merge pull request #8243 from ThomasWaldmann/buster-revival-1.4
revive the buster64 box (1.4-maint)
2024-06-03 12:14:45 +02:00
Thomas Waldmann
18a8baf8e2
revive the buster64 box
RHEL8 and clones are on this glibc version.
2024-06-02 19:23:49 +02:00
TW
10fda2f39b
Merge pull request #8241 from ThomasWaldmann/fix-pytest-1.4
fix pytest_report_header, fixes #8232
2024-05-31 15:58:49 +02:00
Thomas Waldmann
99ee635f8d
fix pytest_report_header, fixes #8232 2024-05-31 14:29:07 +02:00
TW
ecf7efc923
Merge pull request #8231 from ThomasWaldmann/rel140rc1
release candidate 1.4.0rc1
2024-05-26 21:19:23 +02:00
Thomas Waldmann
860732cb0b
build_usage, build_man 2024-05-26 13:33:02 +02:00
Thomas Waldmann
b89173c2fb
update CHANGES 2024-05-26 13:27:31 +02:00
Thomas Waldmann
e32b915645
use PyInstaller 6.7.0 for binary builds 2024-05-26 13:21:08 +02:00
Thomas Waldmann
96ff5e0f68
use Python 3.11.9 for binary builds 2024-05-26 13:18:31 +02:00
TW
83f0aed565
Merge pull request #8226 from ThomasWaldmann/allow-msgpack110-1.4
allow msgpack 1.1.0
2024-05-20 19:45:57 +02:00
Thomas Waldmann
249b9e804f
allow msgpack 1.1.0
tests with 1.1.0rc1 were successful, thus I assume 1.1.0 will also work ok.
2024-05-20 14:13:40 +02:00
TW
d541ef3593
Merge pull request #8224 from ThomasWaldmann/try-msgpack-110rc1-1.4
try msgpack 1.1.0rc1
2024-05-19 20:10:51 +02:00
Thomas Waldmann
9c262ccbfb
try msgpack 1.1.0rc1 2024-05-19 16:15:22 +02:00
TW
1b3755ca03
Merge pull request #8223 from ThomasWaldmann/changes-1.4
update CHANGES (1.4)
2024-05-19 16:02:20 +02:00
Thomas Waldmann
12bc736122
update CHANGES 2024-05-19 16:01:37 +02:00
TW
053b09da9b
Merge pull request #8210 from ThomasWaldmann/fix-cythonize-import-error-reporting-1.4
setup.py: fix import error reporting for cythonize import, see #8208
2024-05-12 18:41:27 +02:00
TW
2c3350851a
Merge pull request #8216 from ThomasWaldmann/fix-docs-cygwin-python-1.4
docs: use python 3.9 in cygwin install docs, fixes #8196
2024-05-12 16:46:57 +02:00
Thomas Waldmann
8dcd8daab1
setup.py: detect noexec build fs issue, see #8208
That "failed to map segment from shared object" error msg is not
very helpful. Add a hint that the filesystem needs to be +exec
(== not noexec mounted, like it might be the case for /tmp on
some systems).
2024-05-12 16:38:13 +02:00
Thomas Waldmann
d033f6cd87
setup.py: fix import error reporting for cythonize import, see #8208
Looks like borg's setup.py has hidden the real cause of a cythonize ImportError.

There are basically 2 cases:
- either there is no Cython installed, then the import fails because the module can not be found, or
- there is some issue within Cython and the import fails due to that.

It's important not to hide the real cause, especially if we run into case 2.

case 1 is kind of expected and frequent, case 2 is rare.
2024-05-12 16:38:11 +02:00
TW
504f169efa
Merge pull request #8212 from ThomasWaldmann/fix-ci-macos-openssl-1.4
github CI: fix PKG_CONFIG_PATH for openssl 3.0
2024-05-12 16:37:09 +02:00
Thomas Waldmann
b5dddcddfc
docs: use python 3.9 in cygwin install docs, fixes #8196 2024-05-12 16:35:38 +02:00
Thomas Waldmann
b9617928d4
github CI: fix PKG_CONFIG_PATH for openssl 3.0 2024-05-12 14:59:02 +02:00
TW
87aab31302
Merge pull request #8197 from ThomasWaldmann/remove-experimental-1.4
recreate: remove experimental status
2024-04-22 15:06:13 +02:00
TW
41a2f0a7be
Merge pull request #8195 from ThomasWaldmann/drop-buster-1.4
vagrant: drop buster VM, fixes #8171
2024-04-22 15:05:26 +02:00
Thomas Waldmann
8dcf279972
recreate: remove experimental status
We have this since quite a while and there were no major complaints.
2024-04-22 15:03:53 +02:00
Thomas Waldmann
90ea511bb7
vagrant: drop buster VM, fixes #8171 2024-04-22 14:41:44 +02:00
TW
30fd46a53c
Merge pull request #8184 from galqiwi/log_message_len_fix_1_4_x
changed insufficiently reserved length for log message (copy of #8152 for 1.4)
2024-04-06 14:05:08 +02:00
Vladimir Malinovskii
a8777e912a changed log message reserved length (40->44) 2024-04-06 01:31:00 +03:00
TW
60e2e1aa25
Merge pull request #8170 from ThomasWaldmann/rel140b2
release 1.4.0b2
2024-03-31 19:50:01 +02:00
Thomas Waldmann
9bee7d433c
set release date 2024-03-30 22:19:41 +01:00
Thomas Waldmann
a41cc1750f
build_usage ; build_man 2024-03-30 22:19:39 +01:00
Thomas Waldmann
7794573d5c
use Cython 3.0.10 2024-03-30 22:19:25 +01:00
Thomas Waldmann
218892fcaf
vagrant: add a ubuntu noble box (24.04) 2024-03-30 22:19:21 +01:00
TW
bf80376c77
Merge pull request #8169 from ThomasWaldmann/update-changes-1.4
update CHANGES
2024-03-30 19:42:49 +01:00
Thomas Waldmann
907cd112d3
update CHANGES 2024-03-30 19:33:37 +01:00
TW
00fd3708ac
Merge pull request #8167 from ThomasWaldmann/debounce-sigint-1.4
fix Ctrl-C / SIGINT behaviour for pyinstaller-made binaries, fixes #8155
2024-03-30 19:32:39 +01:00
TW
f1a3be149f
Merge pull request #8168 from ThomasWaldmann/vagrant-updates-1.4
vagrant: openindiana updates (1.4-maint)
2024-03-30 18:27:50 +01:00
Thomas Waldmann
a5de0c9d96
vagrant: openindiana: pkg set-mediator for openssl 2024-03-30 18:15:32 +01:00
Thomas Waldmann
5b26bdd475
vagrant: openindiana updates
- use openindiana/hipster box
- install missing packages
- also install libxxhash, so it does not use the bundled code
2024-03-30 18:02:36 +01:00
TW
5ad7364209
Merge pull request #8125 from ThomasWaldmann/acl-error-handling-1.4
improve acl_get / acl_set error handling (1.4-maint)
2024-03-30 17:56:16 +01:00
Thomas Waldmann
dfb8d2054e
fix Ctrl-C / SIGINT behaviour for pyinstaller-made binaries, fixes #8155 2024-03-30 17:47:39 +01:00
TW
af8e512d59
Merge pull request #8160 from ThomasWaldmann/repo-upgrade-helper-1.4
repo upgrade helpers (1.4-maint)
2024-03-28 22:13:22 +01:00
Thomas Waldmann
736ffbdada
docs: simplify TAM-related upgrade docs using the new commands 2024-03-28 14:47:16 +01:00
Thomas Waldmann
262d6109a9
upgrade --check-tam: check manifest TAM auth issues
exit with rc=1 if there are issues.
2024-03-28 14:46:56 +01:00
Thomas Waldmann
69a4ea7682
upgrade: auto-use ignore_invalid_archive_tam 2024-03-28 14:46:34 +01:00
Thomas Waldmann
ed80bf2c37
do a late lookup on helpers.workarounds
so we can modify it within borg code elsewhere and it
will get the updated value when it accesses the attribute.
2024-03-28 14:44:38 +01:00
Thomas Waldmann
8643576755
upgrade --check-archives-tam: rc=1 for archive TAM issues
also: better messages about global status.
2024-03-28 14:44:25 +01:00
Thomas Waldmann
965c87bb0a
upgrade --check-archives-tam: check archives tam status
This is a read-only variation of --archives-tam:
- it only checks / displays the current status
- it does not add / fix the archive TAMs
2024-03-28 14:44:00 +01:00
TW
b1053eefa5
Merge pull request #8154 from ThomasWaldmann/update-changes-1.4
update CHANGES
2024-03-21 18:57:38 +01:00
Thomas Waldmann
0648a6dfcc
update CHANGES 2024-03-17 19:56:40 +01:00
Thomas Waldmann
7ba843b8c0
Linux: refactor acl_get 2024-03-17 18:51:08 +01:00
Thomas Waldmann
54c7da3c64
Linux: acl tests: move ACCESS_ACL and DEFAULT_ACL constants
They are only used at one place, move them there rather than having globals.
2024-03-17 18:51:07 +01:00
Thomas Waldmann
b45f572a95
FreeBSD: check first if kind of ACL can be set on a file 2024-03-17 18:51:06 +01:00
Thomas Waldmann
47e291eec1
FreeBSD: simplify numeric_ids check 2024-03-17 18:51:04 +01:00
Thomas Waldmann
b0607909e1
FreeBSD: added tests, only get default ACL from dirs 2024-03-17 18:51:03 +01:00
Thomas Waldmann
f8e8608488
FreeBSD: acl_get: raise OSError if lpathconf fails
Previously:
- acl_get just returned for lpathconf returning EINVAL
- acl_get silently ignored all other lpathconf errors and
  implied it is not a NFS4 acl

Now:
- not sure why the EINVAL silent return was done, but it seems
  wrong. guess it could be the system not implementing a check
  for nfs4. but in that case guess we still would like to get
  the default and access ACL!? Thus, I removed the silent return.
- raise OSError for all lpathconf errors

Cosmetic: add a nfs4_acl boolean, so the code reads better.
2024-03-17 18:50:58 +01:00
Thomas Waldmann
9d638e8d62
FreeBSD: acl_get: add an acl_extended_* call
... to implement same semantics as on linux (only store ACL
if it defines permissions other than those defined by the
traditional file permissions).

Looks like there is no call working with an fd on FreeBSD.
2024-03-17 18:50:55 +01:00
Thomas Waldmann
6ad1ad67d5
Linux: acl_set bug fix: always fsencode path
We use path when raising OSErrors, even if we have an fd.
2024-03-17 18:50:54 +01:00
Thomas Waldmann
17e3cee604
Linux: acl_get: use "nofollow" variant of acl_extended_file call
This is NOT a bug fix, because the previous code contained a
check for symlinks before that line - because symlinks can not
have ACLs under Linux.

Now, this "is it a symlink" check is removed to simplify the
code and the "nofollow" variant of acl_extended_file* is used
to look at the symlink fs object (in the symlink case).

It then should tell us that this does NOT have an extended ACL
(because symlinks can't have ACLs) and so we return there.

Overall the code gets simpler and looks less suspect.
2024-03-17 18:50:53 +01:00
Thomas Waldmann
519e3ebce8
Linux: acl_get: raise OSError for errors in acl_extended_* call
Previously, these conditions were handled the same (just return):
- no extended acl here
- some error happened (e.g. ACLs unsupported, bad file descriptor, file not found, permission error, ...)

Now there will be OSErrors for the error cases.
2024-03-17 18:50:49 +01:00
Thomas Waldmann
33f1ba699e
create/extract: ignore OSError if ACLs are not supported (ENOTSUP)
but do not silence other OSErrors.
2024-03-17 18:49:37 +01:00
Thomas Waldmann
d870c58e61
platform tests: misc. minor cleanups
- remove unused global / import
- use is_linux and is_darwin
- rename darwin acl test method
2024-03-16 15:58:50 +01:00
Thomas Waldmann
926b5a6b08
improve are_acls_working function
- ACLs are not working, if ENOTSUP ("Operation not supported") happens
- fix check for macOS
  On macOS borg uses "acl_extended", not "acl_access" and
  also the ACL text format is a bit different.
2024-03-16 15:56:58 +01:00
Thomas Waldmann
2c53a63a1c
raise OSError if acl_to_text / acl_from_text returns NULL
Also did a small structural refactors there.
2024-03-16 15:56:57 +01:00
Thomas Waldmann
d4a83edfdd
improve acl_get / acl_set error handling, see #4049 2024-03-16 15:56:55 +01:00
TW
702560d9f2
Merge pull request #8148 from ThomasWaldmann/gh-actions-update-1.4
github CI: misc. updates (1.4-maint)
2024-03-15 18:10:02 +01:00
Thomas Waldmann
28bd4d1bc6
github CI: misc updates
- move most Linux tests to ubuntu 22.04
- macOS: run on macos-14 (on Apple Silicon!)
- macOS: use OpenSSL 3.0 from brew
- macOS: run with Python 3.11
- pip install -e .: add -v
- trigger CI run for changed .toml files
- use up-to-date github actions
- remove libb2 references - since borg 1.2, we use blake2 indirectly via python stdlib
2024-03-15 16:38:30 +01:00
TW
d84118a9db
Merge pull request #8141 from ThomasWaldmann/vagrant-updates
Vagrant updates (1.4-maint)
2024-03-10 18:39:03 +01:00
Thomas Waldmann
ac1b28241d
vagrant: use python 3.11.8 2024-03-10 18:37:47 +01:00
Thomas Waldmann
86aeb1c253
vagrant: use pyinstaller 6.5.0 2024-03-10 18:37:46 +01:00
Thomas Waldmann
41520c2937
vagrant: add xxhash for macOS 2024-03-10 18:37:44 +01:00
Thomas Waldmann
0912e1b3e1
vagrant: add libxxhash-dev for debianoid systems 2024-03-10 16:26:06 +01:00
TW
3647426940
Merge pull request #8135 from ThomasWaldmann/msgpack-cython-updates-1.4
msgpack and cython updates (1.4-maint)
2024-03-02 16:35:21 +01:00
Thomas Waldmann
30ad61ae5a
require Cython 3.0.3 at least, fixes #8133
The fix for the Python 3.12 memory leak issue was
in Cython 3.0.3+.
2024-03-02 14:21:48 +01:00
Thomas Waldmann
0151c9a38f
allow msgpack 1.0.8, fixes #8133 2024-03-02 14:19:11 +01:00
TW
5017d92505
Merge pull request #8127 from ThomasWaldmann/ebusy-1.4
create: deal with EBUSY, fixes #8123
2024-02-25 13:26:00 +01:00
TW
a7329314ab
Merge pull request #8130 from ThomasWaldmann/fix-docs-1.4
docs: remove tabs
2024-02-25 12:20:01 +01:00
Thomas Waldmann
7e166598d5
docs: remove tabs 2024-02-25 12:19:06 +01:00
Thomas Waldmann
a88c3d9e25
create: deal with EBUSY, fixes #8123
I put it into same class as EPERM and EACCES:
BackupPermissionError: borg is not permitted to access the file.
2024-02-25 11:21:44 +01:00
TW
21deeaf208
Merge pull request #8126 from bket/use_libressl
[1.4-maint] No need to use OpenSSL 3.0 on OpenBSD
2024-02-25 11:10:09 +01:00
Björn Ketelaars
2d13e8efa0 No need to use OpenSSL 3.0 on OpenBSD
borg-2.0 requires EVP_aes_256_ocb, which is not provided by LibreSSL. As
such, OpenSSL-3.0 is needed on OpenBSD. borg-1.4 however does not use
EVP_aes_256_ocb thus there is no need to use OpenSSL. Instead rely on
LibreSSL.

Undo part of ccb1e92. Tested on OpenBSD -current.
2024-02-25 07:10:02 +01:00
TW
92da66dcfc
Merge pull request #8122 from ThomasWaldmann/update-changes-1.4
update CHANGES
2024-02-24 21:19:50 +01:00
Thomas Waldmann
01965c4993
update CHANGES 2024-02-24 20:43:59 +01:00
TW
17e64224e0
Merge pull request #8114 from stephan13360/1.4-maint
add non-root deployment strategy
2024-02-24 19:39:08 +01:00
TW
4522c040f3
Merge pull request #8117 from ThomasWaldmann/remove-bundled-3rd-party-1.4
Remove bundled 3rd party sw (1.4-maint)
2024-02-24 19:38:29 +01:00
TW
9fca0b2f42
Merge pull request #8118 from ThomasWaldmann/benchmark-crud-options-1.4
benchmark: inherit options --rsh --remote-path, fixes #8099
2024-02-22 22:01:12 +01:00
Thomas Waldmann
5a8a505046
benchmark: inherit options --rsh --remote-path, fixes #8099 2024-02-22 21:35:11 +01:00
Thomas Waldmann
30dc27b11a
remove bundled lz4/zstd/xxhash code
Also adapt our Cython code to directly use the lib headers,
remove our wrappers.
2024-02-22 13:48:51 +01:00
Thomas Waldmann
ccb1e92362
setup.py: remove support for bundled 3rd party src, fixes #8094
Took the setup.py from master branch and only slightly modified the
"checksums" module path for now.

This removes support for all BORG_USE_BUNDLED_*=YES env vars.
"NO" has been the default since quite a while anyway.

This adds support for:
- using OpenSSL 3.0 on OpenBSD
- locating libacl via pkgconfig

Formatting changes came due to master branch using "black".
2024-02-22 13:38:31 +01:00
Stephan Herbers
3172bda300 add restore considerations paragraph 2024-02-22 11:56:28 +01:00
Stephan Herbers
9abacabf51
Apply suggestions from code review
Co-authored-by: NetSysFire <59517351+NetSysFire@users.noreply.github.com>
2024-02-21 13:00:27 +01:00
Stephan Herbers
599514a247 add non-root deployment strategy 2024-02-21 12:25:48 +01:00
TW
4abc7f18cd
Merge pull request #8108 from ThomasWaldmann/new-rc-fix-1.4
fix: Error/CommandError have a output format for 1 argument
2024-02-20 12:54:53 +01:00
Thomas Waldmann
420ef45f63
fix: Error/CommandError have a output format for 1 argument
(backport from master, only thing todo here was that one place)
2024-02-20 04:23:02 +01:00
TW
52c90786fc
Merge pull request #8095 from ThomasWaldmann/update-changes-1.4
update CHANGES
2024-02-13 19:29:39 +01:00
Thomas Waldmann
9eec58bc18
update CHANGES 2024-02-13 18:03:14 +01:00
TW
8c8b5714d4
Merge pull request #8090 from ThomasWaldmann/fix-long-desc-1.4
sdist: dynamically compute readme (long_description)
2024-02-10 18:20:46 +01:00
Thomas Waldmann
2bb4b3d650
sdist: dynamically compute readme (long_description)
The long_desc_from_readme() was not called, it just read the
whole README.rst until "readme" was declared dynamic.
2024-02-10 18:04:28 +01:00
TW
767504b954
Merge pull request #8089 from ThomasWaldmann/fix-docs-1.4
development.rst: fix markup
2024-02-10 17:30:34 +01:00
Thomas Waldmann
9057c7ba4d
development.rst: fix markup 2024-02-10 17:25:43 +01:00
TW
92669abf13
Merge pull request #8077 from ThomasWaldmann/index-check-value-errors-1.4
check: fix return code for index entry value discrepancies
2024-02-08 21:56:20 +01:00
Thomas Waldmann
97fe48c44c
check: fix return code for index entry value discrepancies
Also: use ERROR loglevel for these (not WARNING).

A different amount of index entries was already logged as error
and led to "error_found = True" in repository.check.

Different values in the rebuilt index vs. the on-disk index were
only logged on warning level, but did not lead to error_found = True.

Guess there is no reason why these should not be errors and lead to
error_found = True, so this was fixed in this commit.

Minor related change: change report_error function args, so it can be
called like logger.error - including giving a format AND args.
2024-02-08 19:31:24 +01:00
TW
6292db2866
Merge pull request #8076 from ThomasWaldmann/docs-tests-pypi-pkg-1.4
docs: how to run the testsuite using the dist package
2024-02-08 18:27:00 +01:00
Thomas Waldmann
79268ec951
docs: how to run the testsuite using the dist package 2024-02-08 17:23:52 +01:00
TW
3a81da1154
Merge pull request #8072 from ThomasWaldmann/update-changes-1.4
Update changes (1.4-maint)
2024-02-03 01:43:17 +01:00
Thomas Waldmann
acebc0bb70
update CHANGES 2024-02-02 13:59:18 +01:00
TW
6221de8cbb
Merge pull request #8071 from ThomasWaldmann/init-message-1.4
init: improve message (1.4-maint)
2024-02-01 17:24:15 +01:00
Thomas Waldmann
fb44fb5011
init: better borg key export instructions 2024-02-01 12:55:31 +01:00
Thomas Waldmann
f2b3a3a383
init: remove compatibility warning for borg <=1.0.8
The warning refers to a compatibility issue not relevant any
more since borg 1.0.9 (released 2016-12).

It is now better to remove this warning to make the output
of borg init shorter and maybe get more people to completely
read the important parts of it.
2024-01-31 17:26:20 +01:00
TW
ad44430b2a
Merge pull request #8069 from ThomasWaldmann/less-setuppy2-1.4
Use less setup.py, continued (1.4-maint)
2024-01-31 17:18:10 +01:00
Thomas Waldmann
ef2728a6a6
scripts: make.py clean implementation 2024-01-30 23:55:18 +01:00
Thomas Waldmann
8bf7ad575a
scripts: update docs about make.py 2024-01-30 23:21:04 +01:00
Thomas Waldmann
fd1dad2f79
scripts: rename gendocs.py to make.py 2024-01-30 23:19:09 +01:00
TW
376ad6dc57
Merge pull request #8060 from ThomasWaldmann/slashdot-hack-1.4
slashdot hack (1.4-maint)
2024-01-30 18:28:45 +01:00
Thomas Waldmann
5b96d5acc3
create: add the slashdot hack, fixes #4685 2024-01-30 14:01:40 +01:00
TW
e744e04821
Merge pull request #8067 from ThomasWaldmann/increase-gh-actions-timeout-1.4
gh actions: increase timeout (1.4-maint)
2024-01-30 13:00:10 +01:00
Thomas Waldmann
009a88626f
gh actions: increase timeout
especially the macos workers are sometimes
extremely slow.
2024-01-30 00:39:53 +01:00
TW
e0de99307a
Merge pull request #8066 from ThomasWaldmann/upgrade-bundled-code-1.4
Upgrade bundled code (1.4-maint)
2024-01-30 00:37:24 +01:00
TW
206f90f56a
Merge pull request #8065 from ThomasWaldmann/less-setuppy-1.4
Use less setup.py (1.4-maint)
2024-01-30 00:36:25 +01:00
Thomas Waldmann
b1a8924c0e
upgrade bundled zstd to 1.5.5 2024-01-29 22:30:04 +01:00
Thomas Waldmann
c8007e45ce
upgrade bundled xxhash to 0.8.2 2024-01-29 22:07:39 +01:00
Thomas Waldmann
a3998d0663
upgrade bundled lz4 to 1.9.4 2024-01-29 21:54:49 +01:00
Thomas Waldmann
e80d922730
requirements are defined in pyproject.toml 2024-01-29 20:44:37 +01:00
Thomas Waldmann
4a65f9c943
docs: do not refer to setup.py installation method
Invoking setup.py is deprecated.
2024-01-29 20:39:02 +01:00
Thomas Waldmann
6e228e371b
scripts/gendocs.py: make it work
remove unused html_write function

gendocs: update developer docs
2024-01-29 20:31:56 +01:00
Thomas Waldmann
e2860318fa
git mv setup_docs.py scripts/gendocs.py 2024-01-29 19:56:38 +01:00
Thomas Waldmann
1220a96d1d
setup.py: remove build_usage and build_man as setup.py command 2024-01-29 19:53:22 +01:00
Thomas Waldmann
19718c3c55
setup.py: move long_desc_from_readme definition to here 2024-01-29 19:50:14 +01:00
TW
bb473e2daa
Merge pull request #8055 from ThomasWaldmann/rel140b1
release 1.4.0b1
2024-01-21 14:01:18 +01:00
Thomas Waldmann
bdacc84ceb
build_usage build_man 2024-01-20 19:25:08 +01:00
Thomas Waldmann
1445a2b3ff
update CHANGES 2024-01-20 19:13:40 +01:00
TW
a423092337
Merge pull request #8054 from ThomasWaldmann/more-errors-msgids-1.4
more specific errors / msgids (1.4-maint)
2024-01-20 18:32:08 +01:00
Thomas Waldmann
5d956b9655
add ConnectionBrokenWithHint for BrokenPipeErrors and similar, see #7016
no traceback, but error message and specific exit code.
2024-01-19 18:38:07 +01:00
Thomas Waldmann
45f65f7c57
check_can_create_repository: deal with PermissionErrors, see #7016
borg init calls this. If there is a PermissionError, it is
usually fs permission issue at path or its parent directory.

Don't give a traceback, but rather an error msg and a specific exit code.
2024-01-19 18:13:53 +01:00
TW
266c9f6e8f
Merge pull request #8053 from ThomasWaldmann/remote-version-1.4
implement "borg version", fixes #7829
2024-01-19 15:26:28 +01:00
Thomas Waldmann
8d72927e77
implement "borg version", fixes #7829
Additional new command "borg version <REPO>" which shows client and server version.
2024-01-19 13:01:14 +01:00
TW
c7cef548a9
Merge pull request #8052 from ThomasWaldmann/update-changes-1.4
update CHANGES (1.4-maint)
2024-01-19 10:58:28 +01:00
Thomas Waldmann
9ec454727d
update CHANGES 2024-01-19 00:54:45 +01:00
TW
319441e75d
Merge pull request #8051 from ThomasWaldmann/corrupted-key-errmsg-1.4
better error msg for corrupted key data, fixes #8016
2024-01-19 00:45:59 +01:00
Thomas Waldmann
dbbccf9451
better error msg for corrupted key data, fixes #8016 2024-01-18 23:59:35 +01:00
TW
06f6136a60
Merge pull request #8049 from ThomasWaldmann/bump-api-version-1.4
bump api version to 1.4
2024-01-18 23:13:28 +01:00
TW
49d21f95f5
Merge pull request #8050 from ThomasWaldmann/use-cy308-1.4
use cython 3.0.8
2024-01-18 23:13:10 +01:00
Thomas Waldmann
c457dbcd8a
use cython 3.0.8 2024-01-18 20:17:11 +01:00
TW
1cffe6906d
Merge pull request #8048 from ThomasWaldmann/y2024-1.4
Y2024 (1.4-maint)
2024-01-18 20:10:24 +01:00
Thomas Waldmann
98fa922904
bump api version to 1.4 2024-01-18 20:09:49 +01:00
Thomas Waldmann
2b7f3adec9
it's 2024 2024-01-18 20:02:56 +01:00
TW
a73f74e0b0
Merge pull request #8047 from ThomasWaldmann/error-msg-bad-nonce-file-1.4
repository: give clean error msg for invalid nonce file, see #7967
2024-01-18 19:53:02 +01:00
Thomas Waldmann
742bd0c097
refactor: use less binascii
our own hex_to_bin / bin_to_hex is more comfortable to use.

also: optimize remaining binascii usage / imports.
2024-01-18 18:13:40 +01:00
Thomas Waldmann
d9132a34d4
give clean error msg for invalid nonce file, see #7967
That rather long traceback was not pretty.

Of course users should usually have valid nonce files,
but multiple users managed to corrupt the file contents somehow.

Also:
- add Error / ErrorWithTraceback exception classes to RPC layer.
- add hex_to_bin helper
- also call hex_to_bin for the local nonce file (security dir)
2024-01-17 15:31:03 +01:00
TW
5b77bfef06
Merge pull request #8045 from ThomasWaldmann/brewfile-1.4
Create and use Brewfile (1.4-maint)
2024-01-15 01:14:34 +01:00
Thomas Waldmann
700d588e05
Brewfile: don't install osxfuse cask by default. 2024-01-14 19:58:49 +01:00
Stephen
e3ba364933
docs: add brew bundle instructions (macOS) 2024-01-14 19:54:30 +01:00
Stephen
bfe81ab1d7
github CI (macOS): use brew bundle install (uses Brewfile) 2024-01-14 19:53:45 +01:00
Stephen
ed188d528f
Create Brewfile 2024-01-14 19:51:43 +01:00
TW
5dc8ddc8bd
Merge pull request #8043 from ThomasWaldmann/logging-msg-param-1.4
logger method params (1.4-maint)
2024-01-14 14:42:42 +01:00
Thomas Waldmann
884668ee07
borg.logger: use same method params as python logging 2024-01-14 00:07:05 +01:00
TW
046a5ecdc7
Merge pull request #8033 from ThomasWaldmann/fix-commanderror-args-1.4
fix CommandError args, fixes #8029
2024-01-08 12:46:37 +01:00
TW
9644cd1181
Merge pull request #8034 from ThomasWaldmann/fetch-binaries-1.4
add script for fetching borg binaries from VMs, fixes #7989
2024-01-07 21:12:07 +01:00
Thomas Waldmann
37e1ec5e4a
add script for fetching borg binaries from VMs, fixes #7989 2024-01-07 21:00:16 +01:00
Thomas Waldmann
c2e7b7cc96
fix CommandError args, fixes 8029 2024-01-07 19:47:50 +01:00
TW
2812f992b8
Merge pull request #8032 from kmille/1.4-maint
improve docs for borg with-lock, 1.4-maint backport
2024-01-07 19:08:40 +01:00
TW
531d32460e
Merge pull request #8030 from ThomasWaldmann/fix-with-lock-exceptions-1.4
borg with-lock: catch exception, print error msg, fixes #8022
2024-01-07 19:06:47 +01:00
kmille
0c3c1e6e3c
improve docs for borg with-lock
discussion in #8022
port of #8024
2024-01-07 18:32:40 +01:00
kmille
aae959cd6d
borg with-lock: catch exception, print error msg, fixes #8022 2024-01-07 17:14:26 +01:00
TW
499e07d842
Merge pull request #8028 from ThomasWaldmann/modernize-msgpack-1.4
modernize msgpack wrapper
2024-01-07 05:39:12 +01:00
Thomas Waldmann
46fc887647
msgpack: remove max_*_len, msgpack has builtin DoS-protection
msgpack >= 0.6.1 has DoS-protection and we require >= 1.0.3 anyway.

This is now a bit closer to what master branch uses.
2024-01-04 20:14:59 +01:00
Thomas Waldmann
8cb7bc3d32
msgpack: simplify, we are always using msgpack >= 1.0.3 now
see requirements, we also check this in is_supported_msgpack().

update comment about msgpack api change.
2024-01-04 20:14:34 +01:00
TW
0f23bda622
Merge pull request #8011 from ThomasWaldmann/rel140a1
release 1.4.0a1
2024-01-01 21:56:56 +01:00
Thomas Waldmann
f6963df380
build_man 2024-01-01 21:16:56 +01:00
Thomas Waldmann
5eaabea647
build_usage 2024-01-01 21:16:30 +01:00
Thomas Waldmann
3426f91241
update release checklist 2024-01-01 21:13:27 +01:00
Thomas Waldmann
520870329f
update CHANGES 2024-01-01 21:11:07 +01:00
TW
3a269d9552
Merge pull request #8010 from ThomasWaldmann/update-docs-1.4
update CHANGES
2024-01-01 20:53:39 +01:00
Thomas Waldmann
7e2a1c4ea7 update CHANGES 2024-01-01 20:21:15 +01:00
TW
de5bfdbd7a
Merge pull request #7976 from ThomasWaldmann/new-rc-1.4
optional more specific return codes
2024-01-01 19:53:25 +01:00
TW
0e5f6e6a4e
Merge pull request #8006 from ThomasWaldmann/update-changes-1.4
update CHANGES
2023-12-27 19:28:38 +01:00
Thomas Waldmann
649537f951
update CHANGES 2023-12-27 19:22:04 +01:00
Thomas Waldmann
2b6dd14464
use get_reset_ec to internally re-init ec/warnings
if we do multiple calls to Archiver.do_something(),
we need to reset the ec / warnings after each call,
otherwise they will keep growing (in severity, in length).
2023-12-27 13:49:11 +01:00
Thomas Waldmann
242b8f9f51
refactor (re-)init of exit_code and warnings_list globals
stop directly accessing the variables from other modules.

prefix with underscore to indicate that these shall
only be used within this module and every other user
shall call the respective functions.
2023-12-27 13:49:08 +01:00
Thomas Waldmann
4bb42d2a00
do not return the rc from Archiver methods
this is not needed and getting rid of it makes
the code / behaviour simpler to understand:

if a fatal error is detected, we throw an exception.

if we encounter something warning worthy, we emit and collect the warning.

in a few cases, we directly call set_ec to set the
exit code as needed, e.g. if passing it through
from a subprocess.

also:
- get rid of Archiver.exit_code
- assert that return value of archiver methods is None
- fix a print_warning call to use the correct formatting method
2023-12-27 13:49:06 +01:00
Thomas Waldmann
f81c6b5963
update frontends.rst error/warning list 2023-12-27 13:48:58 +01:00
Thomas Waldmann
69072f0c17
BackupErrors get caught and give warning RCs
also: use more union operators rather than .union()
2023-12-27 13:48:55 +01:00
Thomas Waldmann
83bf4d8c7d
raise BackupOSError subclasses 2023-12-27 13:48:52 +01:00
Thomas Waldmann
82f147f63f
add NotFoundWarning 2023-12-27 13:48:49 +01:00
Thomas Waldmann
ead9fe4763
print_warning*: support warning msgids, fixes #7080 2023-12-27 13:48:47 +01:00
Thomas Waldmann
775b9f5560
extend errorlist script to warnings, update docs 2023-12-27 13:48:41 +01:00
Thomas Waldmann
2fda86ce61
more detailled warnings for source file OSErrors 2023-12-27 13:48:14 +01:00
Thomas Waldmann
53291a4ac0
BackupError->BackupWarning, BackupOSError->BackupOSWarning 2023-12-27 13:48:12 +01:00
Thomas Waldmann
c2e8bb0468
move Backup*Error to errors module 2023-12-27 13:48:11 +01:00
Thomas Waldmann
482ac47ed8
new warnings infrastructure to support modern exit codes
- implement updating exit code based on severity, including modern codes
- extend print_warning with kwargs wc (warning code) and wt (warning type)
- update a global warnings_list with warning_info elements
- create a class hierarchy below BorgWarning class similar to Error class
- diff: change harmless warnings about speed to rc == 0
- delete --force --force: change harmless warnings to rc == 0

Also:

- have BackupRaceConditionError as a more precise subclass of BackupError
2023-12-27 13:47:52 +01:00
Thomas Waldmann
770f4117a5
get rid of some rare error classes, use RTError instead 2023-12-27 13:46:15 +01:00
Thomas Waldmann
f6279eac9d
shorten TAMRequiredError error msg
Users using recently created repos (after borg 1.0.9) or
who followed the upgrade procedure of 1.0.9 or >1.2.4 will
never see this error msg.

So, have it as short as and similar to the ArchiveTAMRequiredError.
2023-12-27 13:46:13 +01:00
Thomas Waldmann
41351af635
fix dealing with remote repo Locking Exceptions
previously, this was handled in RPCError handler and always resulted in rc 2.

now re-raise Lock Exceptions locally, so it gives rc 2 (legacy) or 7x (modern).
2023-12-27 13:46:12 +01:00
Thomas Waldmann
add7a22581
refactor set_ec usage
- _export_tar: remove unneeded call to set_ec
  print_warning() already sets the exit code to EXIT_WARNING.

- msgpack version check: raise Error instead of calling set_ec
2023-12-27 13:46:11 +01:00
Thomas Waldmann
74954faec8
use print_warning also in borg delete ::archive --force --force 2023-12-27 13:46:04 +01:00
Thomas Waldmann
62ad0369ef
update "modern" error RCs (docs and code) 2023-12-27 13:45:48 +01:00
Thomas Waldmann
422d1ba4ca
scripts/errorlist.py: improve error list docs generation
- also output modern rc and traceback yes/no
- recursive list of Error subclasses
2023-12-27 13:41:58 +01:00
Thomas Waldmann
2a13268e05
BORG_EXIT_CODES=modern can be set to get more specific process exit codes
If not set, it will default to "legacy" (always return 2 for errors).

This commit only changes the Error exception class and its subclasses.

The more specific exit codes need to be defined via .exit_mcode in the subclasses.
2023-12-27 13:41:46 +01:00
TW
94c461974d
Merge pull request #8005 from ThomasWaldmann/do-not-accept-empty-paths-1.4
PATH: do not accept empty strings, fixes #4221
2023-12-27 09:51:12 +01:00
Thomas Waldmann
8ad7a9ae77
PATH: do not accept empty strings, fixes #4221 2023-12-27 04:14:58 +01:00
TW
82cba46345
Merge pull request #8003 from ThomasWaldmann/vagrant-updates3-1.4
vagrant updates3 (1.4-maint)
2023-12-26 23:57:51 +01:00
Thomas Waldmann
bd9d12c0ba
vagrant: use openssl 3.0 on macOS 2023-12-26 22:15:59 +01:00
Thomas Waldmann
8ce3f23dca
vagrant: use openssl 3.0 on openbsd 2023-12-26 22:11:00 +01:00
Thomas Waldmann
f2b70cf0a6
vagrant: remove outdated comment 2023-12-26 22:10:59 +01:00
Thomas Waldmann
e80fa4a189
vagrant: use generic/openbsd7 box 2023-12-26 22:10:53 +01:00
TW
60dcd53130
Merge pull request #8001 from ThomasWaldmann/move-conftest.py-1.4
move conftest.py to src/borg/testsuite, see #6386
2023-12-26 20:59:41 +01:00
TW
6ce8bd6fff
Merge pull request #8002 from ThomasWaldmann/remove-dateutil-1.4
get rid of dateutil
2023-12-26 20:59:02 +01:00
Thomas Waldmann
2d97dcf250
get rid of dateutil 2023-12-26 19:12:00 +01:00
Thomas Waldmann
63848349c0
move conftest.py to src/borg/testsuite, see #6386
there, it will be packaged and installed, so
that when one runs the testsuite after installing
the pypi package, it will find the conftest.py.
2023-12-26 18:55:05 +01:00
TW
68109e7af9
Merge pull request #7999 from ThomasWaldmann/fix-msgpack-version-check
msgpack version check: use same versions as pyproject.toml
2023-12-26 18:49:47 +01:00
Thomas Waldmann
0ab26a04b7
msgpack version check: use same versions as pyproject.toml 2023-12-26 17:15:45 +01:00
TW
218160a914
Merge pull request #7993 from ThomasWaldmann/latest-pyinstaller-1.4
use pyinstaller 6.3.0
2023-12-26 01:01:12 +01:00
Thomas Waldmann
7efef07db1
use pyinstaller 6.3.0 2023-12-26 00:08:37 +01:00
TW
453c8eda07
Merge pull request #7998 from ThomasWaldmann/vagrant-updates2-1.4
Vagrant updates2 (1.4-maint)
2023-12-25 20:34:11 +01:00
Thomas Waldmann
b3d0b0abf6
vagrant: netbsd: test on py311 only
the netbsd vagrant machine tends to segfault, guess due to some kernel or virtualbox issue.

 thus, rather only do 1 tox run, so there is less output to review.
2023-12-25 20:02:08 +01:00
Thomas Waldmann
0c89f2c07b
vagrant: use python 3.9.4
3.9.3 was withdrawn by python.org and pyenv devs.
2023-12-25 19:25:49 +01:00
Thomas Waldmann
8088125f5c
vagrant: remove ubuntu 20.04 "focal" box
there are multiple issues with that box:
- it has a OpenSSL 1.x that is out of upstream support.
- has python 3.8

also:
- conservative users who did not upgrade yet to 22.04 will likely not jump onto borg 1.4 soon, but rather use borg 1.2.x.

We'll try to keep this box alive for borg 1.2.x, but for borg 1.4 it is "game over".
2023-12-25 16:36:13 +01:00
Thomas Waldmann
3d856a1c1e
vagrant: remove debian 9 "stretch" box
there are multiple issues with that box:
- debian 9 is out of support by debian, out of even lts support since 2022
- it has a OpenSSL 1.x that is also out of support and noone will care for it.
- has python 3.5

We'll try to keep this box alive for borg 1.2.x, but for borg 1.4 it is "game over".
2023-12-25 16:24:56 +01:00
TW
4f958ab5fe
Merge pull request #7997 from ThomasWaldmann/move-to-pyproj-1.4
move to pyproject.toml (1.4-maint)
2023-12-25 15:57:49 +01:00
TW
043294226b
Merge pull request #7996 from ThomasWaldmann/pyupgrade-py39-1.4
pyupgrade py39 (1.4-maint)
2023-12-25 04:21:41 +01:00
Thomas Waldmann
915e58653b
replace flake8 by ruff 2023-12-25 04:20:15 +01:00
Thomas Waldmann
d4f6f137c3
move most settings to pyproject.toml
except: flake8, which will get replaced by ruff.
2023-12-25 03:45:44 +01:00
TW
0e446170a6
Merge pull request #7994 from ThomasWaldmann/update-docs-1.4
Update docs (1.4-maint)
2023-12-25 02:55:07 +01:00
TW
c7ae13cb2a
Merge pull request #7995 from ThomasWaldmann/require-cython3-1.4
require Cython>=3 (1.4-maint)
2023-12-25 02:54:47 +01:00
Thomas Waldmann
e3198b1df3
cosmetic: make code closer to master branch 2023-12-25 02:44:15 +01:00
Thomas Waldmann
8d2fd2a30e
ran pyupgrade --py39-plus ./**/*.py 2023-12-25 02:34:44 +01:00
Thomas Waldmann
2f541cedee
require Cython>=3 2023-12-25 02:20:50 +01:00
Thomas Waldmann
d9061e9a07
update CHANGES 2023-12-25 02:13:05 +01:00
Thomas Waldmann
c4571f4181
update SECURITY.md about borg 1.4 2023-12-25 02:13:03 +01:00
TW
c14af19576
Merge pull request #7977 from ThomasWaldmann/drop-setup-requires-1.4
drop setup_requires, use pip and build
2023-12-25 01:36:57 +01:00
Thomas Waldmann
84df37a19f
use less setup.py, use pip and build
todo: clean clean2 build_usage build_man
2023-12-24 20:50:19 +01:00
Thomas Waldmann
021d08c8be
fix msys build
build the sdist and wheel using "build", avoid using setup.py.

use "pip install -e ." before invoking pyinstaller, so we have
the _version.py built by setuptools_scm and also it compiled the
extension modules, so pyinstaller can pick them up.

msys2 hack:

https://stackoverflow.com/questions/76079590/error-plat-name-must-be-one-of-win32-win-amd64-win-arm32-win-arm6
2023-12-24 20:45:12 +01:00
Thomas Waldmann
5919b63e9a
setup.py: drop deprecated setup_requires, fixes #6068
We now use pyproject.toml for that.

Also, drop the >=1.7 requirement for setuptools_scm,
that was from 2015 and we assume it is not used any
more anyway.
2023-12-24 18:23:07 +01:00
TW
121c870242
Merge pull request #7980 from ThomasWaldmann/drop-py38-1.4
require python >= 3.9, drop py38, fixes #6383
2023-12-24 15:07:19 +01:00
Thomas Waldmann
6b402896a6
require python >= 3.9, drop py38, fixes #6383 2023-12-24 03:01:21 +01:00
TW
fb1a327008
Merge pull request #7991 from ThomasWaldmann/cleanup-requirements-1.4
Cleanup requirements (1.4-maint)
2023-12-24 02:57:15 +01:00
Thomas Waldmann
299237e401
tox: remove min_version
3.2 is ancient, so we can just use the default now.
2023-12-24 02:33:43 +01:00
Thomas Waldmann
6284053562
update requirements.d/development.lock.txt 2023-12-24 02:33:41 +01:00
Thomas Waldmann
a550fbd845
msgpack: require >= 1.0.2 <= 1.0.7
that simplifies things:
- no broken release in that range
- no ancient stuff < 1.0 any more
2023-12-24 02:09:58 +01:00
Thomas Waldmann
548db7dc90
adjust requirements/development.txt to pyproject.toml 2023-12-24 02:06:25 +01:00
TW
6db2bbf635
Merge pull request #7988 from ThomasWaldmann/vagrant-updates-1.4
Vagrant updates (1.4-maint)
2023-12-24 01:24:56 +01:00
Thomas Waldmann
1081825fe0
vagrant: use freebsd 14, fixes #6871 2023-12-24 01:21:42 +01:00
Thomas Waldmann
5d8bf2f81d
vagrant: update python versions, use 3.11 for binary build, fixes #7987 2023-12-24 01:19:13 +01:00
Thomas Waldmann
35eca86f23
vagrant: update python versions, use 3.11 for binary build, fixes #7987 2023-12-24 00:40:35 +01:00
TW
58ec02d641
Merge pull request #7983 from ThomasWaldmann/setuptools_scm-pyproj-1.4
move setuptools_scm configuration to pyproject.toml (1.4-maint)
2023-12-24 00:30:15 +01:00
TW
057818659a
Merge pull request #7982 from ThomasWaldmann/fix-openssl-warnings-1.4
crypto: get rid of deprecated HMAC_* functions
2023-12-24 00:26:50 +01:00
TW
72e249e243
Merge pull request #7984 from ThomasWaldmann/upgrade-pyinstaller-1.4
use pyinstaller==5.13.2
2023-12-24 00:26:22 +01:00
Thomas Waldmann
b0dd5ca280
use pyinstaller==5.13.2
this is the version we use in master branch since longer.
2023-12-23 23:16:30 +01:00
Thomas Waldmann
15e847d4d9
require recent setuptools and setuptools_scm 2023-12-23 23:00:04 +01:00
Thomas Waldmann
5049817d43
move setuptools_scm configuration to pyproject.toml 2023-12-23 22:57:53 +01:00
TW
94c37858d0
Merge pull request #7979 from ThomasWaldmann/use-cython3-1.4
cython: use 3str language level (default in cython3), fixes #7978
2023-12-23 22:32:28 +01:00
Thomas Waldmann
c813f9ba59
crypto: get rid of deprecated HMAC_* functions
These were deprecated in OpenSSL 3.0.
Instead, use hmac.digest from Python stdlib.
2023-12-23 22:20:50 +01:00
Thomas Waldmann
ed584554d0
cython: use 3str language level (default in cython3), fixes #7978
drop support for Cython 0.29.x, use Cython 3.0.x from now on.
2023-12-23 21:27:31 +01:00
TW
ae6be14822
Merge pull request #7981 from ThomasWaldmann/fix-ci-config-1.4
github ci / codeql: use 1.4-maint branch
2023-12-23 21:25:28 +01:00
Thomas Waldmann
39a48d50cd
github ci / codeql: use 1.4-maint branch 2023-12-23 21:24:44 +01:00
TW
fe624870f5
Merge pull request #7974 from ThomasWaldmann/sync-with-1.2-maint
sync with 1.2-maint
2023-12-23 19:14:34 +01:00
Marc Kohaupt
05fa6fcab2
typo: notes.rst 2023-12-23 18:58:47 +01:00
Thomas Waldmann
49d7e74f97
change log: more infos about #6687 fix 2023-12-23 18:58:20 +01:00
TW
c675a00aa5
Merge pull request #7958 from ThomasWaldmann/rel127
release 1.2.7
2023-12-02 14:00:35 +01:00
Thomas Waldmann
4c131059bc
build_man 2023-12-02 01:02:03 +01:00
Thomas Waldmann
46c1e01d29
build_usage 2023-12-02 01:01:46 +01:00
Thomas Waldmann
8b70bb36c6
update CHANGES 2023-12-02 00:59:58 +01:00
TW
e50d26757c
Merge pull request #7956 from ThomasWaldmann/py313-unistdh-1.2
include unistd.h in _chunker.c
2023-12-02 00:44:59 +01:00
TW
d3e412c7c6
Merge pull request #7957 from ThomasWaldmann/docs-rebuild-refcounts-1.2
docs: Add "check.rebuild_refcounts" message
2023-12-02 00:23:25 +01:00
Sophie Herold
897f88dfaa
docs: Add "check.rebuild_refcounts" message 2023-12-02 00:01:12 +01:00
Felix Schwarz
13c6d1b710
include unistd.h in _chunker.c
With Python 3.13, Python.h no longer includes the <unistd.h> standard header file:
https://docs.python.org/3.13/whatsnew/3.13.html#id8
2023-12-01 23:58:24 +01:00
TW
b2929b4cee
Merge pull request #7952 from ThomasWaldmann/update-changes-1.2
update CHANGES (1.2-maint)
2023-11-29 16:48:38 +01:00
Kevin Yin
bcc8f74b56
Remove period that could be interpreted as part of the command (#7946)
CVE upgrade steps: remove period that could be interpreted as part of the command
2023-11-27 18:12:26 +01:00
Thomas Waldmann
4bdd404949
update CHANGES 2023-11-27 18:08:00 +01:00
TW
70eed5e9fc
Merge pull request #7939 from ThomasWaldmann/lockroster-remove-fix-1.2
LockRoster.modify fix (1.2-maint)
2023-11-18 17:35:22 +01:00
Thomas Waldmann
ec17b00411
LockRoster.modify: no KeyError if element was already gone, fixes #7937
The intention of LockRoster.modify(key, REMOVE) is to remove self.id.

Using set.discard will just ignore it if self.id is not present there anymore.

Previously, using set.remove triggered a KeyError that has been frequently
seen in tracebacks of teardowns involving Repository.__del__ and Repository.__exit__.

I added a REMOVE2 op to serve one caller that needs to get the KeyError if
self.id was not present.

Thanks to @herrmanntom for the workaround!
2023-11-16 15:59:28 +01:00
zDEFz
2cefe8f53d
Add backup by SERIAL method (#7934)
docs: create disk/partition sector backup by disk serial number
2023-11-13 22:03:09 +01:00
TW
2341ba62d3
Merge pull request #7926 from ThomasWaldmann/update-requirements-1.2
update requirements.d/development.lock.txt
2023-11-09 02:19:51 +01:00
Thomas Waldmann
f08386f699
update requirements.d/development.lock.txt 2023-11-09 00:46:12 +01:00
TW
4e349cf3fa
Merge pull request #7915 from ThomasWaldmann/fix-subcommand-env-1.2
create --*-from-command: run subcommands with a clean environment
2023-11-07 03:42:25 +01:00
Thomas Waldmann
e006a6f368
create --*-from-command: run subcommands with a clean environment, fixes #7916
When borg invokes a system command, it needs to prepare the environment
for that. This is especially important when using a pyinstaller-made
borg fat binary that works with a modified env var LD_LIBRARY_PATH -
system commands may crash with that.

borg already had calls to prepare_subprocess_env at some places (e.g.
when invoking ssh for the remote repo connection), but they were
missing for:

borg create --content-from-command ...
borg create --paths-from-command ...
2023-11-06 17:41:52 +01:00
TW
11af475308
Merge pull request #7913 from ThomasWaldmann/docs-upgrade-chkpt-tam-1.2
CVE-2023-36811 upgrade docs: consider checkpoint archives, fixes #7802
2023-11-06 14:30:36 +01:00
Thomas Waldmann
9c886330d3
CVE-2023-36811 upgrade docs: consider checkpoint archives, fixes #7802
The traceback seen there came from borg not showing a .checkpoint archive
that didn't have a valid TAM and thus the user not see it / fix it:

https://github.com/borgbackup/borg/issues/7802#issuecomment-1793860606
2023-11-06 13:31:02 +01:00
TW
21d67b8295
Merge pull request #7904 from ThomasWaldmann/fix-shadow-index2-1.2
shadow index: add more comments
2023-11-05 01:24:47 +01:00
Thomas Waldmann
2a2b750b9d
shadow index: add more comments 2023-11-05 01:22:07 +01:00
TW
a13504c6d5
Merge pull request #7903 from ThomasWaldmann/update-changes-1.2
update CHANGES (1.2-maint)
2023-11-04 18:05:03 +01:00
Thomas Waldmann
d9a0deb227
update CHANGES 2023-11-03 18:43:22 +01:00
TW
686714c366
Merge pull request #7896 from ThomasWaldmann/fix-shadow-index-1.2
fix shadow index update for double-put (1.2-maint)
2023-11-03 14:42:57 +01:00
TW
55ee4e2d28
Merge pull request #7897 from ThomasWaldmann/check-fixes-shadow-index-1.2
check --repair fixes shadow_index (1.2-maint)
2023-11-03 14:42:39 +01:00
TW
cf5c61644b
Merge pull request #7901 from ThomasWaldmann/update-docs-1.2
Update docs (1.2-maint)
2023-11-03 14:35:02 +01:00
Thomas Waldmann
6b928dac93
docs: not only attack/unsafe, can also be a fs issue, fixes #7853 2023-11-01 18:04:24 +01:00
Thomas Waldmann
da4fcc5a66
docs: point to CVE-2023-36811 upgrade steps from borg 1.1 to 1.2 upgrade steps, fixes #7899
also: use 1.2.6 to refer to the fixed version

1.2.5 had issues and was superseded by 1.2.6 just 1 day later,
so we do not need to talk about that.

Also, the docs point out that:
"""
Below, if we speak of borg 1.2.6, we mean a borg version >= 1.2.6 **or** a
borg version that has the relevant security patches for this vulnerability applied
(could be also an older version in that case).
"""

So, it now just talks about "1.2.6" at the relevant places.
2023-11-01 18:04:22 +01:00
Thomas Waldmann
136e3ed1d6
docs: upgrade steps needed for all kinds of repos, fixes #7813
Repos with encryption=none of course can not do real authentication (because there is no secret key material to work with).

But the code always works the same way and such repos also use (and expect) TAMs, so the instructions should be followed in any case.
2023-11-01 18:04:20 +01:00
Thomas Waldmann
a19ebe5ecc
docs: consequences of borg check, fixes #7816 2023-11-01 17:18:32 +01:00
Thomas Waldmann
a7809429b3
check --repair: recreate shadow index, see #6687
before this fix, borg check --repair just created an
empty shadow index, which can lead to incomplete
entries if entries are added later.

and such incomplete (but present) entries can lead to
compact_segments() resurrecting old PUTs by accidentally
dropping related DELs.
2023-10-29 01:12:15 +02:00
Thomas Waldmann
48feb28e11
check --repair: test if shadow index is recreated
still failing here, because it is not.
2023-10-29 01:01:17 +02:00
Thomas Waldmann
8a8837e255
shadow index updates: simplify and more comments
no functional change here.
2023-10-28 17:36:10 +02:00
Thomas Waldmann
b21ed3c658
update shadow index when doing a double-put, fixes #5661
this fixes the test added in previous commit (avoids
that the PUT reappears after index rebuild).
2023-10-28 17:17:47 +02:00
Thomas Waldmann
8e6449f28e
test the shadowing-by-double-put behaviour, see #5661
the new test is currently failing due to a bug in the
repository code.
2023-10-28 17:13:13 +02:00
TW
c35cddeb7e
Merge pull request #7892 from ThomasWaldmann/ci-timeout-1.2
CI: increase timeout to 90min
2023-10-28 01:25:33 +02:00
Thomas Waldmann
dff29627dc
CI: increase timeout to 90min
esp. the macOS CI jobs are sometimes rather slow,
so 40min were not enough.
2023-10-27 22:30:52 +02:00
TW
b83525cd4c
Merge pull request #7891 from ThomasWaldmann/sort-by-aliases-1.2
--sort-by: support "archive" as alias of "name", fixes #7873 (1.2-maint)
2023-10-27 22:27:59 +02:00
TW
fa7a6d7095
Merge pull request #7890 from ThomasWaldmann/doc-updates-1.2
doc updates (1.2-maint)
2023-10-27 22:25:59 +02:00
Thomas Waldmann
0e3773d8c7
--sort-by: support "archive" as alias of "name", fixes #7873 2023-10-27 20:46:03 +02:00
TW
75bc62e2ae
Merge pull request #7889 from ThomasWaldmann/udev-uuid-1.2
automated-local.rst: use GPT UUID for consistent udev rule (1.2-maint)
2023-10-27 20:33:57 +02:00
Thomas Waldmann
05cec8b3b0
docs: OS X -> macOS 2023-10-27 20:31:26 +02:00
Thomas Waldmann
7a4d53c2f5
docs: update macOS hint about full disk access 2023-10-27 20:28:57 +02:00
Thomas Waldmann
661e5cccd3
docs: minor fixes/update to notes 2023-10-27 20:27:51 +02:00
Thomas Waldmann
d13b5d5557
docs: remove info about borg 1.0 files per dir 2023-10-27 20:26:27 +02:00
Thomas Waldmann
e0f80812a5
docs: no signature upload on pypi any more
but we upload it on github releases,
so people can check it.
2023-10-27 20:25:29 +02:00
Thomas Waldmann
7640973d6b
docs: CI only on github actions now 2023-10-27 20:23:00 +02:00
azrdev
69d71d99bf
automated-local.rst: Partition table UUID match works for MBR too
not only GPT
2023-10-27 20:15:05 +02:00
azrdev
b61a565fbc
automated-local.rst: use GPT UUID for consistent udev rule 2023-10-27 20:14:50 +02:00
TW
4a8f8bf789
Merge pull request #7886 from ThomasWaldmann/fix-inv-pattern-errorhandling-1.2
Fix arg parsing error handling (1.2-maint)
2023-10-27 20:09:36 +02:00
Thomas Waldmann
5bfa417546
fix rc and msg if arg parsing throws an exception, fixes #7885
get_args() exception handling before this fix only dealt with
subclasses of "Error", but we have to expect other exceptions
there, too.

In any case, if we have some fatal exception here, we must
terminate with rc 2.

ArgumentTypeError: emit a short error message - usually this is
a user error, invoking borg in a wrong way.

Other exceptions: full info and traceback.
2023-10-25 12:06:27 +02:00
Thomas Waldmann
1bd015cdb5
fix invalid pattern argument error msg 2023-10-25 11:06:37 +02:00
Johannes Lade
d482251a3e
Clarify borg prune -a option description (#7871)
Clarify borg prune -a option description.

The -a option for borg prune accepts only glob pattern (i.e. sh:)
but not the actual prefix sh: which can be confusing especially for
people who don't actually know what glob pattern is.
2023-10-14 22:00:19 +02:00
TW
e2ea9b6547
Merge pull request #7864 from ThomasWaldmann/remove-bountysource-badge-1.2
remove bountysource badge (1.2-maint)
2023-10-10 21:01:56 +02:00
Thomas Waldmann
b8d950d097
remove bountysource badge 2023-10-10 20:32:11 +02:00
TW
bc553b1258
Merge pull request #7857 from ThomasWaldmann/misc-updates-1.2
misc. updates (1.2-maint)
2023-10-09 20:57:11 +02:00
Felix Schwarz
90de901ff4
replace "datetime.utcfromtimestamp" with custom helper to avoid deprecation warnings when using Python 3.12 2023-10-09 20:23:10 +02:00
Thomas Waldmann
f72adc9e61
vagrant: use generic/debian9 box, fixes #7579
the debian/stretch64 was broken and never fixed.
2023-10-08 21:53:32 +02:00
Thomas Waldmann
37c3177e90
update development.lock.txt
there was some breakage with py312, so better
use current tooling.
2023-10-08 21:42:04 +02:00
Thomas Waldmann
d6ad508a29
github CI: test on py312 (w/o -dev) 2023-10-08 00:20:31 +02:00
Thomas Waldmann
3c4b1090a6
vagrant: install py312 via pyenv 2023-10-08 00:15:39 +02:00
Thomas Waldmann
9df5e6c139
allow setuptools_scm >= 8 again
8.0.4 got a fix, so we can allow it again.
2023-10-08 00:10:36 +02:00
TW
8c267e5c3c
Merge pull request #7850 from ThomasWaldmann/allow-msgpack107-1.2
allow msgpack 1.0.7 (1.2-maint)
2023-09-28 15:36:54 +02:00
Thomas Waldmann
39761ebadc
allow msgpack 1.0.7 2023-09-28 14:26:48 +02:00
TW
e936d53a66
Merge pull request #7845 from ThomasWaldmann/update-changes-1.2
update CHANGES
2023-09-26 00:03:10 +02:00
Thomas Waldmann
5195fb2dcf
update CHANGES 2023-09-25 17:07:28 +02:00
TW
c9c495db0a
Merge pull request #7839 from ThomasWaldmann/traceback-for-tam-exceptions-1.2
activate tracebacks for TAM exceptions
2023-09-25 16:53:22 +02:00
TW
4188e03ffc
Merge pull request #7844 from ThomasWaldmann/py312-1.2
support python 3.12 (1.2-maint)
2023-09-25 16:45:04 +02:00
Thomas Waldmann
1a24c2fa43
avoid tarfile deprecation warning for py312 2023-09-25 15:36:43 +02:00
Thomas Waldmann
8f820d1f32
add wheel to build-system requirements 2023-09-25 15:34:44 +02:00
Thomas Waldmann
906962b631
setup.py: declare python 3.12 support 2023-09-25 15:33:02 +02:00
Thomas Waldmann
ce274aa11c
test on python 3.12-dev 2023-09-25 15:29:30 +02:00
TW
47e65d9c00
Merge pull request #7842 from ThomasWaldmann/offline-docs2-1.2
offline docs: remove epub, pdf. only build htlmzip.
2023-09-25 15:15:10 +02:00
Thomas Waldmann
377d02c4b2
offline docs: remove epub, pdf. only build htlmzip.
something is broken with these, build fails.
2023-09-25 15:14:26 +02:00
TW
20f9953293
Merge pull request #7841 from ThomasWaldmann/offline-docs-1.2
readthedocs: also build offline docs, fixes #7835 (1.2-maint)
2023-09-25 14:47:52 +02:00
TW
6170ab7d34
Merge pull request #7840 from ThomasWaldmann/allow-msgpack106-1.2
Allow msgpack 1.0.6 (1.2-maint)
2023-09-25 14:47:28 +02:00
Thomas Waldmann
da2b885179
readthedocs: also build offline docs, fixes #7835 2023-09-25 14:24:13 +02:00
Thomas Waldmann
ed87e45bab
use setuptools_scm < 8 for now
https://github.com/pypa/setuptools_scm/issues/938
2023-09-25 14:14:51 +02:00
Thomas Waldmann
93b93abc18
allow msgpack 1.0.6 2023-09-25 14:10:32 +02:00
Thomas Waldmann
6dbfbd6a19
activate tracebacks for TAM exceptions 2023-09-25 11:01:38 +02:00
TW
0c198160f8
Merge pull request #7821 from ThomasWaldmann/vagrant-updates-1.2
vagrant: add VM with debian bookworm
2023-09-12 22:07:41 +02:00
Thomas Waldmann
3d367b0fd3
vagrant: add VM with debian bookworm
this also brings OpenSSL 3.0.x.
2023-09-12 18:41:38 +02:00
TW
83028c0b89
Merge pull request #7820 from ThomasWaldmann/remove-twine-1.2
remove twine from requirements
2023-09-12 17:37:34 +02:00
Thomas Waldmann
1407a2b2fb
remove twine from requirements
twine is only needed at release time, no need
for all developers or all test runs to install
this.

also, some requirement of twine needs a rust
compiler, so if there is no rust compiler,
automated runs will abort due to that.
2023-09-12 17:05:02 +02:00
TW
a0069d4b45
Merge pull request #7801 from ThomasWaldmann/fix-upgrade-docs-1.2
docs: minor fix to CVE-2023-36811 related upgrade instructions
2023-09-02 15:53:06 +02:00
Thomas Waldmann
97b8eb2104
docs: minor fix to CVE-2023-36811 related upgrade instructions
while the main issue in the code has been fixed since 1.2.5,
let's better refer to 1.2.6, which has fixes in upgrade docs and code.
2023-09-02 15:47:13 +02:00
TW
ee6a8e2ddf
Merge pull request #7798 from ThomasWaldmann/check-improve-TAM-logging-1.2
check: improve logging for TAM issues, fixes #7797
2023-09-01 22:07:59 +02:00
Thomas Waldmann
7fd91f4bb8
check: improve logging for TAM issues, fixes #7797 2023-09-01 15:35:04 +02:00
TW
65d75c167a
Merge pull request #7792 from ThomasWaldmann/rel126
Release 1.2.6
2023-08-31 23:37:46 +02:00
Thomas Waldmann
98dea3e876
requirements: remove cython restrictions 2023-08-31 01:03:34 +02:00
Thomas Waldmann
c133303243
update CHANGES, incl. upgrade instructions 2023-08-31 01:02:44 +02:00
Thomas Waldmann
d05d02f238
implement BORG_WORKAROUNDS=ignore_invalid_archive_tam, see #7791 2023-08-31 00:36:04 +02:00
TW
7ab28487ce
Merge pull request #7790 from ThomasWaldmann/fix-changelog-markup-1.2
fix changelog markup
2023-08-30 18:41:31 +02:00
Thomas Waldmann
447d0df1bb
fix changelog markup 2023-08-30 18:36:42 +02:00
TW
f7a93b30fa
Merge pull request #7788 from ThomasWaldmann/fix-cve-docs-1.2
fix CVE timeline
2023-08-30 14:41:05 +02:00
Thomas Waldmann
ee33c22443
fix CVE timeline 2023-08-30 14:39:41 +02:00
TW
c8b9d72bdb
Merge pull request from GHSA-8fjr-hghr-4m99
Release 1.2.5 incl. archives TAM security fix
2023-08-30 14:18:17 +02:00
Thomas Waldmann
509a5fd71c
build_usage / build_man 2023-08-30 03:49:47 +02:00
Thomas Waldmann
ed1ab84cc7
update CHANGES 2023-08-30 03:47:35 +02:00
Thomas Waldmann
9e63abb679
document vulnerability, repo upgrade procedure 2023-08-30 02:52:34 +02:00
Thomas Waldmann
5e0632a3d0
add tests for archive TAMs, upgrade 2023-08-30 02:52:32 +02:00
Thomas Waldmann
d78ed697ae
rebuild_refcounts: keep archive ID, if possible
rebuild_refcounts verifies and recreates the TAM.
Now it re-uses the salt, so that the archive ID does not change
just because of a new salt if the archive has still the same data.
2023-08-30 02:52:27 +02:00
Thomas Waldmann
85b173d3d1
TAM msgs: be more specific: archives vs. manifest 2023-08-30 02:52:23 +02:00
Thomas Waldmann
7d0d11b979
upgrade: allow enable/disable manifest TAM for unencrypted repos
Recent borg wrote TAM authenticated **archives**
even for unencrypted repos (encryption "none"),
so we also do that for the manifest.

It's kind of fake as there is no secret key involved then,
but it simplifies the code.
2023-08-30 02:52:21 +02:00
Thomas Waldmann
19a7809fe8
upgrade --archives-tam: make sure all archives are TAM authenticated
borg check (rebuild_manifest and rebuild_refcounts) drops archives without TAM,
so let's just always add the TAM.

for unencrypted repos (encryption=none) the TAM is insecure,
but without encryption and authentication, there is no expectation
of security anyway.
2023-08-30 02:52:17 +02:00
Thomas Waldmann
75518d945c
list: support {tam} placeholder. check archive TAM.
list: shows either "verified" or "none", depending on
whether a TAM auth tag could be verified or was
missing (old archives from borg < 1.0.9).

when loading an archive, we now try to verify the archive
TAM, but we do not require it. people might still have
old archives in their repos and we want to be able to
list such repos without fatal exceptions.
2023-08-30 02:52:15 +02:00
Thomas Waldmann
155d8ee23b
cache sync: check archive TAM 2023-08-30 02:52:13 +02:00
Thomas Waldmann
7da8738513
check: rebuild_refcounts verify and recreate TAM
This part of the archive checker recreates the Archive
items (always, just in case some missing chunks needed
repairing).

When loading the Archive item, we now verify the TAM.
When saving the (potentially modified) Archive item,
we now (re-)generate the TAM.

Archives without a valid TAM are dropped rather than TAM-authenticated
when saving them. There shouldn't be any archives without a valid TAM:

- borg writes an archive TAM since long (1.0.9)
- users are expected to TAM-authenticate archives created
  by older borg when upgrading to borg 1.2.5.

Also:

Archive.set_meta: TAM-authenticate new archive

This is also used by Archive.rename and .recreate.
2023-08-30 02:51:52 +02:00
Thomas Waldmann
1fd94bd38f
check: rebuild_manifest must verify archive TAM 2023-08-29 21:59:04 +02:00
TW
8ae06199ef
Merge pull request #7784 from ThomasWaldmann/vagrant-updates-1.2
Vagrant updates (1.2-maint)
2023-08-29 20:49:13 +02:00
Thomas Waldmann
de51c67085
vagrant: fix netbsd pkgsrc
9.0_current stopped working, let's use 9.3.
2023-08-29 19:45:39 +02:00
Thomas Waldmann
055fb76bd8
vagrant / binary build: use python 3.9.18 2023-08-29 19:44:30 +02:00
TW
9133cf5f36
Merge pull request #7783 from ThomasWaldmann/rel-prep-1.2
build_man / build_usage
2023-08-29 18:50:04 +02:00
Thomas Waldmann
62ca4f666b
build_man / build_usage 2023-08-29 18:48:41 +02:00
TW
a034febe55
Merge pull request #7782 from ThomasWaldmann/update-changes-1.2
update CHANGES
2023-08-29 18:34:10 +02:00
Thomas Waldmann
7f07f81e39
update CHANGES 2023-08-29 18:24:56 +02:00
TW
186b3d1319
Merge pull request #7781 from ThomasWaldmann/cy029-1.2
require cython <3 for now
2023-08-29 16:15:33 +02:00
Thomas Waldmann
f8673a379c
require cython <3 for now
also in requirements.d/development.txt.
2023-08-29 16:09:08 +02:00
TW
844ca6b61e
Merge pull request #7772 from ThomasWaldmann/cython029-1.2
use cython 0.29 (1.2-maint)
2023-08-26 14:44:54 +02:00
Thomas Waldmann
37a813f7f8
stay on latest Cython 0.29 (0.29.36) for borg 1.2.x
master branch tests Cython 3 now.
1.2-maint branch may or may not upgrade to build with Cython 3 later.
2023-08-26 14:18:23 +02:00
TW
0493695b6c
Merge pull request #7769 from ThomasWaldmann/docs-fix-rcreate-1.2
docs: fix borg init command in environment.rst.inc
2023-08-24 12:04:38 +02:00
Thomas Waldmann
276781081b
docs: fix borg init command in environment.rst.inc
It mentioned borg rcreate due to a bad backport from master.
Must be borg init in borg 1.2.x.
2023-08-24 11:49:17 +02:00
TW
f6e20fb07c
Merge pull request #7751 from ThomasWaldmann/fix-unreadable-parent-of-root-fixup-1.2
remove unused "flags_root" open flags
2023-07-30 01:18:40 +02:00
Thomas Waldmann
de7e7e2a95
remove unused "flags_root" open flags 2023-07-29 23:42:36 +02:00
TW
ef52d13608
Merge pull request #7749 from ThomasWaldmann/fix-unreadable-parent-of-root-1.2
do not try to read parent dir of recursion root (1.2-maint)
2023-07-29 23:38:28 +02:00
Thomas Waldmann
ed7a410084
create: do not try to read parent dir of recursion root, fixes #7746 2023-07-29 23:04:44 +02:00
Thomas Waldmann
453d35fa47
add a test for unreadable parent dir, see #7746 2023-07-29 22:48:13 +02:00
TW
a3f4adf899
Merge pull request #7732 from ThomasWaldmann/update-docs-1.2
update docs 1.2
2023-07-23 15:08:31 +02:00
Thomas Waldmann
dedbd38844
1.1.x upgrade notes: more precise borg upgrade instructions, fixes #3396 2023-07-22 18:08:35 +02:00
Thomas Waldmann
b419b6ad03
build_man / build_usage 2023-07-20 18:57:22 +02:00
Thomas Waldmann
65a2fe8502
update CHANGES 2023-07-20 18:54:55 +02:00
TW
aa913cda83
Merge pull request #7731 from ThomasWaldmann/authenticated_no_key-fix-1.2
bugfix: skip TAM check with BORG_WORKAROUNDS=authenticated_no_key
2023-07-20 18:34:05 +02:00
Thomas Waldmann
104cc196fc
bugfix: skip TAM check with BORG_WORKAROUNDS=authenticated_no_key
This is an emergency workaround for authenticated repos
if the user has lost the borg key.

We can't compute the TAM key without the borg key, so just
skip all the TAM stuff.
2023-07-20 18:02:58 +02:00
TW
b067aeadef
Merge pull request #7723 from PhrozenByte/docs-check-rewrite-1.2
Backport: Docs: Rewrite `borg check` docs
2023-07-11 00:49:20 +02:00
Daniel Rudolf
c6a6161c60
Docs: Rewrite borg check docs
This commit is a backport of the following commits to the `1.2-maint` branch:

    commit 9edbf4e931
    Author: Daniel Rudolf

        Docs: Rewrite `borg check` docs

    commit 9d59146de4
    Author: Daniel Rudolf

        Docs: Remove technical description from `borg check` docs

    commit a661da13ee
    Author: Daniel Rudolf

        Docs: Improve explanation of `borg check --max-duration`'s side effects

    commit 2647673dc8
    Author: Daniel Rudolf

        Docs: Improve `borg check` docs
2023-07-10 23:03:47 +02:00
TW
4721a35863
Merge pull request #7702 from ThomasWaldmann/authenticated-no-key-1.2
BORG_WORKAROUNDS=authenticated_no_key, fixes #7700
2023-07-07 01:02:41 +02:00
TW
c79e970b78
Merge pull request #7717 from FelixSchwarz/borg12-py312
add `utcnow()` helper function to avoid `datetime.utcnow()`
2023-07-06 23:45:58 +02:00
Felix Schwarz
98f547f278 add utcnow() helper function to avoid datetime.utcnow()
`datetime.utcnow()` is deprecated since Python 3.12. This causes additional lines of
output and thus breaks the test suite. I was not sure if all borg internals are
ready to deal with timezone-aware datetime instances so tried to keep the changes
minimal.
2023-07-06 21:44:09 +02:00
TW
e5cc2e16e3
Merge pull request #7709 from TimWolla/backport-doc-keep-within
Backport: Improve the documentation for `--keep-within`
2023-07-04 14:11:55 +02:00
Tim Düsterhus
b58c6ce3b1 Improve the documentation for --keep-within
Move the explanation below the general explanation of the `--keep-*` option
behavior rephrase the last sentence to make it clear that it works like the
other options that were explained in the previous paragraph.

Resolves #7687

(cherry picked from commit a6f452a050)
2023-07-04 09:16:39 +02:00
TW
dfa04eece6
Merge pull request #7704 from ThomasWaldmann/remove-coala-1.2
remove coala (outdated)
2023-07-03 19:11:27 +02:00
Thomas Waldmann
ae7fd9b101
remove coala (outdated)
last coala release (0.11.0) is now over 6y old.

when using pip install coala, a ton of stuff gets installed (expected)
and a part of that downgrades some stuff we use to outdated, incompatible
versions.

when trying to run coala with python 3.11, it just crashes because the
last release was made for py35/py36 (as seen in their setup.py).

a lot of PRs and tickets pile up at the coala project on github,
but noone is maintaining it.
2023-07-03 18:40:52 +02:00
Thomas Waldmann
8ac7178ab9
BORG_WORKAROUNDS=authenticated_no_key to extract from authenticated repos without key, fixes #7700 2023-07-03 15:38:21 +02:00
TW
6b0dc53f11
Merge pull request #7693 from ThomasWaldmann/docs-1.2
docs updates (1.2-maint)
2023-07-01 15:07:04 +02:00
Thomas Waldmann
d01852c100
update CHANGES 2023-07-01 14:49:28 +02:00
Thomas Waldmann
6d9e8e5483
docs: move upgrade notes to own section, see #7546 2023-07-01 14:43:44 +02:00
Thomas Waldmann
55af769b71
mount -olocal: how to show mount in finder's sidebar, fixes #5321 2023-07-01 14:43:35 +02:00
TW
98cea7516b
Merge pull request #7691 from ThomasWaldmann/macfuse-volname-1.2
mount: make up volname if not given (macOS), fixes #7690
2023-07-01 14:41:14 +02:00
TW
230065f52b
Merge pull request #7694 from ThomasWaldmann/fix-4110-1.2
extract: fix false warning about pattern never matching, fixes #4110
2023-07-01 14:40:24 +02:00
Thomas Waldmann
f5f5311e2b
extract: fix false warning about pattern never matching, fixes #4110 2023-07-01 02:05:58 +02:00
Thomas Waldmann
7c82969761
mount: make up volname if not given (macOS), fixes #7690
macFUSE supports a volname mount option to give what
finder displays on desktop / in directory list.

if the user did not specify it, we make something up,
because otherwise it would be "macFUSE Volume 0 (Python)".
2023-06-30 22:10:32 +02:00
TW
de53164931
Merge pull request #7689 from ThomasWaldmann/diff-surrogate-escape-1.2
diff: remove surrogates before output, fixes #7535
2023-06-30 17:52:26 +02:00
Thomas Waldmann
c587e88830
diff: remove surrogates before output, fixes #7535 2023-06-30 17:10:56 +02:00
TW
a0bc7b9434
Merge pull request #7683 from ThomasWaldmann/fix-repo-reopen-1.2
tests: fix repo reopen 1.2 (backport from master)
2023-06-28 00:24:41 +02:00
Thomas Waldmann
fbf287f825
RepositoryTestCaseBase: remove need for "with" after reopen()
setUp enters the context manager, so let's .reopen() leave it.
then create a fresh Repository instance in self.repository and
enter the context manager again. tearDown then will leave that.
2023-06-27 22:43:00 +02:00
Thomas Waldmann
0ac07b7a3f
RepositoryTestCaseBase: call __exit__
As we call __enter__ in setUp,
let's call __exit__ in tearDown.
2023-06-27 22:34:11 +02:00
Thomas Waldmann
83b2b5cb6c
fix RepositoryTestCaseBase.reopen method
"if self.repository" did not work as expected:
- Repository has a __len__ method, so the boolean evaluation was calling that.
- self.repository is also not set to None anywhere.
2023-06-27 22:33:57 +02:00
TW
0947e9fb22
Merge pull request #7680 from ThomasWaldmann/improve-key-sanity-check-1.2
keyfile: improve key sanity check, fixes #7561
2023-06-27 22:32:30 +02:00
TW
edc9a0843a
Merge pull request #7682 from ThomasWaldmann/test-cosmetics-1.2
tests: avoid long ids in pytest output
2023-06-27 22:30:16 +02:00
Thomas Waldmann
81588989fe
tests: avoid long ids in pytest output
sometimes the automatically computed IDs are just too long,
so rather give IDs directly or avoid them otherwise.
2023-06-27 21:50:51 +02:00
Thomas Waldmann
a5801b9971
keyfile: improve key sanity check, fixes #7561
check key file structure, make sure the binary key
is not way too short (or zero) length.

if key file looks strange, emit warnings.
2023-06-27 21:39:43 +02:00
TW
1da5beacb3
Merge pull request #7678 from ThomasWaldmann/fix-crash-files-cache-size-1.2
--files-cache=size: fix crash, fixes #7658
2023-06-27 21:36:49 +02:00
TW
c7d5fff200
Merge pull request #7679 from ThomasWaldmann/pypi-no-upload-signature-1.2
do not upload gpg signature to pypi, fixes #7649
2023-06-27 21:36:05 +02:00
TW
f6805f3850
Merge pull request #7681 from ThomasWaldmann/fix-list-pattern-docs-1.2
list: fix --pattern examples, fixes #7611
2023-06-27 21:35:41 +02:00
Thomas Waldmann
893530fdac
list: fix --pattern examples, fixes #7611
- pattern needs to start with + - !
- first match wins
- the default is to list everything, thus a 2nd pattern
  is needed to exclude everything not matched by 1st pattern.
2023-06-27 21:00:05 +02:00
Thomas Waldmann
de643c211b
do not upload gpg signature to pypi, fixes #7649 2023-06-27 18:45:41 +02:00
Thomas Waldmann
45aa88dbec
--files-cache=size: fix crash, fixes #7658 2023-06-27 18:40:48 +02:00
TW
f73eb5c6c2
Merge pull request #7644 from eoli3n/1.2-maint
Backport of improvements in patterns help
2023-06-11 17:10:47 +02:00
eoli3n
91547e9844 improve patterns help: declarative includes sample 2023-06-11 11:14:24 +02:00
eoli3n
099f23032e improve patterns help: added pattern prefixes 2023-06-11 11:12:49 +02:00
eoli3n
3e8b0da8be improve patterns help: define a pattern style 2023-06-11 11:09:41 +02:00
eoli3n
952d778bc1 Improve patterns help 2023-06-11 11:08:31 +02:00
TW
c5d579be39
Merge pull request #7628 from ThomasWaldmann/fix-chunksize-distrib-test-1.2
fix test_buzhash_chunksize_distribution
2023-06-06 11:25:11 +02:00
Thomas Waldmann
796d8e2d13
fix test_buzhash_chunksize_distribution
the last chunk can be smaller than 2**min_exp.
2023-06-06 10:46:53 +02:00
TW
742d0df294
Merge pull request #7613 from ThomasWaldmann/1.2-clear-empty-dirs-typo
fix typo
2023-05-29 15:22:15 +02:00
Thomas Waldmann
e711df20e2
fix typo 2023-05-29 14:56:47 +02:00
TW
0bfbd8d8fc
Merge pull request #7595 from nain-F49FF806/1.2-clear-empty-dirs
clear empty directories at end of compact process, fixes #6823
2023-05-27 11:04:13 +02:00
nain
aac55b1ad2 Add function to clear empty directories at end of compact process.
Compact moves data to new segments, and then removes the old segments.
When enough segments are moved, directories holding the now cleared segments
may thus become empty.

With this commit any empty directories are cleared after segments compacting.
Fixes #6823
2023-05-27 04:41:50 -04:00
TW
912dc81a2d
Merge pull request #7606 from nain-F49FF806/data-scandir-unify
unify scanning and listing of segment dirs, files and apply good practices
2023-05-27 10:31:36 +02:00
nain
edb5e749f5 Move value bounds of segment (index) into constants module and use them instead 2023-05-27 02:33:53 -04:00
nain
3f2da1bba9 Simplify generator functions and make get_segment_dirs inputs consistent with get_segment_files
+ get_segment_dirs explicitly takes data_dir for clarity and future flexibility
+ removed multiple cases for one general purpose generator expression
2023-05-26 02:42:13 -04:00
nain
c9f35a16e9 unify scanning and listing of segment dirs / segment files and apply good practices
+ os.scandir instead of os.listdir
  Improved speed and added flexibility with attributes (name,path,is_dir(),is_file())
+ use is_dir / is_file to make sure  we're reading only dirs / files respectively
+ Filtering to particular start, end index range built in
+ Have seperate generators to avoid unnecessary less_than / greater_than checking for every dir/file when not required

Resolves #7597
2023-05-25 11:55:16 -04:00
TW
53bedfb63b
Merge pull request #7589 from ThomasWaldmann/chunker-params-1.2
relax chunker params validation, tests (1.2-maint)
2023-05-19 19:19:04 +02:00
Thomas Waldmann
5cc4e3af6a
tests: only warn about "invalid" chunker params, fixes #7590
we previously allowed some weird chunker params, so we just warn
about them instead of rejecting them.

e.g. one could use super small chunk sizes. borg can do that,
but you'll end up with a huge amount of chunks and very large
hash tables (RAM usage) to manage them.

also, one can violate the min <= mask <= max chunker param condition
as in #7590 and it will somehow work, but will likely not dedup as good
as when not violating that.

borg2 **will** reject such strange chunker params, see #7586 for
some ideas how to "fix" your repos.
2023-05-19 18:05:26 +02:00
Thomas Waldmann
965e8a957f
tests: check buzhash chunksize distribution, see #7586 2023-05-19 18:05:20 +02:00
TW
ce50796b1f
Merge pull request #7568 from elandorr/1.2-maint
incl./excl. options, path-from-stdin exclusiveness
2023-05-15 21:48:29 +02:00
elandorr
8e41831f4c
Update archiver.py 2023-05-15 11:39:10 +00:00
elandorr
dedd58969c
mention paths-from-stdin's exclusiveness 2023-05-14 20:28:52 +00:00
elandorr
4047076e70
include/exclude options 2023-05-14 20:21:41 +00:00
TW
823f0fb9e0
Merge pull request #7564 from elandorr/1.2-maint
doc markup fixes
2023-05-14 18:58:42 +02:00
elandorr
6b38207aeb
markup fix + note about MAX_DATA_SIZE 2023-05-14 07:57:12 +00:00
elandorr
f1bd5da8c9
escape 2023-05-14 07:50:27 +00:00
TW
dbc142a32a
Merge pull request #7549 from sashadev-sky/fix-progress-msg-width-1.2
ProgressIndicatorPercent: fix space computation for wide chars, fixes #3027
2023-05-08 19:46:17 +02:00
Thomas Waldmann
21a9458848 ProgressIndicatorPercent: fix space computation for wide chars, fixes #3027
needs to use swidth() in case there are wide chars (like CJK)
in the left part of the msg (e.g. an archive name).

(cherry picked from commit 3dd14f4855)
2023-05-08 12:44:05 -04:00
TW
2181103c3f
Merge pull request #7550 from sashadev-sky/fix-argparse-errors-1.2
Fix argparse error messages
2023-05-08 15:39:11 +02:00
TW
1d4a7803dd
Merge pull request #7547 from sashadev-sky/issue-7476-1.2
Added pre-commit for linting purposes. Fixes #7476
2023-05-08 15:28:51 +02:00
Thomas Waldmann
3e9668ecbc fix CompressionSpec validator and tests
(cherry picked from commit 0b40e038ad)
2023-05-07 21:03:56 -04:00
Thomas Waldmann
de199c23ab fix SortBySpec validator
(cherry picked from commit 05bf29f504)
2023-05-07 21:01:10 -04:00
Thomas Waldmann
7707ea538c fix FilesCacheMode validator
(cherry picked from commit 0f923c8c4a)
2023-05-07 21:00:20 -04:00
Thomas Waldmann
067e36bccc fix ChunkerParams validator and tests
(cherry picked from commit 6d38530ff1)
2023-05-07 20:59:17 -04:00
Sasha Boginsky
5b5e9282e3 Remove black formatting from cherry-pick 2023-05-07 18:52:42 -04:00
TW
d1ddd57eaf
Merge pull request #7548 from sashadev-sky/option-x-help-1.2
improve --one-file-system help /docs
2023-05-06 20:35:59 +02:00
Thomas Waldmann
49b58bfbab --one-file-system: add macOS apfs notes, fixes #4876
(cherry picked from commit b7adee65bd)
2023-05-06 03:00:08 -04:00
Thomas Waldmann
8cbaaf6113 improve --one-file-system help string, fixes #5618
(cherry picked from commit 6ca1ed9716)
2023-05-06 02:59:14 -04:00
Michael Deyaso
172b03ef4d Changed black rev in pre-commit yaml. Fixes #7476
(cherry picked from commit 20a4c960ec)
2023-05-05 23:51:14 -04:00
Michael Deyaso
9e7b20660b Installed and configured pre-commit to lint and format code. Fixes #7476
(cherry picked from commit 98d1c65b40)
2023-05-05 23:51:09 -04:00
TW
0067077f7a
Merge pull request #7480 from ThomasWaldmann/update-codeql-1.2
codeql action: upgrade to v2 (1.2-maint)
2023-03-27 20:07:28 +02:00
Thomas Waldmann
54d8636a2d
codeql action: upgrade to v2 2023-03-27 16:28:19 +02:00
TW
7d649faea1
Merge pull request #7466 from ThomasWaldmann/rel124
release 1.2.4
2023-03-23 23:24:34 +01:00
Thomas Waldmann
35d76f9a64
build_man 2023-03-23 22:11:46 +01:00
Thomas Waldmann
0d14f3b36f
build_usage 2023-03-23 22:11:45 +01:00
Thomas Waldmann
b81b1abdeb
it's 2023! 2023-03-23 22:11:44 +01:00
Thomas Waldmann
73ee704afa
security: 1.1.x is EOL, please upgrade to 1.2.x.
That means I won't make new 1.1.x releases.

In case there would be a major security or other issue,

I might still make a fix commit to the 1.1-maint branch,
where dist package maintainers or other interested
parties could find it.
2023-03-23 22:11:42 +01:00
Thomas Waldmann
391e05a6c6
change log: set release date, remove outdated stuff 2023-03-23 22:11:41 +01:00
Thomas Waldmann
af16e19933
update CHANGES 2023-03-23 22:10:22 +01:00
TW
05bfed50c3
Merge pull request #7471 from Michael-Girma/bug/issue-7470
Resolved mode bug and added sleep clause for darwin systems. Fixes #7470
2023-03-23 21:29:56 +01:00
Michael Deyaso
ea08161fe8 Resolved mode bug and added sleep clause for darwin systems. Fixes #7470 2023-03-23 21:39:55 +03:00
TW
de2f894bcc
Merge pull request #7464 from ThomasWaldmann/docs-borg-key-file-1.2
BORG_KEY_FILE: clarify docs, fixes #7444
2023-03-22 22:33:42 +01:00
TW
4c230a52e6
Merge pull request #7465 from dotdoom/1.2-maint-ignorezeros
Add --ignore-zeros flag to import-tar
2023-03-22 22:31:37 +01:00
Artem Sheremet
3b7c0605af Add --ignore-zeros flag to import-tar
Fixes #7432.

Backport of #7460.
2023-03-22 18:59:58 +01:00
Thomas Waldmann
e6b8e03e19
BORG_KEY_FILE: clarify docs, fixes #7444 2023-03-22 18:52:58 +01:00
TW
671c66361d
Merge pull request #7456 from ThomasWaldmann/update-changes-1.2
update CHANGES (1.2-maint)
2023-03-20 23:00:03 +01:00
Thomas Waldmann
3cffbbd3af
update CHANGES 2023-03-20 22:10:49 +01:00
TW
2ab678ca93
Merge pull request #7414 from Michael-Girma/backport/1.2-maint-7248
Show ctime and mtime on borg diff, fixes #7248 (Backport of #7335)
2023-03-20 14:14:30 +01:00
Michael Deyaso
19bb27741d bugfix: Fixes path related bug seen when addressing deferred items. 2023-03-20 07:28:59 +03:00
TW
62080a5737
Merge pull request #7441 from ThomasWaldmann/vagrant-updates2-1.2
Vagrant updates2 (1.2-maint)
2023-03-13 09:56:20 +01:00
Thomas Waldmann
cfd6f035ff
vagrant: use pythons working on freebsd
pyenv could not install py38, thus using the fbsd system py38.

removed py38 from pyenv completely.
2023-03-13 04:05:24 +01:00
Thomas Waldmann
96dff54655
vagrant: freebsd: kldload fusefs 2023-03-13 00:56:09 +01:00
TW
6354e4ada4
Merge pull request #7435 from ThomasWaldmann/freebsd-xattr-fixes-1.2
xattrs: fix namespace processing on FreeBSD, fixes #6997
2023-03-13 00:54:13 +01:00
Thomas Waldmann
bd299f1ff3
xattrs: fix namespace processing on FreeBSD, fixes #6997 2023-03-12 23:35:52 +01:00
TW
f623742709
Merge pull request #7439 from ThomasWaldmann/vagrant-updates-1.2
Vagrant updates (1.2-maint)
2023-03-12 23:34:22 +01:00
Thomas Waldmann
e3f887414b
vagrant: pyenv: also install python 3.11.1 for testing 2023-03-12 23:07:11 +01:00
Thomas Waldmann
96a8a98d97
vagrant: pyenv: use python 3.10.1, 3.10.0 build is broken on freebsd 2023-03-12 23:06:57 +01:00
Thomas Waldmann
fc0c594ff6
vagrant: local freebsd 12.1 box -> generic/freebsd13 box
Building in the 12.1 box stopped working, thus replaced it
with the same box as we use in master.
2023-03-12 23:06:50 +01:00
TW
0a61c7b767
Merge pull request #7428 from ThomasWaldmann/update-changes-1.2
update CHANGES (1.2-maint)
2023-03-11 19:17:52 +01:00
TW
c9190fe867
Merge pull request #7434 from ThomasWaldmann/remove-os-directory-1.2
docs: remove OS directory from install docs
2023-03-11 19:15:58 +01:00
Thomas Waldmann
f2825a96d2
docs: remove OS directory from install docs
- links did not work that way
- each OS has its own headline and can be discovered easily without
  that directory
2023-03-11 18:52:58 +01:00
TW
019a999364
Merge pull request #7429 from abebeos/1.2-maint
docs: installation: add link to OS dependencies
2023-03-11 01:16:36 +01:00
abebeos
9e464edb7c docs: installation: add link to OS dependencies 2023-03-10 22:39:30 +00:00
Thomas Waldmann
cd35e22d06
update CHANGES 2023-03-10 22:46:10 +01:00
TW
134583a7bf
Merge pull request #7412 from ThomasWaldmann/hardlink-master-bug-1.2
set .hardlink_master for hardlinkable items, fixes #7175
2023-03-10 19:49:12 +01:00
TW
7643f39901
Merge pull request #7419 from ThomasWaldmann/msgpack105-1.2
allow msgpack 1.0.5 also
2023-03-09 22:20:22 +01:00
Thomas Waldmann
2842463f21
allow msgpack 1.0.5 also 2023-03-09 17:58:57 +01:00
Michael Deyaso
6c042d7192 Show ctime and mtime on borg diff. Fixes #7248
---

- Added assert_line_exists helper in BaseTestCase
- JSON strings in diff output are now sorted alphabetically
- Modified diff test cases to confirm to new output format
- Added a test case to test ctime and mtime inclusion
- Mode, ctime & mtime are now only displayed on diff if --content-only flag is used
2023-03-07 10:54:29 +03:00
Thomas Waldmann
843c12cbea
set .hardlink_master for hardlinkable items, fixes #7175
before, we only set this for regular files, but we better also set this
for block/char devices and fifos also, so we have it for all item types
borg 1.x considers "hardlinkable".

this is important for sequential processing of items in an archive:
if we encounter a hardlinkable item and .source is not set (that
would make it a hardlink slave), it could be a hardlink master or not:

- hardlink_master == False  # this item is not part of a hl group
- hardlink_master == True  # this item is a hardlink master

This will also be important when items are converted to borg2.
2023-03-05 19:20:22 +01:00
TW
d44cf3482e
Merge pull request #7402 from ThomasWaldmann/unicode-filenames-docs-1.2
update FAQ about locale/unicode issues, fixes #6999
2023-03-05 17:48:55 +01:00
TW
d5e4f3f8cf
Merge pull request #7404 from ThomasWaldmann/mount-docs-1.2
docs: improve mount options rendering, fixes #7359
2023-03-05 17:48:01 +01:00
TW
ca1fea37a6
Merge pull request #7410 from snsmac/backport/autocompressortests
"auto" compressor tests: don't assume a specific size (backport to 1.2-maint)
2023-03-05 17:22:45 +01:00
TW
831c9d0c40
Merge pull request #7408 from snsmac/backport/hashindex
hashindex bugfix and refactoring (backport to 1.2-maint)
2023-03-05 15:29:50 +01:00
Thomas Waldmann
8cf313900a "auto" compressor tests: do not assume zlib is better than lz4, fixes #7363
while that might be true for many cases, we can not assume it is
always true, as proven by the failing test on S390.
2023-03-05 11:05:47 +01:00
Thomas Waldmann
a7ce1db529 add num_entries assertion 2023-03-05 11:01:44 +01:00
Thomas Waldmann
f1d68fe4b3 hashindex: simplify size_idx function
Thanks to @jdchristensen for the code.
2023-03-05 11:01:44 +01:00
Thomas Waldmann
d57fafadad Simplify full HT scan assertion 2023-03-05 11:01:44 +01:00
Thomas Waldmann
ec32413b5e hashindex: always have at least 1 empty bucket
avoid rounding / integer conversion issues bringing this down to 0.
2023-03-05 11:01:44 +01:00
Thomas Waldmann
241eaec413 implement ht idx wrap around less strangely, add comment 2023-03-05 11:01:44 +01:00
Thomas Waldmann
51189e1383 if HT is full with entries and tombstones: give up/fail early 2023-03-05 11:01:44 +01:00
Thomas Waldmann
32519617b6 more comments for hashindex_lookup 2023-03-05 11:01:44 +01:00
Thomas Waldmann
3a44894aca bugfix: do not resize hashindex with wrong num_empty
otherwise we would lose the decrement operation on num_empty.
2023-03-05 11:01:44 +01:00
Thomas Waldmann
ecf5ad43ed hashindex: simplify assert 2023-03-05 11:01:44 +01:00
Thomas Waldmann
b2117bfd4f _hashindex: easier to understand code, dubious loops removed, asserts
hashindex_index returns the perfect hashtable index, but does not
check what's in the bucket there, so we had these loops afterwards
to search for an empty or deleted bucket.

problem: if the HT were completely filled with no empty and no deleted
buckets, that loop would never end. due to our HT resizing, it can
never happen, but still not pretty.

when using hashindex_lookup (as also used some lines above), the code
is easier to understand, because (after we resized the HT), we freshly
create the same situation as after the first call of that function:
- return value < 0, because we (still) can not find the key
- start_idx will point to an empty bucket

Thus, we do not need the problematic loops we had there.

Modified the checks to make sure we really have an empty or deleted
bucket before overwriting it with data.

Added some additional asserts to make sure the code behaves.
2023-03-05 11:01:44 +01:00
TW
cef5b446ac
Merge pull request #7411 from ThomasWaldmann/debug-id-hash-command-1.2
debug id-hash: implement file content id-hash computation, see #7406
2023-03-04 22:33:20 +01:00
TW
42fb72c022
Merge pull request #7409 from snsmac/backport/hashtablestress
Add hashtable stress test (backport to 1.2-maint)
2023-03-04 22:14:47 +01:00
Thomas Waldmann
eb8ccd22d4
debug id-hash: implement file content id-hash computation, see #7406 2023-03-04 21:47:14 +01:00
snsmac
c339d2e9e2 Update hashtable stress test to work with borg < 2 where NSIndex has 2-tuples as values 2023-03-04 21:41:31 +01:00
TW
086a4237ca
Merge pull request #7407 from ThomasWaldmann/fix-debug-cmds-1.2
fix borg debug get-obj/put-obj (1.2-maint)
2023-03-04 21:21:09 +01:00
Thomas Waldmann
356ba3cc79
debug put-obj: id must be given on commandline, fixes #7290
this is an incompatible change:

before:
borg debug put-obj :: path1 path2 ...
(and borg computed all IDs automatically) (*)

after:
borg debug put-obj :: id path
(id must be given)

(*) the code just using sha256(data) was outdated and incorrect anyway.

also: fix error handling in get-obj.
2023-03-04 20:25:45 +01:00
Thomas Waldmann
8fe980c94e "auto" compressor tests: don't assume a specific size, fixes #7363
The tests assumed a specific compressed results size, which
is bad, because it might vary depending on the zlib implementation.

Now the "auto" compressor tests just check if it is the same size
as when unconditionally using the zlib compressor.
2023-03-04 20:19:08 +01:00
Thomas Waldmann
b7d70f2064 add comment about how to provoke more collisions 2023-03-04 20:09:58 +01:00
Thomas Waldmann
b33fea7718 add hashtable stress tests
Using NSIndex (repo index) HashIndex, but they all are very similar.
2023-03-04 20:09:50 +01:00
Thomas Waldmann
d739f29657
docs: improve mount options rendering, fixes #7359 2023-03-04 17:08:15 +01:00
Thomas Waldmann
e8573193e4
update FAQ about locale/unicode issues, fixes #6999 2023-03-04 16:32:48 +01:00
TW
6e8c58c125
Merge pull request #7395 from helmutg/reproducible-docs-1.2
Make timestamps in manual pages reproducible
2023-03-01 09:20:20 +01:00
Vagrant Cascadian
a5c9231e88 Make timestamps in manual pages reproducible
Refer to https://reproducible-builds.org/docs/source-date-epoch/ for
documentation on SOURCE_DATE_EPOCH.

Bug-Debian: https://bugs.debian.org/1029807
Signed-off-by: Helmut Grohne <helmut@subdivi.de>

(cherry picked from commit 98352cf667)
2023-02-28 13:38:45 +01:00
TW
a7a80ffe90
Merge pull request #7382 from ThomasWaldmann/rtd-config-1.2
add .readthedocs.yaml
2023-02-25 18:51:51 +01:00
Thomas Waldmann
fd6e76e6a4
add .readthedocs.yaml
same as in master branch
2023-02-25 18:42:49 +01:00
TW
836bd0339d
Merge pull request #7361 from voegelas/1.2-maint-fedora
Installation: Update Fedora in distribution list
2023-02-17 18:58:50 +01:00
Andreas Vögele
007b73ac67 Installation: Update Fedora in distribution list
Fixes #7357
2023-02-17 14:45:17 +01:00
TW
1dce595d8f
Merge pull request #7338 from ThomasWaldmann/timestamp-comparisons-1.2
relaxed timestamp comparisons (1.2)
2023-02-11 00:19:08 +01:00
TW
e41466dbb3
Merge pull request #7337 from ThomasWaldmann/fix-recreate-rechunkify-1.2
recreate without --chunker-params shall not rechunk (1.2)
2023-02-10 18:37:11 +01:00
Thomas Waldmann
6bf818a2a1
tests: use same_ts_ns for all timestamp comparisons 2023-02-10 18:36:10 +01:00
Thomas Waldmann
b3da30e6d4
same_ts_ns: add a timestamp comparison function
also: HAVE_UTIMES means 1us resolution (*)

(*) at least as far as the utimes() call is concerned.

Whether there are other limitations has to be seen.
2023-02-10 17:58:46 +01:00
Thomas Waldmann
cd23d810d3
recreate: --chunker-params must default to None, fixes #7337
before the fix, it triggered rechunking if an archive
was created with non-default chunker params.

but it only should rechunk if borg recreate is invoked with explicitly giving --chunker-params=....
2023-02-10 14:12:08 +01:00
Thomas Waldmann
dbe1c98cb3
tests: recreate without --chunker-params shall not rechunk 2023-02-10 14:01:04 +01:00
TW
5ccfaa7a45
Merge pull request #7299 from ThomasWaldmann/test-dir-mtime-1.2
add test for extracted directory mtime
2023-01-28 00:59:59 +01:00
Thomas Waldmann
b472768b57
more directory timestamp tests 2023-01-24 15:58:30 +01:00
Thomas Waldmann
2291ab755d
add test for extracted directory mtime 2023-01-24 15:22:01 +01:00
TW
48713ebff6
Merge pull request #7265 from ThomasWaldmann/fix-locking-1.2
Fix locking (1.2-maint)
2023-01-19 18:15:28 +01:00
TW
dbba9cbc53
Merge pull request #7268 from ThomasWaldmann/recreate-with-target-1.2
recreate: when --target is given, do not detect "nothing to do"
2023-01-16 17:22:50 +01:00
Thomas Waldmann
c43140686b
recreate: when --target is given, do not detect "nothing to do"
use case:

borg recreate -a src --target dst can be used to make a copy
of an archive inside the same repository, see #7254.
2023-01-14 19:28:57 +01:00
Thomas Waldmann
16eb38e20a
fix host, pid, tid order
using "differenthost" (== not the current hostname) makes
the process_alive check always return True (to play safe,
because in can not check for processes on other hosts).
2023-01-13 22:36:50 +01:00
Thomas Waldmann
91104674e7
use os.replace not os.rename 2023-01-13 22:32:11 +01:00
Thomas Waldmann
804867d3ad
bugfix: thread id must be parsed as hex from lock file name 2023-01-13 22:30:28 +01:00
TW
1dcaa74788
Merge pull request #7252 from puetzk/test_size_on_disk_accurate-flush-1.2
[1.2-maint] Fix test_size_on_disk_accurate for large st_blksize, fixes #7250
2023-01-12 20:52:48 +01:00
Kevin Puetz
fe3775cf80 Fix test_size_on_disk_accurate for large st_blksize, fixes #7250
python's io.BufferedWriter sizes its buffer based on st_blksize.
If the write fits in this buffer, then it's possible the data from
idx.write() has not been flushed through to ,the underlying filesystem,
and getsize(fileno()) sees a too-short (or even empty) file.

Also, getsize is only documented as accepting path-like objects;
passing a fileno seems to work only because the implementation
blindly forwards everything through to os.stat without checking.

Passing unopened_tempfile avoids all three problems
- on windows, it doesn't rely on re-opening NamedTemporaryFile
  (the issue which led to 8ddfd94f09)
- we're following the documented API of getsize(path-like)
- the file is closed (thus flushed) inside idx.write, before getsize()
2023-01-11 19:19:43 -06:00
TW
2336c4b5a6
Merge pull request #7256 from ThomasWaldmann/docs-libb2-1.2
remove BORG_LIBB2_PREFIX (not used any more)
2023-01-10 14:24:38 +01:00
Thomas Waldmann
47e0f9bcb0
remove BORG_LIBB2_PREFIX (not used any more)
the code now uses hashlib.blake2b from python stdlib.
2023-01-10 14:09:28 +01:00
TW
1ac19b034b
Merge pull request #7242 from ThomasWaldmann/upgrade-cython-1.2
upgrade Cython to 0.29.33
2023-01-06 22:04:18 +01:00
TW
32b5f44bbb
Merge pull request #7235 from ThomasWaldmann/macos-resourcefork-mtime-1.2
extract: fix mtime when ResourceFork xattr is set (macOS specific)
2023-01-06 21:41:29 +01:00
Thomas Waldmann
497eca0faf
upgrade Cython to 0.29.33 2023-01-06 21:39:22 +01:00
Thomas Waldmann
b96781b9dd
macOS: fix mtime timestamp extraction if ResourceFork xattr is present, fixes #7234
setting the timestamps after xattrs helps for correct mtime,
but atime is still broken in this case.
2022-12-30 00:08:06 +01:00
Thomas Waldmann
c095e2ef61
macOS: test correct timestamp extraction if ResourceFork xattr is present, see #7234 2022-12-30 00:08:04 +01:00
TW
4ca090bb8c
Merge pull request #7229 from ThomasWaldmann/update-requirements-1.2
update development.lock.txt
2022-12-27 20:31:28 +01:00
Thomas Waldmann
3837e0233d
update development.lock.txt
including a setuptools security fix, see #7227

also:
- cosmetic change in development.txt to have same order
- removed exclusion of broken Cython release (outdated since long)
2022-12-27 19:32:20 +01:00
TW
06716a7971
Merge pull request #7224 from ThomasWaldmann/rel123
release 1.2.3
2022-12-24 23:25:57 +01:00
Thomas Waldmann
ae8b0eb3e4
avoid git complaining about dubious ownership
this lead to breakage of setuptools_scm, it could not
determine the borg version under fakeroot due to this
git / ownership issue.
2022-12-24 18:15:49 +01:00
Thomas Waldmann
795a553675
build_man 2022-12-24 18:09:33 +01:00
Thomas Waldmann
ec49d60d1f
build_usage 2022-12-24 18:09:32 +01:00
Thomas Waldmann
e4cbd28bdd
update CHANGES 2022-12-24 18:09:30 +01:00
Thomas Waldmann
ffd6d51b2c
fix xattr tests
while fixing #6988, some strings were changed, but the tests not adapted:

a758fda089
2022-12-24 18:09:28 +01:00
TW
ab0f0d24fd
Merge pull request #7223 from ThomasWaldmann/fix-docs-prefix-1.2
Fix docs / tests concerning deprecated --prefix (1.2-maint)
2022-12-24 15:10:56 +01:00
Thomas Waldmann
90a937e1ed
remove deprecated --prefix from docs, fixes #7109
also:

- rather mention --glob-archives
- rather test --glob-archives
2022-12-24 14:44:05 +01:00
TW
2f78e545e0
Merge pull request #7222 from ThomasWaldmann/vagrant-updates
Vagrant updates
2022-12-24 04:40:08 +01:00
Thomas Waldmann
aa2c2a4e34
vagrant: use python 3.9.16 for the binary build 2022-12-24 02:24:35 +01:00
Thomas Waldmann
5a442743f5
vagrant: use the openbsd 7.1 box
openbsd 6.9 was removed from the ftp servers and
the vagrant box stopped working. so we now use the
same box as in master branch.
2022-12-24 02:21:50 +01:00
TW
5704c4d887
Merge pull request #7215 from ThomasWaldmann/github-actions-updates-1.2
GitHub actions updates (1.2-maint)
2022-12-17 21:39:07 +01:00
TW
28bf1db2c8
Merge pull request #7216 from ThomasWaldmann/fix-ci-1.2
fix github CI (1.2-maint)
2022-12-17 21:38:05 +01:00
Thomas Waldmann
96d36e32cb
github CI: set PKG_CONFIG_PATH again
strange: on macOS, the globally set PKG_CONFIG_PATH was overwritten,
thus the borg build did not find openssl any more. setting it here
locally again works around the issue.
2022-12-17 20:27:49 +01:00
Thomas Waldmann
dd8e5487e3
github CI: prepend to PKG_CONFIG_PATH 2022-12-17 20:27:39 +01:00
Thomas Waldmann
0849ef25de
github CI: use actions/cache@v3 2022-12-17 20:02:52 +01:00
Thomas Waldmann
b69721ebe0
no fail fast, just checking... 2022-12-17 18:48:41 +01:00
Thomas Waldmann
fe1c89e5ba
github CI: use actions/setup-python@v4
should fix the node12 deprecation warning.
2022-12-17 18:33:26 +01:00
Thomas Waldmann
bb1c132bbc
github CI: use actions/checkout@v3
should fix the node12 deprecation warning.
2022-12-17 18:33:12 +01:00
Thomas Waldmann
73097c8162
github CI: use ubuntu-20.04 (not -latest)
affects only the lint job, everything else used 20.04 already.
2022-12-17 18:33:06 +01:00
TW
d61f5eba63
Merge pull request #7214 from ThomasWaldmann/update-changes-1.2
update CHANGES (1.2-maint)
2022-12-17 18:13:31 +01:00
Thomas Waldmann
9c8ac57399
update CHANGES 2022-12-17 18:04:03 +01:00
TW
2aae879c49
Merge pull request #7210 from ThomasWaldmann/list-directories-dry-run-1.2
create: --list --dry-run output for directories, fixes #7209
2022-12-15 17:50:46 +01:00
Thomas Waldmann
dca6792850
create: --list --dry-run output for directories, fixes #7209
Without the status being set no output was generated in
dry-run mode, confusing users about whether borg would back
up directories (in non-dry-run mode).

- == item not backed up just because of dry-run mode
x == item excluded
2022-12-15 17:26:36 +01:00
TW
6d5c4a48b3
Merge pull request #7201 from ThomasWaldmann/fix-macos-ci-1.2
fix tox4 issue on macOS CI (1.2-maint)
2022-12-10 20:52:29 +01:00
Thomas Waldmann
e66d8c399b
fix tox4 passenv issue, fixes #7199
also: have a setting to disable fail-fast, in case something breaks again.
2022-12-10 19:55:28 +01:00
TW
79b6c985a1
Merge pull request #7169 from ThomasWaldmann/update-twine-upload-1.2
twine: use repo name instead of url
2022-11-27 01:25:39 +01:00
Thomas Waldmann
3984a7e3da
twine: use repo name instead of url 2022-11-27 01:24:25 +01:00
TW
d4cd911d46
Merge pull request #7150 from ThomasWaldmann/fix-nfiles-1.2
Fix nfiles 1.2
2022-11-20 20:30:25 +01:00
TW
889c8778a1
Merge pull request #7149 from ThomasWaldmann/py311-1.2
update python 3.9 and 3.11
2022-11-20 18:48:25 +01:00
Thomas Waldmann
f6419ee28b
archive.save(): always use metadata from stats, fixes #7072
e.g. nfiles, size, etc.

fixes:
- checkpoint archives did not have this metadata yet
- borg import-tar did not have this metadata yet
2022-11-20 18:33:53 +01:00
Thomas Waldmann
822a23da92
github CI: use python 3.11 (not -dev) 2022-11-20 17:58:43 +01:00
Thomas Waldmann
d623b47da9
vagrant: use python 3.9.15 for tests/binary build 2022-11-20 17:57:16 +01:00
TW
c233a964f2
Merge pull request #7147 from ThomasWaldmann/fix-repository-tests-1.2
Fix repository tests 1.2
2022-11-19 17:06:07 +01:00
Rayyan Ansari
d0b09dfc95
testsuite: repository: skip some tests for RemoteRepository 2022-11-19 14:30:41 +01:00
Rayyan Ansari
82f6adf8e0
file_integrity.py: make sure file_fd is always closed on exit 2022-11-19 14:30:16 +01:00
Rayyan Ansari
9cff9d96fa
testsuite: repository: close fd before deleting segment
See last commit.
2022-11-19 14:28:14 +01:00
Rayyan Ansari
05505de807
repository: cleanup(): close segment before unlinking
On Windows, trying to delete a file that is already open raises an exception.

Docs:
https://docs.python.org/3/library/os.html#os.remove
2022-11-19 14:25:01 +01:00
Rayyan Ansari
6a97e936ac
repository: use os.replace instead of os.rename
On Windows, os.rename raises an exception if the destination file already exists, unlike os.replace which replaces the destination file.

Docs:
https://docs.python.org/3/library/os.html#os.rename
https://docs.python.org/3/library/os.html#os.replace
2022-11-19 14:22:35 +01:00
TW
97261dbb22
Merge pull request #7135 from ThomasWaldmann/remove-py37-compat-code-1.2
remove python < 3.7 compatibility code
2022-11-07 09:04:37 +01:00
Thomas Waldmann
f1a355f3d9
remove python < 3.7 compatibility code
not supported any more in master and 1.2-maint branches.
2022-11-06 18:25:03 +01:00
TW
fff56edd8a
Merge pull request #7130 from ThomasWaldmann/fix-6070-1.2
improve/clarify strange test code, fixes #6070
2022-11-05 02:15:25 +01:00
TW
2ae41fd9e6
Merge pull request #7128 from ThomasWaldmann/fix-root-paths-1.2
fix args.paths related argparsing, fixes #6994
2022-11-05 01:12:45 +01:00
Thomas Waldmann
25d6649bcd improve/clarify strange test code, fixes #6070 2022-11-05 00:51:44 +01:00
Thomas Waldmann
d57ed9dd98 fix args.paths related argparsing, fixes #6994
argparse: the default action is "store" and that overwrote an already
existing list in args.paths (e.g. from --pattern="R someroot") when it
started to process the positional PATH args.

with "extend" it now extends the existing args.paths with the list of
positional PATH arguments (which can be 0..N elements long, nargs="*").

note: "extend" is new since python 3.8, thus this can only be backported
to 1.2-maint, but not to 1.1-maint.
2022-11-05 00:30:46 +01:00
TW
01b921dea4
Merge pull request #7124 from ThomasWaldmann/hashindex-test-win-1.2
Fix test_size_on_disk_accurate on Windows
2022-11-04 21:12:58 +01:00
Thomas Waldmann
8ddfd94f09 Fix test_size_on_disk_accurate on Windows
Pass the python file object / fd instead of the file path.

On Windows, a tempfile cannot be opened again, unlike on Unix systems.
See https://docs.python.org/3/library/tempfile.html#tempfile.NamedTemporaryFile
2022-11-04 20:28:09 +01:00
TW
3d1f11b2e2
Merge pull request #7106 from RayyanAnsari/msys2-ci-backport
Use MSYS2 for Windows CI [backport]
2022-10-28 19:11:39 +02:00
Rayyan Ansari
e8dd5c449e
Add new Windows build scripts and CI with GitHub and MSYS2 [backport]
Fixes #7105
2022-10-27 16:56:08 +01:00
Rayyan Ansari
4b27ab2fbd
Remove old Windows scripts and Appveyor CI [backport] 2022-10-27 16:33:02 +01:00
nyuszika7h
84f4c12752
tar_filter: recognize .tar.zst as zstd (backport) (#7095)
tar_filter: recognize .tar.zst as zstd

Backported from #7093
2022-10-22 13:41:11 +02:00
TW
0989cb4040
Merge pull request #7082 from ThomasWaldmann/fix-chunker-params-comparison
Fix chunker params comparison
2022-10-11 20:29:32 +02:00
Thomas Waldmann
3200190cdb diff: more precise warning msgs for different chunker params
if we know both archives' chunker params, use "are different" not "might be".
also do not recommend to enforce it using --same-chunker-params in this case.
2022-10-11 19:44:11 +02:00
Thomas Waldmann
651bef450d recreate: normalize chunker params before comparing them, see #7079
borg < 1.2 did not have the chunker name as first element in the tuple,
but it always was buzhash, because there was no other chunker.
2022-10-10 22:41:39 +02:00
Thomas Waldmann
a6207370a9 diff: normalize chunker params before comparing them, fixes #7079
borg < 1.2 did not have the chunker name as first element in the tuple,
but it always was buzhash, because there was no other chunker.
2022-10-10 22:27:12 +02:00
TW
773f3bff77
Merge pull request #7057 from ThomasWaldmann/fix-recreate-fixed-chunker-1.2
get_chunker: fix missing sparse=False argument, fixes #7056
2022-09-27 22:36:03 +02:00
Thomas Waldmann
875c94a20d get_chunker: fix missing sparse=False argument, fixes #7056 2022-09-27 21:45:51 +02:00
Thomas Waldmann
2f6197df0c add test for recreate with "fixed" chunker 2022-09-27 21:45:51 +02:00
Emmanuel Tanimowo
f49133da04
update automating backups script (#7039)
update "automating backups" script: remove last backslash, use --glob-archives
2022-09-21 00:54:38 +02:00
TW
44b4791e98
Merge pull request #7035 from ThomasWaldmann/fix-7034-1.2
check: fix uninitialised variable if repo is completely empty, fixes #7034
2022-09-17 20:51:48 +02:00
Thomas Waldmann
4f4be2631a check: fix uninitialised variable if repo is completely empty, fixes #7034 2022-09-17 20:29:34 +02:00
TW
9fe7339a39
Merge pull request #7024 from ThomasWaldmann/compat-setuptoolsscm4-1.2
do not use version_tuple placeholder in setuptools_scm template
2022-09-17 13:26:55 +02:00
Thomas Waldmann
3404346419 do not use version_tuple placeholder in setuptools_scm template
that would require setuptools_scm>=5.0.0 but some dists do not have that yet.
also, we do not use the version_tuple from _version.py, so it is not required
anyway.
2022-09-13 21:25:35 +02:00
TW
1b3cbe2461
Merge pull request #6990 from ThomasWaldmann/more-fine-grained-extended-stat-1.2
xattrs / extended stat: improve exception handling (1.2-maint)
2022-09-05 16:06:10 +02:00
Thomas Waldmann
3bd3600875 cosmetic: add missing blank to xattr test skip reason 2022-08-24 14:19:53 +02:00
Thomas Waldmann
a758fda089 xattrs: improve error handling, fixes #6988
looks like we can not rely on listxattr only returning
valid, acceptable names for getxattr.

so getxattr can still fail with EINVAL.

also:

- do not raise an exception if getting a single
xattr fails, rather emit a warning and continue
processing the remaining xattrs (and also the whole
file). we already had that for EPERM (and similar
for ENOATTR), just do it for all errors.

- _check, when it raises OSError, gives us a nice
exception object, use it.

- more helpful error msgs, try not to lose error information
2022-08-24 12:03:51 +02:00
Thomas Waldmann
b5d43506d3 extended stat: more fine grained exception handling
see #6988: it was unclear where exactly the error came from (flags, xattrs or ACLs getting?).
2022-08-23 21:56:31 +02:00
TW
9bfe210948
Merge pull request #6980 from ThomasWaldmann/rel122
release 1.2.2
2022-08-20 21:04:12 +02:00
Thomas Waldmann
101a06234f netbsd vagrant box: py310, remove some outdated fixes
seems like that stuff got fixed in 9.3.
2022-08-20 16:52:31 +02:00
Thomas Waldmann
e87c54a0a1 update CHANGES 2022-08-20 15:38:10 +02:00
Thomas Waldmann
bbad73502b requirements.lock.txt: upgrade Cython and tox 2022-08-20 15:38:10 +02:00
Thomas Waldmann
92ca260fd8 build_usage ; build_man 2022-08-20 15:38:10 +02:00
Thomas Waldmann
b024379904 fix sphinx warnings 2022-08-20 15:38:10 +02:00
TW
4e2cc02e86
Merge pull request #6967 from ThomasWaldmann/fix-versionpy-format-1.2
_version.py: remove trailing blank, add LF at EOF
2022-08-15 15:17:56 +02:00
TW
ec389edc83
Merge pull request #6944 from ThomasWaldmann/fix-ctrl-c-remote-repo-1.2
ctrl-c must not kill important subprocesses, fixes #6912
2022-08-13 23:23:50 +02:00
Thomas Waldmann
6799405012 _version.py: remove trailing blank, add LF at EOF 2022-08-13 20:35:03 +02:00
Thomas Waldmann
7a31978c55 ctrl-c must not kill other subprocesses, fixes #6912
There are some other places with subprocesses:

- borg create --content-from-command
- borg create --paths-from-command
- (de)compression filter process of import-tar / export-tar
2022-08-06 12:48:54 +02:00
Thomas Waldmann
5d83d5f153 ctrl-c must not kill the ssh subprocess, fixes #6912 2022-08-06 12:29:10 +02:00
TW
e215525339
Merge pull request #6943 from ThomasWaldmann/update-changes-1.2
update CHANGES
2022-08-06 11:37:07 +02:00
Thomas Waldmann
c4e552a1e5 update CHANGES 2022-08-06 10:41:58 +02:00
TW
fa8d8ecd73
Merge pull request #6918 from ThomasWaldmann/compaction-debugging-1.2
repository: add debug logging for issue #6687
2022-08-06 10:27:23 +02:00
TW
52cba940ea
Merge pull request #6936 from ThomasWaldmann/mkstemp_mode-1.2
use a custom mkstemp with mode support, fixes #6933, fixes #6400
2022-08-05 13:04:24 +02:00
TW
28bc76a916
Merge pull request #6937 from ThomasWaldmann/fix-warnings-1.2
Fix warnings (1.2-maint)
2022-08-04 18:23:34 +02:00
Thomas Waldmann
69f4898460 make setuptools happy, fixes #6874
work around setuptools puking about:

          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'borg.cache_sync' as an importable package,
          but it is not listed in the `packages` configuration of setuptools.

          'borg.cache_sync' has been automatically added to the distribution only
          because it may contain data files, but this behavior is likely to change
          in future versions of setuptools (and therefore is considered deprecated).

          Please make sure that 'borg.cache_sync' is included as a package by using
          the `packages` configuration field or the proper discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).

          You can read more about "package discovery" and "data files" on setuptools
          documentation page.
2022-08-04 15:50:05 +02:00
Thomas Waldmann
b3b5602ac2 _chunker.c: fix warnings on macOS
macOS does not have POSIX_FADV_DONTNEED, thus some variables are not
needed.
2022-08-04 14:51:06 +02:00
Thomas Waldmann
b0eee13124 use a custom mkstemp with mode support, fixes #6933, fixes #6400
hopefully this is the final fix.

after first fixing of #6400 (by using os.umask after mkstemp), there
was a new problem that chmod was not supported on some fs.

even after fixing that, there were other issues, see the ACLs issue
documented in #6933.

the root cause of all this is tempfile.mkstemp internally using a
very secure, but hardcoded and for our use case problematic mode
of 0o600.

mkstemp_mode (mosty copy&paste from python stdlib tempfile module +
"black" formatting applied) supports giving the mode via the api,
that is the only change needed.

slightly dirty due to the _xxx imports from tempfile, but hopefully
this will be supported in some future python version.
2022-08-04 14:22:56 +02:00
Thomas Waldmann
e5b6670bc6 repository: add debug logging for issue #6687
i suspect that compact_segments wrongly drops some DEL tags.

this could make already deleted chunks re-appear, leading to:
- suddenly more repo index entries than previously (check, repo part)
- suddenly orphaned chunks appearing (check, archives part)

if this is the case, the issue is harmless, but annoying/confusing.

to fix the issue, i need the complete log line "dropping DEL for id X".
the id X it mentions is one of the chunk ids borg check complains about.
2022-07-31 19:05:54 +02:00
TW
4b54cfef45
Merge pull request #6914 from ThomasWaldmann/update-changes-1.2
update CHANGES
2022-07-30 18:32:58 +02:00
Thomas Waldmann
fe27be9f3e update CHANGES 2022-07-30 17:57:57 +02:00
TW
ade2241163
Merge pull request #6911 from horazont/docs/internal-details-1.2
docs(1.2): provide more details on object layout
2022-07-30 17:43:27 +02:00
Jonas Schäfer
bd89fb717e docs: provide more details on object layout
While writing my own out-of-band decoder, I had a hard time figuring out
how to unpack the manifest. From the description, I was only able to
read that the manifest is msgpack'd, but I had not been able to figure
out that it's also going through the same encryption+compression logic
as all other things do.

This should make it a little clearer and provide the necessary
information to understand how the compression works.
2022-07-30 17:32:36 +02:00
TW
01f87c6ca1
Merge pull request #6887 from ThomasWaldmann/prune-checkpointing-1.2
prune/delete --checkpoint-interval=1800 and ctrl-c/SIGINT support, fixes #6284
2022-07-30 16:54:50 +02:00
TW
5e291ff5e1
Merge pull request #6905 from ThomasWaldmann/fix-flags-formatting-1.2
list: fix {flags:<WIDTH>} formatting, fixes #6081
2022-07-30 15:34:19 +02:00
TW
78ae0114d4
Merge pull request #6909 from ThomasWaldmann/update-changes-1.2
update CHANGES
2022-07-30 15:30:48 +02:00
TW
32616111e0
Merge pull request #6901 from ThomasWaldmann/fix-5719-1.2
check: try harder to create the key, fixes #5719
2022-07-30 15:30:32 +02:00
Thomas Waldmann
e7bd9e8442 update CHANGES 2022-07-29 11:48:18 +02:00
Thomas Waldmann
e81f6f34cd list: fix {flags:<WIDTH>} formatting, fixes #6081
item.bsdflags is either not present or an int, thus we default to 0 (== no flags) if not present.
2022-07-29 10:09:00 +02:00
Thomas Waldmann
ea4c47ec7a check: try harder to create the key, fixes #5719
the old code did just 1 attempt to detect the repo decryption key.
if the first chunkid we got from the chunks hashtable iterator was accidentally
the id of the chunk we intentionally corrupted in test_delete_double_force,
setup of the key failed and that made the test crash.

in practice, this could of course also happen if chunks are corrupted, thus
we now do many retries with other chunks before giving up.

error handling was improved: do not return None (instead of a key), it just
leads to weird crashes elsewhere, but fail early with IntegrityError and a
reasonable error msg.

rename method to make_key to avoid confusion with borg.crypto.key.identify_key.
2022-07-29 09:08:03 +02:00
Thomas Waldmann
f8e33cfdf1 prune/delete --checkpoint-interval=1800 and ctrl-c/SIGINT support, fixes #6284
manifest, repo and cache are committed every checkpoint interval.

also, when ctrl-c is pressed, finish deleting the current archive, commit and then terminate.
2022-07-28 19:15:23 +02:00
TW
3d94c595b0
Merge pull request #6896 from ThomasWaldmann/ci-macos-1.2
CI: test on macOS 12 without fuse / fuse tests
2022-07-28 00:09:59 +02:00
Thomas Waldmann
9fb9508f4b CI: test on macOS 12 without fuse / fuse tests
too troublesome on github CI due to kernel extensions needed by macFUSE.
2022-07-27 23:10:10 +02:00
remyabel2
c56025000b
docs: add info on man page installation (#6894) 2022-07-27 13:46:48 +02:00
TW
4c3da429fb
Merge pull request #6890 from ThomasWaldmann/archive-progress-json-docs-1.2
docs: update archive_progress json description about "finished"
2022-07-27 12:46:59 +02:00
Thomas Waldmann
404fe33a8d docs: update archive_progress json description about "finished", see #6570 2022-07-25 13:06:06 +02:00
TW
a4a2530a29
Merge pull request #6877 from LocutusOfBorg/fix-template-1.2
Fix pyproject.toml to create a fixed _version.py file, compatible wi…
2022-07-19 16:41:41 +02:00
Gianfranco Costamagna
ad8555e921 Fix pyproject.toml to create a fixed _version.py file, compatible with both old and new setuptools_scm version (see: #6875) 2022-07-18 19:51:04 +02:00
TW
907764e018
Merge pull request #6850 from ThomasWaldmann/deprecated-prefix-option-1.2
deprecate --prefix, use -a / --glob-archives, see #6806
2022-07-09 16:57:58 +02:00
Thomas Waldmann
c265370cff deprecate --prefix, use -a / --glob-archives, see #6806 2022-07-09 16:12:34 +02:00
TW
a8c556b5e9
Merge pull request #6843 from ThomasWaldmann/expect-1.2
Automate asciinema screencasts (1.2-maint)
2022-07-07 00:36:59 +02:00
TW
8a42d88120 Merge pull request #6197 from hexagonrecursion/expect
Automate asciinema screencasts
2022-07-07 00:01:28 +02:00
TW
1fba67efee
Merge pull request #6824 from ThomasWaldmann/fix-progress-percent-docs-1.2
docs: json progress_percent: some values are optional, fixes #4074
2022-07-02 20:21:36 +02:00
Thomas Waldmann
bfae252593 docs: json progress_percent: some values are optional, fixes #4074
in the finished == true message, these are missing:
- message
- current / total
- info

This is to be somewhat consistent with #6683 by only providing a
minimal set of values for the finished case.

The finished messages is primarily intended for cleanup purposes,
e.g. clearing the progress display.
2022-07-01 14:31:23 +02:00
TW
9c463dc46b
Merge pull request #6816 from ThomasWaldmann/fix-hashindex-compact-1.2
hashindex_compact: fix eval order (check idx before use), fixes #5899
2022-06-30 20:57:38 +02:00
Thomas Waldmann
3f517b019a hashindex_compact: fix eval order (check idx before use), fixes #5899
also: fix "off by one" comment
2022-06-29 18:58:26 +02:00
TW
25c9826456
Merge pull request #6814 from fantasya-pbem/docs/5960_FAQ-quota-size_1.2-maint
[DOCS] #5960 - FAQ: Full quota / full disk
2022-06-29 18:55:36 +02:00
Thalian
6990aca7ac [DOCS] #5960 - FAQ: Full quota / full disk
Backport from master.
2022-06-29 17:56:23 +02:00
TW
0cac871b34
Merge pull request #6787 from ThomasWaldmann/chmod-optional-1.2
SaveFile: the chmod is optional, fixes #6786
2022-06-23 18:07:02 +02:00
Thomas Waldmann
89e24847b9 SaveFile: the chmod is optional, fixes #6786
some filesystems do not support chmod, just ignore if it is failing.
2022-06-23 17:43:21 +02:00
TW
7ee4b7035b
Merge pull request #6779 from ThomasWaldmann/paths-from-normpath-1.2
create --paths-from-(stdin|command): normalize paths, fixes #6778
2022-06-22 00:20:20 +02:00
Thomas Waldmann
ec2bf8e45d create --paths-from-(stdin|command): normalize paths, fixes #6778 2022-06-20 13:44:35 +02:00
TW
97c5c54a0c
Merge pull request #6775 from wizeman/fix-obfuscate-1.2
fix test_obfuscate byte accounting (backport #6773 to 1.2-maint)
2022-06-14 17:29:24 +02:00
Ricardo M. Correia
333320ad56 fix test_obfuscate byte accounting 2022-06-14 16:06:49 +02:00
TW
bab00fb47b
Merge pull request #6769 from ThomasWaldmann/safe-secure-erase-1.2
secure_erase: avoid collateral damage, fixes #6768
2022-06-13 19:41:38 +02:00
Thomas Waldmann
afcebf32f7 secure_erase: avoid collateral damage, fixes #6768
if a hardlink copy of a repo was made and a new repo config
shall be saved, do NOT fill in random garbage before deleting
the previous repo config, because that would damage the hardlink
copy.
2022-06-13 15:47:50 +02:00
TW
73b97f339b
Merge pull request #6765 from targhs/doc/correct-installation-shell-syntax
Correct shell syntax for installation using git
2022-06-12 19:48:03 +02:00
targhs
08629e4d83 Commit shell syntax for installation using git 2022-06-12 22:37:37 +05:30
TW
7f8a5bb9c3
Merge pull request #6753 from ThomasWaldmann/rel121
Release 1.2.1
2022-06-05 23:43:32 +02:00
Thomas Waldmann
9fad4ba1cb build_man 2022-06-05 20:14:26 +02:00
Thomas Waldmann
a7c1c1cb16 build_usage 2022-06-05 20:13:46 +02:00
Thomas Waldmann
6c0842ec42 update CHANGES 2022-06-05 20:09:06 +02:00
TW
b0c726abae
Merge pull request #6752 from ThomasWaldmann/vagrant-updates
Vagrant updates (1.2)
2022-06-05 20:03:03 +02:00
Thomas Waldmann
d8102774b6 vagrant: increase VMCPUS and xdistn from 4 to 16
esp. the darwin64 box based tests take ages, let's see
if they get faster with this.
2022-06-05 19:56:29 +02:00
TW
7475e661bf
Merge pull request #6751 from ThomasWaldmann/update-changes-1.2
update CHANGES (1.2)
2022-06-05 18:36:37 +02:00
TW
eeb55c10c4
Merge pull request #6749 from fantasya-pbem/docs/4776_FAQ_swapped-targets_main
[DOCS] #4776 - FAQ: How to swap backup media
2022-06-05 18:33:50 +02:00
Thalian
cfcf99ccb5 [DOCS] #4776 - FAQ: How to swap backup media
Backport from master.
2022-06-05 18:22:23 +02:00
Thomas Waldmann
4ae08ee972 update CHANGES 2022-06-05 18:18:49 +02:00
TW
cfc623e250
Merge pull request #6748 from ThomasWaldmann/update-changes-rel1118-1.2
update CHANGES with 1.1.18 changelog
2022-06-05 18:08:52 +02:00
Thomas Waldmann
677db66e9d update CHANGES with 1.1.18 changelog 2022-06-05 17:45:48 +02:00
Thomas Waldmann
96c748c1d2 vagrant: add ubuntu "jammy" 22.04 box 2022-06-05 17:35:05 +02:00
TW
7ab36ceb76
Merge pull request #6742 from ThomasWaldmann/msgpack104-1.2
msgpack 1.0.4 / py 3.11 (1.2-maint)
2022-06-03 10:37:31 +02:00
Thomas Waldmann
6d024961aa add python 3.11 to pypi metadata 2022-06-03 10:05:56 +02:00
Thomas Waldmann
d1ef35b4f0 CI: also test on python 3.11-dev 2022-06-03 10:05:56 +02:00
Thomas Waldmann
de61d4953b allow msgpack 1.0.4, fixes #6716 2022-06-03 10:05:44 +02:00
TW
03827bc684
Merge pull request #6738 from ThomasWaldmann/vagrant-updates-1.2
Vagrant updates (1.2-maint)
2022-06-01 20:19:55 +02:00
Thomas Waldmann
354b014b19 vagrant/testing: upgrade development.lock.txt
esp. the Cython version upgrade to 0.29.30 is important for python 3.11.
2022-06-01 19:51:55 +02:00
Thomas Waldmann
6b9a66b5fa vagrant: use pyinstaller 4.10
when installed via pip, this automatically build the bootloader now.
2022-06-01 19:49:37 +02:00
Thomas Waldmann
6063c21bd9 vagrant: use python 3.9.13 for binary build 2022-06-01 19:48:10 +02:00
TW
8370eea9e0
Merge pull request #6731 from KN4CK3R/backport-6724-1.2
Backport make constants for files cache mode more clear (#6724, 1.2)
2022-05-30 16:02:53 +02:00
Elmar Hoffmann
bf31030d05 make constants for files cache mode more clear (#6724)
* make constants for files cache mode more clear

Traditionally, DEFAULT_FILES_CACHE_MODE_UI and DEFAULT_FILES_CACHE_MODE
were - as the naming scheme implies - the same setting, one being the UI
representation as given to the --files-cache command line option and the
other being the same default value in the internal representation.

It happended that the actual value used in borg create always comes from
DEFAULT_FILES_CACHE_MODE_UI (because that does have the --files-cache
option) whereas for all other commands (that do not use the files cache) it
comes from DEFAULT_FILES_CACHE_MODE.

PR #5777 then abused this fact to implement the optimisation to skip loading
of the files cache in those other commands by changing the value of
DEFAULT_FILES_CACHE_MODE to disabled.
This however also changes the meaning of that variable and thus redesignates
it to something not matching the original naming anymore.

Anyone not aware of this change and the intention behind it looking at the
code would have a hard time figuring this out and be easily mislead.

This does away with the confusion making the code more maintainable by
renaming DEFAULT_FILES_CACHE_MODE to FILES_CACHE_MODE_DISABLED, making the
new intention of that internal default clear.

* make constant for files cache mode UI default match naming scheme
2022-05-30 15:31:26 +02:00
TW
5679eb16df
Merge pull request #6722 from ThomasWaldmann/debug-get-chunk-1.2
borg debug dump-repo-objs --ghost: new --segment=S --offset=O options
2022-05-28 00:33:49 +02:00
Thomas Waldmann
49f191bc4c borg debug dump-repo-objs: new --segment=S --offset=O options
the new options limit processing to segment S (and, optionally, also
to offset O), so you do not need to dump your complete repo to only
look at 1 segment (or 1 object).

note: there is no rpc api for scan_low_level, so this only works locally.
2022-05-27 23:29:45 +02:00
TW
1b231e79e2
Merge pull request #6720 from ThomasWaldmann/update-changes-1.2
update CHANGES (1.2)
2022-05-27 21:54:23 +02:00
Thomas Waldmann
b55bb4694d update CHANGES 2022-05-27 17:12:56 +02:00
TW
7d14302b12
Merge pull request #6718 from fantasya-pbem/task/6407_document-pattern-changes-backport
#6407 - Document Borg 1.2 pattern behavior change
2022-05-27 15:41:51 +02:00
Thalian
2694652e37 #6407 - Document Borg 1.2 pattern behavior change
Backport from master: Make clear that absolute paths always go into the matcher as if they are relative (without leading slash). Adapt all examples accordingly.
2022-05-27 15:14:01 +02:00
TW
371bb06dd1
Merge pull request #6700 from KN4CK3R/backport-6688-1.2
Backport Fix typos (#6688)
2022-05-16 00:08:41 +02:00
Andrea Gelmini
f4d7c32058 Fix typos (#6688)
fix typos

Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
2022-05-15 23:43:50 +02:00
TW
39708849a2
Merge pull request #6695 from ThomasWaldmann/versions-mount-1archive-error-1.2
mount -o versions: give clear error msg instead of crashing
2022-05-15 19:52:43 +02:00
Thomas Waldmann
99ccd4f792 mount -o versions: give clear error msg instead of crashing
it does not make sense to request versions view if you only
look at 1 archive, but the code shall not crash in that case
as it did, but give a clear error msg.
2022-05-15 00:40:53 +02:00
Christopher Klooz
c3bbb5e57a
docs: authentication primitives: improved security and performance infos (1.2) (#6692)
docs: authentication primitives: improved security and performance infos
2022-05-14 22:56:39 +02:00
TW
fd4503cd30
Merge pull request #6683 from ThomasWaldmann/fix-archive-progress-json-1.2
show_progress: add finished=true/false to archive_progress json
2022-05-08 19:45:53 +02:00
TW
247b3d9b55
Merge pull request #6686 from ThomasWaldmann/fix-create-filter-1.2
fix create_filter_process exception handler, fixes #6681
2022-05-08 19:15:59 +02:00
Thomas Waldmann
b846ae9659 fix create_filter_process exception handler, fixes #6681
if cmd was falsy (e.g. None), there is no proc.
then, if "yield stream" raises an exception, the exception handler crashed at "proc.kill()".
2022-05-08 18:46:42 +02:00
Thomas Waldmann
9dff959f16 show_progress: add finished=true/false to archive_progress json, fixes #6570
also:
- remove empty values from final json
- add test
2022-05-08 18:38:10 +02:00
TW
ae417cccf6
Merge pull request #6679 from KN4CK3R/backport-6670-1.2
Backport check that borg does not require pytest for normal usage (#6670, 1.2)
2022-05-07 19:59:22 +02:00
TW
d72eb3b1cf
Merge pull request #6678 from KN4CK3R/backport-6659-1.2
Backport mention BORG_FILES_CACHE_SUFFIX as alternative to BORG_FILES_CACHE_TTL (#6659)
2022-05-07 19:57:52 +02:00
Thomas Waldmann
c7390383a0 check that borg does not require pytest for normal usage, fixes #6563
also: move the note about this to the very top of the affected modules.
2022-05-07 18:56:36 +02:00
Thomas Waldmann
6d6711768e docs: mention BORG_FILES_CACHE_SUFFIX as alternative to BORG_FILES_CACHE_TTL, fixes #5602 2022-05-07 18:47:41 +02:00
TW
bcac974df7
Merge pull request #6648 from ThomasWaldmann/key-algo-error-msg-1.2
give a more helpful error msg for unsupported encrypted key format
2022-04-27 11:03:15 +02:00
Thomas Waldmann
25390de2c7 give a more helpful error msg for unsupported key format, fixes #6561 2022-04-24 22:03:28 +02:00
Thomas Waldmann
05d7ee9ed8 give a more helpful error msg for unsupported encrypted key format, fixes #6561 2022-04-24 21:55:41 +02:00
TW
1311a7d7a7
Merge pull request #6645 from hexagonrecursion/1.2-faq
[1.2] Add a troubleshooting note to the FAQ
2022-04-24 13:57:02 +02:00
Andrey Bienkowski
622ec1a181 Add a troubleshooting note to the FAQ 2022-04-24 11:27:37 +03:00
TW
65e371999d
Merge pull request #6636 from ThomasWaldmann/docs-files-cache-1.2
FAQ: add a hint about --debug-topic=files_cache
2022-04-20 13:01:56 +02:00
Thomas Waldmann
56262b57bd FAQ: add a hint about --debug-topic=files_cache 2022-04-20 13:00:34 +02:00
TW
26b46dd1f4
Merge pull request #6629 from ThomasWaldmann/fix-iec-1.2
fix --iec mode output for borg delete and prune
2022-04-19 23:39:26 +02:00
TW
b2d0af9487
Merge pull request #6632 from ThomasWaldmann/fix-info-authenticated-1.2
info: fix authenticated mode repo to show "Encrypted: No", fixes #6462
2022-04-19 23:38:54 +02:00
Thomas Waldmann
c5f3214d8a info: fix authenticated mode repo to show "Encrypted: No", fixes #6462 2022-04-19 22:10:44 +02:00
TW
890b627860
Merge pull request #6627 from ThomasWaldmann/fix-files-cache-6353-1.2
fix transaction rollback: use files cache filename as found in txn.active
2022-04-19 21:57:55 +02:00
Thomas Waldmann
cc04253afd prune: fix --stats --iec output to use binary units, see #6606 2022-04-19 21:51:55 +02:00
Thomas Waldmann
0ddb1fa194 delete: fix --stats --iec output to use binary units, see #6606 2022-04-19 21:50:26 +02:00
Thomas Waldmann
8d160b6a8e fix transaction rollback: use files cache filename as found in txn.active, fixes #6353 2022-04-19 21:17:50 +02:00
TW
55f9028611
diff: support presence change for blkdev, chrdev and fifo items (1.2-maint) (#6615)
diff: support presence change for blkdev, chrdev and fifo items

also: refactor / clean up / reuse code.
2022-04-19 16:47:08 +02:00
TW
5619b5e2bd
Merge pull request #6620 from ThomasWaldmann/secure-erase-msg-1.2
remove stray punctuation from secure-erase message
2022-04-18 10:01:40 +02:00
TW
390b3127d0
Merge pull request #6618 from ThomasWaldmann/invalid-repo-msg-1.2
better error msg for defect or unsupported repo configs, fixes #6566
2022-04-18 09:52:52 +02:00
Jakub Wilk
291901717d remove stray punctuation from secure-erase message 2022-04-18 09:33:26 +02:00
Thomas Waldmann
e32760ee47 better error msg for defect or unsupported repo configs, fixes #6566 2022-04-18 09:28:30 +02:00
TW
9dec811077
Merge pull request #6613 from ThomasWaldmann/fix-pre12-meta-1.2
pre12-meta cache: do not use the cache if want_unique is True, fixes #6612
2022-04-18 09:27:02 +02:00
TW
638184ae6f
Merge pull request #6614 from ThomasWaldmann/require-setuptools-scm-via-pyproject-1.2
setuptools_scm: also require it via pyproject.toml
2022-04-18 08:03:02 +02:00
Thomas Waldmann
6ab17ee4e1 setuptools_scm: also require it via pyproject.toml
we have this in setup_requires in setup.py, but i also added it to
pyproject.toml - so we can be sure it is installed no matter what.

**configuration** of setuptools_scm is done the old way in setup.py
here in 1.2-maint, in master we have switched that over to pyproject.toml.
2022-04-18 05:00:17 +02:00
Thomas Waldmann
2a15ccf5ac pre12-meta cache: do not use the cache if want_unique is True, fixes #6612
we can't cache usize, it needs to get dynamically calculated.
2022-04-18 04:37:00 +02:00
TW
6eb0bee328
Merge pull request #6591 from ThomasWaldmann/fix-md-chunks-processing-1.2
metadata stream can produce all-zero chunks, fixes #6587
2022-04-14 03:53:52 +02:00
Thomas Waldmann
e1fbe1782f metadata stream can produce all-zero chunks, fixes #6587
all-zero chunks are propagated as:
CH_ALLOC, data=None, size=len(zeros)

other chunks are:
CH_DATA, data=data, size=len(data)

also: remove the comment with the wrong assumption
2022-04-14 00:09:53 +02:00
TW
d6f7056d9d
Merge pull request #6589 from ThomasWaldmann/docs-virt-1.2
virtualisation speed tips
2022-04-13 23:24:59 +02:00
Thomas Waldmann
08ed786f6d virtualisation speed tips 2022-04-13 23:08:39 +02:00
TW
7a3056092e
Merge pull request #6582 from ThomasWaldmann/deleted-key-1.2
load_key: no key is same as empty key, fixes #6441
2022-04-12 20:09:10 +02:00
Thomas Waldmann
7fc1d75cd3 load_key: no key is same as empty key, fixes #6441
when migrating from repokey to keyfile, we just store an empty key into the repo config,
because we do not have a "delete key" RPC api. thus, empty key means "there is no key".

here we fix load_key, so that it does not behave differently for no key and empty key:
in both cases, it just returns an empty value.

additionally, we strip the value we get from the config, so whitespace does not matter.

All callers now check for the repokey not being empty, otherwise RepoKeyNotFoundError
is raised.
2022-04-12 19:43:48 +02:00
TW
fddd0c93fa
Merge pull request #6581 from ThomasWaldmann/docs-max-duration-1.2
docs: improve borg check --max-duration description
2022-04-12 19:40:51 +02:00
TW
063adc1799
Merge pull request #6579 from ThomasWaldmann/compact-docs-1.2
docs: borg compact --cleanup-commits also runs a normal compaction
2022-04-12 18:53:53 +02:00
Thomas Waldmann
ec90a065fb docs: improve borg check --max-duration description 2022-04-12 18:45:03 +02:00
TW
8d632c0fcc
Merge pull request #6574 from ThomasWaldmann/more-robust-iterator-1.2
borg check improvements (1.2)
2022-04-12 18:13:36 +02:00
TW
19f9247f65
Merge pull request #6576 from ThomasWaldmann/repo-tags-docs-1.2
docs: fix values of TAG bytes, fixes #6515
2022-04-12 18:13:05 +02:00
Thomas Waldmann
3bacf4ac39 docs: borg compact --cleanup-commits also runs a normal compaction, fixes #6324 2022-04-12 18:11:38 +02:00
Thomas Waldmann
bb6188da49 docs: fix values of TAG bytes, fixes #6515 2022-04-12 17:58:13 +02:00
Thomas Waldmann
c8f160068e check archives: improve error handling for corrupt archive metadata block
this is similar to #4777.

borg check must not crash if an archive metadata block does not decrypt.

Instead, report the archive_id, remove the archive from the manifest and skip to the next archive.
2022-04-12 17:45:48 +02:00
Thomas Waldmann
151fe5f348 check archive: make robust_iterator more robust, fixes #4777
borg check must not crash if an archive metadata chunk does not decrypt.

Instead, report the chunk and skip to the next one.
2022-04-12 17:45:17 +02:00
TW
70e30dea4d
Merge pull request #6558 from bcat/1.2-maint
(docs) 1.2: Recommend umask for passphrase file perms
2022-04-10 14:00:37 +02:00
Jonathan Rascher
4b58dafd31 (docs) Recommend umask for passphrase file perms
The previous sample for creating a ~/.borg-passphrase file creates it first and then chmod's it to 400 permissions. That's probably fine in practice, but means there's a tiny window where the passphrase file is sitting with default permissions (likely world readable, depending on the system umask).

It seems safer to first change the umask to remove all group & world bits (0077) _before_ creating the file. To be polite and avoid messing with the user's previous umask, we do this in a subshell. (Note that umask 0077 leads to a mode of 600 rather than the previous 400, because removing the owner write bit doesn't seem to buy much since the owner can just chmod the file anyway.)
2022-04-09 23:58:45 -05:00
TW
b602352d34
Merge pull request #6545 from ThomasWaldmann/add-pyproject.toml-1.2
add pyproject.toml, fix sys.path, fixes #6466
2022-04-07 21:27:07 +02:00
TW
aa55aef2b9
Merge pull request #6543 from ThomasWaldmann/fix-progress-archivename-1.2
escape % chars in archive name, fixes #6500
2022-04-07 20:22:06 +02:00
Thomas Waldmann
82a80bf760 add pyproject.toml, fix sys.path, fixes #6466
setup.py: add parent to sys.path

When using pyproject.toml the parent of setup.py is not on sys.path by default.

See <pypa/setuptools#3134>.
2022-04-07 18:48:48 +02:00
Thomas Waldmann
a4f7eeb179 escape % chars in archive name, fixes #6500
also: fix percentage format for float value.
2022-04-07 18:03:25 +02:00
TW
a7328edd58
Merge pull request #6532 from bket/1.2_fix_#2055
1.2 - fix OpenBSD symlink mode test failure (#2055)
2022-04-04 21:41:07 +02:00
TW
f5927b6b49
Merge pull request #6528 from ThomasWaldmann/fix-ipv6-url-parsing-1.2
fix scp repo url parsing for ip v6 addrs, fixes #6526
2022-04-04 20:39:28 +02:00
Björn Ketelaars
b467d50c35 1.2 - Fix OpenBSD symlink mode test failure (#2055)
OpenBSD does not have `lchmod()` causing `os.lchmod` to be unavailable
on this platform. As a result ArchiverTestCase::test_basic_functionality
fails when run manually (#2055).

OpenBSD does have `fchmodat()`, which has a flag that makes it behave
like `lchmod()`. In Python this can be used via `os.chmod(path, mode,
follow_symlinks=False)`.

As of Python 3.3 `os.lchmod(path, mode)` is equivalent to
`os.chmod(path, mode, follow_symlinks=False)`. As such, switching to the
latter is preferred as it enables more platforms to do the right thing.
2022-04-04 19:24:28 +02:00
Thomas Waldmann
40f1f9ddee use same host regex for ssh and scp style, refactor/clean up
although bug #6526 did not show with ssh style URLs, we should
not have different regexes for the host part for ssh and scp style.

thus i extracted the host_re from both and also cleaned up a bit.
2022-04-03 21:33:26 +02:00
Thomas Waldmann
e332acb334 fix scp repo url parsing for ip v6 addrs, fixes #6526
added a negative lookahead/lookbehind to make sure an ipv6 addr
(enclosed in square brackets) does not get badly matched by the
regex part intended for hostnames and ipv4 addrs only.

the other part of that regex which is actually intended to match
ipv6 addrs only matches if they are enclosed in square brackets.

also added tests for ssh and scp style repo URLs with ipv6 addrs
in brackets.

also: made regex more readable, putting these 2 cases on separate lines.
2022-04-03 21:30:28 +02:00
TW
e5e360d5b6
Merge pull request #6507 from ThomasWaldmann/update-security-support-1.2
borg 1.2 is security supported
2022-03-30 00:11:57 +02:00
Thomas Waldmann
017f9b2102 borg 1.2 is security supported 2022-03-29 23:55:25 +02:00
TW
28c7494455
Merge pull request #6503 from aspargas2/1.2-maint
1.2-maint backport: delete: add repository id and location to prompt
2022-03-28 18:42:36 +02:00
Thomas Deutschmann
821d814af3 delete: add repository id and location to prompt
Closes: https://github.com/borgbackup/borg/issues/6453
2022-03-28 10:16:53 -04:00
TW
b5443da6cb
Merge pull request #6493 from ThomasWaldmann/use-compare_digest-1.2
use hmac.compare_digest instead of ==, fixes #6470
2022-03-26 19:48:05 +01:00
TW
4ecf0d4d78
Merge pull request #6491 from ThomasWaldmann/crypto-fixes-1.2
avoid losing the key (old crypto)
2022-03-26 19:05:54 +01:00
Thomas Waldmann
b75079c0f8 use hmac.compare_digest instead of ==, fixes #6470 2022-03-26 18:52:41 +01:00
Thomas Waldmann
3e91694a0a avoid losing the key (old crypto)
if we just have a pointer to a bytes object which might go out of scope, we can lose it.

also: cython can directly assign a bytes object into a same-size char array.
2022-03-26 18:32:44 +01:00
TW
2e45e19e02
Merge pull request #6488 from ThomasWaldmann/update-ubuntu-link-1.2
docs: update link to ubuntu packages, fixes #6485
2022-03-26 18:20:54 +01:00
Thomas Waldmann
c14bc52189 docs: update link to ubuntu packages, fixes #6485 2022-03-26 18:07:03 +01:00
TW
df9322c4a9
Merge pull request #6475 from bket/unused-code-1.2
1.2 - Remove unused code
2022-03-21 19:51:09 +01:00
Björn Ketelaars
71fa056c47 1.2 - Remove unused code
#6473 Removed AEAD ciphers AES-OCB and CHACHA20-POLY1305 from 1.2. As a
result some additional code can be removed.

Passes regression tests.
2022-03-21 18:54:50 +01:00
TW
87f0b9803b
Merge pull request #6473 from ThomasWaldmann/remove-aead-1.2
remove AEAD ciphers AES-OCB and CHACHA20-POLY1305, fixes #6472
2022-03-20 23:44:40 +01:00
Thomas Waldmann
aa57a3cb57 remove AEAD ciphers AES-OCB and CHACHA20-POLY1305, fixes #6472
not used in 1.2.x, we'll start using them in 1.3.
2022-03-20 23:05:30 +01:00
TW
d41908166f
Merge pull request #6435 from hexagonrecursion/1-2-fallthrough
1.2: Make switch fallthrough explicit
2022-03-10 12:04:12 +01:00
TW
e25d6c8c4b
Merge pull request #6436 from hexagonrecursion/1-2-numeric
1.2: use --numeric-ids in pull doc
2022-03-10 11:55:42 +01:00
Michael Bauer
1345f06f2b use --numeric-ids in pull doc
replace flag --numeric-owner with --numeric-ids
2022-03-10 08:34:27 +03:00
Andrey Bienkowski
a58fd3db5c 1.2: Make switch fallthrough explicit 2022-03-10 08:24:23 +03:00
TW
791720407b
Merge pull request #6432 from hexagonrecursion/1-2-cflags
1.2: Allow extra compiler flags for every extension build
2022-03-10 00:13:44 +01:00
James Buren
793d543328 Allow extra compiler flags for every extension build
This is mainly intended for explicit warnings but it can be
used for other flags as well.

Conflicts:
	setup.py
2022-03-10 00:50:59 +03:00
TW
f698c12129
Merge pull request #6403 from ThomasWaldmann/fix-savefile-mode-1.2
respect umask for files / dirs (1.2)
2022-03-09 16:53:35 +01:00
braewoods
8b4c501d8d
[1.2-maint] Backport implicit warning patches from master (#6427)
* suppress -Wimplicit-fallthrough warning

These instances of implicit switch case fallthrough appear to be
intentional. Add comments that the compiler understands to suppress
the false positive warning.
2022-03-09 16:10:21 +01:00
TW
450610b168
Merge pull request #6423 from ThomasWaldmann/fix-cython-warnings-1.2
fix "useless trailing comma" cython warnings
2022-03-08 22:19:47 +01:00
Thomas Waldmann
dc811bda6f fix "'HMAC_CTX' already defined" cython warning 2022-03-08 21:47:14 +01:00
Thomas Waldmann
cd629d32cb fix "useless trailing comma" cython warnings 2022-03-08 21:27:20 +01:00
TW
7a1d12d007
Merge pull request #6418 from hexagonrecursion/bp12-unicode
1.2-maint: ‘PyUnicode_AsUnicode’ is deprecated
2022-03-07 20:14:15 +01:00
Andrey Bienkowski
f05036cbcb ‘PyUnicode_AsUnicode’ is deprecated
Fix compilation warnings:
```
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9 -c src/borg/platform/posix.c -o build/temp.linux-x86_64-3.9/src/borg/platform/posix.o
src/borg/platform/posix.c: In function ‘__pyx_pf_4borg_8platform_5posix_2swidth’:
src/borg/platform/posix.c:1572:3: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations]
 1572 |   __pyx_t_2 = __Pyx_PyUnicode_AsUnicode(__pyx_v_s); if (unlikely((!__pyx_t_2) && PyErr_Occurred())) __PYX_ERR(0, 19, __pyx_L1_error)
      |   ^~~~~~~~~
In file included from /opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9/unicodeobject.h:1026,
                 from /opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9/Python.h:93,
                 from src/borg/platform/posix.c:19:
/opt/hostedtoolcache/Python/3.9.10/x64/include/python3.9/cpython/unicodeobject.h:580:45: note: declared here
  580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode(
      |                                             ^~~~~~~~~~~~~~~~~~~
```

Ref: https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_AsUnicode
2022-03-07 21:40:39 +03:00
TW
20a9df18f7
Merge pull request #6413 from aspargas2/1.2-maint
Backport: fix codeql complaining about missing headers needed by borg
2022-03-07 17:59:53 +01:00
aspargas2
2425f54bdb tweak what is backported, as requested
- don't backport `pyproject.toml` or related changes
- install some additional -dev packages with `apt-get` for CodeQL
- run CodeQL in a python venv
2022-03-07 10:14:22 -05:00
TW
bb76dca114
Merge pull request #6414 from hexagonrecursion/bp-blake
1.2-maint: hmac and blake2b minor optimizations and cleanups
2022-03-06 14:39:51 +01:00
Thomas Waldmann
ce72a87f2d fixup: fix SELFTEST_COUNT 2022-03-06 11:31:05 +03:00
Thomas Waldmann
65ccbc2c69 fixup: remove now unused import 2022-03-06 11:31:05 +03:00
Thomas Waldmann
8dd2612fc6 remove the blake2b compat checker script
this was used to compare compatibility of our vendored
blake2b code (which we do not have any more) against the
python stdlib blake2b code (which we always use now anyway).
2022-03-06 11:31:05 +03:00
Thomas Waldmann
45ef108d46 remove our blake2b_128/256 tests
as we're just calling python stdlib there,
we can assume that they have tests for it.

 Conflicts:
	src/borg/testsuite/crypto.py
2022-03-06 11:30:58 +03:00
Thomas Waldmann
c2211a7ef9 remove our hmac_sha256 tests
as we're just calling python stdlib there,
we can assume that they have tests for it.

 Conflicts:
	src/borg/testsuite/crypto.py
2022-03-06 11:28:27 +03:00
Thomas Waldmann
72ccafb4b8 hmac.digest: some more minor optimizations
also some cosmetic changes:
- import hmac module
- use hmac.compare_digest
2022-03-06 11:26:04 +03:00
Thomas Waldmann
bcd1d49d3b hmac_sha256: replace own cython wrapper code by hmac.digest python stdlib (since py38)
i measured performance of both: pretty much the same.
2022-03-06 11:26:04 +03:00
Thomas Waldmann
4fb6df813a fix codeql complaining about missing headers needed by borg
thus:
- install requirements first
- codeql init
- build borg
- codeql analyzse
2022-03-05 23:28:37 -05:00
aspargas2
aa27294927 fix codeql complaining about missing cython
backports commits 6cbb7d6, 142bb0e, 4df8255, and 30c145e
2022-03-05 23:21:20 -05:00
TW
d0fb234f46
Merge pull request #6404 from ThomasWaldmann/import-tar-fixes-1.2
import-tar fixes
2022-03-05 14:26:59 +01:00
TW
8abb0aa7ab
Merge pull request #6375 from ThomasWaldmann/fix-6374-1.2
create: skip with warning if opening the parent dir of recursion root fails, fixes #6374
2022-03-05 14:18:40 +01:00
Thomas Waldmann
7dc800fecd kill filter process in case of borg exceptions, fixes #6401
in the finally-block, we wait for the filter process to die. but it only dies
voluntarily if all data was processed by the filter and it terminates due to EOF.

otoh, if borg has thrown an early exception, e.g. "archive already exists",
we need to kill the filter process to bring it to an early end. in that
case, we also do not need to check the filter rc, because we know we killed it.
2022-03-04 23:33:53 +01:00
Thomas Waldmann
33192e5e5d import-tar: fix mtime type bug
looks like with a .tar file created by the tar tool,
tarinfo.mtime is a float [s]. So, after converting to
nanoseconds, we need to cast to int because that's what
Item.mtime wants.

also added a safe_ns() there to clip values to the safe range.
2022-03-04 23:33:53 +01:00
TW
ea668417c9
Merge pull request #6405 from ThomasWaldmann/fix-ci
fix wrong branch name in github actions
2022-03-04 23:32:47 +01:00
Thomas Waldmann
412e39b898 fix wrong branch name in github actions 2022-03-04 22:50:23 +01:00
Thomas Waldmann
f5a47d86e3 ensure_dir: respect umask for created directory modes, fixes #6400
we tried to be very private / secure here, but that created the issue
that a less secure umask (like e.g. 0o007) just did not work.

to make the umask work, we must start from 0o777 mode and let the
umask do its work, like e.g. 0o777 & ~0o007 --> 0o770.

with borg's default umask of 0o077, it usually ends up being 0o700,
so only permissions for the user (not group, not others).
2022-03-04 21:24:42 +01:00
Thomas Waldmann
9ea1d2a7d6 SaveFile: respect umask for final file mode, fixes #6400 2022-03-04 20:53:10 +01:00
TW
057ae69537
Merge pull request #6384 from ThomasWaldmann/pyupgrade-38-1.2
run pyupgrade (1.2-maint)
2022-02-27 22:52:43 +01:00
Thomas Waldmann
8edb1dc085 manual formatting fixes (cosmetic / pep8) 2022-02-27 21:13:04 +01:00
Thomas Waldmann
c4ed840c7b pyupgrade --py38-plus ./**/*.py 2022-02-27 21:12:44 +01:00
TW
6c243b2c28
Merge pull request #6381 from ThomasWaldmann/remove-blake2-docs-1.2
remove blake2 docs, blake2 code not bundled any more, fixes #6371
2022-02-27 14:44:11 +01:00
Thomas Waldmann
51a676cc6f remove blake2 docs, blake2 code not bundled any more, fixes #6371 2022-02-27 14:43:27 +01:00
TW
97de7e1fe2
Merge pull request #6380 from ThomasWaldmann/fix-zstd-include-1.2
fix zstd.h include for bundled zstd, fixes #6369
2022-02-27 14:39:09 +01:00
Thomas Waldmann
9390d2712b fix zstd.h include for bundled zstd, fixes #6369 2022-02-27 14:35:19 +01:00
Thomas Waldmann
da0ddacc54 create: skip with warning if opening the parent dir of recursion root fails, fixes #6374 2022-02-27 02:40:15 +01:00
TW
c2b76adb19
Merge pull request #6373 from braewoods/warnings
[1.2-maint] fix multiple warnings related to _hashindex.c
2022-02-26 23:42:22 +01:00
James Buren
1618c98975 src/borg/cache_sync/unpack.h: fix compiler warnings
The key argument being sent to hashindex_get and hashindex_set by
multiple functions is a different signedness from what the functions
expect. This resolves the issue by changing the key type in the
unpack_user struct to unsigned char.
2022-02-26 16:13:07 -06:00
James Buren
4427c16d31 src/borg/_hashindex.c: fix compiler warnings
The value argument of hashindex_set is causing harmless pointer type
mismatches. This resolves the issue by changing the type to void*
which silences these types of warnings.
2022-02-26 16:13:02 -06:00
TW
d88d1674b7
Merge pull request #6367 from braewoods/warnings
[1.2-maint] src/borg/compress.pyx: fix compiler warning, closes #6365
2022-02-26 20:42:04 +01:00
James Buren
8e640ae0f3 src/borg/compress.pyx: fix compiler warning, closes #6365
This resolves a compiler warning from the generated code that
resulted from a comparison of two local variables of different
signedness. The issue is resolved by changing the type of both
to int since this seems like the safest choice available.
2022-02-26 13:20:39 -06:00
TW
c6f284d43c
Merge pull request #6364 from braewoods/warnings
[1.2-maint] src/borg/crypto/low_level.pyx: fix compiler warning
2022-02-26 00:40:55 +01:00
James Buren
067dd10027 src/borg/crypto/low_level.pyx: fix compiler warning
The generated source code was producing a compiler warning due to
the pointers differing in constness. The called function expects
a non-const pointer while the generated code produces a const pointer
via a cast. This changes the cast to drop 'const' to make the compiler
happy.
2022-02-25 17:36:48 -06:00
Thomas Waldmann
f81c62f894 vagrant: darwin64: remove fakeroot, fixes #6314
deprecated / unsupported / 404.
2022-02-25 23:48:03 +01:00
TW
0fb546978a
Merge pull request #6358 from hexagonrecursion/apt-get-1.2
s/apt/apt-get/ (1.2)
2022-02-25 21:02:39 +01:00
TW
04712a76dc
Merge pull request #6359 from hexagonrecursion/order-1.2
docs: Clarify on-disk order and size of log entry fields (1.2)
2022-02-25 20:50:59 +01:00
Julian Picht
e49a1b5d9b docs: Clarify on-disk order and size of log entry fields (#6357)
docs: clarify on-disk order and size of log entry fields

The order of the fields of a log entry on disk is CRC32 first, the docs had the size first.

I tried to make this list similar to the HashIndex struct description.
2022-02-25 15:03:21 +03:00
Andrey Bienkowski
29ce1fc818 s/apt/apt-get/
> WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
2022-02-25 14:44:11 +03:00
TW
6f18167873
Merge pull request #6348 from ThomasWaldmann/nodashes-1.2
docs: do not transform --/--- to unicode dashes
2022-02-23 20:58:44 +01:00
Thomas Waldmann
f5116a7e67 docs: do not transform --/--- to unicode dashes 2022-02-23 20:08:27 +01:00
352 changed files with 20891 additions and 90919 deletions

View file

@ -1,27 +0,0 @@
version: '{build}'
environment:
matrix:
- PYTHON: C:\Python38-x64
# Disable automatic builds
build: off
# Build artifacts: all wheel and exe files in the dist folder
artifacts:
- path: 'dist\*.whl'
- path: 'dist\*.exe'
install:
- ps: scripts\win-download-openssl.ps1
- ps: |
& $env:PYTHON\python.exe -m venv borg-env
borg-env\Scripts\activate.ps1
python -m pip install -U pip
pip install -r requirements.d/development.txt
pip install wheel pyinstaller
build_script:
- ps: |
borg-env\Scripts\activate.ps1
scripts\win-build.ps1

View file

@ -1,41 +0,0 @@
[all]
# note: put developer specific settings into ~/.coarc (e.g. editor = ...)
max_line_length = 255
use_spaces = True
ignore = src/borg/(chunker.c|compress.c|hashindex.c|item.c),
src/borg/algorithms/(checksums.c|crc32*.c),
src/borg/algorithms/blake2/**,
src/borg/algorithms/lz4/**,
src/borg/algorithms/xxh64/**,
src/borg/algorithms/zstd/**,
src/borg/crypto/low_level.c,
src/borg/platform/*.c
[all.general]
files = src/borg/**/*.(py|pyx|c)
bears = SpaceConsistencyBear, FilenameBear, InvalidLinkBear, LineLengthBear
file_naming_convention = snake
[all.python]
files = src/borg/**/*.py
bears = PEP8Bear, PyDocStyleBear, PyLintBear
pep_ignore = E123,E125,E126,E127,E128,E226,E301,E309,E402,F401,F405,F811,W690
pylint_disable = C0103, C0111, C0112, C0122, C0123, C0301, C0302, C0325, C0330, C0411, C0412, C0413, C1801,
I1101,
W0102, W0104, W0106, W0108, W0120, W0201, W0212, W0221, W0231, W0401, W0404,
W0511, W0603, W0611, W0612, W0613, W0614, W0621, W0622, W0640, W0702, W0703,
W1201, W1202, W1401,
R0101, R0201, R0204, R0901, R0902, R0903, R0904, R0911, R0912, R0913, R0914, R0915,
R0916, R1701, R1704, R1705, R1706, R1710,
E0102, E0202, E0401, E0601, E0611, E0702, E1101, E1102, E1120, E1129, E1130
pydocstyle_ignore = D100, D101, D102, D103, D104, D105, D200, D201, D202, D203, D204, D205, D209, D210,
D212, D213, D300, D301, D400, D401, D402, D403, D404
[all.c]
files = src/borg/**/*.c
bears = CPPCheckBear
[all.html]
files = src/borg/**/*.html
bears = HTMLLintBear
htmllint_ignore = *

View file

@ -1,24 +0,0 @@
[run]
branch = True
disable_warnings = module-not-measured
source = src/borg
omit =
*/borg/__init__.py
*/borg/__main__.py
*/borg/_version.py
*/borg/fuse.py
*/borg/support/*
*/borg/testsuite/*
*/borg/hash_sizes.py
[report]
exclude_lines =
pragma: no cover
pragma: freebsd only
pragma: unknown platform only
def __repr__
raise AssertionError
raise NotImplementedError
if 0:
if __name__ == .__main__.:
ignore_errors = True

View file

@ -1,56 +1,55 @@
<!--
Thank you for reporting an issue.
*IMPORTANT* - *before* creating a new issue please look around:
- Borgbackup documentation: http://borgbackup.readthedocs.io/en/stable/index.html
*IMPORTANT* before creating a new issue, please look around:
- BorgBackup documentation: https://borgbackup.readthedocs.io/en/stable/index.html
- FAQ: https://borgbackup.readthedocs.io/en/stable/faq.html
and
- open issues in Github tracker: https://github.com/borgbackup/borg/issues
- Open issues in the GitHub tracker: https://github.com/borgbackup/borg/issues
If you cannot find a similar problem, then create a new issue.
Please fill in as much of the template as possible.
-->
## Have you checked borgbackup docs, FAQ, and open Github issues?
## Have you checked the BorgBackup docs, FAQ, and open GitHub issues?
No
## Is this a BUG / ISSUE report or a QUESTION?
## Is this a bug/issue report or a question?
Invalid
Bug/Issue/Question
## System information. For client/server mode post info for both machines.
## System information. For client/server mode, post info for both machines.
#### Your borg version (borg -V).
#### Operating system (distribution) and version.
#### Hardware / network configuration, and filesystems used.
#### Hardware/network configuration and filesystems used.
#### How much data is handled by borg?
#### Full borg commandline that lead to the problem (leave away excludes and passwords)
#### Full borg command line that led to the problem (leave out excludes and passwords).
## Describe the problem you're observing.
#### Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
#### Include any warning/errors/backtraces from the system logs
#### Include any warnings/errors/backtraces from the system logs
<!--
If this complaint relates to borg performance, please include CRUD benchmark
results and any steps you took to troubleshoot.
How to run benchmark: http://borgbackup.readthedocs.io/en/stable/usage/benchmark.html
How to run the benchmark: https://borgbackup.readthedocs.io/en/stable/usage/benchmark.html
*IMPORTANT* - Please mark logs and text output from terminal commands
or else Github will not display them correctly.
*IMPORTANT* Please mark logs and terminal command output, otherwise GitHub will not display them correctly.
An example is provided below.
Example:
```
this is an example how log text should be marked (wrap it with ```)
this is an example of how log text should be marked (wrap it with ```)
```
-->

View file

@ -1,8 +0,0 @@
Thank you for contributing code to Borg, your help is appreciated!
Please, before you submit a pull request, make sure it complies with the
guidelines given in our documentation:
https://borgbackup.readthedocs.io/en/latest/development.html#contributions
**Please remove all above text before submitting your pull request.**

18
.github/PULL_REQUEST_TEMPLATE.md vendored Normal file
View file

@ -0,0 +1,18 @@
<!--
Thank you for contributing to BorgBackup!
Please make sure your PR complies with our contribution guidelines:
https://borgbackup.readthedocs.io/en/latest/development.html#contributions
-->
## Description
<!-- What does this PR do? Reference any related issues with "fixes #XXXX". -->
## Checklist
- [ ] PR is against `master` (or maintenance branch if only applicable there)
- [ ] New code has tests and docs where appropriate
- [ ] Tests pass (run `tox` or the relevant test subset)
- [ ] Commit messages are clean and reference related issues

View file

@ -1,28 +1,32 @@
# badge: https://github.com/borgbackup/borg/workflows/CI/badge.svg?branch=master
# badge: https://github.com/borgbackup/borg/workflows/CI/badge.svg?branch=1.4-maint
name: CI
on:
push:
branches: [ master ]
branches: [ 1.4-maint ]
tags:
- '1.*'
paths:
- '**.py'
- '**.pyx'
- '**.c'
- '**.h'
- '**.yml'
- '**.toml'
- '**.cfg'
- '**.ini'
- 'requirements.d/*'
- '!docs/**'
pull_request:
branches: [ master ]
branches: [ 1.4-maint ]
paths:
- '**.py'
- '**.pyx'
- '**.c'
- '**.h'
- '**.yml'
- '**.toml'
- '**.cfg'
- '**.ini'
- 'requirements.d/*'
@ -31,58 +35,136 @@ on:
jobs:
lint:
runs-on: ubuntu-latest
timeout-minutes: 10
runs-on: ubuntu-22.04
timeout-minutes: 5
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
- uses: actions/checkout@v4
- uses: chartboost/ruff-action@v1
asan_ubsan:
runs-on: ubuntu-24.04
timeout-minutes: 25
needs: [lint]
steps:
- uses: actions/checkout@v4
with:
python-version: 3.8
- name: Lint with flake8
# Just fetching one commit is not enough for setuptools-scm, so we fetch all.
fetch-depth: 0
fetch-tags: true
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install system packages
run: |
pip install flake8
flake8 src scripts conftest.py
sudo apt-get update
sudo apt-get install -y pkg-config build-essential
sudo apt-get install -y libssl-dev libacl1-dev libxxhash-dev liblz4-dev libzstd-dev
pytest:
- name: Install Python dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.d/development.txt
needs: lint
- name: Build Borg with ASan/UBSan
# Build the C/Cython extensions with AddressSanitizer and UndefinedBehaviorSanitizer enabled.
# How this works:
# - The -fsanitize=address,undefined flags inject runtime checks into our native code. If a bug is hit
# (e.g., buffer overflow, use-after-free, out-of-bounds, or undefined behavior), the sanitizer prints
# a detailed error report to stderr, including a stack trace, and forces the process to exit with
# non-zero status. In CI, this will fail the step/job so you will notice.
# - ASAN_OPTIONS/UBSAN_OPTIONS configure the sanitizers' runtime behavior (see below for meanings).
env:
CFLAGS: "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined"
CXXFLAGS: "-O1 -g -fno-omit-frame-pointer -fsanitize=address,undefined"
LDFLAGS: "-fsanitize=address,undefined"
# ASAN_OPTIONS controls AddressSanitizer runtime tweaks:
# - detect_leaks=0: Disable LeakSanitizer to avoid false positives with CPython/pymalloc in short-lived tests.
# - strict_string_checks=1: Make invalid string operations (e.g., over-reads) more likely to be detected.
# - check_initialization_order=1: Catch uses that depend on static initialization order (C++).
# - detect_stack_use_after_return=1: Detect stack-use-after-return via stack poisoning (may increase overhead).
ASAN_OPTIONS: "detect_leaks=0:strict_string_checks=1:check_initialization_order=1:detect_stack_use_after_return=1"
# UBSAN_OPTIONS controls UndefinedBehaviorSanitizer runtime:
# - print_stacktrace=1: Include a stack trace for UB reports to ease debugging.
# Note: UBSan is recoverable by default (process may continue after reporting). If you want CI to
# abort immediately and fail on the first UB, add `halt_on_error=1` (e.g., UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1").
UBSAN_OPTIONS: "print_stacktrace=1"
# PYTHONDEVMODE enables additional Python runtime checks and warnings.
PYTHONDEVMODE: "1"
run: pip install -e .
- name: Run tests under sanitizers
env:
ASAN_OPTIONS: "detect_leaks=0:strict_string_checks=1:check_initialization_order=1:detect_stack_use_after_return=1"
UBSAN_OPTIONS: "print_stacktrace=1"
PYTHONDEVMODE: "1"
# Ensure the ASan runtime is loaded first to avoid "ASan runtime does not come first" warnings.
# We discover libasan/libubsan paths via gcc and preload them for the Python test process.
# the remote tests are slow and likely won't find anything useful
run: |
set -euo pipefail
export LD_PRELOAD="$(gcc -print-file-name=libasan.so):$(gcc -print-file-name=libubsan.so)"
echo "Using LD_PRELOAD=$LD_PRELOAD"
pytest -v --benchmark-skip -k "not remote"
native_tests:
needs: [lint]
permissions:
contents: read
id-token: write
attestations: write
strategy:
matrix:
include:
- os: ubuntu-20.04
python-version: '3.8'
toxenv: py38-fuse2
- os: ubuntu-20.04
python-version: '3.9'
toxenv: py39-fuse3
- os: ubuntu-20.04
python-version: '3.10'
toxenv: py310-fuse3
- os: macos-10.15 # macos-latest is macos 11.6.2 and hanging at test_fuse, #6099
python-version: '3.8'
toxenv: py38-fuse2
fail-fast: false
# noinspection YAMLSchemaValidation
matrix: >-
${{ fromJSON(
github.event_name == 'pull_request' && '{
"include": [
{"os": "ubuntu-22.04", "python-version": "3.10", "toxenv": "py310-fuse2"},
{"os": "ubuntu-24.04", "python-version": "3.14", "toxenv": "py314-fuse3"}
]
}' || '{
"include": [
{"os": "ubuntu-22.04", "python-version": "3.10", "toxenv": "py310-fuse2"},
{"os": "ubuntu-22.04", "python-version": "3.11", "toxenv": "py311-fuse3", "binary": "borg-linux-glibc235-x86_64-gh"},
{"os": "ubuntu-22.04-arm", "python-version": "3.11", "toxenv": "py311-fuse3", "binary": "borg-linux-glibc235-arm64-gh"},
{"os": "ubuntu-24.04", "python-version": "3.12", "toxenv": "py312-fuse3"},
{"os": "ubuntu-24.04", "python-version": "3.13", "toxenv": "py313-fuse3"},
{"os": "ubuntu-24.04", "python-version": "3.14", "toxenv": "py314-fuse3"},
{"os": "macos-15-intel", "python-version": "3.11", "toxenv": "py311-none", "binary": "borg-macos-15-x86_64-gh"},
{"os": "macos-15", "python-version": "3.11", "toxenv": "py311-none", "binary": "borg-macos-15-arm64-gh"}
]
}'
) }}
env:
# Configure pkg-config to use OpenSSL from Homebrew
PKG_CONFIG_PATH: /usr/local/opt/openssl@1.1/lib/pkgconfig
TOXENV: ${{ matrix.toxenv }}
runs-on: ${{ matrix.os }}
timeout-minutes: 40
# macOS machines can be slow, if overloaded.
timeout-minutes: 360
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
with:
# just fetching 1 commit is not enough for setuptools-scm, so we fetch all
# Just fetching one commit is not enough for setuptools-scm, so we fetch all
fetch-depth: 0
fetch-tags: true
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip
uses: actions/cache@v2
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('requirements.d/development.txt') }}
@ -95,38 +177,286 @@ jobs:
run: |
sudo apt-get update
sudo apt-get install -y libssl-dev libacl1-dev liblz4-dev libzstd-dev pkg-config build-essential
sudo apt-get install -y libxxhash-dev || true
sudo apt-get install -y libb2-dev || true
sudo apt-get install -y libfuse-dev fuse || true # Required for Python llfuse module
sudo apt-get install -y libfuse3-dev fuse3 || true # Required for Python pyfuse3 module
sudo apt-get install -y libxxhash-dev
if [[ "$TOXENV" == *"fuse2"* ]]; then
sudo apt-get install -y libfuse-dev fuse # Required for Python llfuse module
elif [[ "$TOXENV" == *"fuse3"* ]]; then
sudo apt-get install -y libfuse3-dev fuse3 # Required for Python pyfuse3 module
fi
- name: Install macOS packages
if: ${{ runner.os == 'macOS' }}
run: |
brew install pkg-config || brew upgrade pkg-config
brew install zstd || brew upgrade zstd
brew install lz4 || brew upgrade lz4
brew install openssl@1.1 || brew upgrade openssl@1.1
brew install --cask macfuse || brew upgrade --cask macfuse # Required for Python llfuse module
run: brew bundle install
- name: Install Python requirements
run: |
python -m pip install --upgrade pip setuptools wheel
pip install -r requirements.d/development.txt
- name: Install borgbackup
- name: Install BorgBackup
run: |
# pip install -e .
python setup.py -v develop
- name: run pytest via tox
if [[ "$TOXENV" == *"fuse2"* ]]; then
pip install -ve ".[llfuse]"
elif [[ "$TOXENV" == *"fuse3"* ]]; then
pip install -ve ".[pyfuse3]"
else
pip install -ve .
fi
- name: Build Borg fat binaries (${{ matrix.binary }})
if: ${{ matrix.binary && startsWith(github.ref, 'refs/tags/') }}
run: |
# do not use fakeroot, but run as root. avoids the dreaded EISDIR sporadic failures. see #2482.
pip install 'pyinstaller==6.14.2'
mkdir -p dist/binary
# Ensure locally built binaries in ./dist/binary are found during tox tests
echo "$GITHUB_WORKSPACE/dist/binary" >> "$GITHUB_PATH"
pyinstaller --clean --distpath=dist/binary scripts/borg.exe.spec
- name: Smoke-test the built binary (${{ matrix.binary }})
if: ${{ matrix.binary && startsWith(github.ref, 'refs/tags/') }}
run: |
pushd dist/binary
echo "single-file binary"
chmod +x borg.exe
./borg.exe -V
echo "single-directory binary"
chmod +x borg-dir/borg.exe
./borg-dir/borg.exe -V
tar czf borg.tgz borg-dir
popd
echo "borg.exe binary in PATH"
borg.exe -V
- name: Prepare binaries (${{ matrix.binary }})
if: ${{ matrix.binary && startsWith(github.ref, 'refs/tags/') }}
run: |
mkdir -p artifacts
if [ -f dist/binary/borg.exe ]; then
cp dist/binary/borg.exe artifacts/${{ matrix.binary }}
fi
if [ -f dist/binary/borg.tgz ]; then
cp dist/binary/borg.tgz artifacts/${{ matrix.binary }}.tgz
fi
echo "binary files"
ls -l artifacts/
- name: Attest binaries provenance (${{ matrix.binary }})
if: ${{ matrix.binary && startsWith(github.ref, 'refs/tags/') }}
uses: actions/attest-build-provenance@v3
with:
subject-path: 'artifacts/*'
- name: Upload binaries (${{ matrix.binary }})
if: ${{ matrix.binary && startsWith(github.ref, 'refs/tags/') }}
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.binary }}
path: artifacts/*
if-no-files-found: error
- name: Run pytest via tox
run: |
# Do not use fakeroot; run as root. Avoids the dreaded sporadic EISDIR failures; see #2482.
#sudo -E bash -c "tox -e py"
tox --skip-missing-interpreters
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v1
uses: codecov/codecov-action@v4
env:
OS: ${{ runner.os }}
python: ${{ matrix.python-version }}
with:
token: ${{ secrets.CODECOV_TOKEN }}
env_vars: OS, python
vm_tests:
# Cross-OS tests running inside VMs, aligned with master branch structure.
# Uses cross-platform-actions/action to run on FreeBSD, NetBSD, OpenBSD, Haiku.
permissions:
contents: read
id-token: write
attestations: write
runs-on: ubuntu-24.04
timeout-minutes: 90
needs: [lint]
continue-on-error: true
strategy:
fail-fast: false
matrix:
include:
- os: freebsd
version: '14.3'
display_name: FreeBSD
# Controls binary build and provenance attestation on tags
do_binaries: true
artifact_prefix: borg-freebsd-14-x86_64-gh
- os: netbsd
version: '10.1'
display_name: NetBSD
do_binaries: false
- os: openbsd
version: '7.7'
display_name: OpenBSD
do_binaries: false
- os: haiku
version: 'r1beta5'
display_name: Haiku
do_binaries: false
steps:
- name: Check out repository
uses: actions/checkout@v4
with:
fetch-depth: 0
fetch-tags: true
- name: Test on ${{ matrix.display_name }}
id: cross_os
uses: cross-platform-actions/action@v0.29.0
env:
DO_BINARIES: ${{ matrix.do_binaries }}
with:
operating_system: ${{ matrix.os }}
version: ${{ matrix.version }}
shell: bash
run: |
set -euxo pipefail
case "${{ matrix.os }}" in
freebsd)
# Ensure a proper hostname/FQDN is set (VMs may not have one by default)
sudo -E /bin/sh -c 'grep -q "freebsd\.local" /etc/hosts || echo "127.0.0.1 freebsd.local freebsd" >> /etc/hosts'
sudo -E hostname freebsd.local
hostname
export IGNORE_OSVERSION=yes
sudo -E pkg update -f
sudo -E pkg install -y xxhash liblz4 zstd pkgconf
sudo -E pkg install -y fusefs-libs
sudo -E kldload fusefs
sudo -E sysctl vfs.usermount=1
sudo -E chmod 666 /dev/fuse
sudo -E pkg install -y rust
sudo -E pkg install -y git
sudo -E pkg install -y python310 py310-sqlite3
sudo -E pkg install -y python311 py311-sqlite3 py311-pip py311-virtualenv
sudo ln -sf /usr/local/bin/python3.11 /usr/local/bin/python3
sudo ln -sf /usr/local/bin/python3.11 /usr/local/bin/python
sudo ln -sf /usr/local/bin/pip3.11 /usr/local/bin/pip3
sudo ln -sf /usr/local/bin/pip3.11 /usr/local/bin/pip
python -m venv .venv
. .venv/bin/activate
python -V
pip -V
python -m pip install --upgrade pip wheel
pip install -r requirements.d/development.txt
pip install -e ".[llfuse]"
if [[ "${{ matrix.do_binaries }}" == "true" && "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
python -m pip install 'pyinstaller==6.14.2'
mkdir -p dist/binary
pyinstaller --clean --distpath=dist/binary scripts/borg.exe.spec
pushd dist/binary
echo "single-file binary"
chmod +x borg.exe
./borg.exe -V
echo "single-directory binary"
chmod +x borg-dir/borg.exe
./borg-dir/borg.exe -V
tar czf borg.tgz borg-dir
popd
mkdir -p artifacts
if [ -f dist/binary/borg.exe ]; then
cp -v dist/binary/borg.exe artifacts/${{ matrix.artifact_prefix }}
fi
if [ -f dist/binary/borg.tgz ]; then
cp -v dist/binary/borg.tgz artifacts/${{ matrix.artifact_prefix }}.tgz
fi
echo "binary files"
ls -l artifacts/
fi
export PATH="$(pwd)/dist/binary:$PATH"
tox -e py311-fuse2
;;
netbsd)
# Ensure a proper hostname/FQDN is set (VMs may not have one by default)
sudo -E /bin/sh -c 'grep -q "netbsd\.local" /etc/hosts || echo "127.0.0.1 netbsd.local netbsd" >> /etc/hosts'
sudo -E hostname netbsd.local
hostname
arch="$(uname -m)"
sudo -E mkdir -p /usr/pkg/etc/pkgin
echo "http://ftp.NetBSD.org/pub/pkgsrc/packages/NetBSD/${arch}/10.1/All" | sudo tee /usr/pkg/etc/pkgin/repositories.conf > /dev/null
sudo -E pkgin update
sudo -E pkgin -y upgrade
sudo -E pkgin -y install zstd lz4 xxhash git
sudo -E pkgin -y install rust
sudo -E pkgin -y install pkg-config
sudo -E pkgin -y install py311-pip py311-virtualenv py311-tox
sudo -E ln -sf /usr/pkg/bin/python3.11 /usr/pkg/bin/python3
sudo -E ln -sf /usr/pkg/bin/pip3.11 /usr/pkg/bin/pip3
sudo -E ln -sf /usr/pkg/bin/virtualenv-3.11 /usr/pkg/bin/virtualenv3
sudo -E ln -sf /usr/pkg/bin/tox-3.11 /usr/pkg/bin/tox3
# Ensure base system admin tools are on PATH for the non-root shell
export PATH="/sbin:/usr/sbin:$PATH"
echo "--- Preparing an extattr-enabled filesystem ---"
# On many NetBSD setups /tmp is tmpfs without extended attributes.
# Create a FFS image with extended attributes enabled and use it for TMPDIR.
VNDDEV="vnd0"
IMGFILE="/tmp/fs.img"
sudo -E dd if=/dev/zero of=${IMGFILE} bs=1m count=1024
sudo -E vndconfig -c "${VNDDEV}" "${IMGFILE}"
sudo -E newfs -O 2ea /dev/r${VNDDEV}a
MNT="/mnt/eafs"
sudo -E mkdir -p ${MNT}
sudo -E mount -t ffs -o extattr /dev/${VNDDEV}a $MNT
export TMPDIR="${MNT}/tmp"
sudo -E mkdir -p ${TMPDIR}
sudo -E chmod 1777 ${TMPDIR}
touch ${TMPDIR}/testfile
lsextattr user ${TMPDIR}/testfile && echo "[xattr] *** xattrs SUPPORTED on ${TMPDIR}! ***"
tox3 -e py311-none
;;
openbsd)
sudo -E pkg_add xxhash lz4 zstd git
sudo -E pkg_add rust
sudo -E pkg_add openssl%3.4
sudo -E pkg_add py3-pip py3-virtualenv py3-tox
export BORG_OPENSSL_NAME=eopenssl34
tox -e py312-none
;;
haiku)
pkgman refresh
pkgman install -y git pkgconfig zstd lz4 xxhash
pkgman install -y openssl3
pkgman install -y rust_bin
pkgman install -y python3.10
pkgman install -y cffi
pkgman install -y lz4_devel zstd_devel xxhash_devel openssl3_devel libffi_devel
# there is no pkgman package for tox, so we install it into a venv
python3 -m ensurepip --upgrade
python3 -m pip install --upgrade pip wheel
python3 -m venv .venv
. .venv/bin/activate
export PKG_CONFIG_PATH="/system/develop/lib/pkgconfig:/system/lib/pkgconfig:${PKG_CONFIG_PATH:-}"
export BORG_LIBLZ4_PREFIX=/system/develop
export BORG_LIBZSTD_PREFIX=/system/develop
export BORG_LIBXXHASH_PREFIX=/system/develop
export BORG_OPENSSL_PREFIX=/system/develop
pip install -r requirements.d/development.txt
pip install -e .
# troubles with either tox or pytest xdist, so we run pytest manually:
pytest -v -rs --benchmark-skip -k "not remote and not socket"
;;
esac
- name: Upload artifacts
if: startsWith(github.ref, 'refs/tags/') && matrix.do_binaries
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.artifact_prefix }}
path: artifacts/*
if-no-files-found: ignore
- name: Attest provenance
if: startsWith(github.ref, 'refs/tags/') && matrix.do_binaries
uses: actions/attest-build-provenance@v3
with:
subject-path: 'artifacts/*'

View file

@ -4,17 +4,17 @@ name: "CodeQL"
on:
push:
branches: [ master ]
branches: [ 1.4-maint ]
pull_request:
# The branches below must be a subset of the branches above
branches: [ master ]
branches: [ 1.4-maint ]
schedule:
- cron: '39 2 * * 5'
jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
runs-on: ubuntu-22.04
permissions:
actions: read
contents: read
@ -29,38 +29,41 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v2
uses: actions/checkout@v4
with:
# just fetching 1 commit is not enough for setuptools-scm, so we fetch all
fetch-depth: 0
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v1
- name: Set up Python
uses: actions/setup-python@v5
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main
python-version: 3.11
- name: Cache pip
uses: actions/cache@v2
uses: actions/cache@v4
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('requirements.d/development.txt') }}
restore-keys: |
${{ runner.os }}-pip-
${{ runner.os }}-
# Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl
- name: Install requirements, build and install Borg
- name: Install requirements
run: |
sudo apt install libacl1-dev
pip3 install -r requirements.d/development.txt
pip3 install -e .
sudo apt-get update
sudo apt-get install -y pkg-config build-essential
sudo apt-get install -y libssl-dev libacl1-dev libxxhash-dev liblz4-dev libzstd-dev
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main
- name: Build and install Borg
run: |
python3 -m venv ../borg-env
source ../borg-env/bin/activate
pip3 install -r requirements.d/development.txt
pip3 install -ve .
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v1
uses: github/codeql-action/analyze@v2

28
.github/workflows/windows.yml vendored Normal file
View file

@ -0,0 +1,28 @@
name: Windows CI
on: [push, pull_request]
jobs:
msys2-ucrt64:
if: false # build is broken, thus disabled, see #8264
runs-on: windows-latest
defaults:
run:
shell: msys2 {0}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: msys2/setup-msys2@v2
with:
msystem: UCRT64
update: true
- name: Install dependencies
run: ./scripts/msys2-install-deps
- name: Build
run: ./scripts/msys2-build
- name: Test
run: ./dist/borg.exe -V
- uses: actions/upload-artifact@v3
with:
name: borg-windows
path: dist/borg.exe

1
.gitignore vendored
View file

@ -32,3 +32,4 @@ borg.exe
.coverage.*
.vagrant
.eggs
.DS_Store

5
.pre-commit-config.yaml Normal file
View file

@ -0,0 +1,5 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.0.287
hooks:
- id: ruff

33
.readthedocs.yaml Normal file
View file

@ -0,0 +1,33 @@
# .readthedocs.yaml - Read the Docs configuration file.
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details.
version: 2
build:
os: ubuntu-22.04
tools:
python: "3.11"
jobs:
post_checkout:
- git fetch --unshallow
apt_packages:
- build-essential
- pkg-config
- libacl1-dev
- libssl-dev
- liblz4-dev
- libzstd-dev
- libxxhash-dev
python:
install:
- requirements: requirements.d/development.lock.txt
- requirements: requirements.d/docs.txt
- method: pip
path: .
sphinx:
configuration: docs/conf.py
formats:
- htmlzip

12
Brewfile Normal file
View file

@ -0,0 +1,12 @@
brew 'pkg-config'
brew 'zstd'
brew 'lz4'
brew 'xxhash'
brew 'openssl@3'
# osxfuse (aka macFUSE) is only required for "borg mount",
# but won't work on GitHub Actions' workers.
# it requires installing a kernel extension, so some users
# may want it and some won't.
#cask 'osxfuse'

View file

@ -1,4 +1,4 @@
Copyright (C) 2015-2022 The Borg Collective (see AUTHORS file)
Copyright (C) 2015-2025 The Borg Collective (see AUTHORS file)
Copyright (C) 2010-2014 Jonas Borgström <jonas@borgstrom.se>
All rights reserved.

View file

@ -1,7 +1,7 @@
# stuff we need to include into the sdist is handled automatically by
# setuptools_scm - it includes all git-committed files.
# but we want to exclude some committed files/dirs not needed in the sdist:
exclude .coafile .editorconfig .gitattributes .gitignore .mailmap Vagrantfile
# Files to include into the sdist are handled automatically by
# setuptools_scm — it includes all Git-committed files.
# But we want to exclude some committed files/dirs not needed in the sdist:
exclude .editorconfig .gitattributes .gitignore .mailmap Vagrantfile
prune .github
include src/borg/platform/darwin.c src/borg/platform/freebsd.c src/borg/platform/linux.c src/borg/platform/posix.c
include src/borg/platform/syncfilerange.c

View file

@ -18,7 +18,7 @@ See the `installation manual`_ or, if you have already
downloaded Borg, ``docs/installation.rst`` to get started with Borg.
There is also an `offline documentation`_ available, in multiple formats.
.. _installation manual: https://borgbackup.readthedocs.org/en/stable/installation.html
.. _installation manual: https://borgbackup.readthedocs.io/en/stable/installation.html
.. _offline documentation: https://readthedocs.org/projects/borgbackup/downloads
Main features
@ -57,10 +57,10 @@ Main features
**Data encryption**
All data can be protected using 256-bit AES encryption, data integrity and
authenticity is verified using HMAC-SHA256. Data is encrypted clientside.
authenticity is verified using HMAC-SHA256. Data is encrypted client-side.
**Obfuscation**
Optionally, borg can actively obfuscate e.g. the size of files / chunks to
Optionally, Borg can actively obfuscate, e.g., the size of files/chunks to
make fingerprinting attacks more difficult.
**Compression**
@ -73,24 +73,24 @@ Main features
* lzma (low speed, high compression)
**Off-site backups**
Borg can store data on any remote host accessible over SSH. If Borg is
installed on the remote host, big performance gains can be achieved
compared to using a network filesystem (sshfs, nfs, ...).
Borg can store data on any remote host accessible over SSH. If Borg is
installed on the remote host, significant performance gains can be achieved
compared to using a network file system (sshfs, NFS, ...).
**Backups mountable as filesystems**
Backup archives are mountable as userspace filesystems for easy interactive
backup examination and restores (e.g. by using a regular file manager).
**Backups mountable as file systems**
Backup archives are mountable as user-space file systems for easy interactive
backup examination and restores (e.g., by using a regular file manager).
**Easy installation on multiple platforms**
We offer single-file binaries that do not require installing anything -
you can just run them on these platforms:
* Linux
* Mac OS X
* macOS
* FreeBSD
* OpenBSD and NetBSD (no xattrs/ACLs support or binaries yet)
* Cygwin (experimental, no binaries yet)
* Linux Subsystem of Windows 10 (experimental)
* Windows Subsystem for Linux (WSL) on Windows 10/11 (experimental)
**Free and Open Source Software**
* security and functionality can be audited independently
@ -128,9 +128,9 @@ Now doing another backup, just to show off the great deduplication::
-----------------------------------------------------------------------------
For a graphical frontend refer to our complementary project `BorgWeb <https://borgweb.readthedocs.io/>`_.
For a graphical frontend, refer to our complementary project `BorgWeb <https://borgweb.readthedocs.io/>`_.
Helping, Donations and Bounties, becoming a Patron
Helping, donations and bounties, becoming a Patron
--------------------------------------------------
Your help is always welcome!
@ -144,17 +144,17 @@ https://www.borgbackup.org/support/fund.html
Links
-----
* `Main Web Site <https://borgbackup.readthedocs.org/>`_
* `Main website <https://borgbackup.readthedocs.io/>`_
* `Releases <https://github.com/borgbackup/borg/releases>`_,
`PyPI packages <https://pypi.python.org/pypi/borgbackup>`_ and
`ChangeLog <https://github.com/borgbackup/borg/blob/master/docs/changes.rst>`_
* `Offline Documentation <https://readthedocs.org/projects/borgbackup/downloads>`_
`Changelog <https://github.com/borgbackup/borg/blob/master/docs/changes.rst>`_
* `Offline documentation <https://readthedocs.org/projects/borgbackup/downloads>`_
* `GitHub <https://github.com/borgbackup/borg>`_ and
`Issue Tracker <https://github.com/borgbackup/borg/issues>`_.
* `Web-Chat (IRC) <https://web.libera.chat/#borgbackup>`_ and
`Mailing List <https://mail.python.org/mailman/listinfo/borgbackup>`_
* `License <https://borgbackup.readthedocs.org/en/stable/authors.html#license>`_
* `Security contact <https://borgbackup.readthedocs.io/en/latest/support.html#security-contact>`_
`Issue tracker <https://github.com/borgbackup/borg/issues>`_.
* `Web chat (IRC) <https://web.libera.chat/#borgbackup>`_ and
`Mailing list <https://mail.python.org/mailman/listinfo/borgbackup>`_
* `License <https://borgbackup.readthedocs.io/en/stable/authors.html#license>`_
* `Security contact <https://borgbackup.readthedocs.io/en/stable/support.html#security-contact>`_
Compatibility notes
-------------------
@ -171,23 +171,19 @@ see ``docs/support.rst`` in the source distribution).
.. start-badges
|doc| |build| |coverage| |bestpractices| |bounties|
.. |bounties| image:: https://api.bountysource.com/badge/team?team_id=78284&style=bounties_posted
:alt: Bounty Source
:target: https://www.bountysource.com/teams/borgbackup
|doc| |build| |coverage| |bestpractices|
.. |doc| image:: https://readthedocs.org/projects/borgbackup/badge/?version=stable
:alt: Documentation
:target: https://borgbackup.readthedocs.org/en/stable/
:target: https://borgbackup.readthedocs.io/en/stable/
.. |build| image:: https://github.com/borgbackup/borg/workflows/CI/badge.svg?branch=master
:alt: Build Status (master)
.. |build| image:: https://github.com/borgbackup/borg/workflows/CI/badge.svg?branch=1.4-maint
:alt: Build Status (1.4-maint)
:target: https://github.com/borgbackup/borg/actions
.. |coverage| image:: https://codecov.io/github/borgbackup/borg/coverage.svg?branch=master
.. |coverage| image:: https://codecov.io/github/borgbackup/borg/coverage.svg?branch=1.4-maint
:alt: Test Coverage
:target: https://codecov.io/github/borgbackup/borg?branch=master
:target: https://codecov.io/github/borgbackup/borg?branch=1.4-maint
.. |screencast_basic| image:: https://asciinema.org/a/133292.png
:alt: BorgBackup Basic Usage

View file

@ -1,48 +1,48 @@
Borg Native on Windows
Borg native on Windows
======================
Running borg natively on windows is in a early alpha stage. Expect many things to fail.
Do not use the native windows build on any data which you do not want to lose!
Running Borg natively on Windows is in an early alpha stage. Expect many things to fail.
Do not use the native Windows build on any data that you do not want to lose!
Build Requirements
------------------
- VC 14.0 Compiler
- OpenSSL Library v1.1.1c, 64bit (available at https://github.com/python/cpython-bin-deps)
Please use the `win-download-openssl.ps1` script to download and extract the library to
- OpenSSL Library v1.1.1c, 64-bit (available at https://github.com/python/cpython-bin-deps)
Use the `win-download-openssl.ps1` script to download and extract the library to
the correct location. See also the OpenSSL section below.
- Patience and a lot of coffee / beer
- Patience and a lot of coffee/beer
What's working
--------------
.. note::
The following examples assume that the `BORG_REPO` and `BORG_PASSPHRASE` environment variables are set
if the repo or passphrase is not explicitly given.
when the repository or passphrase is not explicitly provided.
- Borg does not crash if called with ``borg``
- ``borg init --encryption repokey-blake2 ./demoRepo`` runs without an error/warning.
Note that absolute paths only work if the protocol is explicitly set to file://
- ``borg init --encryption repokey-blake2 ./demoRepo`` runs without errors or warnings.
Note that absolute paths only work if the protocol is explicitly set to ``file://``
- ``borg create ::backup-{now} D:\DemoData`` works as expected.
- ``borg list`` works as expected.
- ``borg extract --strip-components 1 ::backup-XXXX`` works.
If absolute paths are extracted, it's important to pass ``--strip-components 1`` as
- ``borg extract --strip-components 1 ::backup-XXXX`` works.
If absolute paths are extracted, it is important to pass ``--strip-components 1``,
otherwise the data is restored to the original location!
What's NOT working
------------------
- Extracting a backup which was created on windows machine on a non windows machine will fail.
- And many things more.
- Extracting a backup created on a Windows machine on a non-Windows machine will fail.
- Many other things.
OpenSSL, Windows and Python
---------------------------
Windows does not ship OpenSSL by default, so we need to get the library from somewhere else.
However, a default python installation does include `libcrypto` which is required by borg.
The only things which are missing to build borg are the header and `*.lib` files.
Luckily the python developers provide all required files in a separate repository.
However, a default Python installation does include `libcrypto`, which is required by Borg.
The only things missing to build Borg are the header and `*.lib` files.
Luckily, the Python developers provide all required files in a separate repository.
The `win-download-openssl.ps1` script can be used to download the package from
https://github.com/python/cpython-bin-deps and extract the files to the correct location.
For Anaconda, the required libraries can be installed with `conda install -c anaconda openssl`.
For Anaconda, the required libraries can be installed with ``conda install -c anaconda openssl``.

View file

@ -5,8 +5,10 @@
These borg releases are currently supported with security updates.
| Version | Supported |
| ------- | ------------------ |
| 1.1.x | :white_check_mark: |
|---------|--------------------|
| 1.4.x | :white_check_mark: |
| 1.2.x | :white_check_mark: |
| 1.1.x | :x: |
| < 1.1 | :x: |
## Reporting a Vulnerability

295
Vagrantfile vendored
View file

@ -3,22 +3,24 @@
# Automated creation of testing environments / binaries on misc. platforms
$cpus = Integer(ENV.fetch('VMCPUS', '4')) # create VMs with that many cpus
$xdistn = Integer(ENV.fetch('XDISTN', '4')) # dispatch tests to that many pytest workers
$cpus = Integer(ENV.fetch('VMCPUS', '16')) # create VMs with that many cpus
$xdistn = Integer(ENV.fetch('XDISTN', '16')) # dispatch tests to that many pytest workers
$wmem = $xdistn * 256 # give the VM additional memory for workers [MB]
def packages_debianoid(user)
return <<-EOF
export DEBIAN_FRONTEND=noninteractive
# this is to avoid grub asking about which device it should install to:
# This is to avoid GRUB asking which device it should install to:
echo "set grub-pc/install_devices /dev/sda" | debconf-communicate
apt-get -y -qq update
apt-get -y -qq dist-upgrade
# for building borgbackup and dependencies:
apt install -y libssl-dev libacl1-dev liblz4-dev libzstd-dev pkg-config
# For building BorgBackup and dependencies:
apt install -y libssl-dev libacl1-dev liblz4-dev libzstd-dev libxxhash-dev pkg-config
apt install -y libfuse-dev fuse || true
apt install -y libfuse3-dev fuse3 || true
apt install -y locales || true
# We need to give the prefix to support Debian Buster (no libxxhash.pc for pkg-config there):
echo 'export BORG_LIBXXHASH_PREFIX=/usr' >> ~vagrant/.bash_profile
sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
usermod -a -G fuse #{user}
chgrp fuse /dev/fuse
@ -36,15 +38,18 @@ def packages_freebsd
hostname freebsd
# install all the (security and other) updates, base system
freebsd-update --not-running-from-cron fetch install
# for building borgbackup and dependencies:
# For building BorgBackup and dependencies:
pkg install -y liblz4 zstd pkgconf
pkg install -y fusefs-libs || true
pkg install -y fusefs-libs3 || true
pkg install -y rust
pkg install -y git bash # fakeroot causes lots of troubles on freebsd
# for building python (for the tests we use pyenv built pythons):
pkg install -y python38 py38-sqlite3 py38-virtualenv py38-pip
pkg install -y python310 py310-sqlite3
# make sure there is a python3 command
ln -sf /usr/local/bin/python3.8 /usr/local/bin/python3
ln -sf /usr/local/bin/python3.10 /usr/local/bin/python3
python3 -m ensurepip
pip3 install virtualenv
# make bash default / work:
chsh -s bash vagrant
mount -t fdescfs fdesc /dev/fd
@ -52,7 +57,7 @@ def packages_freebsd
# make FUSE work
echo 'fuse_load="YES"' >> /boot/loader.conf
echo 'vfs.usermount=1' >> /etc/sysctl.conf
kldload fuse
kldload fusefs
sysctl vfs.usermount=1
pw groupmod operator -M vagrant
# /dev/fuse has group operator
@ -66,11 +71,23 @@ end
def packages_openbsd
return <<-EOF
hostname "openbsd77.localdomain"
echo "$(hostname)" > /etc/myname
echo "127.0.0.1 localhost" > /etc/hosts
echo "::1 localhost" >> /etc/hosts
echo "127.0.0.1 $(hostname) $(hostname -s)" >> /etc/hosts
echo "https://ftp.eu.openbsd.org/pub/OpenBSD" > /etc/installurl
ftp https://cdn.openbsd.org/pub/OpenBSD/$(uname -r)/$(uname -m)/comp$(uname -r | tr -d .).tgz
tar -C / -xzphf comp$(uname -r | tr -d .).tgz
rm comp$(uname -r | tr -d .).tgz
pkg_add bash
chsh -s bash vagrant
pkg_add xxhash
pkg_add lz4
pkg_add zstd
pkg_add git # no fakeroot
pkg_add rust
pkg_add openssl%3.4
pkg_add py3-pip
pkg_add py3-virtualenv
EOF
@ -78,11 +95,11 @@ end
def packages_netbsd
return <<-EOF
# use the latest stuff, some packages in "9.2" are quite broken
echo 'http://ftp.NetBSD.org/pub/pkgsrc/packages/NetBSD/$arch/9.0_current/All' > /usr/pkg/etc/pkgin/repositories.conf
echo 'http://ftp.NetBSD.org/pub/pkgsrc/packages/NetBSD/$arch/9.3/All' > /usr/pkg/etc/pkgin/repositories.conf
pkgin update
pkgin -y upgrade
pkg_add zstd lz4 xxhash git
pkg_add rust
pkg_add bash
chsh -s bash vagrant
echo "export PROMPT_COMMAND=" >> ~vagrant/.bash_profile # bug in netbsd 9.2, .bash_profile broken for screen
@ -90,45 +107,36 @@ def packages_netbsd
pkg_add pkg-config
# pkg_add fuse # llfuse supports netbsd, but is still buggy.
# https://bitbucket.org/nikratio/python-llfuse/issues/70/perfuse_open-setsockopt-no-buffer-space
pkg_add python38 py38-sqlite3 py38-pip py38-virtualenv py38-expat
ln -s /usr/pkg/lib/python3.8/_sysconfigdata_netbsd9.py /usr/pkg/lib/python3.8/_sysconfigdata__netbsd9_.py # bug in netbsd 9.2, expected filename not there.
pkg_add python39 py39-sqlite3 py39-pip py39-virtualenv py39-expat
ln -s /usr/pkg/bin/python3.9 /usr/pkg/bin/python
ln -s /usr/pkg/bin/python3.9 /usr/pkg/bin/python3
ln -s /usr/pkg/bin/pip3.9 /usr/pkg/bin/pip
ln -s /usr/pkg/bin/pip3.9 /usr/pkg/bin/pip3
ln -s /usr/pkg/bin/virtualenv-3.9 /usr/pkg/bin/virtualenv
ln -s /usr/pkg/bin/virtualenv-3.9 /usr/pkg/bin/virtualenv3
ln -s /usr/pkg/lib/python3.9/_sysconfigdata_netbsd9.py /usr/pkg/lib/python3.9/_sysconfigdata__netbsd9_.py # bug in netbsd 9.2, expected filename not there.
pkg_add py311-sqlite3 py311-pip py311-virtualenv py311-expat
ln -s /usr/pkg/bin/python3.11 /usr/pkg/bin/python
ln -s /usr/pkg/bin/python3.11 /usr/pkg/bin/python3
ln -s /usr/pkg/bin/pip3.11 /usr/pkg/bin/pip
ln -s /usr/pkg/bin/pip3.11 /usr/pkg/bin/pip3
ln -s /usr/pkg/bin/virtualenv-3.11 /usr/pkg/bin/virtualenv
ln -s /usr/pkg/bin/virtualenv-3.11 /usr/pkg/bin/virtualenv3
EOF
end
def packages_darwin
def package_update_openindiana
return <<-EOF
# install all the (security and other) updates
sudo softwareupdate --ignore iTunesX
sudo softwareupdate --ignore iTunes
sudo softwareupdate --ignore Safari
sudo softwareupdate --ignore "Install macOS High Sierra"
sudo softwareupdate --install --all
which brew || CI=1 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
brew update > /dev/null
brew install pkg-config readline openssl@1.1 zstd lz4 xz fakeroot
brew install --cask macfuse
# brew upgrade # upgrade everything (takes rather long)
echo 'export PKG_CONFIG_PATH=/usr/local/opt/openssl@1.1/lib/pkgconfig' >> ~vagrant/.bash_profile
echo "nameserver 1.1.1.1" > /etc/resolv.conf
# needs separate provisioning step + reboot to become effective:
pkg update
EOF
end
def packages_openindiana
return <<-EOF
# needs separate provisioning step + reboot:
#pkg update
#pkg install gcc-7 python-39 setuptools-39
ln -sf /usr/bin/python3.9 /usr/bin/python3
pkg install gcc-13 git
pkg install pkg-config libxxhash
pkg install python-313
ln -sf /usr/bin/python3.13 /usr/bin/python3
ln -sf /usr/bin/python3.13-config /usr/bin/python3-config
python3 -m ensurepip
ln -sf /usr/bin/pip3.9 /usr/bin/pip3
ln -sf /usr/bin/pip3.13 /usr/bin/pip3
pip3 install virtualenv
# let borg's pkg-config find openssl:
pfexec pkg set-mediator -V 3 openssl
EOF
end
@ -147,18 +155,10 @@ def install_pyenv(boxname)
EOF
end
def fix_pyenv_darwin(boxname)
return <<-EOF
echo 'export PYTHON_CONFIGURE_OPTS="--enable-framework"' >> ~/.bash_profile
EOF
end
def install_pythons(boxname)
return <<-EOF
. ~/.bash_profile
pyenv install 3.10.0 # tests, version supporting openssl 1.1
pyenv install 3.9.10 # tests, version supporting openssl 1.1, binary build
pyenv install 3.8.0 # tests, version supporting openssl 1.1
pyenv install 3.11.14 # tests, binary build
pyenv rehash
EOF
end
@ -175,9 +175,9 @@ def build_pyenv_venv(boxname)
return <<-EOF
. ~/.bash_profile
cd /vagrant/borg
# use the latest 3.9 release
pyenv global 3.9.10
pyenv virtualenv 3.9.10 borg-env
# use the latest 3.11 release
pyenv global 3.11.14
pyenv virtualenv 3.11.14 borg-env
ln -s ~/.pyenv/versions/borg-env .
EOF
end
@ -190,8 +190,7 @@ def install_borg(fuse)
pip install -U wheel # upgrade wheel, might be too old
cd borg
pip install -r requirements.d/development.lock.txt
python setup.py clean
python setup.py clean2
python3 scripts/make.py clean
pip install -e .[#{fuse}]
EOF
end
@ -201,10 +200,7 @@ def install_pyinstaller()
. ~/.bash_profile
cd /vagrant/borg
. borg-env/bin/activate
git clone https://github.com/thomaswaldmann/pyinstaller.git
cd pyinstaller
git checkout v4.7-maint
python setup.py install
pip install 'pyinstaller==6.7.0'
EOF
end
@ -227,10 +223,12 @@ def run_tests(boxname, skip_env)
. ../borg-env/bin/activate
if which pyenv 2> /dev/null; then
# for testing, use the earliest point releases of the supported python versions:
pyenv global 3.8.0 3.9.10 3.10.0
pyenv local 3.8.0 3.9.10 3.10.0
pyenv global 3.11.14
pyenv local 3.11.14
fi
# otherwise: just use the system python
# avoid that git complains about dubious ownership if we use fakeroot:
git config --global --add safe.directory /vagrant/borg/borg
# some OSes can only run specific test envs, e.g. because they miss FUSE support:
export TOX_SKIP_ENV='#{skip_env}'
if which fakeroot 2> /dev/null; then
@ -269,147 +267,148 @@ Vagrant.configure(2) do |config|
v.cpus = $cpus
end
config.vm.define "focal64" do |b|
b.vm.box = "ubuntu/focal64"
config.vm.define "noble" do |b|
b.vm.box = "bento/ubuntu-24.04"
b.vm.provider :virtualbox do |v|
v.memory = 1024 + $wmem
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages debianoid", :type => :shell, :inline => packages_debianoid("vagrant")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_sys_venv("focal64")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_sys_venv("noble")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("llfuse")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("focal64", ".*none.*")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("noble", ".*none.*")
end
config.vm.define "bullseye64" do |b|
config.vm.define "jammy" do |b|
b.vm.box = "ubuntu/jammy64"
b.vm.provider :virtualbox do |v|
v.memory = 1024 + $wmem
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages debianoid", :type => :shell, :inline => packages_debianoid("vagrant")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_sys_venv("jammy")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("llfuse")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("jammy", ".*none.*")
end
config.vm.define "trixie" do |b|
b.vm.box = "debian/testing64"
b.vm.provider :virtualbox do |v|
v.memory = 1024 + $wmem
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages debianoid", :type => :shell, :inline => packages_debianoid("vagrant")
b.vm.provision "install pyenv", :type => :shell, :privileged => false, :inline => install_pyenv("trixie")
b.vm.provision "install pythons", :type => :shell, :privileged => false, :inline => install_pythons("trixie")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_pyenv_venv("trixie")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("llfuse")
b.vm.provision "install pyinstaller", :type => :shell, :privileged => false, :inline => install_pyinstaller()
b.vm.provision "build binary with pyinstaller", :type => :shell, :privileged => false, :inline => build_binary_with_pyinstaller("trixie")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("trixie", ".*none.*")
end
config.vm.define "bookworm" do |b|
b.vm.box = "debian/bookworm64"
b.vm.provider :virtualbox do |v|
v.memory = 1024 + $wmem
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages debianoid", :type => :shell, :inline => packages_debianoid("vagrant")
b.vm.provision "install pyenv", :type => :shell, :privileged => false, :inline => install_pyenv("bookworm")
b.vm.provision "install pythons", :type => :shell, :privileged => false, :inline => install_pythons("bookworm")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_pyenv_venv("bookworm")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("llfuse")
b.vm.provision "install pyinstaller", :type => :shell, :privileged => false, :inline => install_pyinstaller()
b.vm.provision "build binary with pyinstaller", :type => :shell, :privileged => false, :inline => build_binary_with_pyinstaller("bookworm")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("bookworm", ".*none.*")
end
config.vm.define "bullseye" do |b|
b.vm.box = "debian/bullseye64"
b.vm.provider :virtualbox do |v|
v.memory = 1024 + $wmem
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages debianoid", :type => :shell, :inline => packages_debianoid("vagrant")
b.vm.provision "install pyenv", :type => :shell, :privileged => false, :inline => install_pyenv("bullseye64")
b.vm.provision "install pythons", :type => :shell, :privileged => false, :inline => install_pythons("bullseye64")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_pyenv_venv("bullseye64")
b.vm.provision "install pyenv", :type => :shell, :privileged => false, :inline => install_pyenv("bullseye")
b.vm.provision "install pythons", :type => :shell, :privileged => false, :inline => install_pythons("bullseye")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_pyenv_venv("bullseye")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("llfuse")
b.vm.provision "install pyinstaller", :type => :shell, :privileged => false, :inline => install_pyinstaller()
b.vm.provision "build binary with pyinstaller", :type => :shell, :privileged => false, :inline => build_binary_with_pyinstaller("bullseye64")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("bullseye64", ".*none.*")
b.vm.provision "build binary with pyinstaller", :type => :shell, :privileged => false, :inline => build_binary_with_pyinstaller("bullseye")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("bullseye", ".*none.*")
end
config.vm.define "buster64" do |b|
b.vm.box = "debian/buster64"
b.vm.provider :virtualbox do |v|
v.memory = 1024 + $wmem
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages debianoid", :type => :shell, :inline => packages_debianoid("vagrant")
b.vm.provision "install pyenv", :type => :shell, :privileged => false, :inline => install_pyenv("buster64")
b.vm.provision "install pythons", :type => :shell, :privileged => false, :inline => install_pythons("buster64")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_pyenv_venv("buster64")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("llfuse")
b.vm.provision "install pyinstaller", :type => :shell, :privileged => false, :inline => install_pyinstaller()
b.vm.provision "build binary with pyinstaller", :type => :shell, :privileged => false, :inline => build_binary_with_pyinstaller("buster64")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("buster64", ".*none.*")
end
config.vm.define "stretch64" do |b|
b.vm.box = "debian/stretch64"
b.vm.provider :virtualbox do |v|
v.memory = 1024 + $wmem
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages debianoid", :type => :shell, :inline => packages_debianoid("vagrant")
b.vm.provision "install pyenv", :type => :shell, :privileged => false, :inline => install_pyenv("stretch64")
b.vm.provision "install pythons", :type => :shell, :privileged => false, :inline => install_pythons("stretch64")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_pyenv_venv("stretch64")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("llfuse")
b.vm.provision "install pyinstaller", :type => :shell, :privileged => false, :inline => install_pyinstaller()
b.vm.provision "build binary with pyinstaller", :type => :shell, :privileged => false, :inline => build_binary_with_pyinstaller("stretch64")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("stretch64", ".*(fuse3|none).*")
end
config.vm.define "freebsd64" do |b|
b.vm.box = "freebsd121-64"
config.vm.define "freebsd13" do |b|
b.vm.box = "generic/freebsd13"
b.vm.provider :virtualbox do |v|
v.memory = 1024 + $wmem
end
b.ssh.shell = "sh"
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages freebsd", :type => :shell, :inline => packages_freebsd
b.vm.provision "install pyenv", :type => :shell, :privileged => false, :inline => install_pyenv("freebsd64")
b.vm.provision "install pythons", :type => :shell, :privileged => false, :inline => install_pythons("freebsd64")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_pyenv_venv("freebsd64")
b.vm.provision "install pyenv", :type => :shell, :privileged => false, :inline => install_pyenv("freebsd13")
b.vm.provision "install pythons", :type => :shell, :privileged => false, :inline => install_pythons("freebsd13")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_pyenv_venv("freebsd13")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("llfuse")
b.vm.provision "install pyinstaller", :type => :shell, :privileged => false, :inline => install_pyinstaller()
b.vm.provision "build binary with pyinstaller", :type => :shell, :privileged => false, :inline => build_binary_with_pyinstaller("freebsd64")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("freebsd64", ".*(fuse3|none).*")
b.vm.provision "build binary with pyinstaller", :type => :shell, :privileged => false, :inline => build_binary_with_pyinstaller("freebsd13")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("freebsd13", ".*(fuse3|none).*")
end
config.vm.define "openbsd64" do |b|
b.vm.box = "generic/openbsd6"
config.vm.define "freebsd14" do |b|
b.vm.box = "generic/freebsd14"
b.vm.provider :virtualbox do |v|
v.memory = 1024 + $wmem
end
b.ssh.shell = "sh"
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages freebsd", :type => :shell, :inline => packages_freebsd
b.vm.provision "install pyenv", :type => :shell, :privileged => false, :inline => install_pyenv("freebsd14")
b.vm.provision "install pythons", :type => :shell, :privileged => false, :inline => install_pythons("freebsd14")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_pyenv_venv("freebsd14")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("llfuse")
b.vm.provision "install pyinstaller", :type => :shell, :privileged => false, :inline => install_pyinstaller()
b.vm.provision "build binary with pyinstaller", :type => :shell, :privileged => false, :inline => build_binary_with_pyinstaller("freebsd14")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("freebsd14", ".*(fuse3|none).*")
end
config.vm.define "openbsd7" do |b|
b.vm.box = "l3system/openbsd77-amd64"
b.vm.provider :virtualbox do |v|
v.memory = 1024 + $wmem
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages openbsd", :type => :shell, :inline => packages_openbsd
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_sys_venv("openbsd64")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_sys_venv("openbsd7")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("nofuse")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("openbsd64", ".*fuse.*")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("openbsd7", ".*fuse.*")
end
config.vm.define "netbsd64" do |b|
config.vm.define "netbsd9" do |b|
b.vm.box = "generic/netbsd9"
b.vm.provider :virtualbox do |v|
v.memory = 4096 + $wmem # need big /tmp tmpfs in RAM!
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages netbsd", :type => :shell, :inline => packages_netbsd
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_sys_venv("netbsd64")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_sys_venv("netbsd9")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg(false)
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("netbsd64", ".*fuse.*")
end
config.vm.define "darwin64" do |b|
b.vm.box = "macos-sierra"
b.vm.provider :virtualbox do |v|
v.memory = 4096 + $wmem
v.customize ['modifyvm', :id, '--ostype', 'MacOS_64']
v.customize ['modifyvm', :id, '--paravirtprovider', 'default']
v.customize ['modifyvm', :id, '--nested-hw-virt', 'on']
# Adjust CPU settings according to
# https://github.com/geerlingguy/macos-virtualbox-vm
v.customize ['modifyvm', :id, '--cpuidset',
'00000001', '000306a9', '00020800', '80000201', '178bfbff']
# Disable USB variant requiring Virtualbox proprietary extension pack
v.customize ["modifyvm", :id, '--usbehci', 'off', '--usbxhci', 'off']
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "packages darwin", :type => :shell, :privileged => false, :inline => packages_darwin
b.vm.provision "install pyenv", :type => :shell, :privileged => false, :inline => install_pyenv("darwin64")
b.vm.provision "fix pyenv", :type => :shell, :privileged => false, :inline => fix_pyenv_darwin("darwin64")
b.vm.provision "install pythons", :type => :shell, :privileged => false, :inline => install_pythons("darwin64")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_pyenv_venv("darwin64")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("llfuse")
b.vm.provision "install pyinstaller", :type => :shell, :privileged => false, :inline => install_pyinstaller()
b.vm.provision "build binary with pyinstaller", :type => :shell, :privileged => false, :inline => build_binary_with_pyinstaller("darwin64")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("darwin64", ".*(fuse3|none).*")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("netbsd9", ".*fuse.*")
end
# rsync on openindiana has troubles, does not set correct owner for /vagrant/borg and thus gives lots of
# permission errors. can be manually fixed in the VM by: sudo chown -R vagrant /vagrant/borg ; then rsync again.
config.vm.define "openindiana64" do |b|
b.vm.box = "openindiana"
config.vm.define "openindiana" do |b|
b.vm.box = "openindiana/hipster"
b.vm.provider :virtualbox do |v|
v.memory = 2048 + $wmem
end
b.vm.provision "fs init", :type => :shell, :inline => fs_init("vagrant")
b.vm.provision "package update openindiana", :type => :shell, :inline => package_update_openindiana, :reboot => true
b.vm.provision "packages openindiana", :type => :shell, :inline => packages_openindiana
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_sys_venv("openindiana64")
b.vm.provision "build env", :type => :shell, :privileged => false, :inline => build_sys_venv("openindiana")
b.vm.provision "install borg", :type => :shell, :privileged => false, :inline => install_borg("nofuse")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("openindiana64", ".*fuse.*")
b.vm.provision "run tests", :type => :shell, :privileged => false, :inline => run_tests("openindiana", ".*fuse.*")
end
# TODO: create more VMs with python 3.8+ and openssl 1.1.
# See branch 1.1-maint for a better equipped Vagrantfile (but still on py35 and openssl 1.0).
end

View file

@ -1,5 +0,0 @@
Here we store 3rd party documentation, licenses, etc.
Please note that all files inside the "borg" package directory (except the
stuff excluded in setup.py) will be INSTALLED, so don't keep docs or licenses
there.

View file

@ -1,122 +0,0 @@
Creative Commons Legal Code
CC0 1.0 Universal
CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
HEREUNDER.
Statement of Purpose
The laws of most jurisdictions throughout the world automatically confer
exclusive Copyright and Related Rights (defined below) upon the creator
and subsequent owner(s) (each and all, an "owner") of an original work of
authorship and/or a database (each, a "Work").
Certain owners wish to permanently relinquish those rights to a Work for
the purpose of contributing to a commons of creative, cultural and
scientific works ("Commons") that the public can reliably and without fear
of later claims of infringement build upon, modify, incorporate in other
works, reuse and redistribute as freely as possible in any form whatsoever
and for any purposes, including without limitation commercial purposes.
These owners may contribute to the Commons to promote the ideal of a free
culture and the further production of creative, cultural and scientific
works, or to gain reputation or greater distribution for their Work in
part through the use and efforts of others.
For these and/or other purposes and motivations, and without any
expectation of additional consideration or compensation, the person
associating CC0 with a Work (the "Affirmer"), to the extent that he or she
is an owner of Copyright and Related Rights in the Work, voluntarily
elects to apply CC0 to the Work and publicly distribute the Work under its
terms, with knowledge of his or her Copyright and Related Rights in the
Work and the meaning and intended legal effect of CC0 on those rights.
1. Copyright and Related Rights. A Work made available under CC0 may be
protected by copyright and related or neighboring rights ("Copyright and
Related Rights"). Copyright and Related Rights include, but are not
limited to, the following:
i. the right to reproduce, adapt, distribute, perform, display,
communicate, and translate a Work;
ii. moral rights retained by the original author(s) and/or performer(s);
iii. publicity and privacy rights pertaining to a person's image or
likeness depicted in a Work;
iv. rights protecting against unfair competition in regards to a Work,
subject to the limitations in paragraph 4(a), below;
v. rights protecting the extraction, dissemination, use and reuse of data
in a Work;
vi. database rights (such as those arising under Directive 96/9/EC of the
European Parliament and of the Council of 11 March 1996 on the legal
protection of databases, and under any national implementation
thereof, including any amended or successor version of such
directive); and
vii. other similar, equivalent or corresponding rights throughout the
world based on applicable law or treaty, and any national
implementations thereof.
2. Waiver. To the greatest extent permitted by, but not in contravention
of, applicable law, Affirmer hereby overtly, fully, permanently,
irrevocably and unconditionally waives, abandons, and surrenders all of
Affirmer's Copyright and Related Rights and associated claims and causes
of action, whether now known or unknown (including existing as well as
future claims and causes of action), in the Work (i) in all territories
worldwide, (ii) for the maximum duration provided by applicable law or
treaty (including future time extensions), (iii) in any current or future
medium and for any number of copies, and (iv) for any purpose whatsoever,
including without limitation commercial, advertising or promotional
purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
member of the public at large and to the detriment of Affirmer's heirs and
successors, fully intending that such Waiver shall not be subject to
revocation, rescission, cancellation, termination, or any other legal or
equitable action to disrupt the quiet enjoyment of the Work by the public
as contemplated by Affirmer's express Statement of Purpose.
3. Public License Fallback. Should any part of the Waiver for any reason
be judged legally invalid or ineffective under applicable law, then the
Waiver shall be preserved to the maximum extent permitted taking into
account Affirmer's express Statement of Purpose. In addition, to the
extent the Waiver is so judged Affirmer hereby grants to each affected
person a royalty-free, non transferable, non sublicensable, non exclusive,
irrevocable and unconditional license to exercise Affirmer's Copyright and
Related Rights in the Work (i) in all territories worldwide, (ii) for the
maximum duration provided by applicable law or treaty (including future
time extensions), (iii) in any current or future medium and for any number
of copies, and (iv) for any purpose whatsoever, including without
limitation commercial, advertising or promotional purposes (the
"License"). The License shall be deemed effective as of the date CC0 was
applied by Affirmer to the Work. Should any part of the License for any
reason be judged legally invalid or ineffective under applicable law, such
partial invalidity or ineffectiveness shall not invalidate the remainder
of the License, and in such case Affirmer hereby affirms that he or she
will not (i) exercise any of his or her remaining Copyright and Related
Rights in the Work or (ii) assert any associated claims and causes of
action with respect to the Work, in either case contrary to Affirmer's
express Statement of Purpose.
4. Limitations and Disclaimers.
a. No trademark or patent rights held by Affirmer are waived, abandoned,
surrendered, licensed or otherwise affected by this document.
b. Affirmer offers the Work as-is and makes no representations or
warranties of any kind concerning the Work, express, implied,
statutory or otherwise, including without limitation warranties of
title, merchantability, fitness for a particular purpose, non
infringement, or the absence of latent or other defects, accuracy, or
the present or absence of errors, whether or not discoverable, all to
the greatest extent permissible under applicable law.
c. Affirmer disclaims responsibility for clearing rights of other persons
that may apply to the Work or any use thereof, including without
limitation any person's Copyright and Related Rights in the Work.
Further, Affirmer disclaims responsibility for obtaining any necessary
consents, permissions or other rights required for any use of the
Work.
d. Affirmer understands and acknowledges that Creative Commons is not a
party to this document and has no duty or obligation with respect to
this CC0 or use of the Work.

View file

@ -1,13 +0,0 @@
# BLAKE2
This is the reference source code package of BLAKE2.
All code is triple-licensed under the [CC0](http://creativecommons.org/publicdomain/zero/1.0),
the [OpenSSL Licence](https://www.openssl.org/source/license.html),
or the [Apache Public License 2.0](https://www.apache.org/licenses/LICENSE-2.0),
at your choosing.
More: [https://blake2.net](https://blake2.net). [GitHub repository](https://github.com/BLAKE2/BLAKE2).
Contact: contact@blake2.net

View file

@ -1,24 +0,0 @@
LZ4 Library
Copyright (c) 2011-2016, Yann Collet
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View file

@ -1,30 +0,0 @@
BSD License
For Zstandard software
Copyright (c) 2016-present, Facebook, Inc. All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.
* Neither the name Facebook nor the names of its contributors may be used to
endorse or promote products derived from this software without specific
prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

174
docs/_templates/layout.html vendored Normal file
View file

@ -0,0 +1,174 @@
{%- extends "basic/layout.html" %}
{# Do this so that bootstrap is included before the main css file #}
{%- block htmltitle %}
{% set script_files = script_files + ["_static/myscript.js"] %}
<!-- Licensed under the Apache 2.0 License -->
<link rel="stylesheet" type="text/css" href="{{ pathto('_static/fonts/open-sans/stylesheet.css', 1) }}" />
<!-- Licensed under the SIL Open Font License -->
<link rel="stylesheet" type="text/css" href="{{ pathto('_static/fonts/source-serif-pro/source-serif-pro.css', 1) }}" />
<link rel="stylesheet" type="text/css" href="{{ pathto('_static/css/bootstrap.min.css', 1) }}" />
<link rel="stylesheet" type="text/css" href="{{ pathto('_static/css/bootstrap-theme.min.css', 1) }}" />
<meta name="viewport" content="width=device-width, initial-scale=1.0">
{{ super() }}
{%- endblock %}
{%- block extrahead %}
{% if theme_touch_icon %}
<link rel="apple-touch-icon" href="{{ pathto('_static/' ~ theme_touch_icon, 1) }}" />
{% endif %}
<meta name="readthedocs-addons-api-version" content="1" />
{{ super() }}
{% endblock %}
{# Displays the URL for the homepage if it's set or the master_doc if it is not #}
{% macro homepage() -%}
{%- if theme_homepage %}
{%- if hasdoc(theme_homepage) %}
{{ pathto(theme_homepage) }}
{%- else %}
{{ theme_homepage }}
{%- endif %}
{%- else %}
{{ pathto(master_doc) }}
{%- endif %}
{%- endmacro %}
{# Displays the URL for the tospage if it's set or falls back to homepage macro #}
{% macro tospage() -%}
{%- if theme_tospage %}
{%- if hasdoc(theme_tospage) %}
{{ pathto(theme_tospage) }}
{%- else %}
{{ theme_tospage }}
{%- endif %}
{%- else %}
{{ homepage() }}
{%- endif %}
{%- endmacro %}
{# Displays the URL for the projectpage if it's set or falls back to homepage macro #}
{% macro projectlink() -%}
{%- if theme_projectlink %}
{%- if hasdoc(theme_projectlink) %}
{{ pathto(theme_projectlink) }}
{%- else %}
{{ theme_projectlink }}
{%- endif %}
{%- else %}
{{ homepage() }}
{%- endif %}
{%- endmacro %}
{# Displays the next and previous links both before and after content #}
{% macro render_relations() -%}
{% if prev or next %}
<div class="footer-relations">
{% if prev %}
<div class="pull-left">
<a class="btn btn-default" href="{{ prev.link|e }}" title="{{ _('previous chapter')}} (use the left arrow)">{{ prev.title }}</a>
</div>
{% endif %}
{%- if next and next.title != '&lt;no title&gt;' %}
<div class="pull-right">
<a class="btn btn-default" href="{{ next.link|e }}" title="{{ _('next chapter')}} (use the right arrow)">{{ next.title }}</a>
</div>
{%- endif %}
</div>
<div class="clearer"></div>
{% endif %}
{%- endmacro %}
{%- macro guzzle_sidebar() %}
<div id="left-column">
<div class="sphinxsidebar">
{%- if sidebars != None %}
{#- new style sidebar: explicitly include/exclude templates #}
{%- for sidebartemplate in sidebars %}
{%- include sidebartemplate %}
{%- endfor %}
{% else %}
{% include "logo-text.html" %}
{% include "globaltoc.html" %}
{% include "searchbox.html" %}
{%- endif %}
</div>
</div>
{%- endmacro %}
{%- block content %}
{%- if pagename == 'index' and theme_index_template %}
{% include theme_index_template %}
{%- else %}
<div class="container-wrapper">
<div id="mobile-toggle">
<a href="#"><span class="glyphicon glyphicon-align-justify" aria-hidden="true"></span></a>
</div>
{%- block sidebar1 %}{{ guzzle_sidebar() }}{% endblock %}
{%- block document_wrapper %}
{%- block document %}
<div id="right-column">
{% block breadcrumbs %}
<div role="navigation" aria-label="breadcrumbs navigation">
<ol class="breadcrumb">
<li><a href="{{ pathto(master_doc) }}">Docs</a></li>
{% for doc in parents %}
<li><a href="{{ doc.link|e }}">{{ doc.title }}</a></li>
{% endfor %}
<li>{{ title }}</li>
</ol>
</div>
{% endblock %}
<div class="document clearer body" role="main">
{% block body %} {% endblock %}
</div>
{%- block bottom_rel_links %}
{{ render_relations() }}
{%- endblock %}
</div>
<div class="clearfix"></div>
{%- endblock %}
{%- endblock %}
{%- block comments -%}
{% if theme_disqus_comments_shortname %}
<div class="container comment-container">
{% include "comments.html" %}
</div>
{% endif %}
{%- endblock %}
</div>
{%- endif %}
{%- endblock %}
{%- block footer %}
<script type="text/javascript">
$("#mobile-toggle a").click(function () {
$("#left-column").toggle();
});
</script>
<script type="text/javascript" src="{{ pathto('_static/js/bootstrap.js', 1)}}"></script>
{%- block footer_wrapper %}
<div class="footer">
&copy; Copyright {{ copyright }}. Created using <a href="http://sphinx.pocoo.org/">Sphinx</a>.
</div>
{%- endblock %}
{%- block ga %}
{%- if theme_google_analytics_account %}
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', '{{ theme_google_analytics_account }}']);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
{%- endif %}
{%- endblock %}
{%- endblock %}

29
docs/_templates/versionselector.html vendored Normal file
View file

@ -0,0 +1,29 @@
<div class="version-selector" id="borg-version-selector" style="display:none;">
<label for="version-select">Select your Borg version:</label>
<select id="version-select"></select>
</div>
<script type="text/javascript">
// Populate the version selector using ReadTheDocs data if available.
function borgInitVersionSelector(data) {
var versions = data && data.versions && data.versions.active;
if (!versions || !versions.length) return;
var current = data.versions && data.versions.current && data.versions.current.slug;
var select = document.getElementById("version-select");
if (!select) return;
versions.forEach(function(v) {
var opt = document.createElement("option");
opt.value = v.urls.documentation;
opt.textContent = v.slug;
if (v.slug === current) opt.selected = true;
select.appendChild(opt);
});
select.addEventListener("change", function() {
window.location.href = this.value;
});
document.getElementById("borg-version-selector").style.display = "";
}
document.addEventListener("readthedocs-addons-data-ready", function(event) {
borgInitVersionSelector(event.detail.data());
});
</script>

117
docs/binaries/00_README.txt Normal file
View file

@ -0,0 +1,117 @@
Binary BorgBackup builds
========================
General notes
-------------
The binaries are supposed to work on the specified platform without installing anything else.
There are some limitations, though:
- for Linux, your system must have the same or newer glibc version as the one used for building
- for macOS, you need to have the same or newer macOS version as the one used for building
- for other OSes, there are likely similar limitations
If you don't find something working on your system, check the older borg releases.
*.asc are GnuPG signatures - only provided for locally built binaries.
*.exe (or no extension) is the single-file fat binary.
*.tgz is the single-directory fat binary (extract it once with tar -xzf).
Using the single-directory build is faster and does not require as much space
in the temporary directory as the self-extracting single-file build.
macOS: to avoid issues, download the file via the command line OR remove the
"quarantine" attribute after downloading:
$ xattr -dr com.apple.quarantine borg-macos1012.tgz
Download the correct files
--------------------------
Binaries built on GitHub servers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
borg-linux-glibc235-x86_64-gh Linux AMD/Intel (built on Ubuntu 22.04 LTS with glibc 2.35)
borg-linux-glibc235-arm64-gh Linux ARM (built on Ubuntu 22.04 LTS with glibc 2.35)
borg-macos-15-arm64-gh macOS Apple Silicon (built on macOS 15 w/o FUSE support)
borg-macos-15-x86_64-gh macOS Intel (built on macOS 15 w/o FUSE support)
borg-freebsd-14-x86_64-gh FreeBSD AMD/Intel (built on FreeBSD 14)
Binaries built locally
~~~~~~~~~~~~~~~~~~~~~~
borg-linux-glibc231-x86_64 Linux (built on Debian 11 "Bullseye" with glibc 2.31)
Note: if you don't find a specific binary here, check release 1.4.1 or 1.2.9.
Verifying your download
-----------------------
I provide GPG signatures for files which I have built locally on my machines.
To check the GPG signature, download both the file and the corresponding
signature (*.asc file) and then (on the shell) type, for example:
gpg --recv-keys 9F88FB52FAF7B393
gpg --verify borgbackup.tar.gz.asc borgbackup.tar.gz
The files are signed by:
Thomas Waldmann <tw@waldmann-edv.de>
GPG key fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393
My fingerprint is also in the footer of all my BorgBackup mailing list posts.
Provenance attestations for GitHub-built binaries
-------------------------------------------------
For binaries built on GitHub (files with a "-gh" suffix in the name), we publish
an artifact provenance attestation that proves the binary was built by our
GitHub Actions workflow from a specific commit or tag. You can verify this using
the GitHub CLI (gh). Install it from https://cli.github.com/ and make sure you
use a recent version that supports "gh attestation".
Practical example (Linux, 1.4.3 tag):
curl -LO https://github.com/borgbackup/borg/releases/download/1.4.3/borg-linux-glibc235-x86_64-gh
gh attestation verify --repo borgbackup/borg --source-ref refs/tags/1.4.3 borg-linux-glibc235-x86_64-gh
If verification succeeds, gh prints a summary stating the subject (your file),
that it was attested by GitHub Actions, and the job/workflow reference.
Installing
----------
It is suggested that you rename or symlink the binary to just "borg".
If you need "borgfs", just also symlink it to the same binary, it will
detect internally under which name it was invoked.
On UNIX-like platforms, /usr/local/bin/ or ~/bin/ is a nice place for it,
but you can invoke it from anywhere by providing the full path to it.
Make sure the file is readable and executable (chmod +rx borg on UNIX-like
platforms).
Reporting issues
----------------
Please first check the FAQ and whether a GitHub issue already exists.
If you find a NEW issue, please open a ticket on our issue tracker:
https://github.com/borgbackup/borg/issues/
There, please give:
- the version number (it is displayed if you invoke borg -V)
- the sha256sum of the binary
- a good description of what the issue is
- a good description of how to reproduce your issue
- a traceback with system info (if you have one)
- your precise platform (CPU, 32/64-bit?), OS, distribution, release
- your Python and (g)libc versions

View file

@ -5,7 +5,7 @@
Borg documentation
==================
.. when you add an element here, do not forget to add it to index.rst
.. When you add an element here, do not forget to add it to index.rst.
.. Note: Some things are in appendices (see latex_appendices in conf.py)
.. toctree::

View file

@ -52,8 +52,7 @@ h1 {
}
.container.experimental,
#debugging-facilities,
#borg-recreate {
#debugging-facilities {
/* don't change text dimensions */
margin: 0 -30px; /* padding below + border width */
padding: 0 10px; /* 10 px visual margin between edge of text and the border */
@ -123,10 +122,26 @@ table.docutils.borg-options-table tr td:first-child:not([colspan="3"]) {
margin: 0;
}
.borg-options-table {
.borg-options-table,
.borg-encryption-table {
width: 100%;
}
.borg-encryption-table th {
text-align: center;
}
.borg-encryption-table th,
.borg-encryption-table td {
vertical-align: top;
}
.borg-encryption-table .literal {
display: block;
margin: 0;
white-space: normal;
}
kbd, /* used in usage pages for options */
code,
.rst-content tt.literal,
@ -179,3 +194,45 @@ cite {
#common-options .option {
white-space: nowrap;
}
/* Remove the right-column max-width cap so content fills the full available width. */
#right-column {
max-width: none;
}
/* Hide the default RTD flyout since we show the version selector in the sidebar. */
readthedocs-flyout {
display: none !important;
}
/* Version selector in the sidebar. */
.version-selector {
padding: 0 22px;
margin: 7px 0 7px 0;
font-size: 14px;
}
.version-selector label {
display: block;
margin-bottom: 4px;
color: #000;
}
.version-selector select {
width: 100%;
padding: 4px;
background-color: #fafafa;
color: #000;
border: 1px solid #ccc;
border-radius: 3px;
}
.version-selector::after {
content: '';
display: block;
border-top: 1px solid #ccc;
margin: 7px 0 0 0;
}
/* Reduce top and bottom margin of searchbox block to 7px to match separator spacing. */
.sidebar-block:has(#main-search) {
margin-top: 7px;
margin-bottom: 7px;
}
/* Reduce the separator margin below the search block to 7px. */
.sphinxsidebar > .sidebar-block:has(#main-search):after {
margin: 7px 22px 0 22px;
}

File diff suppressed because it is too large Load diff

View file

@ -12,8 +12,10 @@
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
import sys, os
sys.path.insert(0, os.path.abspath('../src'))
import sys
import os
sys.path.insert(0, os.path.abspath("../src"))
from borg import __version__ as sw_version
@ -40,7 +42,7 @@ master_doc = 'index'
# General information about the project.
project = 'Borg - Deduplicating Archiver'
copyright = u'2010-2014 Jonas Borgström, 2015-2022 The Borg Collective (see AUTHORS file)'
copyright = '2010-2014 Jonas Borgström, 2015-2025 The Borg Collective (see AUTHORS file)'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
@ -111,6 +113,7 @@ def set_rst_settings(app):
def setup(app):
app.setup_extension('sphinxcontrib.jquery')
app.add_css_file('css/borg.css')
app.connect('builder-inited', set_rst_settings)
@ -154,10 +157,11 @@ html_last_updated_fmt = '%Y-%m-%d'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
html_use_smartypants = True
smartquotes_action = 'qe' # no D in there means "do not transform -- and ---"
# Custom sidebar templates, maps document names to template names.
html_sidebars = {
'**': ['logo-text.html', 'searchbox.html', 'globaltoc.html'],
'**': ['logo-text.html', "versionselector.html", 'searchbox.html', 'globaltoc.html'],
}
# Additional templates that should be rendered to pages, maps page names to
@ -255,9 +259,11 @@ extensions = [
'sphinx.ext.todo',
'sphinx.ext.coverage',
'sphinx.ext.viewcode',
'sphinxcontrib.jquery', # jquery is not included anymore by default
'guzzle_sphinx_theme', # register the theme as an extension to generate a sitemap.xml
]
extlinks = {
'issue': ('https://github.com/borgbackup/borg/issues/%s', '#'),
'issue': ('https://github.com/borgbackup/borg/issues/%s', '#%s'),
'targz_url': ('https://pypi.python.org/packages/source/b/borgbackup/%%s-%s.tar.gz' % version, None),
}

View file

@ -14,3 +14,4 @@ This chapter details deployment strategies for the following scenarios.
deployment/automated-local
deployment/image-backup
deployment/pull-backup
deployment/non-root-user

View file

@ -14,8 +14,8 @@ systemd and udev.
Overview
--------
An udev rule is created to trigger on the addition of block devices. The rule contains a tag
that triggers systemd to start a oneshot service. The oneshot service executes a script in
A udev rule is created to trigger on the addition of block devices. The rule contains a tag
that causes systemd to start a oneshot service. The oneshot service executes a script in
the standard systemd service environment, which automatically captures stdout/stderr and
logs it to the journal.
@ -29,30 +29,16 @@ Configuring the system
First, create the ``/etc/backups`` directory (as root).
All configuration goes into this directory.
Then, create ``/etc/backups/40-backup.rules`` with the following content (all on one line)::
Find out the ID of the partition table of your backup disk (here assumed to be /dev/sdz):
lsblk --fs -o +PTUUID /dev/sdz
ACTION=="add", SUBSYSTEM=="bdi", DEVPATH=="/devices/virtual/bdi/*",
TAG+="systemd", ENV{SYSTEMD_WANTS}="automatic-backup.service"
Then, create ``/etc/backups/80-backup.rules`` with the following content (all on one line)::
.. topic:: Finding a more precise udev rule
ACTION=="add", SUBSYSTEM=="block", ENV{ID_PART_TABLE_UUID}=="<the PTUUID you just noted>", TAG+="systemd", ENV{SYSTEMD_WANTS}+="automatic-backup.service"
If you always connect the drive(s) to the same physical hardware path, e.g. the same
eSATA port, then you can make a more precise udev rule.
Execute ``udevadm monitor`` and connect a drive to the port you intend to use.
You should see a flurry of events, find those regarding the `block` subsystem.
Pick the event whose device path ends in something similar to a device file name,
typically`sdX/sdXY`. Use the event's device path and replace `sdX/sdXY` after the
`/block/` part in the path with a star (\*). For example:
`DEVPATH=="/devices/pci0000:00/0000:00:11.0/ata3/host2/target2:0:0/2:0:0:0/block/*"`.
Reboot a few times to ensure that the hardware path does not change: on some motherboards
components of it can be random. In these cases you cannot use a more accurate rule,
or need to insert additional stars for matching the path.
The "systemd" tag in conjunction with the SYSTEMD_WANTS environment variable has systemd
launch the "automatic-backup" service, which we will create next, as the
``/etc/backups/automatic-backup.service`` file:
The "systemd" tag in conjunction with the SYSTEMD_WANTS environment variable causes systemd to
launch the "automatic-backup" service, which we will create next, as the file
``/etc/backups/automatic-backup.service``:
.. code-block:: ini
@ -79,13 +65,13 @@ modify it to suit your needs (e.g. more backup sets, dumping databases etc.).
# Script configuration
#
# The backup partition is mounted there
# The backup partition is mounted here
MOUNTPOINT=/mnt/backup
# This is the location of the Borg repository
TARGET=$MOUNTPOINT/borg-backups/backup.borg
# Archive name schema
# Archive name scheme
DATE=$(date --iso-8601)-$(hostname)
# This is the file that will later contain UUIDs of registered backup drives
@ -107,10 +93,10 @@ modify it to suit your needs (e.g. more backup sets, dumping databases etc.).
echo "Disk $uuid is a backup disk"
partition_path=/dev/disk/by-uuid/$uuid
# Mount file system if not already done. This assumes that if something is already
# Mount the file system if not already done. This assumes that if something is already
# mounted at $MOUNTPOINT, it is the backup drive. It won't find the drive if
# it was mounted somewhere else.
(mount | grep $MOUNTPOINT) || mount $partition_path $MOUNTPOINT
findmnt $MOUNTPOINT >/dev/null || mount $partition_path $MOUNTPOINT
drive=$(lsblk --inverse --noheadings --list --paths --output name $partition_path | head --lines 1)
echo "Drive path: $drive"
@ -136,8 +122,8 @@ modify it to suit your needs (e.g. more backup sets, dumping databases etc.).
# This is just an example, change it however you see fit
borg create $BORG_OPTS \
--exclude /root/.cache \
--exclude /var/lib/docker/devicemapper \
--exclude root/.cache \
--exclude var/lib/docker/devicemapper \
$TARGET::$DATE-$$-system \
/ /boot
@ -145,7 +131,7 @@ modify it to suit your needs (e.g. more backup sets, dumping databases etc.).
# Even if it isn't (add --exclude /home above), it probably makes sense
# to have /home in a separate archive.
borg create $BORG_OPTS \
--exclude 'sh:/home/*/.cache' \
--exclude 'sh:home/*/.cache' \
$TARGET::$DATE-$$-home \
/home/
@ -164,7 +150,7 @@ modify it to suit your needs (e.g. more backup sets, dumping databases etc.).
fi
Create the ``/etc/backups/autoeject`` file to have the script automatically eject the drive
after creating the backup. Rename the file to something else (e.g. ``/etc/backup/autoeject-no``)
after creating the backup. Rename the file to something else (e.g. ``/etc/backups/autoeject-no``)
when you want to do something with the drive after creating backups (e.g running check).
Create the ``/etc/backups/backup-suspend`` file if the machine should suspend after completing
@ -178,7 +164,7 @@ The last part is to actually enable the udev rules and services:
.. code-block:: bash
ln -s /etc/backups/40-backup.rules /etc/udev/rules.d/40-backup.rules
ln -s /etc/backups/80-backup.rules /etc/udev/rules.d/80-backup.rules
ln -s /etc/backups/automatic-backup.service /etc/systemd/system/automatic-backup.service
systemctl daemon-reload
udevadm control --reload
@ -191,7 +177,7 @@ Find the UUID of the file system that backups should be stored on::
lsblk -o+uuid,label
Note the UUID into the ``/etc/backup/backup.disks`` file.
Note the UUID into the ``/etc/backups/backup.disks`` file.
Mount the drive to /mnt/backup.
@ -212,7 +198,7 @@ Security considerations
-----------------------
The script as shown above will mount any file system with an UUID listed in
``/etc/backup/backup.disks``. The UUID check is a safety / annoyance-reduction
``/etc/backups/backup.disks``. The UUID check is a safety / annoyance-reduction
mechanism to keep the script from blowing up whenever a random USB thumb drive is connected.
It is not meant as a security mechanism. Mounting file systems and reading repository
data exposes additional attack surfaces (kernel file system drivers,

View file

@ -4,20 +4,20 @@
Central repository server with Ansible or Salt
==============================================
This section will give an example how to setup a borg repository server for multiple
This section gives an example of how to set up a Borg repository server for multiple
clients.
Machines
--------
There are multiple machines used in this section and will further be named by their
respective fully qualified domain name (fqdn).
respective fully qualified domain name (FQDN).
* The backup server: `backup01.srv.local`
* The clients:
- John Doe's desktop: `johndoe.clnt.local`
- Webserver 01: `web01.srv.local`
- Web server 01: `web01.srv.local`
- Application server 01: `app01.srv.local`
User and group
@ -28,7 +28,7 @@ Recommended user and group with additional settings:
* User: `backup`
* Group: `backup`
* Shell: `/bin/bash` (or other capable to run the `borg serve` command)
* Shell: `/bin/bash` (or another capable of running the `borg serve` command)
* Home: `/home/backup`
Most clients shall initiate a backup from the root user to catch all
@ -79,11 +79,11 @@ The options which are added to the key will perform the following:
3. Restrict ssh and do not allow stuff which imposes a security risk
Due to the ``cd`` command we use, the server automatically changes the current
working directory. Then client doesn't need to have knowledge of the absolute
working directory. The client doesn't need to have knowledge of the absolute
or relative remote repository path and can directly access the repositories at
``<user>@<host>:<repo>``.
``ssh://<user>@<host>/./<repo>``.
.. note:: The setup above ignores all client given commandline parameters
.. note:: The setup above ignores all client-given command-line parameters
which are normally appended to the `borg serve` command.
Client
@ -93,21 +93,21 @@ The client needs to initialize the `pictures` repository like this:
::
borg init backup@backup01.srv.local:pictures
borg init ssh://backup@backup01.srv.local/./pictures
Or with the full path (should actually never be used, as only for demonstrational purposes).
The server should automatically change the current working directory to the `<client fqdn>` folder.
Or with the full path (should actually never be used, as only for demonstration purposes).
The server should automatically change the current working directory to the `<client FQDN>` folder.
::
borg init backup@backup01.srv.local:/home/backup/repos/johndoe.clnt.local/pictures
borg init ssh://backup@backup01.srv.local/home/backup/repos/johndoe.clnt.local/pictures
When `johndoe.clnt.local` tries to access a not restricted path the following error is raised.
When `johndoe.clnt.local` tries to access an unrestricted path, the following error is raised.
John Doe tries to backup into the Web 01 path:
::
borg init backup@backup01.srv.local:/home/backup/repos/web01.srv.local/pictures
borg init ssh://backup@backup01.srv.local/home/backup/repos/web01.srv.local/pictures
::

View file

@ -5,11 +5,11 @@
Hosting repositories
====================
This sections shows how to securely provide repository storage for users.
This section shows how to securely provide repository storage for users.
Optionally, each user can have a storage quota.
Repositories are accessed through SSH. Each user of the service should
have her own login which is only able to access the user's files.
have their own login which is only able to access the user's files.
Technically it would be possible to have multiple users share one login,
however, separating them is better. Separate logins increase isolation
and are thus an additional layer of security and safety for both the

View file

@ -8,6 +8,38 @@ Backing up disk images can still be efficient with Borg because its `deduplicati
technique makes sure only the modified parts of the file are stored. Borg also has
optional simple sparse file support for extract.
It is of utmost importance to pin down the disk you want to back up.
You need to use the SERIAL for that.
Use:
.. code-block:: bash
# You can find the short disk serial by:
# udevadm info --query=property --name=nvme1n1 | grep ID_SERIAL_SHORT | cut -d '=' -f 2
DISK_SERIAL="7VS0224F"
DISK_ID=$(readlink -f /dev/disk/by-id/*"${DISK_SERIAL}") # Returns /dev/nvme1n1
mapfile -t PARTITIONS < <(lsblk -o NAME,TYPE -p -n -l "$DISK_ID" | awk '$2 == "part" {print $1}')
echo "Partitions of $DISK_ID:"
echo "${PARTITIONS[@]}"
echo "Disk Identifier: $DISK_ID"
# Use the following line to perform a borg backup for the full disk:
# borg create --read-special /path/to/repo::{now} "$DISK_ID"
# Use the following to perform a borg backup for all partitions of the disk
# borg create --read-special /path/to/repo::{now} "${PARTITIONS[@]}"
# Example output:
# Partitions of /dev/nvme1n1:
# /dev/nvme1n1p1
# /dev/nvme1n1p2
# /dev/nvme1n1p3
# Disk Identifier: /dev/nvme1n1
# borg create --read-special /path/to/repo::{now} /dev/nvme1n1
# borg create --read-special /path/to/repo::{now} /dev/nvme1n1p1 /dev/nvme1n1p2 /dev/nvme1n1p3
Decreasing the size of image backups
------------------------------------
@ -33,7 +65,7 @@ deduplicating. For backup, save the disk header and the contents of each partiti
PARTNUM=$(echo $x | grep -Eo "[0-9]+$")
ntfsclone -so - $x | borg create repo::hostname-part$PARTNUM -
done
# to backup non-NTFS partitions as well:
# to back up non-NTFS partitions as well:
echo "$PARTITIONS" | grep -v NTFS | cut -d' ' -f1 | while read x; do
PARTNUM=$(echo $x | grep -Eo "[0-9]+$")
borg create --read-special repo::hostname-part$PARTNUM $x
@ -116,4 +148,4 @@ way to create application-consistent backups.
Borg doesn't intend to address these issues due to their huge complexity and
platform/software dependency. Combining Borg with the mechanisms provided by the platform
(snapshots, hypervisor features) will be the best approach to start tackling them.
(snapshots, hypervisor features) will be the best approach to start tackling them.

View file

@ -0,0 +1,66 @@
.. include:: ../global.rst.inc
.. highlight:: none
.. _non_root_user:
================================
Backing up using a non-root user
================================
This section describes how to run Borg as a non-root user and still be able to
back up every file on the system.
Normally Borg is run as the root user to bypass all filesystem permissions and
be able to read all files. But in theory this also allows Borg to modify or
delete files on your system, in case of a bug, for example.
To eliminate this possibility, we can run Borg as a non-root user and give it read-only
permissions to all files on the system.
Using Linux capabilities inside a systemd service
=================================================
One way to do so is to use Linux `capabilities
<https://man7.org/linux/man-pages/man7/capabilities.7.html>`_ within a systemd
service.
Linux capabilities allow us to give some of the privileges that the root user has to
a non-root user. This works on a per-thread level and does not grant these permissions
to the non-root user as a whole.
For this, we need to run our backup script from a systemd service and use the `AmbientCapabilities
<https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#AmbientCapabilities=>`_
option added in systemd 229.
A very basic unit file would look like this:
::
[Unit]
Description=Borg Backup
[Service]
Type=oneshot
User=borg
ExecStart=/usr/local/sbin/backup.sh
AmbientCapabilities=CAP_DAC_READ_SEARCH
The ``CAP_DAC_READ_SEARCH`` capability gives Borg read-only access to all files and directories on the system.
This service can then be started manually using ``systemctl start``, a systemd timer or other methods.
Restore considerations
======================
When restoring files, the root user should be used. When using the non-root user, borg extract will
change all files to be owned by the non-root user. Using borg mount will not allow the non-root user
to access files that it would not have access to on the system itself.
Other than that, the same restore process that would be used when running the backup as root can be used.
.. warning::
When using a local repo and running borg commands as root, make sure to only use commands that do not
modify the repo itself, like extract or mount. Modifying the repo using the root user will break
the repo for the non-root user, since some files inside the repo will now be owned by root.

View file

@ -6,20 +6,20 @@
Backing up in pull mode
=======================
Typically the borg client connects to a backup server using SSH as a transport
Typically the Borg client connects to a backup server using SSH as a transport
when initiating a backup. This is referred to as push mode.
If you however require the backup server to initiate the connection or prefer
If, however, you require the backup server to initiate the connection or prefer
it to initiate the backup run, one of the following workarounds is required to
allow such a pull mode setup.
A common use case for pull mode is to backup a remote server to a local personal
A common use case for pull mode is to back up a remote server to a local personal
computer.
SSHFS
=====
Assuming you have a pull backup system set up with borg, where a backup server
Assuming you have a pull backup system set up with Borg, where a backup server
pulls the data from the target via SSHFS. In this mode, the backup client's file
system is mounted remotely on the backup server. Pull mode is even possible if
the SSH connection must be established by the client via a remote tunnel. Other
@ -58,13 +58,13 @@ completely in every aspect from such a backup.
mappings, assuming they only come from files (/etc/passwd and group).
This assumption might be wrong, e.g. if users/groups also come from
ldap or other providers.
Thus, it might be better to use ``--numeric-owner`` and not archive any
Thus, it might be better to use ``--numeric-ids`` and not archive any
user or group names (but just the numeric IDs) and not use chroot.
Creating a backup
-----------------
Generally, in a pull backup situation there is no direct way for borg to know
Generally, in a pull backup situation there is no direct way for Borg to know
the client's original UID:GID name mapping of files, because Borg would use
``/etc/passwd`` and ``/etc/group`` of the backup server to map the names. To
derive the right names, Borg needs to have access to the client's passwd and
@ -72,7 +72,7 @@ group files and use them in the backup process.
The solution to this problem is chrooting into an sshfs mounted directory. In
this example the whole client root file system is mounted. We use the
stand-alone BorgBackup executable and copy it into the mounted file system to
standalone BorgBackup executable and copy it into the mounted file system to
make Borg available after entering chroot; this can be skipped if Borg is
already installed on the client.
@ -98,9 +98,9 @@ create the backup, retaining the original paths, excluding the repository:
::
borg create --exclude /borgrepo --files-cache ctime,size /borgrepo::archive /
borg create --exclude borgrepo --files-cache ctime,size /borgrepo::archive /
For the sake of simplicity only ``/borgrepo`` is excluded here. You may want to
For the sake of simplicity only ``borgrepo`` is excluded here. You may want to
set up an exclude file with additional files and folders to be excluded. Also
note that we have to modify Borg's file change detection behaviour SSHFS
cannot guarantee stable inode numbers, so we have to supply the
@ -181,13 +181,13 @@ When doing a full restore, we restore all files (including the ones containing
the ID-to-name mapping, ``/etc/passwd`` and ``/etc/group``). Everything will be
consistent automatically if we restore the numeric IDs stored in the archive. So
there is no need for a chroot environment; we just mount the client file system
and extract a backup, utilizing the ``--numeric-owner`` option:
and extract a backup, utilizing the ``--numeric-ids`` option:
::
sshfs root@host:/ /mnt/sshfs
cd /mnt/sshfs
borg extract --numeric-owner /path/to/repo::archive
borg extract --numeric-ids /path/to/repo::archive
cd ~
umount /mnt/sshfs

View file

@ -19,7 +19,7 @@ Some guidance for contributors:
- Discuss changes on the GitHub issue tracker, on IRC or on the mailing list.
- Make your PRs on the ``master`` branch (see `Branching Model`_ for details).
- Make your PRs on the ``master`` branch (see `Branching Model`_ for details and exceptions).
- Do clean changesets:
@ -52,8 +52,8 @@ Borg development happens on the ``master`` branch and uses GitHub pull
requests (if you don't have GitHub or don't want to use it you can
send smaller patches via the borgbackup mailing list to the maintainers).
Stable releases are maintained on maintenance branches named ``x.y-maint``, eg.
the maintenance branch of the 1.0.x series is ``1.0-maint``.
Stable releases are maintained on maintenance branches named ``x.y-maint``, e.g.
the maintenance branch of the 1.4.x series is ``1.4-maint``.
Most PRs should be filed against the ``master`` branch. Only if an
issue affects **only** a particular maintenance branch a PR should be
@ -131,12 +131,9 @@ you run the tests.
Continuous Integration
----------------------
All pull requests go through `GitHub Actions`_, which runs the tests on Linux
and Mac OS X as well as the flake8 style checker. Windows builds run on AppVeyor_,
while additional Unix-like platforms are tested on Golem_.
All pull requests go through `GitHub Actions`_, which runs the tests on misc.
Python versions and on misc. platforms as well as some additional checks.
.. _AppVeyor: https://ci.appveyor.com/project/borgbackup/borg/
.. _Golem: https://golem.enkore.de/view/Borg/
.. _GitHub Actions: https://github.com/borgbackup/borg/actions
Output and Logging
@ -164,6 +161,12 @@ virtual env and run::
pip install -r requirements.d/development.txt
This project utilizes pre-commit to lint code before it is committed.
Although pre-commit is installed when running the command above, the pre-commit hooks
will have to be installed separately. Run this command to install the pre-commit hooks::
pre-commit install
Running the tests
-----------------
@ -171,7 +174,7 @@ The tests are in the borg/testsuite package.
To run all the tests, you need to have fakeroot installed. If you do not have
fakeroot, you still will be able to run most tests, just leave away the
`fakeroot -u` from the given command lines.
``fakeroot -u`` from the given command lines.
To run the test suite use the following command::
@ -182,7 +185,7 @@ Some more advanced examples::
# verify a changed tox.ini (run this after any change to tox.ini):
fakeroot -u tox --recreate
fakeroot -u tox -e py38 # run all tests, but only on python 3.8
fakeroot -u tox -e py310 # run all tests, but only on python 3.10
fakeroot -u tox borg.testsuite.locking # only run 1 test module
@ -194,26 +197,36 @@ Important notes:
- When using ``--`` to give options to py.test, you MUST also give ``borg.testsuite[.module]``.
Running the tests (using the pypi package)
------------------------------------------
Running more checks using coala
-------------------------------
First install coala and some checkers ("bears"):
Since borg 1.4, it is also possible to run the tests without a development
environment, using the borgbackup dist package (downloaded from pypi.org or
github releases page):
::
pip install -r requirements.d/coala.txt
# optional: create and use a virtual env:
python3 -m venv env
. env/bin/activate
You can now run coala from the toplevel directory; it will read its settings
from ``.coafile`` there:
# install packages
pip install borgbackup
pip install pytest pytest-benchmark
::
# method A: use a pytest.ini
coala
cat >pytest.ini <<<EOF
[pytest]
python_files = testsuite/*.py
markers = allow_cache_wipe
addopts = -rs --benchmark-skip
EOF
Some bears have additional requirements and they usually tell you about
them in case they are missing.
pytest --pyargs borg.testsuite
# method B: give the options via the cmdline (each time you invoke the tests):
pytest -rs --benchmark-skip -o 'python_files=testsuite/*.py' -o 'markers=allow_cache_wipe' --pyargs borg.testsuite
Adding a compression algorithm
------------------------------
@ -236,8 +249,8 @@ for easier use by packagers downstream.
When a command is added, a command line flag changed, added or removed,
the usage docs need to be rebuilt as well::
python setup.py build_usage
python setup.py build_man
python scripts/make.py build_usage
python scripts/make.py build_man
However, we prefer to do this as part of our :ref:`releasing`
preparations, so it is generally not necessary to update these when
@ -291,6 +304,49 @@ Usage::
# To copy files from the VM (in this case, the generated binary):
vagrant scp OS:/vagrant/borg/borg.exe .
Using Podman
------------
macOS-based developers (and others who prefer containers) can run the Linux test suite locally using Podman.
Prerequisites:
- Install Podman (e.g., ``brew install podman``).
- Initialize the Podman machine, only once: ``podman machine init``.
- Start the Podman machine, before using it: ``podman machine start``.
Usage::
# Open an interactive shell in the container (default if no command given):
./scripts/linux-run
# Run the default tox environment:
./scripts/linux-run tox
# Run a specific tox environment:
./scripts/linux-run tox -e py311-pyfuse3
# Pass arguments to pytest (e.g., run specific tests):
./scripts/linux-run tox -e py313-pyfuse3 -- -k mount
# Switch base image (temporarily):
./scripts/linux-run --image python:3.11-bookworm tox
Resource Usage
~~~~~~~~~~~~~~
The default Podman VM uses 2GB RAM and half your CPUs.
For heavy tests (parallel execution), this might be tight.
- **Check usage:** Run ``podman stats`` in another terminal while tests are running.
- **Increase resources:**
::
podman machine stop
podman machine set --cpus 6 --memory 4096
podman machine start
Creating standalone binaries
----------------------------
@ -310,7 +366,6 @@ If you encounter issues, see also our `Vagrantfile` for details.
work on same OS, same architecture (x86 32bit, amd64 64bit)
without external dependencies.
.. _releasing:
Creating a new release
@ -326,12 +381,18 @@ Checklist:
- Update ``CHANGES.rst``, based on ``git log $PREVIOUS_RELEASE..``.
- Check version number of upcoming release in ``CHANGES.rst``.
- Render ``CHANGES.rst`` via ``make html`` and check for markup errors.
- Verify that ``MANIFEST.in`` and ``setup.py`` are complete.
- ``python setup.py build_usage ; python setup.py build_man`` and commit.
- Verify that ``MANIFEST.in``, ``pyproject.toml`` and ``setup.py`` are complete.
- Run these commands, check git status for files that might need to be added, and commit::
python scripts/make.py build_usage
python scripts/make.py build_man
- Tag the release::
git tag -s -m "tagged/signed release X.Y.Z" X.Y.Z
- Push the release PR branch to GitHub, make a pull request.
- Also push the release tag.
- Create a clean repo and use it for the following steps::
git clone borg borg-clean
@ -340,8 +401,9 @@ Checklist:
It will also reveal uncommitted required files.
Moreover, it makes sure the vagrant machines only get committed files and
do a fresh start based on that.
- Run tox and/or binary builds on all supported platforms via vagrant,
check for test failures.
- Optional: run tox and/or binary builds on all supported platforms via vagrant,
check for test failures. This is now optional as we do platform testing and
binary building on GitHub.
- Create sdist, sign it, upload release to (test) PyPi:
::
@ -349,26 +411,32 @@ Checklist:
scripts/sdist-sign X.Y.Z
scripts/upload-pypi X.Y.Z test
scripts/upload-pypi X.Y.Z
- Put binaries into dist/borg-OSNAME and sign them:
::
Note: the signature is not uploaded to PyPi any more, but we upload it to
github releases.
scripts/sign-binaries 201912312359
- When GitHub CI looks good on the release PR, merge it and then check "Actions":
GitHub will create binary assets after the release PR is merged within the
CI testing of the merge. Check the "Upload binaries" step on Ubuntu (AMD/Intel
and ARM64) and macOS (Intel and ARM64), fetch the ZIPs with the binaries.
- Unpack the ZIPs and test the binaries, upload the binaries to the GitHub
release page (borg-OS-SPEC-ARCH-gh and borg-OS-SPEC-ARCH-gh.tgz).
- Close the release milestone on GitHub.
- `Update borgbackup.org
<https://github.com/borgbackup/borgbackup.github.io/pull/53/files>`_ with the
new version number and release date.
- Announce on:
- Mailing list.
- Twitter.
- IRC channel (change ``/topic``).
- Mailing list.
- Mastodon / BlueSky / X (aka Twitter).
- IRC channel (change ``/topic``).
- Create a GitHub release, include:
* Standalone binaries (see above for how to create them).
- pypi dist package and signature
- Standalone binaries (see above for how to create them).
+ For OS X, document the OS X Fuse version in the README of the binaries.
OS X FUSE uses a kernel extension that needs to be compatible with the
code contained in the binary.
* A link to ``CHANGES.rst``.
- For macOS binaries **with** FUSE support, document the macFUSE version
in the README of the binaries. macFUSE uses a kernel extension that needs
to be compatible with the code contained in the binary.
- A link to ``CHANGES.rst``.

View file

@ -45,6 +45,12 @@ repository is only modified from one place. Also keep in mind that
Borg will keep an exclusive lock on the repository while creating
or deleting archives, which may make *simultaneous* backups fail.
Can I back up to multiple, swapped backup targets?
--------------------------------------------------
It is possible to swap your backup disks if each backup medium is assigned its
own repository by creating a new one with :ref:`borg_init`.
Can I copy or synchronize my repo to another location?
------------------------------------------------------
@ -107,6 +113,9 @@ run into this by yourself by restoring an older copy of your repository.
"attack": maybe an attacker has replaced your repo by an older copy, trying to
trick you into AES counter reuse, trying to break your repo encryption.
Borg users have also reported that fs issues (like hw issues / I/O errors causing
the fs to become read-only) can cause this warning, see :issue:`7853`.
If you'ld decide to ignore this and accept unsafe operation for this repository,
you could delete the manifest-timestamp and the local cache:
@ -201,15 +210,15 @@ sure you have an up-to-date version of borgbackup if you want to continue
instead of retransferring a huge file. In some cases, there is only an outdated
version shipped with your distribution (e.g. Debian). See :ref:`installation`.
How can I backup huge file(s) over a unstable connection?
---------------------------------------------------------
How can I back up huge file(s) over an unstable connection?
-----------------------------------------------------------
This is not a problem anymore.
For more details, see :ref:`checkpoints_parts`.
How can I switch append-only mode on and off?
-----------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------
You could do that (via borg config REPO append_only 0/1), but using different
ssh keys and different entries in ``authorized_keys`` is much easier and also
@ -219,8 +228,12 @@ maybe has less potential of things going wrong somehow.
My machine goes to sleep causing `Broken pipe`
----------------------------------------------
When backing up your data over the network, your machine should not go to sleep.
On macOS you can use `caffeinate` to avoid that.
While backing up your data over the network, your machine should not go to sleep.
On Linux you can use `systemd-inhibit` to avoid that. On macOS you can use `caffeinate`.
``systemd-inhibit borg create ...``
``caffeinate -i borg create ...``
How can I restore huge file(s) over an unstable connection?
-----------------------------------------------------------
@ -368,9 +381,7 @@ Checking processors
operate hardware outside its specifications for productive use.
Tools to verify correct processor operation include Prime95 (mprime), linpack,
and the `Intel Processor Diagnostic Tool
<https://downloadcenter.intel.com/download/19792/Intel-Processor-Diagnostic-Tool>`_
(applies only to Intel processors).
and stress-ng.
.. rubric:: Repairing a damaged repository
@ -429,18 +440,19 @@ How do I configure different prune policies for different directories?
Say you want to prune ``/var/log`` faster than the rest of
``/``. How do we implement that? The answer is to backup to different
archive *names* and then implement different prune policies for
different prefixes. For example, you could have a script that does::
different --glob-archives matching patterns.
borg create --exclude /var/log $REPOSITORY:main-$(date +%Y-%m-%d) /
For example, you could have a script that does::
borg create --exclude var/log $REPOSITORY:main-$(date +%Y-%m-%d) /
borg create $REPOSITORY:logs-$(date +%Y-%m-%d) /var/log
Then you would have two different prune calls with different policies::
borg prune --verbose --list -d 30 --prefix main- "$REPOSITORY"
borg prune --verbose --list -d 7 --prefix logs- "$REPOSITORY"
borg prune --verbose --list -d 30 --glob-archives 'main-*' "$REPOSITORY"
borg prune --verbose --list -d 7 --glob-archives 'logs-*' "$REPOSITORY"
This will keep 7 days of logs and 30 days of everything else. Borg 1.1
also supports the ``--glob-archives`` parameter.
This will keep 7 days of logs and 30 days of everything else.
How do I remove files from an existing backup?
----------------------------------------------
@ -468,6 +480,24 @@ will only be applied to new chunks, not existing chunks. So it's safe
to change them.
Why is backing up an unmodified FAT filesystem slow on Linux?
-------------------------------------------------------------
By default, the files cache used by BorgBackup considers the inode of files.
When an inode number changes compared to the last backup, it hashes the file
again. The ``vfat`` kernel driver does not produce stable inode numbers by
default. One way to achieve stable inode numbering is mounting the filesystem
using ``nfs=nostale_ro``. Doing so implies mounting the filesystem read-only.
Another option is to not consider inode numbers in the files cache by passing
``--files-cache=ctime,size``.
Why are backups slow on a Linux server that is a member of a Windows domain?
----------------------------------------------------------------------------
If a Linux server is a member of a Windows domain, username to userid resolution might be
performed via ``winbind`` without caching, which can slow down backups significantly.
You can use e.g. ``nscd`` to add caching and improve the speed.
Security
########
@ -514,14 +544,33 @@ The Borg config directory has content that you should take care of:
recovered.
``keys`` subdirectory
All your borg keyfile keys are stored in this directory. Please note that
borg repokey keys are stored inside the repository. You MUST make sure to have an
independent backup of these keyfiles, otherwise you cannot access your backups anymore if you lose
them. You also MUST keep these files secret; everyone who gains access to your repository and has
the corresponding keyfile (and the key passphrase) can extract it.
All your borg keyfile keys are stored in this directory. Please note that borg
repokey keys are stored inside the repository. In any case, you MUST make sure
to have an independent backup of the borg keys, see :ref:`borg_key_export` for
more details.
Make sure that only you have access to the Borg config directory.
Note about creating multiple keyfile repositories at the same path
------------------------------------------------------------------
If you create a new keyfile-encrypted repository at the same filesystem
path multiple times (for example, when a previous repository at that path
was moved away or unmounted), Borg will not overwrite or reuse the existing
key file in your keys directory. Instead, it creates a new key file by
appending a numeric suffix to the base name (e.g., .2, .3, ...).
This means you may see multiple key files like:
- ~/.config/borg/keys/home_user_backup
- ~/.config/borg/keys/home_user_backup.2
- ~/.config/borg/keys/home_user_backup.3
Each of these corresponds to a distinct repository created at the same
path at different times. This behavior avoids accidental key reuse or
overwrite.
.. _cache_security:
Do I need to take security precautions regarding the cache?
@ -564,8 +613,7 @@ Using ``BORG_PASSCOMMAND`` with a properly permissioned file
directory and use permissions to keep anyone else from reading it. For
example, first create a key::
head -c 32 /dev/urandom | base64 -w 0 > ~/.borg-passphrase
chmod 400 ~/.borg-passphrase
(umask 0077; head -c 32 /dev/urandom | base64 -w 0 > ~/.borg-passphrase)
Then in an automated script one can put::
@ -655,7 +703,7 @@ all your files/dirs data and metadata are stored in their encrypted form
into the repository.
Yes, as an attacker with access to the remote server could delete (or
otherwise make unavailable) all your backups.
otherwise make unavailable) all your backups on that server.
How can I protect against a hacked backup client?
-------------------------------------------------
@ -692,7 +740,7 @@ These are your options to protect against that:
How can I protect against theft, sabotage, lightning, fire, ...?
----------------------------------------------------------------
In general: if your only backup medium is nearby the backupped machine and
In general: if your only backup medium is nearby the backed-up machine and
always connected, you can easily get into trouble: they likely share the same
fate if something goes really wrong.
@ -727,7 +775,7 @@ Since the nonce is not necessary to read the data that is already encrypted,
``borg info``, ``borg list``, ``borg extract`` and ``borg mount`` should work
just fine without it.
If the the nonce file stored in the repo is lost, but you still have your local copy,
If the nonce file stored in the repo is lost, but you still have your local copy,
borg will recreate the repository nonce file the next time you run ``borg create``.
This should be safe for repositories that are only used from one user account
on one machine.
@ -739,31 +787,34 @@ the nonce is deleted or if you suspect it may have been tampered with. See :ref:
Common issues
#############
Why does Borg extract hang after some time?
-------------------------------------------
/path/to/repo is not a valid repository. Check repo config.
-----------------------------------------------------------
When I do a ``borg extract``, after a while all activity stops, no cpu usage,
no downloads.
There can be many causes of this error. E.g. you have incorrectly specified the repository path.
This may happen when the SSH connection is stuck on server side. You can
configure SSH on client side to prevent this by sending keep-alive requests,
for example in ~/.ssh/config:
You will also get this error if you try to access a repository that uses the argon2 key algorithm using an old version of borg.
We recommend upgrading to the latest stable version and trying again. We are sorry. We should have thought about forward
compatibility and implemented a more helpful error message.
::
Why am I seeing idle borg serve processes on the repo server?
-------------------------------------------------------------
Host borg.example.com
# Client kills connection after 3*30 seconds without server response:
ServerAliveInterval 30
ServerAliveCountMax 3
Please see the next question.
You can also do the opposite and configure SSH on server side in
/etc/ssh/sshd_config, to make the server send keep-alive requests to the client:
Why does Borg disconnect or hang when backing up to a remote server?
--------------------------------------------------------------------
::
Communication with the remote server (using an ssh: repo URL) happens via an SSH
connection. This can lead to some issues that would not occur during a local backup:
# Server kills connection after 3*30 seconds without client response:
ClientAliveInterval 30
ClientAliveCountMax 3
- Since Borg does not send data all the time, the connection may get closed, leading
to errors like "connection closed by remote".
- On the other hand, network issues may lead to a dysfunctional connection
that is only detected after some time by the server, leading to stale ``borg serve``
processes and locked repositories.
To fix such problems, please apply these :ref:`SSH settings <ssh_configuration>` so that
keep-alive requests are sent regularly.
How can I deal with my very unstable SSH connection?
----------------------------------------------------
@ -779,32 +830,6 @@ could try to work around:
to do any more. Due to the way borg mount works, this might be less efficient
than borg extract for bigger volumes of data.
Why do I get "connection closed by remote" after a while?
---------------------------------------------------------
When doing a backup to a remote server (using a ssh: repo URL), it sometimes
stops after a while (some minutes, hours, ... - not immediately) with
"connection closed by remote" error message. Why?
That's a good question and we are trying to find a good answer in :issue:`636`.
Why am I seeing idle borg serve processes on the repo server?
-------------------------------------------------------------
Maybe the ssh connection between client and server broke down and that was not
yet noticed on the server. Try these settings:
::
# /etc/ssh/sshd_config on borg repo server - kill connection to client
# after ClientAliveCountMax * ClientAliveInterval seconds with no response
ClientAliveInterval 20
ClientAliveCountMax 3
If you have multiple borg create ... ; borg create ... commands in a already
serialized way in a single script, you need to give them ``--lock-wait N`` (with N
being a bit more than the time the server needs to terminate broken down
connections and release the lock).
.. _disable_archive_chunks:
@ -822,18 +847,8 @@ will make the subsequent rebuilds faster (because it needs to transfer less data
from the repository). While being faster, the cache needs quite some disk space,
which might be unwanted.
There is a temporary (but maybe long lived) hack to avoid using lots of disk
space for chunks.archive.d (see :issue:`235` for details):
::
# this assumes you are working with the same user as the backup.
cd ~/.cache/borg/$(borg config /path/to/repo id)
rm -rf chunks.archive.d ; touch chunks.archive.d
This deletes all the cached archive chunk indexes and replaces the directory
that kept them with a file, so borg won't be able to store anything "in" there
in future.
You can disable the cached archive chunk indexes by setting the environment
variable ``BORG_USE_CHUNKS_ARCHIVE`` to ``no``.
This has some pros and cons, though:
@ -863,6 +878,12 @@ Check if your encoding is set correctly. For most POSIX-like systems, try::
export LANG=en_US.UTF-8 # or similar, important is correct charset
If that does not help:
- check for typos, check if you really used ``export``.
- check if you have set ``LC_ALL`` - if so, try not setting it.
- check if you generated the respective locale via ``locale-gen``.
I can't extract non-ascii filenames by giving them on the commandline!?
-----------------------------------------------------------------------
@ -933,6 +954,24 @@ Then you do the backup and look at the log output:
details and potential issues).
You can use the ``stat`` command on files to manually look at fs metadata to debug if
there is any unexpected change triggering the ``M`` status.
Also, the ``--debug-topic=files_cache`` option of ``borg create`` provides a lot of debug
output helping to analyse why the files cache does not give its expected high performance.
When borg runs inside a virtual machine, there are some more things to look at:
Some hypervisors (e.g. kvm on older proxmox) give some broadly compatible CPU type to the
VM (usually to ease migration between VM hosts of potentially different hardware CPUs).
It is broadly compatible because they leave away modern CPU features that could be
not present in older or other CPUs, e.g. hardware acceleration for AES crypto, for
sha2 hashes, for (P)CLMUL(QDQ) computations useful for crc32.
So, basically you pay for compatibility with bad performance. If you prefer better
performance, you should try to use the x86-64-v2-AES VCPU or the "host" VCPU,
exposing misc. hw acceleration features to the VM which runs borg.
On Linux, check ``/proc/cpuinfo`` for the CPU flags inside the VM.
For kvm check the docs about "Host model" and "Host passthrough".
See also the next few FAQ entries for more details.
@ -1007,6 +1046,10 @@ will be slow because it would chunk all the files each time. If you set
BORG_FILES_CACHE_TTL to at least 26 (or maybe even a small multiple of that),
it would be much faster.
Besides using a higher BORG_FILES_CACHE_TTL (which also increases memory usage),
there is also BORG_FILES_CACHE_SUFFIX which can be used to have separate (smaller)
files caches for each backup set instead of the default one (big) unified files cache.
Another possible reason is that files don't always have the same path, for
example if you mount a filesystem without stable mount points for each backup
or if you are running the backup from a filesystem snapshot whose name is not
@ -1020,6 +1063,32 @@ A workaround is to set the option ``--files-cache=ctime,size`` to exclude the in
number comparison from the files cache check so that files with different inode
numbers won't be treated as modified.
Using a pure-python msgpack! This will result in lower performance.
-------------------------------------------------------------------
borg uses `msgpack` to serialize/deserialize data.
`msgpack` has 2 implementations:
- a fast one (C code compiled into a platform specific binary), and
- a slow pure-python one.
The slow one is used if it can't successfully import the fast one.
If you use the pyinstaller-made borg "fat binary" which we offer on github
releases, it could be that you downloaded a binary that does not match the
(g)libc on your system.
Binaries made for an older glibc than the one you have on your system usually
just work, but the opposite is not necessarily the case and can lead to misc.
issues - like failing to load the fast msgpack code or not working at all.
So: try a binary made for an older glibc.
If you see this without using a "fat binary" from us, it usually means that
msgpack is not built / installed correctly. It could be also that the platform
is not fully supported (so the python code works, but there is no fast binary
code).
Is there a way to limit bandwidth with Borg?
--------------------------------------------
@ -1178,10 +1247,58 @@ It may be useful to set ``BORG_RELOCATED_REPO_ACCESS_IS_OK=yes`` to avoid the
prompts when renaming multiple repositories or in a non-interactive context
such as a script. See :doc:`deployment` for an example.
The repository quota size is reached, what can I do?
----------------------------------------------------
The simplest solution is to increase or disable the quota and resume the backup:
::
borg config /path/to/repo storage_quota 0
If you are bound to the quota, you have to free repository space. The first to
try is running :ref:`borg_compact` to free unused backup space (see also
:ref:`separate_compaction`):
::
borg compact /path/to/repo
If your repository is already compacted, run :ref:`borg_prune` or
:ref:`borg_delete` to delete archives that you do not need anymore, and then run
``borg compact`` again.
My backup disk is full, what can I do?
--------------------------------------
Borg cannot work if you really have zero free space on the backup disk, so the
first thing you must do is deleting some files to regain free disk space. See
:ref:`about_free_space` for further details.
Some Borg commands that do not change the repository might work under disk-full
conditions, but generally this should be avoided. If your backup disk is already
full when Borg starts a write command like `borg create`, it will abort
immediately and the repository will stay as-is.
If you run a backup that stops due to a disk running full, Borg will roll back,
delete the new new segment file and thus freeing disk space automatically. There
may be a checkpoint archive left that has been saved before the disk got full.
You can keep it to speed up the next backup or delete it to get back more disk
space.
Miscellaneous
#############
macOS: borg mounts not shown in Finder's side bar
-------------------------------------------------
https://github.com/macfuse/macfuse/wiki/Mount-options#local
Read the above first and use this on your own risk::
borg mount -olocal REPO MOUNTPOINT
Requirements for the borg single-file binary, esp. (g)libc?
-----------------------------------------------------------
@ -1295,7 +1412,7 @@ There are some caveats:
Why is my backup bigger than with attic?
----------------------------------------
Attic was rather unflexible when it comes to compression, it always
Attic was rather inflexible when it comes to compression, it always
compressed using zlib level 6 (no way to switch compression off or
adjust the level or algorithm).

View file

@ -13,9 +13,9 @@
.. _ACL: https://en.wikipedia.org/wiki/Access_control_list
.. _libacl: https://savannah.nongnu.org/projects/acl/
.. _libattr: https://savannah.nongnu.org/projects/attr/
.. _libxxhash: https://github.com/Cyan4973/xxHash
.. _liblz4: https://github.com/Cyan4973/lz4
.. _libzstd: https://github.com/facebook/zstd
.. _libb2: https://github.com/BLAKE2/libb2
.. _OpenSSL: https://www.openssl.org/
.. _`Python 3`: https://www.python.org/
.. _Buzhash: https://en.wikipedia.org/wiki/Buzhash

View file

@ -6,7 +6,7 @@ Borg Documentation
.. include:: ../README.rst
.. when you add an element here, do not forget to add it to book.rst
.. When you add an element here, do not forget to add it to book.rst.
.. toctree::
:maxdepth: 2

View file

@ -42,7 +42,7 @@ package which can be installed with the package manager.
Distribution Source Command
============ ============================================= =======
Alpine Linux `Alpine repository`_ ``apk add borgbackup``
Arch Linux `[community]`_ ``pacman -S borg``
Arch Linux `[extra]`_ ``pacman -S borg``
Debian `Debian packages`_ ``apt install borgbackup``
Gentoo `ebuild`_ ``emerge borgbackup``
GNU Guix `GNU Guix`_ ``guix package --install borg``
@ -63,9 +63,9 @@ Ubuntu `Ubuntu packages`_, `Ubuntu PPA`_ ``apt install borgbac
============ ============================================= =======
.. _Alpine repository: https://pkgs.alpinelinux.org/packages?name=borgbackup
.. _[community]: https://www.archlinux.org/packages/?name=borg
.. _[extra]: https://www.archlinux.org/packages/?name=borg
.. _Debian packages: https://packages.debian.org/search?keywords=borgbackup&searchon=names&exact=1&suite=all&section=all
.. _Fedora official repository: https://apps.fedoraproject.org/packages/borgbackup
.. _Fedora official repository: https://packages.fedoraproject.org/pkgs/borgbackup/borgbackup/
.. _FreeBSD ports: https://www.freshports.org/archivers/py-borgbackup/
.. _ebuild: https://packages.gentoo.org/packages/app-backup/borgbackup
.. _GNU Guix: https://www.gnu.org/software/guix/package-list.html#borg
@ -78,7 +78,7 @@ Ubuntu `Ubuntu packages`_, `Ubuntu PPA`_ ``apt install borgbac
.. _Homebrew: https://formulae.brew.sh/formula/borgbackup
.. _private Tap: https://github.com/borgbackup/homebrew-tap
.. _Raspbian testing: https://archive.raspbian.org/raspbian/pool/main/b/borgbackup/
.. _Ubuntu packages: https://packages.ubuntu.com/xenial/borgbackup
.. _Ubuntu packages: https://launchpad.net/ubuntu/+source/borgbackup
.. _Ubuntu PPA: https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/borgbackup
Please ask package maintainers to build a package or, if you can package /
@ -105,22 +105,20 @@ Standalone Binary
.. note:: Releases are signed with an OpenPGP key, see
:ref:`security-contact` for more instructions.
Borg x86/x64 amd/intel compatible binaries (generated with `pyinstaller`_)
are available on the releases_ page for the following platforms:
Borg x86-64 AMD/Intel-compatible binaries (generated with `pyinstaller`_)
are available on the releases_ page for the following platforms (for more
details see the ``00_README.txt`` file there):
* **Linux**: glibc >= 2.28 (ok for most supported Linux releases).
Older glibc releases are untested and may not work.
* **MacOS**: 10.12 or newer (To avoid signing issues download the file via
command line **or** remove the ``quarantine`` attribute after downloading:
``$ xattr -dr com.apple.quarantine borg-macosx64.tgz``)
* **FreeBSD**: 12.1 (unknown whether it works for older releases)
* Linux
* FreeBSD
* MacOS
ARM binaries are built by Johann Bauer, see: https://borg.bauerj.eu/
To install such a binary, just drop it into a directory in your ``PATH``,
make borg readable and executable for its users and then you can run ``borg``::
sudo cp borg-linux64 /usr/local/bin/borg
sudo cp borg-linux /usr/local/bin/borg
sudo chown root:root /usr/local/bin/borg
sudo chmod 755 /usr/local/bin/borg
@ -157,30 +155,26 @@ Dependencies
~~~~~~~~~~~~
To install Borg from a source package (including pip), you have to install the
following dependencies first:
following dependencies first. For the libraries you will also need their
development header files (sometimes in a separate `-dev` or `-devel` package).
* `Python 3`_ >= 3.8.0, plus development headers.
* OpenSSL_ >= 1.0.0, plus development headers.
* libacl_ (which depends on libattr_), both plus development headers.
* We have bundled code of the following packages, but borg by default (see
setup.py if you want to change that) prefers a shared library if it can
be found on the system (lib + dev headers) at build time:
- liblz4_ >= 1.7.0 (r129)
- libzstd_ >= 1.3.0
- libxxhash >= 0.8.1 (0.8.0 might work also)
* pkg-config (cli tool) and pkgconfig python package (borg uses these to
discover header and library location - if it can't import pkgconfig and
is not pointed to header/library locations via env vars [see setup.py],
it will fall back to using the bundled code, see above).
**These must be present before invoking setup.py!**
* some other Python dependencies, pip will automatically install them for you.
* optionally, if you wish to mount an archive as a FUSE filesystem, you need
* `Python 3`_ >= 3.10.0
* OpenSSL_ >= 1.0.0
* libacl_ (which depends on libattr_)
* libxxhash_ >= 0.8.1
* liblz4_ >= 1.7.0 (r129)
* libzstd_ >= 1.3.0
* pkg-config (cli tool) - Borg uses this to discover header and library
locations automatically. Alternatively, you can also point to them via some
environment variables, see setup.py.
* Some other Python dependencies, pip will automatically install them for you.
* Optionally, if you wish to mount an archive as a FUSE filesystem, you need
a FUSE implementation for Python:
- Either pyfuse3_ (preferably, newer and maintained) or llfuse_ (older,
unmaintained now). See also the BORG_FUSE_IMPL env variable.
- See setup.py about the version requirements.
- pyfuse3_ >= 3.1.1 (for fuse 3, use `pip install borgbackup[pyfuse3]`), or
- llfuse_ >= 1.3.8 (for fuse 2, use `pip install borgbackup[llfuse]`).
- Additionally, your OS will need to have FUSE support installed
(e.g. a package `fuse` for fuse 2 or a package `fuse3` for fuse 3 support).
If you have troubles finding the right package names, have a look at the
distribution specific sections below or the Vagrantfile in the git repository,
@ -188,24 +182,34 @@ which contains installation scripts for a number of operating systems.
In the following, the steps needed to install the dependencies are listed for a
selection of platforms. If your distribution is not covered by these
instructions, try to use your package manager to install the dependencies. On
FreeBSD, you may need to get a recent enough OpenSSL version from FreeBSD
ports.
instructions, try to use your package manager to install the dependencies.
After you have installed the dependencies, you can proceed with steps outlined
under :ref:`pip-installation`.
Arch Linux
++++++++++
Install the runtime and build dependencies::
pacman -S python python-pip python-virtualenv openssl acl xxhash lz4 zstd base-devel
pacman -S fuse2 # needed for llfuse
pacman -S fuse3 # needed for pyfuse3
Note that Arch Linux specifically doesn't support
`partial upgrades <https://wiki.archlinux.org/title/Partial_upgrade>`__,
so in case some packages cannot be retrieved from the repo, run with ``pacman -Syu``.
Debian / Ubuntu
+++++++++++++++
Install the dependencies with development headers::
sudo apt-get install python3 python3-dev python3-pip python3-virtualenv \
libacl1-dev libacl1 \
libacl1-dev \
libssl-dev \
liblz4-dev libzstd-dev libxxhash-dev \
build-essential \
pkg-config python3-pkgconfig
build-essential pkg-config
sudo apt-get install libfuse-dev fuse # needed for llfuse
sudo apt-get install libfuse3-dev fuse3 # needed for pyfuse3
@ -219,10 +223,10 @@ Fedora
Install the dependencies with development headers::
sudo dnf install python3 python3-devel python3-pip python3-virtualenv \
libacl-devel libacl \
libacl-devel \
openssl-devel \
lz4-devel libzstd-devel xxhash-devel \
pkgconf python3-pkgconfig
pkgconf
sudo dnf install gcc gcc-c++ redhat-rpm-config
sudo dnf install fuse-devel fuse # needed for llfuse
sudo dnf install fuse3-devel fuse3 # needed for pyfuse3
@ -247,16 +251,10 @@ Alternatively, you can enumerate all build dependencies in the command line::
macOS
+++++
When installing via Homebrew_, dependencies are installed automatically. To install
dependencies manually::
When installing borgbackup via Homebrew_, the basic dependencies are installed automatically.
brew install python3 openssl zstd lz4 xxhash
brew install pkg-config
pip3 install virtualenv pkgconfig
For FUSE support to mount the backup archives, you need at least version 3.0 of
macFUSE, which is available via `github
<https://github.com/osxfuse/osxfuse/releases/latest>`__, or Homebrew::
For FUSE support to mount the backup archives, you need macFUSE, which is available
via `github <https://github.com/osxfuse/osxfuse/releases/latest>`__, or Homebrew::
brew install --cask macfuse
@ -266,7 +264,14 @@ the installed ``openssl`` formula, point pkg-config to the correct path::
PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig" pip install borgbackup[llfuse]
For OS X Catalina and later, be aware that you must authorize full disk access.
When working from a borg git repo workdir, you can install dependencies using the
Brewfile::
brew install python@3.11 # can be any supported python3 version
brew bundle install # install requirements from borg repo's ./Brewfile
pip3 install virtualenv
Be aware that for all recent macOS releases you must authorize full disk access.
It is no longer sufficient to run borg backups as root. If you have not yet
granted full disk access, and you run Borg backup from cron, you will see
messages such as::
@ -315,8 +320,8 @@ Cygwin
Use the Cygwin installer to install the dependencies::
python38 python38-devel python38-pkgconfig
python38-setuptools python38-pip python38-wheel python38-virtualenv
python39 python39-devel
python39-setuptools python39-pip python39-wheel python39-virtualenv
libssl-devel libxxhash-devel liblz4-devel libzstd-devel
binutils gcc-g++ git make openssh
@ -330,6 +335,8 @@ Virtualenv_ can be used to build and install Borg without affecting
the system Python or requiring root access. Using a virtual environment is
optional, but recommended except for the most simple use cases.
Ensure to install the dependencies as described within :ref:`source-install`.
.. note::
If you install into a virtual environment, you need to **activate** it
first (``source borg-env/bin/activate``), before running ``borg``.
@ -344,9 +351,6 @@ This will use ``pip`` to install the latest release from PyPi::
# might be required if your tools are outdated
pip install -U pip setuptools wheel
# pkgconfig MUST be available before borg is installed!
pip install pkgconfig
# install Borg + Python dependencies into virtualenv
pip install borgbackup
# or alternatively (if you want FUSE support):
@ -358,6 +362,19 @@ activating your virtual environment::
pip install -U borgbackup # or ... borgbackup[llfuse/pyfuse3]
When doing manual pip installation, man pages are not automatically
installed. You can run these commands to install the man pages
locally::
# get borg from github
git clone https://github.com/borgbackup/borg.git borg
# Install the files with proper permissions
install -D -m 0644 borg/docs/man/borg*.1* $HOME/.local/share/man/man1/borg.1
# Update the man page cache
mandb
.. _git-installation:
Using git
@ -366,20 +383,30 @@ Using git
This uses latest, unreleased development code from git.
While we try not to break master, there are no guarantees on anything.
Ensure to install the dependencies as described within :ref:`source-install`.
Version metadata is obtained dynamically at install time using ``setuptools-scm``.
Please ensure that your git repo either has correct tags, or provide the version
manually using the ``SETUPTOOLS_SCM_PRETEND_VERSION`` environment variable.
::
# get borg from github
git clone https://github.com/borgbackup/borg.git
# create a virtual environment
virtualenv --python=${which python3} borg-env
virtualenv --python=$(which python3) borg-env
source borg-env/bin/activate # always before using!
# install borg + dependencies into virtualenv
# install borg dependencies into virtualenv
cd borg
pip install -r requirements.d/development.txt
pip install -r requirements.d/docs.txt # optional, to build the docs
# set a borg version if setuptools-scm fails to do so automatically
export SETUPTOOLS_SCM_PRETEND_VERSION=
# install borg into virtualenv
pip install -e . # in-place editable mode
or
pip install -e .[pyfuse3] # in-place editable mode, use pyfuse3
@ -397,9 +424,9 @@ If you need to use a different version of Python you can install this using ``py
...
# create a virtual environment
pyenv install 3.8.0 # minimum, preferably use something more recent!
pyenv global 3.8.0
pyenv local 3.8.0
pyenv install 3.10.0 # minimum, preferably use something more recent!
pyenv global 3.10.0
pyenv local 3.10.0
virtualenv --python=${pyenv which python} borg-env
source borg-env/bin/activate # always before using!
...

View file

@ -4,7 +4,7 @@
Internals
=========
The internals chapter describes and analyses most of the inner workings
The internals chapter describes and analyzes most of the inner workings
of Borg.
Borg uses a low-level, key-value store, the :ref:`repository`, and
@ -19,12 +19,12 @@ specified when the backup was performed.
Deduplication is performed globally across all data in the repository
(multiple backups and even multiple hosts), both on data and file
metadata, using :ref:`chunks` created by the chunker using the
Buzhash_ algorithm ("buzhash" chunker) or a simpler fixed blocksize
Buzhash_ algorithm ("buzhash" chunker) or a simpler fixed block size
algorithm ("fixed" chunker).
To actually perform the repository-wide deduplication, a hash of each
chunk is checked against the :ref:`chunks cache <cache>`, which is a
hash-table of all chunks that already exist.
hash table of all chunks that already exist.
.. figure:: internals/structure.png
:figwidth: 100%

View file

@ -21,29 +21,29 @@ Repository
.. Some parts of this description were taken from the Repository docstring
Borg stores its data in a `Repository`, which is a file system based
Borg stores its data in a `Repository`, which is a filesystem-based
transactional key-value store. Thus the repository does not know about
the concept of archives or items.
Each repository has the following file structure:
README
simple text file telling that this is a Borg repository
Simple text file telling that this is a Borg repository
config
repository configuration
Repository configuration
data/
directory where the actual data is stored
Directory where the actual data is stored
hints.%d
hints for repository compaction
Hints for repository compaction
index.%d
repository index
Repository index
lock.roster and lock.exclusive/*
used by the locking system to manage shared and exclusive locks
Used by the locking system to manage shared and exclusive locks
Transactionality is achieved by using a log (aka journal) to record changes. The log is a series of numbered files
called segments_. Each segment is a series of log entries. The segment number together with the offset of each
@ -94,13 +94,13 @@ this value in a non-empty repository, you may also need to relocate the segment
files manually.
A segment starts with a magic number (``BORG_SEG`` as an eight byte ASCII string),
followed by a number of log entries. Each log entry consists of:
followed by a number of log entries. Each log entry consists of: (in this order)
* 32-bit size of the entry
* CRC32 of the entire entry (for a PUT this includes the data)
* entry tag: PUT, DELETE or COMMIT
* PUT and DELETE follow this with the 32 byte key
* PUT follow the key with the data
* First, unsigned 32-bit number, the CRC32 of the entire entry (for a PUT including the DATA) excluding the CRC32 field
* Second, unsigned 32-bit size of the entry (including the whole header)
* Third, unsigned 8-bit entry tag: PUT(0), DELETE(1) or COMMIT(2)
* Fourth, on PUT or DELETE, 32 byte key
* Fifth, PUT only, (size - 41) bytes of data (length = size - sizeof(CRC32) - sizeof(size) - sizeof(entry tag) - sizeof(key))
Those files are strictly append-only and modified only once.
@ -121,6 +121,14 @@ partial/uncommitted transaction.
The size of individual segments is limited to 4 GiB, since the offset of entries
within segments is stored in a 32-bit unsigned integer in the repository index.
Objects
~~~~~~~
All objects (the manifest, archives, archive item streams chunks and file data
chunks) are encrypted and/or compressed. See :ref:`data-encryption` for a
graphic outlining the anatomy of an object in Borg. The `type` for compression
is explained in :ref:`data-compression`.
Index, hints and integrity
~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -649,7 +657,7 @@ can be used to tune the chunker parameters, the default is:
- CHUNK_MIN_EXP = 19 (minimum chunk size = 2^19 B = 512 kiB)
- CHUNK_MAX_EXP = 23 (maximum chunk size = 2^23 B = 8 MiB)
- HASH_MASK_BITS = 21 (target chunk size ~= 2^21 B = 2 MiB)
- HASH_WINDOW_SIZE = 4095 [B] (`0xFFF`)
- HASH_WINDOW_SIZE = 4095 [B] (`0xFFF`) (must be an odd number)
The buzhash table is altered by XORing it with a seed randomly generated once
for the repository, and stored encrypted in the keyfile. This is to prevent
@ -810,7 +818,7 @@ bucket is reached.
This particular mode of operation is open addressing with linear probing.
When the hash table is filled to 75%, its size is grown. When it's
emptied to 25%, its size is shrinked. Operations on it have a variable
emptied to 25%, its size is shrunk. Operations on it have a variable
complexity between constant and linear with low factor, and memory overhead
varies between 33% and 300%.
@ -860,6 +868,8 @@ HashIndex is implemented in C and wrapped with Cython in a class-based interface
The Cython wrapper checks every passed value against these reserved values and
raises an AssertionError if they are used.
.. _data-encryption:
Encryption
----------
@ -961,18 +971,23 @@ key file, wrapped using the standard ``textwrap`` module with a header.
The header is a single line with a MAGIC string, a space and a hexadecimal
representation of the repository id.
.. _data-compression:
Compression
-----------
Borg supports the following compression methods:
Borg supports the following compression methods, each identified by two bytes:
- none (no compression, pass through data 1:1)
- lz4 (low compression, but super fast)
- none (no compression, pass through data 1:1), identified by ``\x00\x00``
- lz4 (low compression, but super fast), identified by ``\x01\x00``
- zstd (level 1-22 offering a wide range: level 1 is lower compression and high
speed, level 22 is higher compression and lower speed) - since borg 1.1.4
speed, level 22 is higher compression and lower speed) - since borg 1.1.4,
identified by ``\x03\x00``
- zlib (level 0-9, level 0 is no compression [but still adding zlib overhead],
level 1 is low, level 9 is high compression)
- lzma (level 0-9, level 0 is low, level 9 is high compression).
level 1 is low, level 9 is high compression), identified by a zlib header
(``\x.8\x..``)
- lzma (level 0-9, level 0 is low, level 9 is high compression), identified
by ``\x02\x00``.
Speed: none > lz4 > zlib > lzma, lz4 > zstd
Compression: lzma > zlib > lz4 > none, zstd > lz4
@ -1132,7 +1147,7 @@ cache ``config``, since all three are transacted together.
The ``[integrity]`` section is used:
.. code-block:: ini
.. code-block:: none
[cache]
version = 1

View file

@ -10,18 +10,18 @@ Borg does not have a public API on the Python level. That does not keep you from
but does mean that there are no release-to-release guarantees on what you might find in that package, not
even for point releases (1.1.x), and there is no documentation beyond the code and the internals documents.
Borg does on the other hand provide an API on a command-line level. In other words, a frontend should to
(for example) create a backup archive just invoke :ref:`borg_create`, give commandline parameters/options
as needed and parse JSON output from borg.
Borg does, on the other hand, provide an API on a command-line level. In other words, a frontend should
(for example) create a backup archive by invoking :ref:`borg_create`, pass command-line parameters/options
as needed, and parse JSON output from Borg.
Important: JSON output is expected to be UTF-8, but currently borg depends on the locale being configured
for that (must be a UTF-8 locale and *not* "C" or "ascii"), so that Python will choose to encode to UTF-8.
The same applies to any inputs read by borg, they are expected to be UTF-8 encoded also.
Important: JSON output is expected to be UTF-8, but currently Borg depends on the locale being configured
for that (must be a UTF-8 locale and not "C" or "ASCII"), so that Python will choose to encode to UTF-8.
The same applies to any inputs read by Borg; they are expected to be UTF-8 encoded also.
We consider this a bug (see :issue:`2273`) and might fix it later, so borg will use UTF-8 independent of
the locale.
On POSIX systems, you can usually set environment vars to choose a UTF-8 locale:
On POSIX systems, you can usually set environment variables to choose a UTF-8 locale:
::
@ -50,17 +50,20 @@ archive_progress
The following keys exist, each represents the current progress.
original_size
Original size of data processed so far (before compression and deduplication)
Original size of data processed so far (before compression and deduplication, may be empty/absent)
compressed_size
Compressed size
Compressed size (may be empty/absent)
deduplicated_size
Deduplicated size
Deduplicated size (may be empty/absent)
nfiles
Number of (regular) files processed so far
Number of (regular) files processed so far (may be empty/absent)
path
Current path
Current path (may be empty/absent)
time
Unix timestamp (float)
finished
boolean indicating whether the operation has finished, only the last object for an *operation*
can have this property set to *true*.
progress_message
A message-based progress information with no concrete progress information, just a message
@ -90,12 +93,14 @@ progress_percent
can have this property set to *true*.
message
A formatted progress message, this will include the percentage and perhaps other information
(absent for finished == true)
current
Current value (always less-or-equal to *total*)
Current value (always less-or-equal to *total*, absent for finished == true)
info
Array that describes the current item, may be *null*, contents depend on *msgid*
(absent for finished == true)
total
Total value
Total value (absent for finished == true)
time
Unix timestamp (float)
@ -465,13 +470,13 @@ changes:
A list of *Change* objects describing the changes made to the item in the two archives. For example,
there will be two changes if the contents of a file are changed, and its ownership are changed.
The *Change* object can contain a number of properties depending on the type of change that occured.
The *Change* object can contain a number of properties depending on the type of change that occurred.
If a 'property' is not required for the type of change, it is not output.
The possible properties of a *Change* object are:
type:
The **type** property is always present. It identifies the type of change and will be one of these values:
- *modified* - file contents changed.
- *added* - the file was added.
- *removed* - the file was removed.
@ -495,26 +500,26 @@ added:
removed:
See **added** property.
old_mode:
If **type** == '*mode*', then **old_mode** and **new_mode** provide the mode and permissions changes.
new_mode:
See **old_mode** property.
old_user:
If **type** == '*owner*', then **old_user**, **new_user**, **old_group** and **new_group** provide the user
and group ownership changes.
old_group:
See **old_user** property.
new_user:
See **old_user** property.
new_group:
See **old_user** property.
Example (excerpt) of ``borg diff --json-lines``::
@ -533,92 +538,182 @@ Message IDs are strings that essentially give a log message or operation a name,
full text, since texts change more frequently. Message IDs are unambiguous and reduce the need to parse
log messages.
Assigned message IDs are:
Assigned message IDs and related error RCs (exit codes) are:
.. See scripts/errorlist.py; this is slightly edited.
Errors
Archive.AlreadyExists
Archive {} already exists
Archive.DoesNotExist
Archive {} does not exist
Archive.IncompatibleFilesystemEncodingError
Failed to encode filename "{}" into file system encoding "{}". Consider configuring the LANG environment variable.
Cache.CacheInitAbortedError
Cache initialization aborted
Cache.EncryptionMethodMismatch
Repository encryption method changed since last access, refusing to continue
Cache.RepositoryAccessAborted
Repository access aborted
Cache.RepositoryIDNotUnique
Cache is newer than repository - do you have multiple, independently updated repos with same ID?
Cache.RepositoryReplay
Cache is newer than repository - this is either an attack or unsafe (multiple repos with same ID)
Buffer.MemoryLimitExceeded
Error rc: 2 traceback: no
Error: {}
ErrorWithTraceback rc: 2 traceback: yes
Error: {}
Buffer.MemoryLimitExceeded rc: 2 traceback: no
Requested buffer size {} is above the limit of {}.
ExtensionModuleError
The Borg binary extension modules do not seem to be properly installed
IntegrityError
Data integrity error: {}
NoManifestError
Repository has no manifest.
PlaceholderError
EfficientCollectionQueue.SizeUnderflow rc: 2 traceback: no
Could not pop_front the first {} elements; collection only has {} elements.
RTError rc: 2 traceback: no
Runtime Error: {}
CancelledByUser rc: 3 traceback: no
Cancelled by user.
CommandError rc: 4 traceback: no
Command Error: {}
PlaceholderError rc: 5 traceback: no
Formatting Error: "{}".format({}): {}({})
KeyfileInvalidError
Invalid key file for repository {} found in {}.
KeyfileMismatchError
Mismatch between repository {} and key file {}.
KeyfileNotFoundError
No key file for repository {} found in {}.
PassphraseWrong
passphrase supplied in BORG_PASSPHRASE is incorrect
PasswordRetriesExceeded
exceeded the maximum password retries
RepoKeyNotFoundError
No key entry found in the config of repository {}.
UnsupportedManifestError
InvalidPlaceholder rc: 6 traceback: no
Invalid placeholder "{}" in string: {}
Repository.AlreadyExists rc: 10 traceback: no
A repository already exists at {}.
Repository.AtticRepository rc: 11 traceback: no
Attic repository detected. Please run "borg upgrade {}".
Repository.CheckNeeded rc: 12 traceback: yes
Inconsistency detected. Please run "borg check {}".
Repository.DoesNotExist rc: 13 traceback: no
Repository {} does not exist.
Repository.InsufficientFreeSpaceError rc: 14 traceback: no
Insufficient free space to complete transaction (required: {}, available: {}).
Repository.InvalidRepository rc: 15 traceback: no
{} is not a valid repository. Check repo config.
Repository.InvalidRepositoryConfig rc: 16 traceback: no
{} does not have a valid configuration. Check repo config [{}].
Repository.ObjectNotFound rc: 17 traceback: yes
Object with key {} not found in repository {}.
Repository.ParentPathDoesNotExist rc: 18 traceback: no
The parent path of the repo directory [{}] does not exist.
Repository.PathAlreadyExists rc: 19 traceback: no
There is already something at {}.
Repository.StorageQuotaExceeded rc: 20 traceback: no
The storage quota ({}) has been exceeded ({}). Try deleting some archives.
Repository.PathPermissionDenied rc: 21 traceback: no
Permission denied to {}.
MandatoryFeatureUnsupported rc: 25 traceback: no
Unsupported repository feature(s) {}. A newer version of Borg is required to access this repository.
NoManifestError rc: 26 traceback: no
Repository has no manifest.
UnsupportedManifestError rc: 27 traceback: no
Unsupported manifest envelope. A newer version is required to access this repository.
UnsupportedPayloadError
Unsupported payload type {}. A newer version is required to access this repository.
NotABorgKeyFile
This file is not a borg key backup, aborting.
RepoIdMismatch
Archive.AlreadyExists rc: 30 traceback: no
Archive {} already exists
Archive.DoesNotExist rc: 31 traceback: no
Archive {} does not exist
Archive.IncompatibleFilesystemEncodingError rc: 32 traceback: no
Failed to encode filename "{}" into file system encoding "{}". Consider configuring the LANG environment variable.
KeyfileInvalidError rc: 40 traceback: no
Invalid key data for repository {} found in {}.
KeyfileMismatchError rc: 41 traceback: no
Mismatch between repository {} and key file {}.
KeyfileNotFoundError rc: 42 traceback: no
No key file for repository {} found in {}.
NotABorgKeyFile rc: 43 traceback: no
This file is not a Borg key backup, aborting.
RepoKeyNotFoundError rc: 44 traceback: no
No key entry found in the config of repository {}.
RepoIdMismatch rc: 45 traceback: no
This key backup seems to be for a different backup repository, aborting.
UnencryptedRepo
Keymanagement not available for unencrypted repositories.
UnknownKeyType
Keytype {0} is unknown.
LockError
UnencryptedRepo rc: 46 traceback: no
Key management not available for unencrypted repositories.
UnknownKeyType rc: 47 traceback: no
Key type {0} is unknown.
UnsupportedPayloadError rc: 48 traceback: no
Unsupported payload type {}. A newer version is required to access this repository.
NoPassphraseFailure rc: 50 traceback: no
Cannot acquire a passphrase: {}
PasscommandFailure rc: 51 traceback: no
Passcommand supplied in BORG_PASSCOMMAND failed: {}
PassphraseWrong rc: 52 traceback: no
Passphrase supplied in BORG_PASSPHRASE, by BORG_PASSCOMMAND, or via BORG_PASSPHRASE_FD is incorrect.
PasswordRetriesExceeded rc: 53 traceback: no
Exceeded the maximum password retries.
Cache.CacheInitAbortedError rc: 60 traceback: no
Cache initialization aborted
Cache.EncryptionMethodMismatch rc: 61 traceback: no
Repository encryption method changed since last access, refusing to continue
Cache.RepositoryAccessAborted rc: 62 traceback: no
Repository access aborted
Cache.RepositoryIDNotUnique rc: 63 traceback: no
Cache is newer than repository - do you have multiple, independently updated repos with same ID?
Cache.RepositoryReplay rc: 64 traceback: no
Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe (multiple repos with same ID)
LockError rc: 70 traceback: no
Failed to acquire the lock {}.
LockErrorT
LockErrorT rc: 71 traceback: yes
Failed to acquire the lock {}.
ConnectionClosed
Connection closed by remote host
InvalidRPCMethod
RPC method {} is not valid
PathNotAllowed
Repository path not allowed
RemoteRepository.RPCServerOutdated
LockFailed rc: 72 traceback: yes
Failed to create/acquire the lock {} ({}).
LockTimeout rc: 73 traceback: no
Failed to create/acquire the lock {} (timeout).
NotLocked rc: 74 traceback: yes
Failed to release the lock {} (was not locked).
NotMyLock rc: 75 traceback: yes
Failed to release the lock {} (was/is locked, but not by me).
ConnectionClosed rc: 80 traceback: no
Connection closed by remote host.
ConnectionClosedWithHint rc: 81 traceback: no
Connection closed by remote host. {}
InvalidRPCMethod rc: 82 traceback: no
RPC method {} is not valid.
PathNotAllowed rc: 83 traceback: no
Repository path not allowed: {}
RemoteRepository.RPCServerOutdated rc: 84 traceback: no
Borg server is too old for {}. Required version {}
UnexpectedRPCDataFormatFromClient
UnexpectedRPCDataFormatFromClient rc: 85 traceback: no
Borg {}: Got unexpected RPC data format from client.
UnexpectedRPCDataFormatFromServer
UnexpectedRPCDataFormatFromServer rc: 86 traceback: no
Got unexpected RPC data format from server:
{}
Repository.AlreadyExists
Repository {} already exists.
Repository.CheckNeeded
Inconsistency detected. Please run "borg check {}".
Repository.DoesNotExist
Repository {} does not exist.
Repository.InsufficientFreeSpaceError
Insufficient free space to complete transaction (required: {}, available: {}).
Repository.InvalidRepository
{} is not a valid repository. Check repo config.
Repository.AtticRepository
Attic repository detected. Please run "borg upgrade {}".
Repository.ObjectNotFound
Object with key {} not found in repository {}.
ConnectionBrokenWithHint rc: 87 traceback: no
Connection to remote host is broken. {}
IntegrityError rc: 90 traceback: yes
Data integrity error: {}
FileIntegrityError rc: 91 traceback: yes
File failed integrity check: {}
DecompressionError rc: 92 traceback: yes
Decompression error: {}
ArchiveTAMInvalid rc: 95 traceback: yes
Data integrity error: {}
ArchiveTAMRequiredError rc: 96 traceback: yes
Archive '{}' is unauthenticated, but it is required for this repository.
TAMInvalid rc: 97 traceback: yes
Data integrity error: {}
TAMRequiredError rc: 98 traceback: yes
Manifest is unauthenticated, but it is required for this repository.
TAMUnsupportedSuiteError rc: 99 traceback: yes
Could not verify manifest: Unsupported suite {!r}; a newer version is needed.
Warnings
BorgWarning rc: 1
Warning: {}
BackupWarning rc: 1
{}: {}
FileChangedWarning rc: 100
{}: file changed while we backed it up
IncludePatternNeverMatchedWarning rc: 101
Include pattern '{}' never matched.
BackupError rc: 102
{}: backup error
BackupRaceConditionError rc: 103
{}: file type or inode changed while we backed it up (race condition, skipped file)
BackupOSError rc: 104
{}: {}
BackupPermissionError rc: 105
{}: {}
BackupIOError rc: 106
{}: {}
BackupFileNotFoundError rc: 107
{}: {}
Operations
- cache.begin_transaction
@ -632,6 +727,7 @@ Operations
- repository.check
- check.verify_data
- check.rebuild_manifest
- check.rebuild_refcounts
- extract
*info* is one string element, the name of the path currently extracted.

View file

@ -30,7 +30,7 @@ Under these circumstances Borg guarantees that the attacker cannot
1. modify the data of any archive without the client detecting the change
2. rename, remove or add an archive without the client detecting the change
3. recover plain-text data
3. recover plaintext data
4. recover definite (heuristics based on access patterns are possible)
structural information such as the object graph (which archives
refer to what chunks)
@ -142,14 +142,17 @@ Depending on the chosen mode (see :ref:`borg_init`) different primitives are use
and is also tracked locally on the client to avoid counter reuse.
- The authentication primitive is either HMAC-SHA-256 or BLAKE2b-256
in a keyed mode. HMAC-SHA-256 uses 256 bit keys, while BLAKE2b-256
uses 512 bit keys.
in a keyed mode.
The latter is secure not only because BLAKE2b itself is not
susceptible to `length extension`_, but also since it truncates the
hash output from 512 bits to 256 bits, which would make the
construction safe even if BLAKE2b were broken regarding length
extension or similar attacks.
Both HMAC-SHA-256 and BLAKE2b have undergone extensive cryptanalysis
and have proven secure against known attacks. The known vulnerability
of SHA-256 against length extension attacks does not apply to HMAC-SHA-256.
The authentication primitive should be chosen based upon SHA hardware support:
all AMD Ryzen, Intel 10th+ generation mobile and Intel 11th+ generation
desktop processors, Apple M1+ and most current ARM64 architectures support
SHA extensions and are likely to perform best with HMAC-SHA-256.
64-bit CPUs without SHA extensions are likely to perform best with BLAKE2b.
- The primitive used for authentication is always the same primitive
that is used for deriving the chunk ID, but they are always

View file

@ -1,8 +1,8 @@
Introduction
============
.. this shim is here to fix the structure in the PDF
rendering. without this stub, the elements in the toctree of
index.rst show up a level below the README file included
.. This shim is here to fix the structure in the PDF
rendering. Without this stub, the elements in the toctree of
index.rst show up a level below the README file included there.
.. include:: ../README.rst

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-BENCHMARK-CRUD" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-benchmark-crud" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-benchmark-crud \- Benchmark Create, Read, Update, Delete for archives.
.SH SYNOPSIS
.sp
@ -37,7 +38,7 @@ borg [common options] benchmark crud [options] REPOSITORY PATH
.sp
This command benchmarks borg CRUD (create, read, update, delete) operations.
.sp
It creates input data below the given PATH and backups this data into the given REPO.
It creates input data below the given PATH and backs up this data into the given REPO.
The REPO must already exist (it could be a fresh empty repo or an existing repo, the
command will create / read / update / delete some archives named borg\-benchmark\-crud* there.
.sp
@ -47,11 +48,9 @@ If your repository is encrypted and borg needs a passphrase to unlock the key, u
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
BORG_PASSPHRASE=mysecret borg benchmark crud REPO PATH
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -62,17 +61,17 @@ Also, due to the kind of content used, no compression is used in these benchmark
.INDENT 0.0
.TP
.B C\- == borg create (1st archive creation, no compression, do not use files cache)
C\-Z\- == all\-zero files. full dedup, this is primarily measuring reader/chunker/hasher.
C\-R\- == random files. no dedup, measuring throughput through all processing stages.
C\-Z\- == all\-zero files. full deduplication; this primarily measures reader/chunker/hasher.
C\-R\- == random files. no deduplication, measuring throughput through all processing stages.
.TP
.B R\- == borg extract (extract archive, dry\-run, do everything, but do not write files to disk)
R\-Z\- == all zero files. Measuring heavily duplicated files.
R\-Z\- == all\-zero files. Measuring heavily duplicated files.
R\-R\- == random files. No duplication here, measuring throughput through all processing
stages, except writing to disk.
.TP
.B U\- == borg create (2nd archive creation of unchanged input files, measure files cache speed)
The throughput value is kind of virtual here, it does not actually read the file.
U\-Z\- == needs to check the 2 all\-zero chunks\(aq existence in the repo.
U\-Z\- == needs to check the two all\-zero chunks\(aq existence in the repo.
U\-R\- == needs to check existence of a lot of different chunks in the repo.
.TP
.B D\- == borg delete archive (delete last remaining archive, measure deletion + compaction)
@ -81,7 +80,7 @@ D\-R\- == many chunks to delete / many segments to compact/remove.
.UNINDENT
.sp
Please note that there might be quite some variance in these measurements.
Try multiple measurements and having a otherwise idle machine (and network, if you use it).
Try multiple measurements and have an otherwise idle machine (and network, if you use it).
.SH OPTIONS
.sp
See \fIborg\-common(1)\fP for common options of Borg commands.
@ -92,12 +91,11 @@ See \fIborg\-common(1)\fP for common options of Borg commands.
repository to use for benchmark (must exist)
.TP
.B PATH
path were to create benchmark input data
path where to create benchmark input data
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-BENCHMARK" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-benchmark" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-benchmark \- benchmark command
.SH SYNOPSIS
.nf
@ -37,11 +38,10 @@ borg [common options] benchmark crud ...
.sp
.SH DESCRIPTION
.sp
These commands do various benchmarks.
These commands perform various benchmarks.
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-benchmark\-crud(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-BREAK-LOCK" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-break-lock" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-break-lock \- Break the repository lock (e.g. in case it was left by a dead borg.
.SH SYNOPSIS
.sp
@ -36,8 +37,8 @@ borg [common options] break\-lock [options] [REPOSITORY]
.SH DESCRIPTION
.sp
This command breaks the repository and cache locks.
Please use carefully and only while no borg process (on any machine) is
trying to access the Cache or the Repository.
Please use with care and only when no borg process (on any machine) is
trying to access the cache or the repository.
.SH OPTIONS
.sp
See \fIborg\-common(1)\fP for common options of Borg commands.
@ -50,7 +51,6 @@ repository for which to break the locks
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,99 +28,131 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-CHECK" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-check" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-check \- Check repository consistency
.SH SYNOPSIS
.sp
borg [common options] check [options] [REPOSITORY_OR_ARCHIVE]
.SH DESCRIPTION
.sp
The check command verifies the consistency of a repository and the corresponding archives.
The check command verifies the consistency of a repository and its archives.
It consists of two major steps:
.INDENT 0.0
.IP 1. 3
Checking the consistency of the repository itself. This includes checking
the segment magic headers, and both the metadata and data of all objects in
the segments. The read data is checked by size and CRC. Bit rot and other
types of accidental damage can be detected this way. Running the repository
check can be split into multiple partial checks using \fB\-\-max\-duration\fP\&.
When checking a remote repository, please note that the checks run on the
server and do not cause significant network traffic.
.IP 2. 3
Checking consistency and correctness of the archive metadata and optionally
archive data (requires \fB\-\-verify\-data\fP). This includes ensuring that the
repository manifest exists, the archive metadata chunk is present, and that
all chunks referencing files (items) in the archive exist. This requires
reading archive and file metadata, but not data. To cryptographically verify
the file (content) data integrity pass \fB\-\-verify\-data\fP, but keep in mind
that this requires reading all data and is hence very time consuming. When
checking archives of a remote repository, archive checks run on the client
machine because they require decrypting data and therefore the encryption
key.
.UNINDENT
.sp
check \-\-repair is a potentially dangerous function and might lead to data loss
(for kinds of corruption it is not capable of dealing with). BE VERY CAREFUL!
Both steps can also be run independently. Pass \fB\-\-repository\-only\fP to run the
repository checks only, or pass \fB\-\-archives\-only\fP to run the archive checks
only.
.sp
The \fB\-\-max\-duration\fP option can be used to split a long\-running repository
check into multiple partial checks. After the given number of seconds the check
is interrupted. The next partial check will continue where the previous one
stopped, until the full repository has been checked. Assuming a complete check
would take 7 hours, then running a daily check with \fB\-\-max\-duration=3600\fP
(1 hour) would result in one full repository check per week. Doing a full
repository check aborts any previous partial check; the next partial check will
restart from the beginning. With partial repository checks you can run neither
archive checks, nor enable repair mode. Consequently, if you want to use
\fB\-\-max\-duration\fP you must also pass \fB\-\-repository\-only\fP, and must not pass
\fB\-\-archives\-only\fP, nor \fB\-\-repair\fP\&.
.sp
\fBWarning:\fP Please note that partial repository checks (i.e. running it with
\fB\-\-max\-duration\fP) can only perform non\-cryptographic checksum checks on the
segment files. A full repository check (i.e. without \fB\-\-max\-duration\fP) can
also do a repository index check. Enabling partial repository checks excludes
archive checks for the same reason. Therefore partial checks may be useful with
very large repositories only where a full check would take too long.
.sp
The \fB\-\-verify\-data\fP option will perform a full integrity verification (as
opposed to checking the CRC32 of the segment) of data, which means reading the
data from the repository, decrypting and decompressing it. It is a complete
cryptographic verification and hence very time consuming, but will detect any
accidental and malicious corruption. Tamper\-resistance is only guaranteed for
encrypted repositories against attackers without access to the keys. You cannot
use \fB\-\-verify\-data\fP with \fB\-\-repository\-only\fP\&.
.SS About repair mode
.sp
The check command is a read\-only task by default. If any corruption is found,
Borg will report the issue and proceed with checking. To actually repair the
issues found, pass \fB\-\-repair\fP\&.
.sp
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
\fB\-\-repair\fP is a \fBPOTENTIALLY DANGEROUS FEATURE\fP and might lead to data
loss! This does not just include data that was previously lost anyway, but
might include more data for kinds of corruption it is not capable of
dealing with. \fBBE VERY CAREFUL!\fP
.UNINDENT
.UNINDENT
.sp
Pursuant to the previous warning it is also highly recommended to test the
reliability of the hardware running this software with stress testing software
such as memory testers. Unreliable hardware can also lead to data loss especially
when this command is run in repair mode.
reliability of the hardware running Borg with stress testing software. This
especially includes storage and memory testers. Unreliable hardware might lead
to additional data loss.
.sp
First, the underlying repository data files are checked:
It is highly recommended to create a backup of your repository before running
in repair mode (i.e. running it with \fB\-\-repair\fP).
.sp
Repair mode will attempt to fix any corruptions found. Fixing corruptions does
not mean recovering lost data: Borg can not magically restore data lost due to
e.g. a hardware failure. Repairing a repository means sacrificing some data
for the sake of the repository as a whole and the remaining data. Hence it is,
by definition, a potentially lossy task.
.sp
In practice, repair mode hooks into both the repository and archive checks:
.INDENT 0.0
.IP \(bu 2
For all segments, the segment magic header is checked.
.IP \(bu 2
For all objects stored in the segments, all metadata (e.g. CRC and size) and
all data is read. The read data is checked by size and CRC. Bit rot and other
types of accidental damage can be detected this way.
.IP \(bu 2
In repair mode, if an integrity error is detected in a segment, try to recover
as many objects from the segment as possible.
.IP \(bu 2
In repair mode, make sure that the index is consistent with the data stored in
the segments.
.IP \(bu 2
If checking a remote repo via \fBssh:\fP, the repo check is executed on the server
without causing significant network traffic.
.IP \(bu 2
The repository check can be skipped using the \fB\-\-archives\-only\fP option.
.IP \(bu 2
A repository check can be time consuming. Partial checks are possible with the
\fB\-\-max\-duration\fP option.
.IP 1. 3
When checking the repository\(aqs consistency, repair mode will try to recover
as many objects from segments with integrity errors as possible, and ensure
that the index is consistent with the data stored in the segments.
.IP 2. 3
When checking the consistency and correctness of archives, repair mode might
remove whole archives from the manifest if their archive metadata chunk is
corrupt or lost. On a chunk level (i.e. the contents of files), repair mode
will replace corrupt or lost chunks with a same\-size replacement chunk of
zeroes. If a previously zeroed chunk reappears, repair mode will restore
this lost chunk using the new chunk. Lastly, repair mode will also delete
orphaned chunks (e.g. caused by read errors while creating the archive).
.UNINDENT
.sp
Second, the consistency and correctness of the archive metadata is verified:
.INDENT 0.0
.IP \(bu 2
Is the repo manifest present? If not, it is rebuilt from archive metadata
chunks (this requires reading and decrypting of all metadata and data).
.IP \(bu 2
Check if archive metadata chunk is present; if not, remove archive from manifest.
.IP \(bu 2
For all files (items) in the archive, for all chunks referenced by these
files, check if chunk is present. In repair mode, if a chunk is not present,
replace it with a same\-size replacement chunk of zeroes. If a previously lost
chunk reappears (e.g. via a later backup), in repair mode the all\-zero replacement
chunk will be replaced by the correct chunk. This requires reading of archive and
file metadata, but not data.
.IP \(bu 2
In repair mode, when all the archives were checked, orphaned chunks are deleted
from the repo. One cause of orphaned chunks are input file related errors (like
read errors) in the archive creation process.
.IP \(bu 2
In verify\-data mode, a complete cryptographic verification of the archive data
integrity is performed. This conflicts with \fB\-\-repository\-only\fP as this mode
only makes sense if the archive checks are enabled. The full details of this mode
are documented below.
.IP \(bu 2
If checking a remote repo via \fBssh:\fP, the archive check is executed on the
client machine because it requires decryption, and this is always done client\-side
as key access is needed.
.IP \(bu 2
The archive checks can be time consuming; they can be skipped using the
\fB\-\-repository\-only\fP option.
.UNINDENT
Most steps taken by repair mode have a one\-time effect on the repository, like
removing a lost archive from the repository. However, replacing a corrupt or
lost chunk with an all\-zero replacement will have an ongoing effect on the
repository: When attempting to extract a file referencing an all\-zero chunk,
the \fBextract\fP command will distinctly warn about it. The FUSE filesystem
created by the \fBmount\fP command will reject reading such a \(dqzero\-patched\(dq
file unless a special mount option is given.
.sp
The \fB\-\-max\-duration\fP option can be used to split a long\-running repository check
into multiple partial checks. After the given number of seconds the check is
interrupted. The next partial check will continue where the previous one stopped,
until the complete repository has been checked. Example: Assuming a full check took 7
hours, then running a daily check with \-\-max\-duration=3600 (1 hour) resulted in one
full check per week.
.sp
Attention: Partial checks can only do way less checking than a full check (only the
CRC32 checks on segment file entries are done), and cannot be combined with the
\fB\-\-repair\fP option. Partial checks may therefore be useful only with very large
repositories where a full check took too long. Doing a full repository check aborts a
partial check; the next partial check will restart from the beginning.
.sp
The \fB\-\-verify\-data\fP option will perform a full integrity verification (as opposed to
checking the CRC32 of the segment) of data, which means reading the data from the
repository, decrypting and decompressing it. This is a cryptographic verification,
which will detect (accidental) corruption. For encrypted repositories it is
tamper\-resistant as well, unless the attacker has access to the keys. It is also very
slow.
As mentioned earlier, Borg might be able to \(dqheal\(dq a \(dqzero\-patched\(dq file in
repair mode, if all its previously lost chunks reappear (e.g. via a later
backup). This is achieved by Borg not only keeping track of the all\-zero
replacement chunks, but also by keeping metadata about the lost chunks. In
repair mode Borg will check whether a previously lost chunk reappeared and will
replace the all\-zero replacement chunk by the reappeared chunk. If all lost
chunks of a \(dqzero\-patched\(dq file reappear, this effectively \(dqheals\(dq the file.
Consequently, if lost chunks were repaired earlier, it is advised to run
\fB\-\-repair\fP a second time after creating some new backups.
.SH OPTIONS
.sp
See \fIborg\-common(1)\fP for common options of Borg commands.
@ -129,7 +162,7 @@ See \fIborg\-common(1)\fP for common options of Borg commands.
.B REPOSITORY_OR_ARCHIVE
repository or archive to check consistency of
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-repository\-only
@ -154,13 +187,13 @@ do only a partial repo check for max. SECONDS seconds (Default: unlimited)
.INDENT 0.0
.TP
.BI \-P \ PREFIX\fR,\fB \ \-\-prefix \ PREFIX
only consider archive names starting with this prefix.
only consider archive names starting with this prefix. (deprecated)
.TP
.BI \-a \ GLOB\fR,\fB \ \-\-glob\-archives \ GLOB
only consider archive names matching the glob. sh: rules apply, see "borg help patterns". \fB\-\-prefix\fP and \fB\-\-glob\-archives\fP are mutually exclusive.
only consider archive names matching the glob. sh: rules apply (without actually using the sh: prefix), see \(dqborg help patterns\(dq.
.TP
.BI \-\-sort\-by \ KEYS
Comma\-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp
Comma\-separated list of sorting keys; valid keys are: timestamp, archive, name, id; default is: timestamp
.TP
.BI \-\-first \ N
consider first N archives after other filters were applied
@ -171,7 +204,6 @@ consider last N archives after other filters were applied
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-COMMON" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-common" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-common \- Common options of Borg commands
.SH SYNOPSIS
.INDENT 0.0
@ -79,7 +80,7 @@ show/log the return code (rc)
set umask to M (local only, default: 0077)
.TP
.BI \-\-remote\-path \ PATH
use PATH as borg executable on the remote (default: "borg")
use PATH as borg executable on the remote (default: \(dqborg\(dq)
.TP
.BI \-\-remote\-ratelimit \ RATE
deprecated, use \fB\-\-upload\-ratelimit\fP instead
@ -97,7 +98,7 @@ set network upload buffer size in MiB. (default: 0=no buffer)
treat part files like normal files (e.g. to list/extract them)
.TP
.BI \-\-debug\-profile \ FILE
Write execution profile in Borg format into FILE. For local use a Python\-compatible file can be generated by suffixing FILE with ".pyprof".
Write execution profile in Borg format into FILE. For local use a Python\-compatible file can be generated by suffixing FILE with \(dq.pyprof\(dq.
.TP
.BI \-\-rsh \ RSH
Use this command to connect to the \(aqborg serve\(aq process (default: \(aqssh\(aq)
@ -105,7 +106,6 @@ Use this command to connect to the \(aqborg serve\(aq process (default: \(aqssh\
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-COMPACT" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-compact" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-compact \- compact segment files in the repository
.SH SYNOPSIS
.sp
@ -51,9 +52,17 @@ A segment is compacted if the amount of saved space is above the percentage valu
given by the \fB\-\-threshold\fP option. If omitted, a threshold of 10% is used.
When using \fB\-\-verbose\fP, borg will output an estimate of the freed space.
.sp
For maximum compaction, use \fB\-\-threshold 0\fP\&. This will compact whenever any
space can be saved and thus rewrites the most data; it can be much slower on
large repositories. Using \fB\-\-threshold 1\fP usually achieves nearly the same
result significantly faster. Higher thresholds (e.g. the default 10) trade
compaction thoroughness for speed. Note: \fB\-\-threshold 100\fP will effectively
compact nothing.
.sp
After upgrading borg (server) to 1.2+, you can use \fBborg compact \-\-cleanup\-commits\fP
to clean up the numerous 17byte commit\-only segments that borg 1.1 did not clean up
due to a bug. It is enough to do that once per repository.
to clean up the numerous 17\-byte commit\-only segments that borg 1.1 did not clean up
due to a bug. It is enough to do that once per repository. After cleaning up the
commits, borg will also do a normal compaction.
.sp
See \fIseparate_compaction\fP in Additional Notes for more details.
.SH OPTIONS
@ -65,9 +74,12 @@ See \fIborg\-common(1)\fP for common options of Borg commands.
.B REPOSITORY
repository to compact
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-n\fP,\fB \-\-dry\-run
do nothing
.TP
.B \-\-cleanup\-commits
cleanup commit\-only 17\-byte segment files
.TP
@ -78,21 +90,18 @@ set minimum threshold for saved space in PERCENT (Default: 10)
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# compact segments and free repo disk space
$ borg compact /path/to/repo
# same as above plus clean up 17byte commit\-only segments
# same as above plus clean up 17\-byte commit\-only segments
$ borg compact \-\-cleanup\-commits /path/to/repo
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-COMPRESSION" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-compression" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-compression \- Details regarding compression
.SH DESCRIPTION
.sp
@ -53,20 +54,20 @@ Do not compress.
Use lz4 compression. Very high speed, very low compression. (default)
.TP
.B zstd[,L]
Use zstd ("zstandard") compression, a modern wide\-range algorithm.
Use zstd (\(dqzstandard\(dq) compression, a modern wide\-range algorithm.
If you do not explicitly give the compression level L (ranging from 1
to 22), it will use level 3.
Archives compressed with zstd are not compatible with borg < 1.1.4.
.TP
.B zlib[,L]
Use zlib ("gz") compression. Medium speed, medium compression.
Use zlib (\(dqgz\(dq) compression. Medium speed, medium compression.
If you do not explicitly give the compression level L (ranging from 0
to 9), it will use level 6.
Giving level 0 (means "no compression", but still has zlib protocol
overhead) is usually pointless, you better use "none" compression.
Giving level 0 (means \(dqno compression\(dq, but still has zlib protocol
overhead) is usually pointless, you better use \(dqnone\(dq compression.
.TP
.B lzma[,L]
Use lzma ("xz") compression. Low speed, high compression.
Use lzma (\(dqxz\(dq) compression. Low speed, high compression.
If you do not explicitly give the compression level L (ranging from 0
to 9), it will use level 6.
Giving levels above 6 is pointless and counterproductive because it does
@ -76,43 +77,91 @@ lots of CPU cycles and RAM.
.B auto,C[,L]
Use a built\-in heuristic to decide per chunk whether to compress or not.
The heuristic tries with lz4 whether the data is compressible.
For incompressible data, it will not use compression (uses "none").
For incompressible data, it will not use compression (uses \(dqnone\(dq).
For compressible data, it uses the given C[,L] compression \- with C[,L]
being any valid compression specifier.
being any valid compression specifier. This can be helpful for media files
which often cannot be compressed much more.
.TP
.B obfuscate,SPEC,C[,L]
Use compressed\-size obfuscation to make fingerprinting attacks based on
the observable stored chunk size more difficult.
Note:
\- you must combine this with encryption or it won\(aqt make any sense.
\- your repo size will be bigger, of course.
the observable stored chunk size more difficult. Note:
.INDENT 7.0
.IP \(bu 2
You must combine this with encryption, or it won\(aqt make any sense.
.IP \(bu 2
Your repo size will be bigger, of course.
.IP \(bu 2
A chunk is limited by the constant \fBMAX_DATA_SIZE\fP (cur. ~20MiB).
.UNINDENT
.sp
The SPEC value will determine how the size obfuscation will work:
The SPEC value determines how the size obfuscation works:
.sp
\fIRelative random reciprocal size variation\fP (multiplicative)
.sp
Relative random reciprocal size variation:
Size will increase by a factor, relative to the compressed data size.
Smaller factors are often used, larger factors rarely.
1: factor 0.01 .. 100.0
2: factor 0.1 .. 1000.0
3: factor 1.0 .. 10000.0
4: factor 10.0 .. 100000.0
5: factor 100.0 .. 1000000.0
6: factor 1000.0 .. 10000000.0
Smaller factors are used often, larger factors rarely.
.sp
Add a randomly sized padding up to the given size:
110: 1kiB
Available factors:
.INDENT 7.0
.INDENT 3.5
.sp
.EX
1: 0.01 .. 100
2: 0.1 .. 1,000
3: 1 .. 10,000
4: 10 .. 100,000
5: 100 .. 1,000,000
6: 1,000 .. 10,000,000
.EE
.UNINDENT
.UNINDENT
.sp
Example probabilities for SPEC \fB1\fP:
.INDENT 7.0
.INDENT 3.5
.sp
.EX
90 % 0.01 .. 0.1
9 % 0.1 .. 1
0.9 % 1 .. 10
0.09% 10 .. 100
.EE
.UNINDENT
.UNINDENT
.sp
\fIRandomly sized padding up to the given size\fP (additive)
.INDENT 7.0
.INDENT 3.5
.sp
.EX
110: 1kiB (2 ^ (SPEC \- 100))
\&...
120: 1MiB
\&...
123: 8MiB (max.)
.EE
.UNINDENT
.UNINDENT
.sp
\fIPadmé padding\fP (deterministic)
.INDENT 7.0
.INDENT 3.5
.sp
.EX
250: pads to sums of powers of 2, max 12% overhead
.EE
.UNINDENT
.UNINDENT
.sp
Uses the Padmé algorithm to deterministically pad the compressed size to a sum of
powers of 2, limiting overhead to 12%. See \%<https://\:lbarman\:.ch/\:blog/\:padme/> for details.
.UNINDENT
.sp
Examples:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
borg create \-\-compression lz4 REPO::ARCHIVE data
borg create \-\-compression zstd REPO::ARCHIVE data
borg create \-\-compression zstd,10 REPO::ARCHIVE data
@ -120,14 +169,13 @@ borg create \-\-compression zlib REPO::ARCHIVE data
borg create \-\-compression zlib,1 REPO::ARCHIVE data
borg create \-\-compression auto,lzma,6 REPO::ARCHIVE data
borg create \-\-compression auto,lzma ...
borg create \-\-compression obfuscate,3,none ...
borg create \-\-compression obfuscate,110,none ...
borg create \-\-compression obfuscate,3,auto,zstd,10 ...
borg create \-\-compression obfuscate,2,zstd,6 ...
.ft P
.fi
borg create \-\-compression obfuscate,250,zstd,3 ...
.EE
.UNINDENT
.UNINDENT
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-CONFIG" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-config" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-config \- get, set, and delete values in a repository or cache config file
.SH SYNOPSIS
.sp
@ -39,10 +40,10 @@ This command gets and sets options in a local repository or cache config file.
For security reasons, this command only works on local repositories.
.sp
To delete a config value entirely, use \fB\-\-delete\fP\&. To list the values
of the configuration file or the default values, use \fB\-\-list\fP\&. To get and existing
of the configuration file or the default values, use \fB\-\-list\fP\&. To get an existing
key, pass only the key name. To set a key, pass both the key name and
the new value. Keys can be specified in the format "section.name" or
simply "name"; the section will default to "repository" and "cache" for
the new value. Keys can be specified in the format \(dqsection.name\(dq or
simply \(dqname\(dq; the section will default to \(dqrepository\(dq and \(dqcache\(dq for
the repo and cache configs, respectively.
.sp
By default, borg config manipulates the repository config file. Using \fB\-\-cache\fP
@ -62,7 +63,7 @@ name of config key
.B VALUE
new value for key
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-c\fP,\fB \-\-cache
@ -76,7 +77,7 @@ list the configuration of the repo
.UNINDENT
.SH EXAMPLES
.sp
\fBNOTE:\fP
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
The repository & cache config files are some of the only directly manipulable
@ -87,8 +88,7 @@ making changes!
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# find cache directory
$ cd ~/.cache/borg/$(borg config /path/to/repo id)
@ -97,14 +97,12 @@ $ borg config /path/to/repo additional_free_space 2G
# make a repo append\-only
$ borg config /path/to/repo append_only 1
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-CREATE" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-create" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-create \- Create new archive
.SH SYNOPSIS
.sp
@ -37,13 +38,18 @@ borg [common options] create [options] ARCHIVE [PATH...]
.sp
This command creates a backup archive containing all files found while recursively
traversing all paths specified. Paths are added to the archive as they are given,
that means if relative paths are desired, the command has to be run from the correct
which means that if relative paths are desired, the command has to be run from the correct
directory.
.sp
When giving \(aq\-\(aq as path, borg will read data from standard input and create a
file \(aqstdin\(aq in the created archive from that data. In some cases it\(aqs more
appropriate to use \-\-content\-from\-command, however. See section \fIReading from
stdin\fP below for details.
The slashdot hack in paths (recursion roots) is triggered by using \fB/./\fP:
\fB/this/gets/stripped/./this/gets/archived\fP means to process that filesystem object, but
strip the prefix on the left side of \fB\&./\fP from the archived items (in this case,
\fBthis/gets/archived\fP will be the path in the archived item).
.sp
When giving \(aq\-\(aq as a path, Borg will read data from standard input and create a
file \(aqstdin\(aq in the created archive from that data. In some cases, it is more
appropriate to use \fB\-\-content\-from\-command\fP\&. See section \(dqReading from stdin\(dq
below for details.
.sp
The archive will consume almost no disk space for files or parts of files that
have already been stored in other archives.
@ -65,7 +71,7 @@ This comparison can operate in different modes as given by \fB\-\-files\-cache\f
.IP \(bu 2
ctime,size,inode (default)
.IP \(bu 2
mtime,size,inode (default behaviour of borg versions older than 1.1.0rc4)
mtime,size,inode (default behavior of borg versions older than 1.1.0rc4)
.IP \(bu 2
ctime,size (ignore the inode number)
.IP \(bu 2
@ -78,42 +84,54 @@ rechunk,mtime (all files are considered modified \- rechunk, cache mtime)
disabled (disable the files cache, all files considered modified \- rechunk)
.UNINDENT
.sp
inode number: better safety, but often unstable on network filesystems
inode number: better safety, but often unstable on network file systems
.sp
Normally, detecting file modifications will take inode information into
consideration to improve the reliability of file change detection.
This is problematic for files located on sshfs and similar network file
systems which do not provide stable inode numbers, such files will always
This is problematic for files located on SSHFS and similar network file
systems which do not provide stable inode numbers; such files will always
be considered modified. You can use modes without \fIinode\fP in this case to
improve performance, but reliability of change detection might be reduced.
improve performance, but the reliability of change detection might be reduced.
.sp
ctime vs. mtime: safety vs. speed
.INDENT 0.0
.IP \(bu 2
ctime is a rather safe way to detect changes to a file (metadata and contents)
as it can not be set from userspace. But, a metadata\-only change will already
as it cannot be set from user space. However, a metadata\-only change will already
update the ctime, so there might be some unnecessary chunking/hashing even
without content changes. Some filesystems do not support ctime (change time).
E.g. doing a chown or chmod to a file will change its ctime.
without content changes. Some file systems do not support ctime (change time).
For example, doing a chown or chmod to a file will change its ctime.
.IP \(bu 2
mtime usually works and only updates if file contents were changed. But mtime
can be arbitrarily set from userspace, e.g. to set mtime back to the same value
can be arbitrarily set from user space, e.g., to set mtime back to the same value
it had before a content change happened. This can be used maliciously as well as
well\-meant, but in both cases mtime based cache modes can be problematic.
well\-meant, but in both cases mtime\-based cache modes can be problematic.
.UNINDENT
.sp
The mount points of filesystems or filesystem snapshots should be the same for every
The \fB\-\-files\-changed\fP option controls how Borg detects if a file has changed during backup:
.INDENT 0.0
.IP \(bu 2
ctime (default): Use ctime to detect changes. This is the safest option.
.IP \(bu 2
mtime: Use mtime to detect changes.
.IP \(bu 2
disabled: Disable the \(dqfile has changed while we backed it up\(dq detection completely.
This is not recommended unless you know what you\(aqre doing, as it could lead to
inconsistent backups if files change during the backup process.
.UNINDENT
.sp
The mount points of file systems or file system snapshots should be the same for every
creation of a new archive to ensure fast operation. This is because the file cache that
is used to determine changed files quickly uses absolute filenames.
is used to determine changed files quickly uses absolute file names.
If this is not possible, consider creating a bind mount to a stable location.
.sp
The \fB\-\-progress\fP option shows (from left to right) Original, Compressed and Deduplicated
(O, C and D, respectively), then the Number of files (N) processed so far, followed by
(O, C and D, respectively), then the number of files (N) processed so far, followed by
the currently processed path.
.sp
When using \fB\-\-stats\fP, you will get some statistics about how much data was
added \- the "This Archive" deduplicated size there is most interesting as that is
how much your repository will grow. Please note that the "All archives" stats refer to
added \- the \(dqThis Archive\(dq deduplicated size there is most interesting as that is
how much your repository will grow. Please note that the \(dqAll archives\(dq stats refer to
the state after creation. Also, the \fB\-\-stats\fP and \fB\-\-dry\-run\fP options are mutually
exclusive because the data is not actually compressed and deduplicated during a dry run.
.sp
@ -132,7 +150,7 @@ name of archive to create (must be also a valid directory name)
.B PATH
paths to archive
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-n\fP,\fB \-\-dry\-run
@ -169,15 +187,15 @@ set mode to M in archive for stdin data (default: 0660)
interpret PATH as command and store its stdout. See also section Reading from stdin below.
.TP
.B \-\-paths\-from\-stdin
read DELIM\-separated list of paths to backup from stdin. Will not recurse into directories.
read DELIM\-separated list of paths to backup from stdin. All control is external: it will back up all files given \- no more, no less.
.TP
.B \-\-paths\-from\-command
interpret PATH as command and treat its output as \fB\-\-paths\-from\-stdin\fP
.TP
.BI \-\-paths\-delimiter \ DELIM
set path delimiter for \fB\-\-paths\-from\-stdin\fP and \fB\-\-paths\-from\-command\fP (default: n)
set path delimiter for \fB\-\-paths\-from\-stdin\fP and \fB\-\-paths\-from\-command\fP (default: \fB\en\fP)
.UNINDENT
.SS Exclusion options
.SS Include/Exclude options
.INDENT 0.0
.TP
.BI \-e \ PATTERN\fR,\fB \ \-\-exclude \ PATTERN
@ -193,7 +211,7 @@ include/exclude paths matching PATTERN
read include/exclude patterns from PATTERNFILE, one per line
.TP
.B \-\-exclude\-caches
exclude directories that contain a CACHEDIR.TAG file (\fI\%http://www.bford.info/cachedir/spec.html\fP)
exclude directories that contain a CACHEDIR.TAG file (\%<http://\:www\:.bford\:.info/\:cachedir/\:spec\:.html>)
.TP
.BI \-\-exclude\-if\-present \ NAME
exclude directories that are tagged by containing a filesystem object with the given NAME
@ -208,7 +226,7 @@ exclude files flagged NODUMP
.INDENT 0.0
.TP
.B \-x\fP,\fB \-\-one\-file\-system
stay in the same file system and do not store mount points of other file systems. This might behave different from your expectations, see the docs.
stay in the same file system and do not store mount points of other file systems \- this might behave different from your expectations, see the description below.
.TP
.B \-\-numeric\-owner
deprecated, use \fB\-\-numeric\-ids\fP instead
@ -246,6 +264,9 @@ detect sparse holes in input (supported only by fixed chunker)
.BI \-\-files\-cache \ MODE
operate files cache in MODE. default: ctime,size,inode
.TP
.BI \-\-files\-changed \ MODE
specify how to detect if a file has changed during backup (ctime, mtime, disabled). default: ctime
.TP
.B \-\-read\-special
open and read block and char device files as well as FIFOs as if they were regular files. Also follows symlinks pointing to these kinds of files.
.UNINDENT
@ -265,20 +286,22 @@ write checkpoint every SECONDS seconds (Default: 1800)
specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE). default: buzhash,19,23,21,4095
.TP
.BI \-C \ COMPRESSION\fR,\fB \ \-\-compression \ COMPRESSION
select compression algorithm, see the output of the "borg help compression" command for details.
select compression algorithm, see the output of the \(dqborg help compression\(dq command for details.
.UNINDENT
.SH EXAMPLES
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
# Backup ~/Documents into an archive named "my\-documents"
.EX
# Backup ~/Documents into an archive named \(dqmy\-documents\(dq
$ borg create /path/to/repo::my\-documents ~/Documents
# same, but list all files as we process them
$ borg create \-\-list /path/to/repo::my\-documents ~/Documents
# Backup /mnt/disk/docs, but strip path prefix using the slashdot hack
$ borg create /path/to/repo::docs /mnt/disk/./docs
# Backup ~/Documents and ~/src but exclude pyc files
$ borg create /path/to/repo::my\-files \e
~/Documents \e
@ -288,17 +311,17 @@ $ borg create /path/to/repo::my\-files \e
# Backup home directories excluding image thumbnails (i.e. only
# /home/<one directory>/.thumbnails is excluded, not /home/*/*/.thumbnails etc.)
$ borg create /path/to/repo::my\-files /home \e
\-\-exclude \(aqsh:/home/*/.thumbnails\(aq
\-\-exclude \(aqsh:home/*/.thumbnails\(aq
# Backup the root filesystem into an archive named "root\-YYYY\-MM\-DD"
# Backup the root filesystem into an archive named \(dqroot\-YYYY\-MM\-DD\(dq
# use zlib compression (good, but slow) \- default is lz4 (fast, low compression ratio)
$ borg create \-C zlib,6 \-\-one\-file\-system /path/to/repo::root\-{now:%Y\-%m\-%d} /
# Backup onto a remote host ("push" style) via ssh to port 2222,
# logging in as user "borg" and storing into /path/to/repo
# Backup onto a remote host (\(dqpush\(dq style) via ssh to port 2222,
# logging in as user \(dqborg\(dq and storing into /path/to/repo
$ borg create ssh://borg@backup.example.org:2222/path/to/repo::{fqdn}\-root\-{now} /
# Backup a remote host locally ("pull" style) using sshfs
# Backup a remote host locally (\(dqpull\(dq style) using sshfs
$ mkdir sshfs\-mount
$ sshfs root@example.com:/ sshfs\-mount
$ cd sshfs\-mount
@ -341,7 +364,7 @@ $ borg create /path/to/repo::{hostname}\-{user}\-{now:%Y\-%m\-%dT%H:%M:%S.%f} ~
# Backing up relative paths by moving into the correct directory first
$ cd /home/user/Documents
# The root directory of the archive will be "projectA"
# The root directory of the archive will be \(dqprojectA\(dq
$ borg create /path/to/repo::daily\-projectA\-{now:%Y\-%m\-%d} projectA
# Use external command to determine files to archive
@ -352,10 +375,9 @@ $ borg create \-\-paths\-from\-command /path/to/repo::joes\-files \-\- find /srv
# Use \-\-paths\-from\-stdin with \-\-paths\-delimiter (for example, for filenames with newlines in them)
$ find ~ \-size \-1000k \-print0 | borg create \e
\-\-paths\-from\-stdin \e
\-\-paths\-delimiter "\e0" \e
\-\-paths\-delimiter \(dq\e0\(dq \e
/path/to/repo::smallfiles\-handle\-newline
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH NOTES
@ -373,16 +395,16 @@ only include the objects specified by \fB\-\-exclude\-if\-present\fP in your bac
and not include any other contents of the containing folder, this can be enabled
through using the \fB\-\-keep\-exclude\-tags\fP option.
.sp
The \fB\-x\fP or \fB\-\-one\-file\-system\fP option excludes directories, that are mountpoints (and everything in them).
It detects mountpoints by comparing the device number from the output of \fBstat()\fP of the directory and its
The \fB\-x\fP or \fB\-\-one\-file\-system\fP option excludes directories that are mount points (and everything in them).
It detects mount points by comparing the device number from the output of \fBstat()\fP of the directory and its
parent directory. Specifically, it excludes directories for which \fBstat()\fP reports a device number different
from the device number of their parent. Be aware that in Linux (and possibly elsewhere) there are directories
with device number different from their parent, which the kernel does not consider a mountpoint and also the
other way around. Examples are bind mounts (possibly same device number, but always a mountpoint) and ALL
subvolumes of a btrfs (different device number from parent but not necessarily a mountpoint). Therefore when
using \fB\-\-one\-file\-system\fP, one should make doubly sure that the backup works as intended especially when using
btrfs. This is even more important, if the btrfs layout was created by someone else, e.g. a distribution
installer.
from the device number of their parent.
In general: be aware that there are directories with device numbers different from their parent, which the kernel
does not consider mount points, and vice versa.
Linux examples for this are bind mounts (possibly same device number, but always a mount point) and all
subvolumes of a Btrfs file system (different device numbers from the parent but not necessarily mount points).
macOS examples are the APFS mounts of a typical macOS installation.
Therefore, when using \fB\-\-one\-file\-system\fP, you should double\-check that the backup works as intended.
.SS Item flags
.sp
\fB\-\-list\fP outputs a list of all files, directories and other
@ -395,7 +417,7 @@ If you are interested only in a subset of that output, you can give e.g.
below).
.sp
A uppercase character represents the status of a regular file relative to the
"files" cache (not relative to the repo \-\- this is an issue if the files cache
\(dqfiles\(dq cache (not relative to the repo \-\- this is an issue if the files cache
is not used). Metadata is stored in any case and for \(aqA\(aq and \(aqM\(aq also new data
chunks are stored. For \(aqU\(aq all data chunks refer to already existing chunks.
.INDENT 0.0
@ -439,18 +461,16 @@ Other flags used include:
.IP \(bu 2
\(aq?\(aq = missing status code (if you see this, please file a bug report!)
.UNINDENT
.SS Reading from stdin
.SS Reading backup data from stdin
.sp
There are two methods to read from stdin. Either specify \fB\-\fP as path and
pipe directly to borg:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
backup\-vm \-\-id myvm \-\-stdout | borg create REPO::ARCHIVE \-
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -461,11 +481,9 @@ to the command:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
borg create \-\-content\-from\-command REPO::ARCHIVE \-\- backup\-vm \-\-id myvm \-\-stdout
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -486,10 +504,22 @@ creation a bit.
.sp
By default, the content read from stdin is stored in a file called \(aqstdin\(aq.
Use \fB\-\-stdin\-name\fP to change the name.
.SS Feeding all file paths from externally
.sp
Usually, you give a starting path (recursion root) to borg and then borg
automatically recurses, finds and backs up all fs objects contained in
there (optionally considering include/exclude rules).
.sp
If you need more control and you want to give every single fs object path
to borg (maybe implementing your own recursion or your own rules), you can use
\fB\-\-paths\-from\-stdin\fP or \fB\-\-paths\-from\-command\fP (with the latter, borg will
fail to create an archive should the command fail).
.sp
Borg supports paths with the slashdot hack to strip path prefixes here also.
So, be careful not to unintentionally trigger that.
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-delete(1)\fP, \fIborg\-prune(1)\fP, \fIborg\-check(1)\fP, \fIborg\-patterns(1)\fP, \fIborg\-placeholders(1)\fP, \fIborg\-compression(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-DELETE" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-delete" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-delete \- Delete an existing repository or archives
.SH SYNOPSIS
.sp
@ -48,15 +49,13 @@ with the \fB\-\-cache\-only\fP option, or keep the security info with the
When in doubt, use \fB\-\-dry\-run \-\-list\fP to see what would be deleted.
.sp
When using \fB\-\-stats\fP, you will get some statistics about how much data was
deleted \- the "Deleted data" deduplicated size there is most interesting as
deleted \- the \(dqDeleted data\(dq deduplicated size there is most interesting as
that is how much your repository will shrink.
Please note that the "All archives" stats refer to the state after deletion.
Please note that the \(dqAll archives\(dq stats refer to the state after deletion.
.sp
You can delete multiple archives by specifying their common prefix, if they
have one, using the \fB\-\-prefix PREFIX\fP option. You can also specify a shell
pattern to match multiple archives using the \fB\-\-glob\-archives GLOB\fP option
(for more info on these patterns, see \fIborg_patterns\fP). Note that these
two options are mutually exclusive.
You can delete multiple archives by specifying a shell pattern to match
multiple archives using the \fB\-\-glob\-archives GLOB\fP option (for more info on
these patterns, see \fIborg_patterns\fP).
.sp
To avoid accidentally deleting archives, especially when using glob patterns,
it might be helpful to use the \fB\-\-dry\-run\fP to test out the command without
@ -73,7 +72,7 @@ repository or archive to delete
.B ARCHIVE
archives to delete
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-n\fP,\fB \-\-dry\-run
@ -96,18 +95,21 @@ keep the local security info when deleting a repository
.TP
.B \-\-save\-space
work slower, but using less space
.TP
.BI \-c \ SECONDS\fR,\fB \ \-\-checkpoint\-interval \ SECONDS
write checkpoint every SECONDS seconds (Default: 1800)
.UNINDENT
.SS Archive filters
.INDENT 0.0
.TP
.BI \-P \ PREFIX\fR,\fB \ \-\-prefix \ PREFIX
only consider archive names starting with this prefix.
only consider archive names starting with this prefix. (deprecated)
.TP
.BI \-a \ GLOB\fR,\fB \ \-\-glob\-archives \ GLOB
only consider archive names matching the glob. sh: rules apply, see "borg help patterns". \fB\-\-prefix\fP and \fB\-\-glob\-archives\fP are mutually exclusive.
only consider archive names matching the glob. sh: rules apply (without actually using the sh: prefix), see \(dqborg help patterns\(dq.
.TP
.BI \-\-sort\-by \ KEYS
Comma\-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp
Comma\-separated list of sorting keys; valid keys are: timestamp, archive, name, id; default is: timestamp
.TP
.BI \-\-first \ N
consider first N archives after other filters were applied
@ -119,20 +121,19 @@ consider last N archives after other filters were applied
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# delete a single backup archive:
$ borg delete /path/to/repo::Monday
# actually free disk space:
$ borg compact /path/to/repo
# delete all archives whose names begin with the machine\(aqs hostname followed by "\-"
$ borg delete \-\-prefix \(aq{hostname}\-\(aq /path/to/repo
# delete all archives whose names begin with the machine\(aqs hostname followed by \(dq\-\(dq
$ borg delete \-\-glob\-archives \(aq{hostname}\-*\(aq /path/to/repo
# delete all archives whose names contain "\-2012\-"
# delete all archives whose names contain \(dq\-2012\-\(dq
$ borg delete \-\-glob\-archives \(aq*\-2012\-*\(aq /path/to/repo
# see what would be deleted if delete was run without \-\-dry\-run
# see what would be deleted if delete were run without \-\-dry\-run
$ borg delete \-\-list \-\-dry\-run \-a \(aq*\-May\-*\(aq /path/to/repo
# delete the whole repository and the related local cache:
@ -142,14 +143,12 @@ repo Mon, 2016\-02\-15 19:26:54
root\-2016\-02\-15 Mon, 2016\-02\-15 19:36:29
newname Mon, 2016\-02\-15 19:50:19
Type \(aqYES\(aq if you understand this and want to continue: YES
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-compact(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-DIFF" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-diff" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-diff \- Diff contents of two archives
.SH SYNOPSIS
.sp
@ -38,17 +39,84 @@ borg [common options] diff [options] REPO::ARCHIVE1 ARCHIVE2 [PATH...]
This command finds differences (file contents, user/group/mode) between archives.
.sp
A repository location and an archive name must be specified for REPO::ARCHIVE1.
ARCHIVE2 is just another archive name in same repository (no repository location
ARCHIVE2 is just another archive name in the same repository (no repository location
allowed).
.SS What is compared
.sp
For archives created with Borg 1.1 or newer diff automatically detects whether
the archives are created with the same chunker params. If so, only chunk IDs
For each matching item in both archives, Borg reports:
.INDENT 0.0
.IP \(bu 2
Content changes: total added/removed bytes within files. If chunker parameters are comparable,
Borg compares chunk IDs quickly; otherwise, it compares the content.
.IP \(bu 2
Metadata changes: user, group, mode, and other metadata shown inline like
\(dq[old_mode \-> new_mode]\(dq for mode changes. Use \fB\-\-content\-only\fP to suppress metadata changes.
.IP \(bu 2
Added/removed items: printed as \(dqadded SIZE path\(dq or \(dqremoved SIZE path\(dq.
.UNINDENT
.SS Output formats
.sp
The default (text) output shows one line per changed path, e.g.:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
+135 B \-252 B [ \-rw\-r\-\-r\-\- \-> \-rwxr\-xr\-x ] path/to/file
.EE
.UNINDENT
.UNINDENT
.sp
JSON Lines output (\fB\-\-json\-lines\fP) prints one JSON object per changed path, e.g.:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
{\(dqpath\(dq: \(dqPATH\(dq, \(dqchanges\(dq: [
{\(dqtype\(dq: \(dqmodified\(dq, \(dqadded\(dq: BYTES, \(dqremoved\(dq: BYTES},
{\(dqtype\(dq: \(dqmode\(dq, \(dqold_mode\(dq: \(dq\-rw\-r\-\-r\-\-\(dq, \(dqnew_mode\(dq: \(dq\-rwxr\-xr\-x\(dq},
{\(dqtype\(dq: \(dqadded\(dq, \(dqsize\(dq: SIZE},
{\(dqtype\(dq: \(dqremoved\(dq, \(dqsize\(dq: SIZE}
]}
.EE
.UNINDENT
.UNINDENT
.sp
Only actual changes are included in the \(dqchanges\(dq list. For example, a modified entry with
added=0 and removed=0 is omitted.
.SS Sorting
.sp
Use \fB\-\-sort\-by FIELDS\fP where FIELDS is a comma\-separated list of fields.
Sorts are applied stably from last to first in the given list. Prepend \(dq>\(dq for
descending, \(dq<\(dq (or no prefix) for ascending, for example \fB\-\-sort\-by=\(dq>size_added,path\(dq\fP\&.
Supported fields include:
.INDENT 0.0
.IP \(bu 2
path: the item path
.IP \(bu 2
size_added: total bytes added for the item content
.IP \(bu 2
size_removed: total bytes removed for the item content
.IP \(bu 2
size_diff: size_added \- size_removed (net content change)
.IP \(bu 2
size: size of the item as stored in ARCHIVE2 (0 for removed items)
.IP \(bu 2
user, group, uid, gid, ctime, mtime: taken from the item state in ARCHIVE2 when present
.IP \(bu 2
ctime_diff, mtime_diff: timestamp difference (archive2 \- archive1)
.UNINDENT
.sp
The \fB\-\-sort\fP option is deprecated and only sorts by path.
.SS Performance considerations
.sp
For archives created with Borg 1.1 or newer, diff automatically detects whether
the archives were created with the same chunker parameters. If so, only chunk IDs
are compared, which is very fast.
.sp
For archives prior to Borg 1.1 chunk contents are compared by default.
If you did not create the archives with different chunker params,
For archives prior to Borg 1.1, chunk contents are compared by default.
If you did not create the archives with different chunker parameters,
pass \fB\-\-same\-chunker\-params\fP\&.
Note that the chunker params changed from Borg 0.xx to 1.0.
Note that the chunker parameters changed from Borg 0.xx to 1.0.
.sp
For more help on include/exclude patterns, see the \fIborg_patterns\fP command output.
.SH OPTIONS
@ -66,7 +134,7 @@ ARCHIVE2 name (no repository location allowed)
.B PATH
paths of items inside the archives to compare; patterns are supported
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-numeric\-owner
@ -79,12 +147,15 @@ only consider numeric user and group identifiers
Override check of chunker parameters.
.TP
.B \-\-sort
Sort the output lines by file path.
.TP
.B \-\-json\-lines
Format output as JSON Lines.
Sort the output by path (deprecated, use \-\-sort\-by=path).
.UNINDENT
.SS Exclusion options
.IP "System Message: WARNING/2 (docs/borg-diff.rst:, line 119)"
Option list ends without a blank line; unexpected unindent.
.sp
\-\-sort\-by FIELD[,FIELD...] Advanced sorting: specify field(s) to sort by. Accepts a comma\-separated list. Prefix with > for descending or < for ascending (default).
\-\-content\-only Only compare differences in content (exclude metadata differences)
\-\-json\-lines Format output as JSON Lines.
.SS Include/Exclude options
.INDENT 0.0
.TP
.BI \-e \ PATTERN\fR,\fB \ \-\-exclude \ PATTERN
@ -103,8 +174,7 @@ read include/exclude patterns from PATTERNFILE, one per line
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg init \-e=none testrepo
$ mkdir testdir
$ cd testdir
@ -114,10 +184,10 @@ $ touch file3
$ borg create ../testrepo::archive1 .
$ chmod a+x file1
$ echo "something" >> file2
$ echo \(dqsomething\(dq >> file2
$ borg create ../testrepo::archive2 .
$ echo "testing 123" >> file1
$ echo \(dqtesting 123\(dq >> file1
$ rm file3
$ touch file4
$ borg create ../testrepo::archive3 .
@ -139,18 +209,24 @@ added 0 B file4
removed 0 B file3
$ borg diff \-\-json\-lines testrepo::archive1 archive3
{"path": "file1", "changes": [{"type": "modified", "added": 17, "removed": 5}, {"type": "mode", "old_mode": "\-rw\-r\-\-r\-\-", "new_mode": "\-rwxr\-xr\-x"}]}
{"path": "file2", "changes": [{"type": "modified", "added": 135, "removed": 252}]}
{"path": "file4", "changes": [{"type": "added", "size": 0}]}
{"path": "file3", "changes": [{"type": "removed", "size": 0}]
.ft P
.fi
{\(dqpath\(dq: \(dqfile1\(dq, \(dqchanges\(dq: [{\(dqtype\(dq: \(dqmodified\(dq, \(dqadded\(dq: 17, \(dqremoved\(dq: 5}, {\(dqtype\(dq: \(dqmode\(dq, \(dqold_mode\(dq: \(dq\-rw\-r\-\-r\-\-\(dq, \(dqnew_mode\(dq: \(dq\-rwxr\-xr\-x\(dq}]}
{\(dqpath\(dq: \(dqfile2\(dq, \(dqchanges\(dq: [{\(dqtype\(dq: \(dqmodified\(dq, \(dqadded\(dq: 135, \(dqremoved\(dq: 252}]}
{\(dqpath\(dq: \(dqfile4\(dq, \(dqchanges\(dq: [{\(dqtype\(dq: \(dqadded\(dq, \(dqsize\(dq: 0}]}
{\(dqpath\(dq: \(dqfile3\(dq, \(dqchanges\(dq: [{\(dqtype\(dq: \(dqremoved\(dq, \(dqsize\(dq: 0}]}
# Use \-\-sort\-by with a comma\-separated list; sorts apply stably from last to first.
# Here: primary by net size change descending, tie\-breaker by path ascending
$ borg diff \-\-sort\-by=\(dq>size_diff,path\(dq testrepo::archive1 archive3
+17 B \-5 B [\-rw\-r\-\-r\-\- \-> \-rwxr\-xr\-x] file1
removed 0 B file3
added 0 B file4
+135 B \-252 B file2
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-EXPORT-TAR" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-export-tar" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-export-tar \- Export archive contents as a tarball
.SH SYNOPSIS
.sp
@ -50,7 +51,7 @@ before writing it to FILE:
.IP \(bu 2
\&.tar.xz or .txz: xz
.IP \(bu 2
\&.tar.zstd: zstd
\&.tar.zstd or .tar.zst: zstd
.IP \(bu 2
\&.tar.lz4: lz4
.UNINDENT
@ -66,7 +67,7 @@ BSD flags, ACLs, extended attributes (xattrs), atime and ctime are not exported.
Timestamp resolution is limited to whole seconds, not the nanosecond resolution
otherwise supported by Borg.
.sp
A \fB\-\-sparse\fP option (as found in borg extract) is not supported.
A \fB\-\-sparse\fP option (as found in \fBborg extract\fP) is not supported.
.sp
By default the entire archive is extracted but a subset of files and directories
can be selected by passing a list of \fBPATHs\fP as arguments.
@ -86,12 +87,12 @@ See \fIborg\-common(1)\fP for common options of Borg commands.
archive to export
.TP
.B FILE
output tar file. "\-" to write to stdout instead.
output tar file. \(dq\-\(dq to write to stdout instead.
.TP
.B PATH
paths to extract; patterns are supported
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-tar\-filter
@ -100,7 +101,7 @@ filter program to pipe data through
.B \-\-list
output verbose list of items (files, dirs, ...)
.UNINDENT
.SS Exclusion options
.SS Include/Exclude options
.INDENT 0.0
.TP
.BI \-e \ PATTERN\fR,\fB \ \-\-exclude \ PATTERN
@ -122,31 +123,28 @@ Remove the specified number of leading path elements. Paths with fewer elements
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
# export as uncompressed tar
.EX
# Export as an uncompressed tar.
$ borg export\-tar /path/to/repo::Monday Monday.tar
# exclude some types, compress using gzip
# Exclude some types; compress using gzip.
$ borg export\-tar /path/to/repo::Monday Monday.tar.gz \-\-exclude \(aq*.so\(aq
# use higher compression level with gzip
$ borg export\-tar \-\-tar\-filter="gzip \-9" testrepo::linux Monday.tar.gz
# Use a higher compression level with gzip.
$ borg export\-tar \-\-tar\-filter=\(dqgzip \-9\(dq testrepo::linux Monday.tar.gz
# export a tar, but instead of storing it on disk,
# Export a tar, but instead of storing it on disk,
# upload it to a remote site using curl.
$ borg export\-tar /path/to/repo::Monday \- | curl \-\-data\-binary @\- https://somewhere/to/POST
# remote extraction via "tarpipe"
$ borg export\-tar /path/to/repo::Monday \- | ssh somewhere "cd extracted; tar x"
.ft P
.fi
# Remote extraction via \(dqtarpipe\(dq.
$ borg export\-tar /path/to/repo::Monday \- | ssh somewhere \(dqcd extracted; tar x\(dq
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,37 +28,41 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-EXTRACT" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-extract" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-extract \- Extract archive contents
.SH SYNOPSIS
.sp
borg [common options] extract [options] ARCHIVE [PATH...]
.SH DESCRIPTION
.sp
This command extracts the contents of an archive. By default the entire
archive is extracted but a subset of files and directories can be selected
by passing a list of \fBPATHs\fP as arguments. The file selection can further
be restricted by using the \fB\-\-exclude\fP option.
This command extracts the contents of an archive.
.sp
By default, the entire archive is extracted, but a subset of files and directories
can be selected by passing a list of \fBPATH\fP arguments. The default interpretation
for the paths to extract is \fIpp:\fP which is a literal path\-prefix match. If you want
to use e.g. a wildcard, you must select a different pattern style such as \fIsh:\fP or
\fIfm:\fP\&. See \fIborg_patterns\fP for more information.
.sp
The file selection can be further restricted by using the \fB\-\-exclude\fP option.
For more help on include/exclude patterns, see the \fIborg_patterns\fP command output.
.sp
By using \fB\-\-dry\-run\fP, you can do all extraction steps except actually writing the
output data: reading metadata and data chunks from the repo, checking the hash/hmac,
decrypting, decompressing.
output data: reading metadata and data chunks from the repository, checking the hash/HMAC,
decrypting, and decompressing.
.sp
\fB\-\-progress\fP can be slower than no progress display, since it makes one additional
pass over the archive metadata.
.sp
\fBNOTE:\fP
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
Currently, extract always writes into the current working directory ("."),
Currently, extract always writes into the current working directory (\(dq.\(dq),
so make sure you \fBcd\fP to the right place before calling \fBborg extract\fP\&.
.sp
When parent directories are not extracted (because of using file/directory selection
or any other reason), borg can not restore parent directories\(aq metadata, e.g. owner,
group, permission, etc.
or any other reason), Borg cannot restore parent directories\(aq metadata, e.g., owner,
group, permissions, etc.
.UNINDENT
.UNINDENT
.SH OPTIONS
@ -72,7 +77,7 @@ archive to extract
.B PATH
paths to extract; patterns are supported
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-list
@ -105,7 +110,7 @@ write all extracted data to stdout
.B \-\-sparse
create holes in output sparse file from all\-zero chunks
.UNINDENT
.SS Exclusion options
.SS Include/Exclude options
.INDENT 0.0
.TP
.BI \-e \ PATTERN\fR,\fB \ \-\-exclude \ PATTERN
@ -127,8 +132,7 @@ Remove the specified number of leading path elements. Paths with fewer elements
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# Extract entire archive
$ borg extract /path/to/repo::my\-files
@ -138,22 +142,23 @@ $ borg extract \-\-list /path/to/repo::my\-files
# Verify whether an archive could be successfully extracted, but do not write files to disk
$ borg extract \-\-dry\-run /path/to/repo::my\-files
# Extract the "src" directory
# Extract the \(dqsrc\(dq directory
$ borg extract /path/to/repo::my\-files home/USERNAME/src
# Extract the "src" directory but exclude object files
# Extract the \(dqsrc\(dq directory but exclude object files
$ borg extract /path/to/repo::my\-files home/USERNAME/src \-\-exclude \(aq*.o\(aq
# Extract only the C files
$ borg extract /path/to/repo::my\-files \(aqsh:home/USERNAME/src/*.c\(aq
# Restore a raw device (must not be active/in use/mounted at that time)
$ borg extract \-\-stdout /path/to/repo::my\-sdx | dd of=/dev/sdx bs=10M
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-mount(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-IMPORT-TAR" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-import-tar" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-import-tar \- Create a backup archive from a tarball
.SH SYNOPSIS
.sp
@ -37,9 +38,9 @@ borg [common options] import\-tar [options] ARCHIVE TARFILE
.sp
This command creates a backup archive from a tarball.
.sp
When giving \(aq\-\(aq as path, Borg will read a tar stream from standard input.
When giving \(aq\-\(aq as a path, Borg will read a tar stream from standard input.
.sp
By default (\-\-tar\-filter=auto) Borg will detect whether the file is compressed
By default (\fB\-\-tar\-filter=auto\fP) Borg will detect whether the file is compressed
based on its file extension and pipe the file through an appropriate filter:
.INDENT 0.0
.IP \(bu 2
@ -49,16 +50,16 @@ based on its file extension and pipe the file through an appropriate filter:
.IP \(bu 2
\&.tar.xz or .txz: xz \-d
.IP \(bu 2
\&.tar.zstd: zstd \-d
\&.tar.zstd or .tar.zst: zstd \-d
.IP \(bu 2
\&.tar.lz4: lz4 \-d
.UNINDENT
.sp
Alternatively, a \-\-tar\-filter program may be explicitly specified. It should
Alternatively, a \fB\-\-tar\-filter\fP program may be explicitly specified. It should
read compressed data from stdin and output an uncompressed tar stream on
stdout.
.sp
Most documentation of borg create applies. Note that this command does not
Most documentation of \fBborg create\fP applies. Note that this command does not
support excluding files.
.sp
import\-tar is a lossy conversion:
@ -70,6 +71,10 @@ A \fB\-\-sparse\fP option (as found in borg create) is not supported.
.sp
import\-tar reads POSIX.1\-1988 (ustar), POSIX.1\-2001 (pax), GNU tar, UNIX V7 tar
and SunOS tar with extended attributes.
.sp
To import multiple tarballs into a single archive, they can be simply
concatenated (e.g. using \(dqcat\(dq) into a single file, and imported with an
\fB\-\-ignore\-zeros\fP option to skip through the stop markers between them.
.SH OPTIONS
.sp
See \fIborg\-common(1)\fP for common options of Borg commands.
@ -80,9 +85,9 @@ See \fIborg\-common(1)\fP for common options of Borg commands.
name of archive to create (must be also a valid directory name)
.TP
.B TARFILE
input tar file. "\-" to read from stdin instead.
input tar file. \(dq\-\(dq to read from stdin instead.
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-tar\-filter
@ -99,6 +104,9 @@ only display items with the given status characters
.TP
.B \-\-json
output stats as JSON (implies \-\-stats)
.TP
.B \-\-ignore\-zeros
ignore zero\-filled blocks in the input tarball
.UNINDENT
.SS Archive options
.INDENT 0.0
@ -116,12 +124,11 @@ write checkpoint every SECONDS seconds (Default: 1800)
specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE). default: buzhash,19,23,21,4095
.TP
.BI \-C \ COMPRESSION\fR,\fB \ \-\-compression \ COMPRESSION
select compression algorithm, see the output of the "borg help compression" command for details.
select compression algorithm, see the output of the \(dqborg help compression\(dq command for details.
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-INFO" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-info" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-info \- Show archive details such as disk space used
.SH SYNOPSIS
.sp
@ -38,12 +39,12 @@ borg [common options] info [options] [REPOSITORY_OR_ARCHIVE]
This command displays detailed information about the specified archive or repository.
.sp
Please note that the deduplicated sizes of the individual archives do not add
up to the deduplicated size of the repository ("all archives"), because the two
are meaning different things:
up to the deduplicated size of the repository (\(dqall archives\(dq), because the two
mean different things:
.sp
This archive / deduplicated size = amount of data stored ONLY for this archive
= unique chunks of this archive.
All archives / deduplicated size = amount of data stored in the repo
All archives / deduplicated size = amount of data stored in the repository
= all chunks in the repository.
.sp
Borg archives can only contain a limited amount of file metadata.
@ -59,7 +60,7 @@ See \fIborg\-common(1)\fP for common options of Borg commands.
.B REPOSITORY_OR_ARCHIVE
repository or archive to display information about
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-json
@ -69,13 +70,13 @@ format output as JSON
.INDENT 0.0
.TP
.BI \-P \ PREFIX\fR,\fB \ \-\-prefix \ PREFIX
only consider archive names starting with this prefix.
only consider archive names starting with this prefix. (deprecated)
.TP
.BI \-a \ GLOB\fR,\fB \ \-\-glob\-archives \ GLOB
only consider archive names matching the glob. sh: rules apply, see "borg help patterns". \fB\-\-prefix\fP and \fB\-\-glob\-archives\fP are mutually exclusive.
only consider archive names matching the glob. sh: rules apply (without actually using the sh: prefix), see \(dqborg help patterns\(dq.
.TP
.BI \-\-sort\-by \ KEYS
Comma\-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp
Comma\-separated list of sorting keys; valid keys are: timestamp, archive, name, id; default is: timestamp
.TP
.BI \-\-first \ N
consider first N archives after other filters were applied
@ -87,8 +88,7 @@ consider last N archives after other filters were applied
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg info /path/to/repo::2017\-06\-29T11:00\-srv
Archive name: 2017\-06\-29T11:00\-srv
Archive fingerprint: b2f1beac2bd553b34e06358afa45a3c1689320d39163890c5bbbd49125f00fe5
@ -141,14 +141,12 @@ All archives: 121.82 TB 112.41 TB 215.42 GB
Unique chunks Total chunks
Chunk index: 1015213 626934122
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-list(1)\fP, \fIborg\-diff(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,6 @@
.\" Man page generated from reStructuredText.
'\" t
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,135 +29,151 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-INIT" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-init" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-init \- Initialize an empty repository
.SH SYNOPSIS
.sp
borg [common options] init [options] [REPOSITORY]
.SH DESCRIPTION
.sp
This command initializes an empty repository. A repository is a filesystem
directory containing the deduplicated data from zero or more archives.
.SS Encryption mode TLDR
This command initializes an empty repository. A repository is a
filesystem directory containing the deduplicated data from zero or more
archives.
.SS Encryption mode TL;DR
.sp
The encryption mode can only be configured when creating a new repository \-
you can neither configure it on a per\-archive basis nor change the
encryption mode of an existing repository.
The encryption mode can only be configured when creating a new
repository. You can neither configure encryption on a per\-archive
basis, nor change the encryption mode of an existing repository. You
should thus take possible future use into account when deciding on an
encryption mode.
.sp
Use \fBrepokey\fP:
As a general rule of thumb, use \fBrepokey\fP with a strong passphrase:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
borg init \-\-encryption repokey /path/to/repo
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Or \fBrepokey\-blake2\fP depending on which is faster on your client machines (see below):
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
borg init \-\-encryption repokey\-blake2 /path/to/repo
.ft P
.fi
.UNINDENT
.UNINDENT
.sp
Borg will:
However, there are many reasons to choose differently. See the section
below for details. In any case, Borg will:
.INDENT 0.0
.IP 1. 3
Ask you to come up with a passphrase.
Ask you to enter a unique and strong passphrase.
.IP 2. 3
Create a borg key (which contains 3 random secrets. See \fIkey_files\fP).
Create a random Borg key (which actually consists of three random
secrets, see \fIkey_files\fP for details).
.IP 3. 3
Encrypt the key with your passphrase.
Encrypt the Borg key with your passphrase.
.IP 4. 3
Store the encrypted borg key inside the repository directory (in the repo config).
This is why it is essential to use a secure passphrase.
Store the encrypted Borg key inside the repository directory (with
\fBrepokey\fP and \fBrepokey\-blake2\fP modes; with \fBkeyfile\fP and
\fBkeyfile\-blake2\fP modes the Borg key is stored in your home
directory instead, see below). Since we usually have to assume that
an attacker could gain access to the repository (that\(aqs why we
encrypt the data in the first place), choosing a strong and unique
passphrase is absolutely crucial.
.IP 5. 3
Encrypt and sign your backups to prevent anyone from reading or forging them unless they
have the key and know the passphrase. Make sure to keep a backup of
your key \fBoutside\fP the repository \- do not lock yourself out by
"leaving your keys inside your car" (see \fIborg_key_export\fP).
For remote backups the encryption is done locally \- the remote machine
never sees your passphrase, your unencrypted key or your unencrypted files.
Chunking and id generation are also based on your key to improve
your privacy.
Encrypt and sign your backups with the Borg key to prevent anyone
from reading or forging them unless they have the Borg key \fIand\fP
know the passphrase.
.IP 6. 3
Use the key when extracting files to decrypt them and to verify that the contents of
the backups have not been accidentally or maliciously altered.
Use the Borg key to decrypt and thus access the data stored in your
repository, e.g. when extracting files. The contents can also be
verified to detect accidental corruption or malicious tampering.
.UNINDENT
.sp
As you can see, you always need \fIboth\fP the Borg key and passphrase to
access your data. Thus it\(aqs crucial to keep a backup of your key
\fIoutside\fP both the repository and the system you create backups of.
You can easily run into a \(dqleaving your keys inside your car\(dq situation
otherwise. See \fIborg_key_export\fP to create a backup of your key
(e.g., by printing it on paper).
.sp
Encryption is done locally \- i.e., if you back up to a remote machine,
the remote machine neither sees your passphrase, nor your unencrypted
Borg key, nor your unencrypted files. Chunking and ID generation are
based on your key to improve privacy.
.sp
\fBAbout hardware acceleration:\fP
.sp
Borg encrypts data with AES, which is pretty fast thanks to hardware
acceleration on basically all modern Intel, AMD, and ARM CPUs since
around the early 2010s (very cheap models since the mid\-2010s).
.sp
As the hashing algorithm, Borg can use either SHA256 or BLAKE2b. ARM
CPUs support hardware\-accelerated SHA256 hashing since ARMv7 with NEON
(around 2011), or ARMv8 (around 2013). AMD CPUs support it since Zen 1
(around 2017), i.e. all AMD Ryzen CPUs. Intel CPUs support it since Ice
Lake on mobile (10th gen, around 2021), and Rocket Lake on desktop
(11th gen, around 2021). Very cheap models have received support a few
years later. If your CPU doesn\(aqt support hardware\-accelerated SHA256
hashing, you might want to give BLAKE2b hashing a try \- it\(aqs likely
faster then. So, instead of \fBrepokey\fP mode, use \fBrepokey\-blake2\fP
(or any of the other \fB\-blake2\fP modes for that matter).
.sp
Hardware acceleration is always used automatically when available.
.SS Picking a passphrase
.sp
Make sure you use a good passphrase. Not too short, not too simple. The real
encryption / decryption key is encrypted with / locked by your passphrase.
If an attacker gets your key, he can\(aqt unlock and use it without knowing the
passphrase.
Make sure you use a good passphrase. Not too short, not too simple. The
real encryption / decryption key is encrypted with / locked by your
passphrase. If an attacker gets your borg key, they can\(aqt unlock and use
it without knowing the passphrase.
.sp
Be careful with special or non\-ascii characters in your passphrase:
Be careful with special or non\-ASCII characters in your passphrase:
.INDENT 0.0
.IP \(bu 2
Borg processes the passphrase as unicode (and encodes it as utf\-8),
so it does not have problems dealing with even the strangest characters.
Borg processes the passphrase as Unicode (and encodes it as UTF\-8), so
it does not have problems dealing with even the strangest characters.
.IP \(bu 2
BUT: that does not necessarily apply to your OS / VM / keyboard configuration.
BUT: that does not necessarily apply to your OS / VM / keyboard
configuration.
.UNINDENT
.sp
So better use a long passphrase made from simple ascii chars than one that
includes non\-ascii stuff or characters that are hard/impossible to enter on
a different keyboard layout.
So it is better to use a long passphrase made from simple ASCII
characters than one that includes non\-ASCII characters or characters
that are hard or impossible to enter on a different keyboard layout.
.sp
You can change your passphrase for existing repos at any time, it won\(aqt affect
the encryption/decryption key or other secrets.
.SS More encryption modes
You can change your passphrase for existing repositories at any time; it
won\(aqt affect the encryption/decryption key or other secrets. See
\fIborg_key_change\-passphrase\fP\&.
.SS More about encryption modes
.sp
Only use \fB\-\-encryption none\fP if you are OK with anyone who has access to
your repository being able to read your backups and tamper with their
contents without you noticing.
Choosing the right encryption mode isn\(aqt always easy and many factors
can change which mode is best for you. However, note that you can\(aqt
really do anything \fIwrong\fP if you choose \fBrepokey\fP with a strong
passphrase. So, if you\(aqre not sure, choose \fBrepokey\fP (or
\fBrepokey\-blake2\fP, depending on your hardware, see above).
.sp
If you want "passphrase and having\-the\-key" security, use \fB\-\-encryption keyfile\fP\&.
The key will be stored in your home directory (in \fB~/.config/borg/keys\fP).
.sp
If you do \fBnot\fP want to encrypt the contents of your backups, but still
want to detect malicious tampering use \fB\-\-encryption authenticated\fP\&.
.sp
If \fBBLAKE2b\fP is faster than \fBSHA\-256\fP on your hardware, use \fB\-\-encryption authenticated\-blake2\fP,
\fB\-\-encryption repokey\-blake2\fP or \fB\-\-encryption keyfile\-blake2\fP\&. Note: for remote backups
the hashing is done on your local machine.
Borg supports the following encryption modes:
.\" nanorst: inline-fill
.
.TS
center;
|l|l|l|l|.
_
box center;
l|l|l|l.
T{
Hash/MAC
T} T{
Not encrypted
no auth
T} T{
Not encrypted,
but authenticated
Not Encrypted
T} T{
Encrypted (AEAD w/ AES)
and authenticated
T}
_
T{
Not Authenticated
T} T{
Authenticated
T}
_
T{
SHA\-256
T} T{
none
\fBnone\fP
T} T{
\fIauthenticated\fP
\fBauthenticated\fP
T} T{
repokey
keyfile
\fBrepokey\fP
\fBkeyfile\fP
T}
_
T{
@ -163,57 +181,106 @@ BLAKE2b
T} T{
n/a
T} T{
\fIauthenticated\-blake2\fP
\fBauthenticated\-blake2\fP
T} T{
\fIrepokey\-blake2\fP
\fIkeyfile\-blake2\fP
\fBrepokey\-blake2\fP
\fBkeyfile\-blake2\fP
T}
_
.TE
.\" nanorst: inline-replace
.
.sp
Modes \fImarked like this\fP in the above table are new in Borg 1.1 and are not
backwards\-compatible with Borg 1.0.x.
Borg 1.0 and older support \fBnone\fP, \fBrepokey\fP, and \fBkeyfile\fP
modes only. If you need such old clients to be able to access your
repo, you can\(aqt use any of the other modes.
.sp
On modern Intel/AMD CPUs (except very cheap ones), AES is usually
hardware\-accelerated.
BLAKE2b is faster than SHA256 on Intel/AMD 64\-bit CPUs
(except AMD Ryzen and future CPUs with SHA extensions),
which makes \fIauthenticated\-blake2\fP faster than \fInone\fP and \fIauthenticated\fP\&.
\fBAbout modes without encryption:\fP
.sp
On modern ARM CPUs, NEON provides hardware acceleration for SHA256 making it faster
than BLAKE2b\-256 there. NEON accelerates AES as well.
Avoid using \fBnone\fP mode. If you think about using \fBnone\fP mode,
please reconsider and be absolutely sure. Using any mode other than
\fBnone\fP allows Borg to detect accidental corruption or malicious
tampering with the repo. It also prevents denial\-of\-service attacks
against clients. Instead of \fBnone\fP mode, you likely want to use
\fBauthenticated\fP mode, or \fBrepokey\fP or \fBkeyfile\fP modes with an
empty passphrase instead (see below).
.sp
Hardware acceleration is always used automatically when available.
If you don\(aqt want to encrypt your data, use \fBauthenticated\fP or
\fBauthenticated\-blake2\fP modes. These modes require a passphrase in
normal operation, but in emergency situations you can access the repo
without the passphrase with \fBBORG_WORKAROUNDS=authenticated_no_key\fP
(see \fIenvironment\-variables\fP).
.sp
\fIrepokey\fP and \fIkeyfile\fP use AES\-CTR\-256 for encryption and HMAC\-SHA256 for
authentication in an encrypt\-then\-MAC (EtM) construction. The chunk ID hash
is HMAC\-SHA256 as well (with a separate key).
These modes are compatible with Borg 1.0.x.
If you just don\(aqt want to choose a passphrase, use \fBkeyfile\fP or
\fBkeyfile\-blake2\fP modes with an empty passphrase. These modes are
generally safe even without a passphrase, but keeping an offsite
backup of the Borg key is also important then. See below for details.
.sp
\fIrepokey\-blake2\fP and \fIkeyfile\-blake2\fP are also authenticated encryption modes,
but use BLAKE2b\-256 instead of HMAC\-SHA256 for authentication. The chunk ID
hash is a keyed BLAKE2b\-256 hash.
These modes are new and \fInot\fP compatible with Borg 1.0.x.
If you can assure that an attacker can\(aqt gain access to your repo, e.g.
when independently encrypting your repository disk or filesystem, you
can think about using \fBrepokey\fP or \fBrepokey\-blake2\fP modes with an
empty passphrase. However, keep in mind that if an attacker still
somehow manages to gain access, they have full access to your repo. In
such situations choosing \fBrepokey\fP over \fBauthenticated\fP mode has
the advantage of allowing you to add a passphrase later using
\fIborg_key_change\-passphrase\fP\&.
.sp
\fIauthenticated\fP mode uses no encryption, but authenticates repository contents
through the same HMAC\-SHA256 hash as the \fIrepokey\fP and \fIkeyfile\fP modes (it uses it
as the chunk ID hash). The key is stored like \fIrepokey\fP\&.
This mode is new and \fInot\fP compatible with Borg 1.0.x.
\fBAbout modes with encryption:\fP
.sp
\fIauthenticated\-blake2\fP is like \fIauthenticated\fP, but uses the keyed BLAKE2b\-256 hash
from the other blake2 modes.
This mode is new and \fInot\fP compatible with Borg 1.0.x.
With \fBrepokey\fP and \fBrepokey\-blake2\fP modes the key is stored with
the repo and encrypted with your passphrase. If an attacker gains
access to your repo and knows the passphrase, he can access and tamper
with the repo. The repo\(aqs security thus relies on the strength of your
passphrase. Creating an offsite backup of your Borg key (e.g., by
printing it on paper) is recommended, see \fIborg_key_export\fP\&.
.sp
\fInone\fP mode uses no encryption and no authentication. It uses SHA256 as chunk
ID hash. This mode is not recommended, you should rather consider using an authenticated
or authenticated/encrypted mode. This mode has possible denial\-of\-service issues
when running \fBborg create\fP on contents controlled by an attacker.
Use it only for new repositories where no encryption is wanted \fBand\fP when compatibility
with 1.0.x is important. If compatibility with 1.0.x is not important, use
\fIauthenticated\-blake2\fP or \fIauthenticated\fP instead.
This mode is compatible with Borg 1.0.x.
If you\(aqre thinking about storing the passphrase on the disk of the
system you\(aqre backing up, consider using the \fBkeyfile\fP method
instead. It generally provides the same or better security then.
.sp
With \fBkeyfile\fP and \fBkeyfile\-blake2\fP modes the key is stored on your
local machine (in \fB~/.config/borg/keys\fP) instead. An attacker gaining
access to your repo then needs both the Borg key, and your passphrase to
access and tamper with the repo. However, if you lose the key, you lose
access to the repo, too. You \fBmust\fP create an offsite backup of your
Borg key, e.g. by printing it on paper. Storing a copy of the Borg key
on the system you\(aqre creating backups of is \fBNOT\fP sufficient. Use
\fIborg_key_export\fP to create the backup.
.sp
The \fBkeyfile\fP and \fBkeyfile\-blake2\fP modes allow for \(dqpassphrase and
having\-the\-key\(dq security when using a strong passphrase, but can also
be used with an empty passphrase. Storing a (easily readable)
passphrase on the disk of the system you\(aqre backing up with
\fBkeyfile\fP and \fBkeyfile\-blake2\fP modes adds no security over using an
empty passphrase.
.sp
\fBTechnical details:\fP
.sp
\fBrepokey\fP and \fBkeyfile\fP use AES\-CTR\-256 for encryption and
HMAC\-SHA256 for authentication in an encrypt\-then\-MAC (EtM)
construction. The chunk ID hash is HMAC\-SHA256 (with a separate key).
These modes are compatible with all Borg versions.
.sp
\fBrepokey\-blake2\fP and \fBkeyfile\-blake2\fP are also authenticated
encryption modes, but use BLAKE2b\-256 instead of HMAC\-SHA256 for
authentication. The chunk ID hash is a keyed BLAKE2b\-256 hash. These
modes are only compatible with Borg 1.1 and later.
.sp
\fBauthenticated\fP mode uses no encryption, but authenticates repo
contents through the same HMAC\-SHA256 hash as the \fBrepokey\fP and
\fBkeyfile\fP modes (it uses it as the chunk ID hash). The key is stored
like \fBrepokey\fP within the repo. This mode is only compatible with
Borg 1.1 and later.
.sp
\fBauthenticated\-blake2\fP is like \fBauthenticated\fP, but uses the keyed
BLAKE2b\-256 hash from the other BLAKE2b modes. This mode is only
compatible with Borg 1.1 and later.
.sp
\fBnone\fP mode uses no encryption and no authentication. It uses SHA256
as chunk ID hash. This mode is not recommended. You should instead
consider using an authenticated or authenticated/encrypted mode. This
mode has possible denial\-of\-service issues when running \fBborg create\fP
on contents controlled by an attacker. See above for alternatives.
This mode is compatible with all Borg versions.
.SH OPTIONS
.sp
See \fIborg\-common(1)\fP for common options of Borg commands.
@ -223,7 +290,7 @@ See \fIborg\-common(1)\fP for common options of Borg commands.
.B REPOSITORY
repository to create
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.BI \-e \ MODE\fR,\fB \ \-\-encryption \ MODE
@ -242,8 +309,7 @@ create the parent directories of the repository directory, if they are missing.
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# Local repository, repokey encryption, BLAKE2b (often faster, since Borg 1.1)
$ borg init \-\-encryption=repokey\-blake2 /path/to/repo
@ -257,14 +323,12 @@ $ borg init \-\-encryption=repokey\-blake2 user@hostname:backup
# Remote repository (accesses a remote borg via ssh)
# keyfile: stores the (encrypted) key into ~/.config/borg/keys/
$ borg init \-\-encryption=keyfile user@hostname:backup
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-create(1)\fP, \fIborg\-delete(1)\fP, \fIborg\-check(1)\fP, \fIborg\-list(1)\fP, \fIborg\-key\-import(1)\fP, \fIborg\-key\-export(1)\fP, \fIborg\-key\-change\-passphrase(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,20 +28,20 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-KEY-CHANGE-PASSPHRASE" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-key-change-passphrase" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-key-change-passphrase \- Change repository key file passphrase
.SH SYNOPSIS
.sp
borg [common options] key change\-passphrase [options] [REPOSITORY]
.SH DESCRIPTION
.sp
The key files used for repository encryption are optionally passphrase
The key files used for repository encryption are optionally passphrase\-
protected. This command can be used to change this passphrase.
.sp
Please note that this command only changes the passphrase, but not any
secret protected by it (like e.g. encryption/MAC keys or chunker seed).
Thus, changing the passphrase after passphrase and borg key got compromised
secret protected by it (e.g., encryption/MAC keys or the chunker seed).
Thus, changing the passphrase after the passphrase and Borg key were compromised
does not protect future (nor past) backups to the same repository.
.SH OPTIONS
.sp
@ -52,15 +53,14 @@ REPOSITORY
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# Create a key file protected repository
$ borg init \-\-encryption=keyfile \-v /path/to/repo
Initializing repository at "/path/to/repo"
Initializing repository at \(dq/path/to/repo\(dq
Enter new passphrase:
Enter same passphrase again:
Remember your passphrase. Your data will be inaccessible without it.
Key in "/root/.config/borg/keys/mnt_backup" created.
Key in \(dq/root/.config/borg/keys/mnt_backup\(dq created.
Keep this key safe. Your data will be inaccessible without it.
Synchronizing chunks cache...
Archives: 0, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 0.
@ -78,8 +78,7 @@ Key updated
# key file (creating or overwriting the output key)
# (keyfile repositories only)
$ BORG_KEY_FILE=/path/to/output\-key borg key import /path/to/repo /path/to/exported
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -87,20 +86,17 @@ Fully automated using environment variables:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ BORG_NEW_PASSPHRASE=old borg init \-e=repokey repo
# now "old" is the current passphrase.
# now \(dqold\(dq is the current passphrase.
$ BORG_PASSPHRASE=old BORG_NEW_PASSPHRASE=new borg key change\-passphrase repo
# now "new" is the current passphrase.
.ft P
.fi
# now \(dqnew\(dq is the current passphrase.
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,19 +28,35 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-KEY-EXPORT" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-key-export" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-key-export \- Export the repository key for backup
.SH SYNOPSIS
.sp
borg [common options] key export [options] [REPOSITORY] [PATH]
.SH DESCRIPTION
.sp
This command backs up the borg key.
.sp
If repository encryption is used, the repository is inaccessible
without the key. This command allows one to backup this essential key.
without the borg key (and the passphrase that protects the borg key).
If a repository is not encrypted, but authenticated, the borg key is
still needed to access the repository normally.
.sp
For repositories using \fBkeyfile\fP encryption the key is kept locally
on the system that is capable of doing backups. To guard against loss
or corruption of this key, the key needs to be backed up independently
of the main data backup.
.sp
For repositories using \fBrepokey\fP encryption or \fBauthenticated\fP mode
the key is kept in the repository. A backup is thus not strictly needed,
but guards against the repository becoming inaccessible if the key is
corrupted or lost.
.sp
Note that the backup produced does not include the passphrase itself
(i.e. the exported key stays encrypted). In order to regain access to a
repository, one needs both the exported key and the original passphrase.
Keep the exported key and the passphrase at safe places.
.sp
There are three backup formats. The normal backup format is suitable for
digital storage as a file. The \fB\-\-paper\fP backup format is optimized
@ -47,22 +64,11 @@ for printing and typing in while importing, with per line checks to
reduce problems with manual input. The \fB\-\-qr\-html\fP creates a printable
HTML template with a QR code and a copy of the \fB\-\-paper\fP\-formatted key.
.sp
For repositories using keyfile encryption the key is saved locally
on the system that is capable of doing backups. To guard against loss
of this key, the key needs to be backed up independently of the main
data backup.
.sp
For repositories using the repokey encryption the key is saved in the
repository in the config file. A backup is thus not strictly needed,
but guards against the repository becoming inaccessible if the file
is damaged for some reason.
.sp
Examples:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
borg key export /path/to/repo > encrypted\-key\-backup
borg key export \-\-paper /path/to/repo > encrypted\-key\-backup.txt
borg key export \-\-qr\-html /path/to/repo > encrypted\-key\-backup.html
@ -70,8 +76,7 @@ borg key export \-\-qr\-html /path/to/repo > encrypted\-key\-backup.html
borg key export /path/to/repo encrypted\-key\-backup
borg key export \-\-paper /path/to/repo encrypted\-key\-backup.txt
borg key export \-\-qr\-html /path/to/repo encrypted\-key\-backup.html
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH OPTIONS
@ -85,7 +90,7 @@ REPOSITORY
.B PATH
where to store the backup
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-paper
@ -97,7 +102,6 @@ Create an html file suitable for printing and later type\-in or qr scan
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-key\-import(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-KEY-IMPORT" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-key-import" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-key-import \- Import the repository key from backup
.SH SYNOPSIS
.sp
@ -39,12 +40,12 @@ This command restores a key previously backed up with the export command.
.sp
If the \fB\-\-paper\fP option is given, the import will be an interactive
process in which each line is checked for plausibility before
proceeding to the next line. For this format PATH must not be given.
proceeding to the next line. For this format, PATH must not be provided.
.sp
For repositories using keyfile encryption, the key file which \fBborg key
import\fP writes to depends on several factors. If the \fBBORG_KEY_FILE\fP
environment variable is set and non\-empty, \fBborg key import\fP creates
or overwrites that file named by \fB$BORG_KEY_FILE\fP\&. Otherwise, \fBborg
or overwrites the file named by \fB$BORG_KEY_FILE\fP\&. Otherwise, \fBborg
key import\fP searches in the \fB$BORG_KEYS_DIR\fP directory for a key file
associated with the repository. If a key file is found in
\fB$BORG_KEYS_DIR\fP, \fBborg key import\fP overwrites it; otherwise, \fBborg
@ -60,7 +61,7 @@ REPOSITORY
.B PATH
path to the backup (\(aq\-\(aq to read from stdin)
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-paper
@ -69,7 +70,6 @@ interactively import from a backup done with \fB\-\-paper\fP
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-key\-export(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-KEY-MIGRATE-TO-REPOKEY" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-key-migrate-to-repokey" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-key-migrate-to-repokey \- Migrate passphrase -> repokey
.SH SYNOPSIS
.sp
@ -38,10 +39,10 @@ borg [common options] key migrate\-to\-repokey [options] [REPOSITORY]
This command migrates a repository from passphrase mode (removed in Borg 1.0)
to repokey mode.
.sp
You will be first asked for the repository passphrase (to open it in passphrase
mode). This is the same passphrase as you used to use for this repo before 1.0.
You will first be asked for the repository passphrase (to open it in passphrase
mode). This is the same passphrase you used for this repository before 1.0.
.sp
It will then derive the different secrets from this passphrase.
The different secrets will then be derived from this passphrase.
.sp
Then you will be asked for a new passphrase (twice, for safety). This
passphrase will be used to protect the repokey (which contains these same
@ -49,7 +50,7 @@ secrets in encrypted form). You may use the same passphrase as you used to
use, but you may also use a different one.
.sp
After migrating to repokey mode, you can change the passphrase at any time.
But please note: the secrets will always stay the same and they could always
Please note: the secrets will always stay the same, and they could always
be derived from your (old) passphrase\-mode passphrase.
.SH OPTIONS
.sp
@ -60,7 +61,6 @@ REPOSITORY
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-KEY" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-key" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-key \- Manage a keyfile or repokey of a repository
.SH SYNOPSIS
.nf
@ -41,7 +42,6 @@ borg [common options] key migrate\-to\-repokey ...
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-key\-export(1)\fP, \fIborg\-key\-import(1)\fP, \fIborg\-key\-change\-passphrase(1)\fP, \fIborg\-key\-migrate\-to\-repokey(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-LIST" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-list" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-list \- List archive or repository contents
.SH SYNOPSIS
.sp
@ -50,7 +51,7 @@ repository or archive to list contents of
.B PATH
paths to list; patterns are supported
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-consider\-checkpoints
@ -60,25 +61,25 @@ Show checkpoint archives in the repository contents list (default: hidden).
only print file/directory names, nothing else
.TP
.BI \-\-format \ FORMAT
specify format for file or archive listing (default for files: "{mode} {user:6} {group:6} {size:8} {mtime} {path}{extra}{NL}"; for archives: "{archive:<36} {time} [{id}]{NL}")
specify format for file or archive listing (default for files: \(dq{mode} {user:6} {group:6} {size:8} {mtime} {path}{extra}{NL}\(dq; for archives: \(dq{archive:<36} {time} [{id}]{NL}\(dq)
.TP
.B \-\-json
Only valid for listing repository contents. Format output as JSON. The form of \fB\-\-format\fP is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A "barchive" key is therefore not available.
Only valid for listing repository contents. Format output as JSON. The form of \fB\-\-format\fP is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A \(dqbarchive\(dq key is therefore not available.
.TP
.B \-\-json\-lines
Only valid for listing archive contents. Format output as JSON Lines. The form of \fB\-\-format\fP is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A "bpath" key is therefore not available.
Only valid for listing archive contents. Format output as JSON Lines. The form of \fB\-\-format\fP is ignored, but keys used in it are added to the JSON output. Some keys are always present. Note: JSON can only represent text. A \(dqbpath\(dq key is therefore not available.
.UNINDENT
.SS Archive filters
.INDENT 0.0
.TP
.BI \-P \ PREFIX\fR,\fB \ \-\-prefix \ PREFIX
only consider archive names starting with this prefix.
only consider archive names starting with this prefix. (deprecated)
.TP
.BI \-a \ GLOB\fR,\fB \ \-\-glob\-archives \ GLOB
only consider archive names matching the glob. sh: rules apply, see "borg help patterns". \fB\-\-prefix\fP and \fB\-\-glob\-archives\fP are mutually exclusive.
only consider archive names matching the glob. sh: rules apply (without actually using the sh: prefix), see \(dqborg help patterns\(dq.
.TP
.BI \-\-sort\-by \ KEYS
Comma\-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp
Comma\-separated list of sorting keys; valid keys are: timestamp, archive, name, id; default is: timestamp
.TP
.BI \-\-first \ N
consider first N archives after other filters were applied
@ -86,7 +87,7 @@ consider first N archives after other filters were applied
.BI \-\-last \ N
consider last N archives after other filters were applied
.UNINDENT
.SS Exclusion options
.SS Include/Exclude options
.INDENT 0.0
.TP
.BI \-e \ PATTERN\fR,\fB \ \-\-exclude \ PATTERN
@ -105,8 +106,7 @@ read include/exclude patterns from PATTERNFILE, one per line
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg list /path/to/repo
Monday Mon, 2016\-02\-15 19:15:11
repo Mon, 2016\-02\-15 19:26:54
@ -122,14 +122,14 @@ lrwxrwxrwx root root 0 Fri, 2015\-03\-27 20:24:26 bin/bzcmp \-> bzdif
\-rwxr\-xr\-x root root 2140 Fri, 2015\-03\-27 20:24:22 bin/bzdiff
\&...
$ borg list /path/to/repo::root\-2016\-02\-15 \-\-pattern "\- bin/ba*"
$ borg list /path/to/repo::root\-2016\-02\-15 \-\-pattern \(dq\- bin/ba*\(dq
drwxr\-xr\-x root root 0 Mon, 2016\-02\-15 17:44:27 .
drwxrwxr\-x root root 0 Mon, 2016\-02\-15 19:04:49 bin
lrwxrwxrwx root root 0 Fri, 2015\-03\-27 20:24:26 bin/bzcmp \-> bzdiff
\-rwxr\-xr\-x root root 2140 Fri, 2015\-03\-27 20:24:22 bin/bzdiff
\&...
$ borg list /path/to/repo::archiveA \-\-format="{mode} {user:6} {group:6} {size:8d} {isomtime} {path}{extra}{NEWLINE}"
$ borg list /path/to/repo::archiveA \-\-format=\(dq{mode} {user:6} {group:6} {size:8d} {isomtime} {path}{extra}{NEWLINE}\(dq
drwxrwxr\-x user user 0 Sun, 2015\-02\-01 11:00:00 .
drwxrwxr\-x user user 0 Sun, 2015\-02\-01 11:00:00 code
drwxrwxr\-x user user 0 Sun, 2015\-02\-01 11:00:00 code/myproject
@ -137,29 +137,27 @@ drwxrwxr\-x user user 0 Sun, 2015\-02\-01 11:00:00 code/myproject
\-rw\-rw\-r\-\- user user 1416192 Sun, 2015\-02\-01 11:00:00 code/myproject/file.text
\&...
$ borg list /path/to/repo/::archiveA \-\-pattern \(aqre:\e.ext$\(aq
$ borg list /path/to/repo/::archiveA \-\-pattern \(aq+ re:\e.ext$\(aq \-\-pattern \(aq\- re:^.*$\(aq
\-rw\-rw\-r\-\- user user 1416192 Sun, 2015\-02\-01 11:00:00 code/myproject/file.ext
\&...
$ borg list /path/to/repo/::archiveA \-\-pattern \(aqre:.ext$\(aq
$ borg list /path/to/repo/::archiveA \-\-pattern \(aq+ re:.ext$\(aq \-\-pattern \(aq\- re:^.*$\(aq
\-rw\-rw\-r\-\- user user 1416192 Sun, 2015\-02\-01 11:00:00 code/myproject/file.ext
\-rw\-rw\-r\-\- user user 1416192 Sun, 2015\-02\-01 11:00:00 code/myproject/file.text
\&...
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH NOTES
.SS The FORMAT specifier syntax
.sp
The \fB\-\-format\fP option uses python\(aqs \fI\%format string syntax\fP\&.
The \fB\-\-format\fP option uses Python\(aqs format string syntax \%<https://\:docs\:.python\:.org/\:3\:.10/\:library/\:string\:.html#\:formatstrings>\&.
.sp
Examples:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg list \-\-format \(aq{archive}{NL}\(aq /path/to/repo
ArchiveFoo
ArchiveBar
@ -179,8 +177,7 @@ $ borg list \-\-format \(aq{mode} {user:6} {group:6} {size:8} {mtime} {path}{ext
$ borg list \-\-format \(aq{mode} {user:>6} {group:>6} {size:<8} {mtime} {path}{extra}{NL}\(aq /path/to/repo::ArchiveFoo
\-rw\-rw\-r\-\- user user 1024 Thu, 2021\-12\-09 10:22:17 file\-foo
\&...
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -207,7 +204,7 @@ Keys available only when listing archives in a repository:
.IP \(bu 2
archive: archive name interpreted as text (might be missing non\-text characters, see barchive)
.IP \(bu 2
name: alias of "archive"
name: alias of \(dqarchive\(dq
.IP \(bu 2
barchive: verbatim archive name, can contain any character except NUL
.IP \(bu 2
@ -217,9 +214,11 @@ bcomment: verbatim archive comment, can contain any character except NUL
.IP \(bu 2
id: internal ID of the archive
.IP \(bu 2
tam: TAM authentication state of this archive
.IP \(bu 2
start: time (start) of creation of the archive
.IP \(bu 2
time: alias of "start"
time: alias of \(dqstart\(dq
.IP \(bu 2
end: time (end) of creation of the archive
.IP \(bu 2
@ -309,14 +308,13 @@ archiveid
.IP \(bu 2
archivename
.IP \(bu 2
extra: prepends {source} with " \-> " for soft links and " link to " for hard links
extra: prepends {source} with \(dq \-> \(dq for soft links and \(dq link to \(dq for hard links
.IP \(bu 2
health: either "healthy" (file ok) or "broken" (if file has all\-zero replacement chunks)
health: either \(dqhealthy\(dq (file ok) or \(dqbroken\(dq (if file has all\-zero replacement chunks)
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-info(1)\fP, \fIborg\-diff(1)\fP, \fIborg\-prune(1)\fP, \fIborg\-patterns(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,18 +28,42 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-MOUNT" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-mount" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-mount \- Mount archive or an entire repository as a FUSE filesystem
.SH SYNOPSIS
.sp
borg [common options] mount [options] REPOSITORY_OR_ARCHIVE MOUNTPOINT [PATH...]
.SH DESCRIPTION
.sp
This command mounts an archive as a FUSE filesystem. This can be useful for
browsing an archive or restoring individual files. Unless the \fB\-\-foreground\fP
option is given the command will run in the background until the filesystem
is \fBumounted\fP\&.
This command mounts a repository or an archive as a FUSE filesystem.
This can be useful for browsing or restoring individual files.
.sp
When restoring, take into account that the current FUSE implementation does
not support special filesystem flags and ACLs.
.sp
When mounting a repository, the top directories will be named like the
archives and the directory structure below these will be loaded on\-demand from
the repository when entering these directories, so expect some delay.
.sp
Care should be taken, as Borg backs up symlinks as\-is. When an archive
or repository is mounted, it is possible to “jump” outside the mount point
by following a symlink. If this happens, files or directories (or versions of them)
that are not part of the archive or repository may appear to be within the mount point.
.sp
Unless the \fB\-\-foreground\fP option is given the command will run in the
background until the filesystem is \fBunmounted\fP\&.
.sp
Performance tips:
.INDENT 0.0
.IP \(bu 2
when doing a \(dqwhole repository\(dq mount:
do not enter archive directories if not needed; this avoids on\-demand loading.
.IP \(bu 2
only mount a specific archive, not the whole repository.
.IP \(bu 2
only mount specific paths in a specific archive, not the complete archive.
.UNINDENT
.sp
The command \fBborgfs\fP provides a wrapper for \fBborg mount\fP\&. This can also be
used in fstab entries:
@ -59,7 +84,7 @@ override the user and group ids of all files (i.e., \fBborg mount \-o
uid=1000,gid=1000\fP).
.sp
The man page references \fBuser_id\fP and \fBgroup_id\fP mount options
(implemented by fuse) which specify the user and group id of the mount owner
(implemented by FUSE) which specify the user and group id of the mount owner
(aka, the user who does the mounting). It is set automatically by libfuse (or
the filesystem if libfuse is not used). However, you should not specify these
manually. Unlike the \fBuid\fP and \fBgid\fP mount options which affect all files,
@ -69,16 +94,16 @@ manually. Unlike the \fBuid\fP and \fBgid\fP mount options which affect all file
Additional mount options supported by borg:
.INDENT 0.0
.IP \(bu 2
versions: when used with a repository mount, this gives a merged, versioned
view of the files in the archives. EXPERIMENTAL, layout may change in future.
\fBversions\fP: when used with a repository mount, this gives a merged, versioned
view of the files in the archives. EXPERIMENTAL; the layout may change in the future.
.IP \(bu 2
allow_damaged_files: by default damaged files (where missing chunks were
replaced with runs of zeros by borg check \fB\-\-repair\fP) are not readable and
\fBallow_damaged_files\fP: by default damaged files (where missing chunks were
replaced with runs of zeros by \fBborg check \-\-repair\fP) are not readable and
return EIO (I/O error). Set this option to read such files.
.IP \(bu 2
ignore_permissions: for security reasons the "default_permissions" mount
option is internally enforced by borg. "ignore_permissions" can be given to
not enforce "default_permissions".
\fBignore_permissions\fP: for security reasons the \fBdefault_permissions\fP mount
option is internally enforced by borg. \fBignore_permissions\fP can be given to
not enforce \fBdefault_permissions\fP\&.
.UNINDENT
.sp
The BORG_MOUNT_DATA_CACHE_ENTRIES environment variable is meant for advanced users
@ -90,8 +115,15 @@ When the daemonized process receives a signal or crashes, it does not unmount.
Unmounting in these cases could cause an active rsync or similar process
to unintentionally delete data.
.sp
When running in the foreground ^C/SIGINT unmounts cleanly, but other
signals or crashes do not.
When running in the foreground, ^C/SIGINT cleanly unmounts the filesystem,
but other signals or crashes do not.
.sp
Debugging:
.sp
\fBborg mount\fP usually daemonizes and the daemon process sends stdout/stderr
to /dev/null. Thus, you need to either use \fB\-f / \-\-foreground\fP to make it stay
in the foreground and not daemonize, or use \fBBORG_LOGGING_CONF\fP to reconfigure
the logger to output to a file.
.SH OPTIONS
.sp
See \fIborg\-common(1)\fP for common options of Borg commands.
@ -107,7 +139,7 @@ where to mount filesystem
.B PATH
paths to extract; patterns are supported
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-consider\-checkpoints
@ -129,13 +161,13 @@ use numeric user and group identifiers from archive(s)
.INDENT 0.0
.TP
.BI \-P \ PREFIX\fR,\fB \ \-\-prefix \ PREFIX
only consider archive names starting with this prefix.
only consider archive names starting with this prefix. (deprecated)
.TP
.BI \-a \ GLOB\fR,\fB \ \-\-glob\-archives \ GLOB
only consider archive names matching the glob. sh: rules apply, see "borg help patterns". \fB\-\-prefix\fP and \fB\-\-glob\-archives\fP are mutually exclusive.
only consider archive names matching the glob. sh: rules apply (without actually using the sh: prefix), see \(dqborg help patterns\(dq.
.TP
.BI \-\-sort\-by \ KEYS
Comma\-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp
Comma\-separated list of sorting keys; valid keys are: timestamp, archive, name, id; default is: timestamp
.TP
.BI \-\-first \ N
consider first N archives after other filters were applied
@ -143,7 +175,7 @@ consider first N archives after other filters were applied
.BI \-\-last \ N
consider last N archives after other filters were applied
.UNINDENT
.SS Exclusion options
.SS Include/Exclude options
.INDENT 0.0
.TP
.BI \-e \ PATTERN\fR,\fB \ \-\-exclude \ PATTERN
@ -164,7 +196,6 @@ Remove the specified number of leading path elements. Paths with fewer elements
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-umount(1)\fP, \fIborg\-extract(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,48 +28,45 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-PATTERNS" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-patterns" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-patterns \- Details regarding patterns
.SH DESCRIPTION
.sp
The path/filenames used as input for the pattern matching start from the
The path/filenames used as input for the pattern matching start with the
currently active recursion root. You usually give the recursion root(s)
when invoking borg and these can be either relative or absolute paths.
.sp
So, when you give \fIrelative/\fP as root, the paths going into the matcher
will look like \fIrelative/.../file.ext\fP\&. When you give \fI/absolute/\fP as
root, they will look like \fI/absolute/.../file.ext\fP\&.
.sp
File paths in Borg archives are always stored normalized and relative.
This means that e.g. \fBborg create /path/to/repo ../some/path\fP will
store all files as \fIsome/path/.../file.ext\fP and \fBborg create
/path/to/repo /home/user\fP will store all files as
\fIhome/user/.../file.ext\fP\&.
Be careful, your patterns must match the archived paths:
.INDENT 0.0
.IP \(bu 2
Archived paths never start with a leading slash (\(aq/\(aq), nor with \(aq.\(aq, nor with \(aq..\(aq.
.INDENT 2.0
.IP \(bu 2
When you back up absolute paths like \fB/home/user\fP, the archived
paths start with \fBhome/user\fP\&.
.IP \(bu 2
When you back up relative paths like \fB\&./src\fP, the archived paths
start with \fBsrc\fP\&.
.IP \(bu 2
When you back up relative paths like \fB\&../../src\fP, the archived paths
start with \fBsrc\fP\&.
.UNINDENT
.UNINDENT
.sp
A directory exclusion pattern can end either with or without a slash (\(aq/\(aq).
If it ends with a slash, such as \fIsome/path/\fP, the directory will be
included but not its content. If it does not end with a slash, such as
\fIsome/path\fP, both the directory and content will be excluded.
.sp
File patterns support these styles: fnmatch, shell, regular expressions,
path prefixes and path full\-matches. By default, fnmatch is used for
\fB\-\-exclude\fP patterns and shell\-style is used for the \fB\-\-pattern\fP
option. For commands that support patterns in their \fBPATH\fP argument
like (\fBborg list\fP), the default pattern is path prefix.
Borg supports different pattern styles. To define a non\-default
style for a specific pattern, prefix it with two characters followed
by a colon \(aq:\(aq (i.e. \fBfm:path/*\fP, \fBsh:path/**\fP).
.sp
Starting with Borg 1.2, for all but regular expression pattern matching
styles, all paths are treated as relative, meaning that a leading path
separator is removed after normalizing and before matching. This allows
you to use absolute or relative patterns arbitrarily.
.sp
If followed by a colon (\(aq:\(aq) the first two characters of a pattern are
used as a style selector. Explicit style selection is necessary when a
non\-default style is desired or when the desired pattern starts with
two alphanumeric characters followed by a colon (i.e. \fIaa:something/*\fP).
The default pattern style for \fB\-\-exclude\fP differs from \fB\-\-pattern\fP, see below.
.INDENT 0.0
.TP
.B \fI\%Fnmatch\fP, selector \fIfm:\fP
.B Fnmatch \%<https://\:docs\:.python\:.org/\:3/\:library/\:fnmatch\:.html>, selector \fIfm:\fP
This is the default style for \fB\-\-exclude\fP and \fB\-\-exclude\-from\fP\&.
These patterns use a variant of shell pattern syntax, with \(aq*\(aq matching
any number of characters, \(aq?\(aq matching any single character, \(aq[...]\(aq
@ -98,8 +96,8 @@ path and any substring match is sufficient. It is strongly recommended to
anchor patterns to the start (\(aq^\(aq), to the end (\(aq$\(aq) or both. Path
separators (backslash for Windows and \(aq/\(aq on other systems) in paths are
always normalized to a forward slash (\(aq/\(aq) before applying a pattern. The
regular expression syntax is described in the \fI\%Python documentation for
the re module\fP\&.
regular expression syntax is described in the Python documentation for
the re module \%<https://\:docs\:.python\:.org/\:3/\:library/\:re\:.html>\&.
.TP
.B Path prefix, selector \fIpp:\fP
This pattern style is useful to match whole sub\-directories. The pattern
@ -122,7 +120,7 @@ Other include/exclude patterns that would normally match will be ignored.
Same logic applies for exclude.
.UNINDENT
.sp
\fBNOTE:\fP
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
\fIre:\fP, \fIsh:\fP and \fIfm:\fP patterns are all implemented on top of the Python SRE
@ -138,6 +136,15 @@ Exclusions can be passed via the command line option \fB\-\-exclude\fP\&. When u
from within a shell, the patterns should be quoted to protect them from
expansion.
.sp
Patterns matching special characters, e.g. white space, within a shell may
require adjustments, such as putting quotation marks around the arguments.
Example:
Using bash, the following command line option would match and exclude \(dqitem name\(dq:
\fB\-\-pattern=\(aq\-path/item name\(aq\fP
Note that when patterns are used within a pattern file directly read by borg,
e.g. when using \fB\-\-exclude\-from\fP or \fB\-\-patterns\-from\fP, there is no shell
involved and thus no quotation marks are required.
.sp
The \fB\-\-exclude\-from\fP option permits loading exclusion patterns from a text
file with one pattern per line. Lines empty or starting with the number sign
(\(aq#\(aq) after removing whitespace on both ends are ignored. The optional style
@ -152,50 +159,80 @@ Examples:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# Exclude a directory anywhere in the tree named \(ga\(gasteamapps/common\(ga\(ga
# (and everything below it), regardless of where it appears:
$ borg create \-e \(aqsh:**/steamapps/common/**\(aq backup /
# Exclude the contents of \(ga\(ga/home/user/.cache\(ga\(ga:
$ borg create \-e \(aqsh:home/user/.cache/**\(aq backup /home/user
$ borg create \-e home/user/.cache/ backup /home/user
# The file \(aq/home/user/.cache/important\(aq is *not* backed up:
$ borg create \-e home/user/.cache/ backup / /home/user/.cache/important
# Exclude \(aq/home/user/file.o\(aq but not \(aq/home/user/file.odt\(aq:
$ borg create \-e \(aq*.o\(aq backup /
# Exclude \(aq/home/user/junk\(aq and \(aq/home/user/subdir/junk\(aq but
# not \(aq/home/user/importantjunk\(aq or \(aq/etc/junk\(aq:
$ borg create \-e \(aq/home/*/junk\(aq backup /
# Exclude the contents of \(aq/home/user/cache\(aq but not the directory itself:
$ borg create \-e home/user/cache/ backup /
# The file \(aq/home/user/cache/important\(aq is *not* backed up:
$ borg create \-e /home/user/cache/ backup / /home/user/cache/important
$ borg create \-e \(aqhome/*/junk\(aq backup /
# The contents of directories in \(aq/home\(aq are not backed up when their name
# ends in \(aq.tmp\(aq
$ borg create \-\-exclude \(aqre:^/home/[^/]+\e.tmp/\(aq backup /
$ borg create \-\-exclude \(aqre:^home/[^/]+\e.tmp/\(aq backup /
# Load exclusions from file
$ cat >exclude.txt <<EOF
# Comment line
/home/*/junk
home/*/junk
*.tmp
fm:aa:something/*
re:^/home/[^/]+\e.tmp/
sh:/home/*/.thumbnails
re:^home/[^/]+\e.tmp/
sh:home/*/.thumbnails
# Example with spaces, no need to escape as it is processed by borg
some file with spaces.txt
EOF
$ borg create \-\-exclude\-from exclude.txt backup /
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
A more general and easier to use way to define filename matching patterns exists
with the \fB\-\-pattern\fP and \fB\-\-patterns\-from\fP options. Using these, you may
specify the backup roots (starting points) and patterns for inclusion/exclusion.
A root path starts with the prefix \fIR\fP, followed by a path (a plain path, not a
file pattern). An include rule starts with the prefix +, an exclude rule starts
with the prefix \-, an exclude\-norecurse rule starts with !, all followed by a pattern.
A more general and easier to use way to define filename matching patterns
exists with the \fB\-\-pattern\fP and \fB\-\-patterns\-from\fP options. Using
these, you may specify the backup roots, default pattern styles and
patterns for inclusion and exclusion.
.INDENT 0.0
.TP
.B Root path prefix \fBR\fP
A recursion root path starts with the prefix \fBR\fP, followed by a path
(a plain path, not a file pattern). Use this prefix to have the root
paths in the patterns file rather than as command line arguments.
.TP
.B Pattern style prefix \fBP\fP (only useful within patterns files)
To change the default pattern style, use the \fBP\fP prefix, followed by
the pattern style abbreviation (\fBfm\fP, \fBpf\fP, \fBpp\fP, \fBre\fP, \fBsh\fP).
All patterns following this line in the same patterns file will use this
style until another style is specified or the end of the file is reached.
When the current patterns file is finished, the default pattern style will
reset.
.TP
.B Exclude pattern prefix \fB\-\fP
Use the prefix \fB\-\fP, followed by a pattern, to define an exclusion.
This has the same effect as the \fB\-\-exclude\fP option.
.TP
.B Exclude no\-recurse pattern prefix \fB!\fP
Use the prefix \fB!\fP, followed by a pattern, to define an exclusion
that does not recurse into subdirectories. This saves time, but
prevents include patterns to match any files in subdirectories.
.TP
.B Include pattern prefix \fB+\fP
Use the prefix \fB+\fP, followed by a pattern, to define inclusions.
This is useful to include paths that are covered in an exclude
pattern and would otherwise not be backed up.
.UNINDENT
.sp
\fBNOTE:\fP
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
Via \fB\-\-pattern\fP or \fB\-\-patterns\-from\fP you can define BOTH inclusion and exclusion
@ -204,13 +241,37 @@ of files using pattern prefixes \fB+\fP and \fB\-\fP\&. With \fB\-\-exclude\fP a
.UNINDENT
.UNINDENT
.sp
Inclusion patterns are useful to include paths that are contained in an excluded
path. The first matching pattern is used so if an include pattern matches before
an exclude pattern, the file is backed up. If an exclude\-norecurse pattern matches
a directory, it won\(aqt recurse into it and won\(aqt discover any potential matches for
include rules below that directory.
The first matching pattern is used, so if an include pattern matches
before an exclude pattern, the file is backed up. Note that a no\-recurse
exclude stops examination of subdirectories so that potential includes
will not match \- use normal excludes for such use cases.
.sp
\fBNOTE:\fP
Example:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
# Define the recursion root
R /
# Exclude all iso files in any directory
\- **/*.iso
# Explicitly include all inside etc and root
+ etc/**
+ root/**
# Exclude a specific directory under each user\(aqs home directories
\- home/*/.cache
# Explicitly include everything in /home
+ home/**
# Explicitly exclude some directories without recursing into them
! re:^(dev|proc|run|sys|tmp)
# Exclude all other files and directories
# that are not specifically included earlier.
\- **
.EE
.UNINDENT
.UNINDENT
.sp
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
It\(aqs possible that a sub\-directory/file is matched while parent directories are not.
@ -231,16 +292,14 @@ Examples:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# backup pics, but not the ones from 2018, except the good ones:
# note: using = is essential to avoid cmdline argument parsing issues.
borg create \-\-pattern=+pics/2018/good \-\-pattern=\-pics/2018 repo::arch pics
# use a file with patterns:
borg create \-\-patterns\-from patterns.lst repo::arch
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -248,26 +307,24 @@ The patterns.lst file could look like that:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
# "sh:" pattern style is the default, so the following line is not needed:
.EX
# \(dqsh:\(dq pattern style is the default, so the following line is not needed:
P sh
R /
# can be rebuild
\- /home/*/.cache
\- home/*/.cache
# they\(aqre downloads for a reason
\- /home/*/Downloads
\- home/*/Downloads
# susan is a nice person
# include susans home
+ /home/susan
+ home/susan
# also back up this exact file
+ pf:/home/bobby/specialfile.txt
+ pf:home/bobby/specialfile.txt
# don\(aqt backup the other home directories
\- /home/*
\- home/*
# don\(aqt even look in /proc
! /proc
.ft P
.fi
! proc
.EE
.UNINDENT
.UNINDENT
.sp
@ -275,13 +332,11 @@ You can specify recursion roots either on the command line or in a patternfile:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# these two commands do the same thing
borg create \-\-exclude /home/bobby/junk repo::arch /home/bobby /home/susan
borg create \-\-exclude home/bobby/junk repo::arch /home/bobby /home/susan
borg create \-\-patterns\-from patternfile.lst repo::arch
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -289,23 +344,20 @@ The patternfile:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# note that excludes use fm: by default and patternfiles use sh: by default.
# therefore, we need to specify fm: to have the same exact behavior.
P fm
R /home/bobby
R /home/susan
\- /home/bobby/junk
.ft P
.fi
\- home/bobby/junk
.EE
.UNINDENT
.UNINDENT
.sp
This allows you to share the same patterns between multiple repositories
without needing to specify them on the command line.
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-PLACEHOLDERS" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-placeholders" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-placeholders \- Details regarding placeholders
.SH DESCRIPTION
.sp
@ -47,11 +48,11 @@ The full name of the machine in reverse domain name notation.
.TP
.B {now}
The current local date and time, by default in ISO\-8601 format.
You can also supply your own \fI\%format string\fP, e.g. {now:%Y\-%m\-%d_%H:%M:%S}
You can also supply your own format string \%<https://\:docs\:.python\:.org/\:3\:.10/\:library/\:datetime\:.html#\:strftime-and-strptime-behavior>, e.g. {now:%Y\-%m\-%d_%H:%M:%S}
.TP
.B {utcnow}
The current UTC date and time, by default in ISO\-8601 format.
You can also supply your own \fI\%format string\fP, e.g. {utcnow:%Y\-%m\-%d_%H:%M:%S}
You can also supply your own format string \%<https://\:docs\:.python\:.org/\:3\:.10/\:library/\:datetime\:.html#\:strftime-and-strptime-behavior>, e.g. {utcnow:%Y\-%m\-%d_%H:%M:%S}
.TP
.B {user}
The user name (or UID, if no name is available) of the user running borg.
@ -76,11 +77,9 @@ If literal curly braces need to be used, double them for escaping:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
borg create /path/to/repo::{{literal_text}}
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -88,17 +87,15 @@ Examples:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
borg create /path/to/repo::{hostname}\-{user}\-{utcnow} ...
borg create /path/to/repo::{hostname}\-{now:%Y\-%m\-%d_%H:%M:%S} ...
borg prune \-\-prefix \(aq{hostname}\-\(aq ...
.ft P
.fi
borg prune \-\-glob\-archives \(aq{hostname}\-*\(aq ...
.EE
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
systemd uses a difficult, non\-standard syntax for command lines in unit files (refer to
@ -111,7 +108,6 @@ double all percent signs (\fB{hostname}\-{now:%Y\-%m\-%d_%H:%M:%S}\fP
becomes \fB{hostname}\-{now:%%Y\-%%m\-%%d_%%H:%%M:%%S}\fP).
.UNINDENT
.UNINDENT
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-PRUNE" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-prune" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-prune \- Prune repository archives according to specified rules
.SH SYNOPSIS
.sp
@ -42,7 +43,7 @@ Important: Repository disk space is \fBnot\fP freed until you run \fBborg compac
.sp
This command is normally used by automated backup scripts wanting to keep a
certain number of historic backups. This retention policy is commonly referred to as
\fI\%GFS\fP
GFS \%<https://\:en\:.wikipedia\:.org/\:wiki/\:Backup_rotation_scheme#\:Grandfather-father-son>
(Grandfather\-father\-son) backup rotation scheme.
.sp
Also, prune automatically removes checkpoint archives (incomplete archives left
@ -61,10 +62,10 @@ If you have multiple sequences of archives with different data sets (e.g.
from different machines) in one shared repository, use one prune call per
data set that matches only the respective archives using the \-P option.
.sp
The \fB\-\-keep\-within\fP option takes an argument of the form "<int><char>",
where char is "H", "d", "w", "m", "y". For example, \fB\-\-keep\-within 2d\fP means
The \fB\-\-keep\-within\fP option takes an argument of the form \(dq<int><char>\(dq,
where char is \(dqH\(dq, \(dqd\(dq, \(dqw\(dq, \(dqm\(dq, \(dqy\(dq. For example, \fB\-\-keep\-within 2d\fP means
to keep all archives that were created within the past 48 hours.
"1m" is taken to mean "31d". The archives kept with this option do not
\(dq1m\(dq is taken to mean \(dq31d\(dq. The archives kept with this option do not
count towards the totals specified by any other options.
.sp
A good procedure is to thin out more and more the older your backups get.
@ -73,21 +74,25 @@ up to 7 most recent days with backups (days without backups do not count).
The rules are applied from secondly to yearly, and backups selected by previous
rules do not count towards those of later rules. The time that each backup
starts is used for pruning purposes. Dates and times are interpreted in
the local timezone, and weeks go from Monday to Sunday. Specifying a
negative number of archives to keep means that there is no limit. As of borg
1.2.0, borg will retain the oldest archive if any of the secondly, minutely,
hourly, daily, weekly, monthly, or yearly rules was not otherwise able to meet
its retention target. This enables the first chronological archive to continue
aging until it is replaced by a newer archive that meets the retention criteria.
the local time zone, and weeks go from Monday to Sunday. Specifying a
negative number of archives to keep means that there is no limit. As of Borg
1.2.0, Borg will retain the oldest archive if any of the secondly, minutely,
hourly, daily, weekly, monthly, quarterly, or yearly rules was not otherwise
able to meet its retention target. This enables the first chronological archive
to continue aging until it is replaced by a newer archive that meets the
retention criteria.
.sp
The \fB\-\-keep\-13weekly\fP and \fB\-\-keep\-3monthly\fP rules are two different
strategies for keeping archives every quarter year.
.sp
The \fB\-\-keep\-last N\fP option is doing the same as \fB\-\-keep\-secondly N\fP (and it will
keep the last N archives under the assumption that you do not create more than one
backup archive in the same second).
.sp
When using \fB\-\-stats\fP, you will get some statistics about how much data was
deleted \- the "Deleted data" deduplicated size there is most interesting as
deleted \- the \(dqDeleted data\(dq deduplicated size there is most interesting as
that is how much your repository will shrink.
Please note that the "All archives" stats refer to the state after pruning.
Please note that the \(dqAll archives\(dq stats refer to the state after pruning.
.SH OPTIONS
.sp
See \fIborg\-common(1)\fP for common options of Borg commands.
@ -97,7 +102,7 @@ See \fIborg\-common(1)\fP for common options of Borg commands.
.B REPOSITORY
repository to prune
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-n\fP,\fB \-\-dry\-run
@ -133,65 +138,71 @@ number of weekly archives to keep
.B \-m\fP,\fB \-\-keep\-monthly
number of monthly archives to keep
.TP
.B \-\-keep\-13weekly
number of quarterly archives to keep (13 week strategy)
.TP
.B \-\-keep\-3monthly
number of quarterly archives to keep (3 month strategy)
.TP
.B \-y\fP,\fB \-\-keep\-yearly
number of yearly archives to keep
.TP
.B \-\-save\-space
work slower, but using less space
.TP
.BI \-c \ SECONDS\fR,\fB \ \-\-checkpoint\-interval \ SECONDS
write checkpoint every SECONDS seconds (Default: 1800)
.UNINDENT
.SS Archive filters
.INDENT 0.0
.TP
.BI \-P \ PREFIX\fR,\fB \ \-\-prefix \ PREFIX
only consider archive names starting with this prefix.
only consider archive names starting with this prefix. (deprecated)
.TP
.BI \-a \ GLOB\fR,\fB \ \-\-glob\-archives \ GLOB
only consider archive names matching the glob. sh: rules apply, see "borg help patterns". \fB\-\-prefix\fP and \fB\-\-glob\-archives\fP are mutually exclusive.
only consider archive names matching the glob. sh: rules apply (without actually using the sh: prefix), see \(dqborg help patterns\(dq.
.UNINDENT
.SH EXAMPLES
.sp
Be careful, prune is a potentially dangerous command, it will remove backup
Be careful: prune is a potentially dangerous command; it will remove backup
archives.
.sp
The default of prune is to apply to \fBall archives in the repository\fP unless
you restrict its operation to a subset of the archives using \fB\-\-prefix\fP\&.
When using \fB\-\-prefix\fP, be careful to choose a good prefix \- e.g. do not use a
prefix "foo" if you do not also want to match "foobar".
By default, prune applies to \fBall archives in the repository\fP unless you
restrict its operation to a subset of the archives using \fB\-\-glob\-archives\fP\&.
When using \fB\-\-glob\-archives\fP, be careful to choose a good matching pattern —
for example, do not use \(dqfoo*\(dq if you do not also want to match \(dqfoobar\(dq.
.sp
It is strongly recommended to always run \fBprune \-v \-\-list \-\-dry\-run ...\fP
first so you will see what it would do without it actually doing anything.
first, so you can see what it would do without actually doing anything.
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
# Keep 7 end of day and 4 additional end of week archives.
.EX
# Keep 7 end\-of\-day and 4 additional end\-of\-week archives.
# Do a dry\-run without actually deleting anything.
$ borg prune \-v \-\-list \-\-dry\-run \-\-keep\-daily=7 \-\-keep\-weekly=4 /path/to/repo
# Same as above but only apply to archive names starting with the hostname
# of the machine followed by a "\-" character:
$ borg prune \-v \-\-list \-\-keep\-daily=7 \-\-keep\-weekly=4 \-\-prefix=\(aq{hostname}\-\(aq /path/to/repo
# actually free disk space:
# of the machine followed by a \(dq\-\(dq character:
$ borg prune \-v \-\-list \-\-keep\-daily=7 \-\-keep\-weekly=4 \-\-glob\-archives=\(aq{hostname}\-*\(aq /path/to/repo
# Actually free disk space:
$ borg compact /path/to/repo
# Keep 7 end of day, 4 additional end of week archives,
# and an end of month archive for every month:
# Keep 7 end\-of\-day, 4 additional end\-of\-week archives,
# and an end\-of\-month archive for every month:
$ borg prune \-v \-\-list \-\-keep\-daily=7 \-\-keep\-weekly=4 \-\-keep\-monthly=\-1 /path/to/repo
# Keep all backups in the last 10 days, 4 additional end of week archives,
# and an end of month archive for every month:
# Keep all backups in the last 10 days, 4 additional end\-of\-week archives,
# and an end\-of\-month archive for every month:
$ borg prune \-v \-\-list \-\-keep\-within=10d \-\-keep\-weekly=4 \-\-keep\-monthly=\-1 /path/to/repo
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
There is also a visualized prune example in \fBdocs/misc/prune\-example.txt\fP\&.
There is also a visual example of pruning in \fBdocs/misc/prune\-example.txt\fP\&.
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-compact(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-RECREATE" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-recreate" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-recreate \- Re-create archives
.SH SYNOPSIS
.sp
@ -37,13 +38,13 @@ borg [common options] recreate [options] [REPOSITORY_OR_ARCHIVE] [PATH...]
.sp
Recreate the contents of existing archives.
.sp
recreate is a potentially dangerous function and might lead to data loss
Recreate is a potentially dangerous operation and might lead to data loss
(if used wrongly). BE VERY CAREFUL!
.sp
Important: Repository disk space is \fBnot\fP freed until you run \fBborg compact\fP\&.
.sp
\fB\-\-exclude\fP, \fB\-\-exclude\-from\fP, \fB\-\-exclude\-if\-present\fP, \fB\-\-keep\-exclude\-tags\fP
and PATH have the exact same semantics as in "borg create", but they only check
and PATH have the exact same semantics as in \(dqborg create\(dq, but they only check
for files in the archives and not in the local file system. If PATHs are specified,
the resulting archives will only contain files from these PATHs.
.sp
@ -55,34 +56,34 @@ Due to how Borg stores compressed size information this might display
incorrect information for archives that were not recreated at the same time.
There is no risk of data loss by this.
.sp
\fB\-\-chunker\-params\fP will re\-chunk all files in the archive, this can be
\fB\-\-chunker\-params\fP will re\-chunk all files in the archive; this can be
used to have upgraded Borg 0.xx or Attic archives deduplicate with
Borg 1.x archives.
.sp
\fBUSE WITH CAUTION.\fP
Depending on the PATHs and patterns given, recreate can be used to permanently
delete files from archives.
When in doubt, use \fB\-\-dry\-run \-\-verbose \-\-list\fP to see how patterns/PATHS are
When in doubt, use \fB\-\-dry\-run \-\-verbose \-\-list\fP to see how patterns/PATHs are
interpreted. See \fIlist_item_flags\fP in \fBborg create\fP for details.
.sp
The archive being recreated is only removed after the operation completes. The
archive that is built during the operation exists at the same time at
"<ARCHIVE>.recreate". The new archive will have a different archive ID.
\(dq<ARCHIVE>.recreate\(dq. The new archive will have a different archive ID.
.sp
With \fB\-\-target\fP the original archive is not replaced, instead a new archive is created.
With \fB\-\-target\fP the original archive is not replaced; instead, a new archive is created.
.sp
When rechunking (or recompressing), space usage can be substantial \- expect
at least the entire deduplicated size of the archives using the previous
chunker (or compression) params.
chunker (or compression) parameters.
.sp
If you recently ran borg check \-\-repair and it had to fix lost chunks with all\-zero
If you recently ran \fBborg check \-\-repair\fP and it had to fix lost chunks with all\-zero
replacement chunks, please first run another backup for the same data and re\-run
borg check \-\-repair afterwards to heal any archives that had lost chunks which are
\fBborg check \-\-repair\fP afterwards to heal any archives that had lost chunks which are
still generated from the input data.
.sp
Important: running borg recreate to re\-chunk will remove the chunks_healthy
Important: running \fBborg recreate\fP to re\-chunk will remove the \fBchunks_healthy\fP
metadata of all items with replacement chunks, so healing will not be possible
any more after re\-chunking (it is also unlikely it would ever work: due to the
anymore after re\-chunking (it is also unlikely it would ever work: due to the
change of chunking parameters, the missing chunk likely will never be seen again
even if you still have the data that produced it).
.SH OPTIONS
@ -97,7 +98,7 @@ repository or archive to recreate
.B PATH
paths to recreate; patterns are supported
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-\-list
@ -112,7 +113,7 @@ do not change anything
.B \-s\fP,\fB \-\-stats
print statistics at end
.UNINDENT
.SS Exclusion options
.SS Include/Exclude options
.INDENT 0.0
.TP
.BI \-e \ PATTERN\fR,\fB \ \-\-exclude \ PATTERN
@ -128,7 +129,7 @@ include/exclude paths matching PATTERN
read include/exclude patterns from PATTERNFILE, one per line
.TP
.B \-\-exclude\-caches
exclude directories that contain a CACHEDIR.TAG file (\fI\%http://www.bford.info/cachedir/spec.html\fP)
exclude directories that contain a CACHEDIR.TAG file (\%<http://\:www\:.bford\:.info/\:cachedir/\:spec\:.html>)
.TP
.BI \-\-exclude\-if\-present \ NAME
exclude directories that are tagged by containing a filesystem object with the given NAME
@ -152,56 +153,53 @@ add a comment text to the archive
manually specify the archive creation date/time (UTC, yyyy\-mm\-ddThh:mm:ss format). alternatively, give a reference file/directory.
.TP
.BI \-C \ COMPRESSION\fR,\fB \ \-\-compression \ COMPRESSION
select compression algorithm, see the output of the "borg help compression" command for details.
select compression algorithm, see the output of the \(dqborg help compression\(dq command for details.
.TP
.BI \-\-recompress \ MODE
recompress data chunks according to \fIMODE\fP and \fB\-\-compression\fP\&. Possible modes are \fIif\-different\fP: recompress if current compression is with a different compression algorithm (the level is not considered); \fIalways\fP: recompress even if current compression is with the same compression algorithm (use this to change the compression level); and \fInever\fP: do not recompress (use this option to explicitly prevent recompression). If no MODE is given, \fIif\-different\fP will be used. Not passing \-\-recompress is equivalent to "\-\-recompress never".
recompress data chunks according to \fIMODE\fP and \fB\-\-compression\fP\&. Possible modes are \fIif\-different\fP: recompress if current compression is with a different compression algorithm (the level is not considered); \fIalways\fP: recompress even if current compression is with the same compression algorithm (use this to change the compression level); and \fInever\fP: do not recompress (use this option to explicitly prevent recompression). If no MODE is given, \fIif\-different\fP will be used. Not passing \-\-recompress is equivalent to \(dq\-\-recompress never\(dq.
.TP
.BI \-\-chunker\-params \ PARAMS
specify the chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or \fIdefault\fP to use the current defaults. default: buzhash,19,23,21,4095
rechunk using given chunker parameters (ALGO, CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS, HASH_WINDOW_SIZE) or \fIdefault\fP to use the chunker defaults. default: do not rechunk
.UNINDENT
.SH EXAMPLES
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
# Make old (Attic / Borg 0.xx) archives deduplicate with Borg 1.x archives.
.EX
# Make old (Attic/Borg 0.xx) archives deduplicate with Borg 1.x archives.
# Archives created with Borg 1.1+ and the default chunker params are skipped
# (archive ID stays the same).
$ borg recreate /mnt/backup \-\-chunker\-params default \-\-progress
# Create a backup with little but fast compression
# Create a backup with low but fast compression.
$ borg create /mnt/backup::archive /some/files \-\-compression lz4
# Then compress it \- this might take longer, but the backup has already completed,
# Then compress it this might take longer, but the backup has already completed,
# so no inconsistencies from a long\-running backup job.
$ borg recreate /mnt/backup::archive \-\-recompress \-\-compression zlib,9
# Remove unwanted files from all archives in a repository.
# Note the relative path for the \-\-exclude option \- archives only contain relative paths.
# Note the relative path for the \-\-exclude option archives only contain relative paths.
$ borg recreate /mnt/backup \-\-exclude home/icke/Pictures/drunk_photos
# Change archive comment
$ borg create \-\-comment "This is a comment" /mnt/backup::archivename ~
# Change the archive comment.
$ borg create \-\-comment \(dqThis is a comment\(dq /mnt/backup::archivename ~
$ borg info /mnt/backup::archivename
Name: archivename
Fingerprint: ...
Comment: This is a comment
\&...
$ borg recreate \-\-comment "This is a better comment" /mnt/backup::archivename
$ borg recreate \-\-comment \(dqThis is a better comment\(dq /mnt/backup::archivename
$ borg info /mnt/backup::archivename
Name: archivename
Fingerprint: ...
Comment: This is a better comment
\&...
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-patterns(1)\fP, \fIborg\-placeholders(1)\fP, \fIborg\-compression(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-RENAME" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-rename" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-rename \- Rename an existing archive
.SH SYNOPSIS
.sp
@ -54,8 +55,7 @@ the new archive name to use
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg create /path/to/repo::archivename ~
$ borg list /path/to/repo
archivename Mon, 2016\-02\-15 19:50:19
@ -63,14 +63,12 @@ archivename Mon, 2016\-02\-15 19:50:19
$ borg rename /path/to/repo::archivename newname
$ borg list /path/to/repo
newname Mon, 2016\-02\-15 19:50:19
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-SERVE" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-serve" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-serve \- Start in server mode. This command is usually not used manually.
.SH SYNOPSIS
.sp
@ -39,7 +40,7 @@ This command starts a repository server process. This command is usually not use
.SH OPTIONS
.sp
See \fIborg\-common(1)\fP for common options of Borg commands.
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.BI \-\-restrict\-to\-path \ PATH
@ -49,14 +50,14 @@ restrict repository access to PATH. Can be specified multiple times to allow the
restrict repository access. Only the repository located at PATH (no sub\-directories are considered) is accessible. Can be specified multiple times to allow the client access to several repositories. Unlike \fB\-\-restrict\-to\-path\fP sub\-directories are not accessible; PATH needs to directly point at a repository location. PATH may be an empty directory or the last element of PATH may not exist, in which case the client may initialize a repository there.
.TP
.B \-\-append\-only
only allow appending to repository segment files. Note that this only affects the low level structure of the repository, and running \fIdelete\fP or \fIprune\fP will still be allowed. See \fIappend_only_mode\fP in Additional Notes for more details.
only allow appending to repository segment files. Note that this only affects the low level structure of the repository, and running \fIdelete\fP or \fIprune\fP or reading from the repository will still be allowed. See \fIappend_only_mode\fP in Additional Notes for more details.
.TP
.BI \-\-storage\-quota \ QUOTA
Override storage quota of the repository (e.g. 5G, 1.5T). When a new repository is initialized, sets the storage quota on the new repository as well. Default: no quota.
.UNINDENT
.SH EXAMPLES
.sp
\fBborg serve\fP has special support for ssh forced commands (see \fBauthorized_keys\fP
\fBborg serve\fP has special support for SSH forced commands (see \fBauthorized_keys\fP
example below): if the environment variable SSH_ORIGINAL_COMMAND is set it will
ignore some options given on the command line and use the values from the
variable instead. This only applies to a carefully controlled allowlist of safe
@ -71,44 +72,43 @@ giving up and aborting the operation when another process is holding a lock.
.UNINDENT
.sp
Environment variables (such as BORG_XXX) contained in the original
command sent by the client are \fInot\fP interpreted, but ignored. If BORG_XXX environment
variables should be set on the \fBborg serve\fP side, then these must be set in system\-specific
command sent by the client are \fInot\fP interpreted; they are ignored. If BORG_XXX environment
variables need to be set on the \fBborg serve\fP side, then these must be set in system\-specific
locations like \fB/etc/environment\fP or in the forced command itself (example below).
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
# Allow an SSH keypair to only run borg, and only have access to /path/to/repo.
.EX
# Allow an SSH key pair to only run borg, and only have access to /path/to/repo.
# Use key options to disable unneeded and potentially dangerous SSH functionality.
# This will help to secure an automated remote backup system.
# This helps secure an automated remote backup system.
$ cat ~/.ssh/authorized_keys
command="borg serve \-\-restrict\-to\-path /path/to/repo",restrict ssh\-rsa AAAAB3[...]
command=\(dqborg serve \-\-restrict\-to\-path /path/to/repo\(dq,restrict ssh\-rsa AAAAB3[...]
# Set a BORG_XXX environment variable on the "borg serve" side
# Set a BORG_XXX environment variable on the \(ga\(gaborg serve\(ga\(ga side.
$ cat ~/.ssh/authorized_keys
command="export BORG_XXX=value; borg serve [...]",restrict ssh\-rsa [...]
.ft P
.fi
command=\(dqBORG_XXX=value borg serve [...]\(dq,restrict ssh\-rsa [...]
.EE
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
The examples above use the \fBrestrict\fP directive. This does automatically
block potential dangerous ssh features, even when they are added in a future
update. Thus, this option should be preferred.
The examples above use the \fBrestrict\fP directive and assume a POSIX\-compliant
shell set as the user\(aqs login shell.
This automatically blocks potentially dangerous SSH features, even when
they are added in a future update. Thus, this option should be preferred.
.sp
If you\(aqre using openssh\-server < 7.2, however, you have to explicitly specify
the ssh features to restrict and cannot simply use the restrict option as it
has been introduced in v7.2. We recommend to use
If you\(aqre using OpenSSH server < 7.2, however, you have to explicitly specify
the SSH features to restrict and cannot simply use the \fBrestrict\fP option as it
was introduced in v7.2. We recommend using
\fBno\-port\-forwarding,no\-X11\-forwarding,no\-pty,no\-agent\-forwarding,no\-user\-rc\fP
in this case.
.UNINDENT
.UNINDENT
.sp
Details about sshd usage: \fI\%sshd(8)\fP
Details about sshd usage: sshd(8) \%<https://\:www\:.openbsd\:.org/\:cgi-bin/\:man\:.cgi/\:OpenBSD-current/\:man8/\:sshd.8>
.SS SSH Configuration
.sp
\fBborg serve\fP\(aqs pipes (\fBstdin\fP/\fBstdout\fP/\fBstderr\fP) are connected to the \fBsshd\fP process on the server side. In the event that the SSH connection between \fBborg serve\fP and the client is disconnected or stuck abnormally (for example, due to a network outage), it can take a long time for \fBsshd\fP to notice the client is disconnected. In the meantime, \fBsshd\fP continues running, and as a result so does the \fBborg serve\fP process holding the lock on the repository. This can cause subsequent \fBborg\fP operations on the remote repository to fail with the error: \fBFailed to create/acquire the lock\fP\&.
@ -119,42 +119,56 @@ Either in the client side\(aqs \fB~/.ssh/config\fP file, or in the client\(aqs \
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
Host backupserver
ServerAliveInterval 10
ServerAliveCountMax 30
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
Replacing \fBbackupserver\fP with the hostname, FQDN or IP address of the borg server.
Replace \fBbackupserver\fP with the hostname, FQDN, or IP address of the Borg server.
.sp
This will cause the client to send a keepalive to the server every 10 seconds. If 30 consecutive keepalives are sent without a response (a time of 300 seconds), the ssh client process will be terminated, causing the borg process to terminate gracefully.
This will cause the client to send a keepalive to the server every 10 seconds. If 30 consecutive keepalives are sent without a response (a time of 300 seconds), the SSH client process will be terminated, causing the Borg process to terminate gracefully.
.sp
On the server side\(aqs \fBsshd\fP configuration file (typically \fB/etc/ssh/sshd_config\fP):
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
ClientAliveInterval 10
ClientAliveCountMax 30
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
This will cause the server to send a keep alive to the client every 10 seconds. If 30 consecutive keepalives are sent without a response (a time of 300 seconds), the server\(aqs sshd process will be terminated, causing the \fBborg serve\fP process to terminate gracefully and release the lock on the repository.
This will cause the server to send a keepalive to the client every 10 seconds. If 30 consecutive keepalives are sent without a response (a time of 300 seconds), the server\(aqs sshd process will be terminated, causing the \fBborg serve\fP process to terminate gracefully and release the lock on the repository.
.sp
If you then run borg commands with \fB\-\-lock\-wait 600\fP, this gives sufficient time for the borg serve processes to terminate after the SSH connection is torn down after the 300 second wait for the keepalives to fail.
If you then run Borg commands with \fB\-\-lock\-wait 600\fP, this gives sufficient time for the \fBborg serve\fP processes to terminate after the SSH connection is torn down following the 300\-second wait for the keepalives to fail.
.sp
You may, of course, modify the timeout values demonstrated above to values that suit your environment and use case.
.sp
When the client is untrusted, it is a good idea to set the backup
user\(aqs shell to a simple implementation (\fB/bin/sh\fP is only an example and may or may
not be such a simple implementation):
.INDENT 0.0
.INDENT 3.5
.sp
.EX
chsh \-s /bin/sh BORGUSER
.EE
.UNINDENT
.UNINDENT
.sp
Because the configured shell is used by openssh \%<https://\:www\:.openssh\:.com/>
to execute the command configured through the \fBauthorized_keys\fP file
using \fB\(dq$SHELL\(dq \-c \(dq$COMMAND\(dq\fP,
setting a minimal shell implementation reduces the attack surface
compared to when a feature\-rich and complex shell implementation is
used.
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,15 +28,15 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-UMOUNT" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-umount" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-umount \- un-mount the FUSE filesystem
.SH SYNOPSIS
.sp
borg [common options] umount [options] MOUNTPOINT
.SH DESCRIPTION
.sp
This command un\-mounts a FUSE filesystem that was mounted with \fBborg mount\fP\&.
This command unmounts a FUSE filesystem that was mounted with \fBborg mount\fP\&.
.sp
This is a convenience wrapper that just calls the platform\-specific shell
command \- usually this is either umount or fusermount \-u.
@ -52,8 +53,7 @@ mountpoint of the filesystem to umount
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# Mounting the repository shows all archives.
# Archives are loaded lazily, expect some delay when navigating to an archive
# for the first time.
@ -69,7 +69,7 @@ bin boot etc home lib lib64 lost+found media mnt opt
root sbin srv tmp usr var
$ borg umount /tmp/mymountpoint
# The "versions view" merges all archives in the repository
# The \(dqversions view\(dq merges all archives in the repository
# and provides a versioned view on files.
$ borg mount \-o versions /path/to/repo /tmp/mymountpoint
$ ls \-l /tmp/mymountpoint/home/user/doc.txt/
@ -79,7 +79,7 @@ total 24
$ borg umount /tmp/mymountpoint
# Archive filters are supported.
# These are especially handy for the "versions view",
# These are especially handy for the \(dqversions view\(dq,
# which does not support lazy processing of archives.
$ borg mount \-o versions \-\-glob\-archives \(aq*\-my\-home\(aq \-\-last 10 /path/to/repo /tmp/mymountpoint
@ -87,42 +87,44 @@ $ borg mount \-o versions \-\-glob\-archives \(aq*\-my\-home\(aq \-\-last 10 /pa
# These can speed up mounting and lower memory needs significantly.
$ borg mount /path/to/repo /tmp/mymountpoint only/that/path
$ borg mount \-\-exclude \(aq...\(aq /path/to/repo /tmp/mymountpoint
.ft P
.fi
# When using BORG_REPO env var, use :: as positional argument:
export BORG_REPO=/path/to/repo
# Mount the whole repo:
borg mount :: /tmp/mymountpoint
# Mount some specific archive:
borg mount ::root\-2016\-02\-15 /tmp/mymountpoint
.EE
.UNINDENT
.UNINDENT
.SS borgfs
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ echo \(aq/mnt/backup /tmp/myrepo fuse.borgfs defaults,noauto 0 0\(aq >> /etc/fstab
$ echo \(aq/mnt/backup::root\-2016\-02\-15 /tmp/myarchive fuse.borgfs defaults,noauto 0 0\(aq >> /etc/fstab
$ mount /tmp/myrepo
$ mount /tmp/myarchive
$ ls /tmp/myrepo
root\-2016\-02\-01 root\-2016\-02\-2015
root\-2016\-02\-01 root\-2016\-02\-15
$ ls /tmp/myarchive
bin boot etc home lib lib64 lost+found media mnt opt root sbin srv tmp usr var
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
\fBborgfs\fP will be automatically provided if you used a distribution
package, \fBpip\fP or \fBsetup.py\fP to install Borg. Users of the
standalone binary will have to manually create a symlink (see
\fIpyinstaller\-binary\fP).
package or \fBpip\fP to install Borg. Users of the standalone binary will have
to manually create a symlink (see \fIpyinstaller\-binary\fP).
.UNINDENT
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP, \fIborg\-mount(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +28,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-UPGRADE" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-upgrade" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-upgrade \- upgrade a repository from a previous version
.SH SYNOPSIS
.sp
@ -53,6 +54,23 @@ except when noted otherwise in the changelog
.UNINDENT
.SS Borg 1.x.y upgrades
.sp
Archive TAM authentication:
.sp
Use \fBborg upgrade \-\-archives\-tam REPO\fP to add archive TAMs to all
archives that are not TAM authenticated yet.
This is a convenient method to just trust all archives present \- if
an archive does not have TAM authentication yet, a TAM will be added.
Archives created by old borg versions < 1.0.9 do not have TAMs.
Archives created by newer borg versions should have TAMs already.
If you have a high\-risk environment, you should not just run this,
but first verify that the archives are authentic and not malicious
(== have good content, have a good timestamp).
Borg 1.2.5+ needs all archives to be TAM authenticated for safety reasons.
.sp
This upgrade needs to be done once per repository.
.sp
Manifest TAM authentication:
.sp
Use \fBborg upgrade \-\-tam REPO\fP to require manifest authentication
introduced with Borg 1.0.9 to address security issues. This means
that modifying the repository after doing this with a version prior
@ -69,7 +87,7 @@ If you routinely do this you might not want to enable this upgrade
reverse the upgrade by issuing \fBborg upgrade \-\-disable\-tam REPO\fP\&.
.sp
See
\fI\%https://borgbackup.readthedocs.io/en/stable/changes.html#pre\-1\-0\-9\-manifest\-spoofing\-vulnerability\fP
\%<https://\:borgbackup\:.readthedocs\:.io/\:en/\:stable/\:changes\:.html#\:pre-1-0-9-manifest-spoofing-vulnerability>
for details.
.SS Attic and Borg 0.xx to Borg 1.x
.sp
@ -84,7 +102,7 @@ with the old chunks in the upgraded repository.
See \fB\-\-chunker\-params\fP option of \fBborg create\fP and \fBborg recreate\fP\&.
.sp
\fBborg upgrade\fP will change the magic strings in the repository\(aqs
segments to match the new Borg magic strings. The keyfiles found in
segments to match the new Borg magic strings. The key files found in
$ATTIC_KEYS_DIR or ~/.attic/keys/ will also be converted and
copied to $BORG_KEYS_DIR or ~/.config/borg/keys.
.sp
@ -92,20 +110,18 @@ The cache files are converted, from $ATTIC_CACHE_DIR or
~/.cache/attic to $BORG_CACHE_DIR or ~/.cache/borg, but the
cache layout between Borg and Attic changed, so it is possible
the first backup after the conversion takes longer than expected
due to the cache resync.
due to the cache re\-sync.
.sp
Upgrade should be able to resume if interrupted, although it
will still iterate over all segments. If you want to start
from scratch, use \fIborg delete\fP over the copied repository to
from scratch, use \fBborg delete\fP over the copied repository to
make sure the cache files are also removed:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
borg delete borg
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -131,7 +147,7 @@ See \fIborg\-common(1)\fP for common options of Borg commands.
.B REPOSITORY
path to the repository to be upgraded
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-n\fP,\fB \-\-dry\-run
@ -146,15 +162,23 @@ Force upgrade
.B \-\-tam
Enable manifest authentication (in key and cache) (Borg 1.0.9 and later).
.TP
.B \-\-check\-tam
check manifest authentication (in key and cache).
.TP
.B \-\-disable\-tam
Disable manifest authentication (in key and cache).
.TP
.B \-\-check\-archives\-tam
check TAM authentication for all archives.
.TP
.B \-\-archives\-tam
add TAM authentication for all archives.
.UNINDENT
.SH EXAMPLES
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
# Upgrade the borg repository to the most recent version.
$ borg upgrade \-v /path/to/repo
making a hardlink copy in /path/to/repo.before\-upgrade\-2016\-02\-15\-20:51:55
@ -164,17 +188,16 @@ converting repo index /path/to/repo/index.0
converting 1 segments...
converting borg 0.xx to borg current
no key file found for repository
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.SS Upgrading a passphrase encrypted attic repo
.SS Upgrading a passphrase\-encrypted Attic repo
.sp
attic offered a "passphrase" encryption mode, but this was removed in borg 1.0
and replaced by the "repokey" mode (which stores the passphrase\-protected
encryption key into the repository config).
Attic offered a \(dqpassphrase\(dq encryption mode, but this was removed in Borg 1.0
and replaced by the \(dqrepokey\(dq mode (which stores the passphrase\-protected
encryption key in the repository config).
.sp
Thus, to upgrade a "passphrase" attic repo to a "repokey" borg repo, 2 steps
Thus, to upgrade a \(dqpassphrase\(dq Attic repo to a \(dqrepokey\(dq Borg repo, two steps
are needed, in this order:
.INDENT 0.0
.IP \(bu 2
@ -185,7 +208,6 @@ borg key migrate\-to\-repokey repo
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

81
docs/man/borg-version.1 Normal file
View file

@ -0,0 +1,81 @@
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
.
.de1 rstReportMargin
\\$1 \\n[an-margin]
level \\n[rst2man-indent-level]
level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
-
\\n[rst2man-indent0]
\\n[rst2man-indent1]
\\n[rst2man-indent2]
..
.de1 INDENT
.\" .rstReportMargin pre:
. RS \\$1
. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin]
. nr rst2man-indent-level +1
.\" .rstReportMargin post:
..
.de UNINDENT
. RE
.\" indent \\n[an-margin]
.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]]
.nr rst2man-indent-level -1
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "borg-version" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-version \- Display the borg client / borg server version
.SH SYNOPSIS
.sp
borg [common options] version [options] [REPOSITORY]
.SH DESCRIPTION
.sp
This command displays the Borg client version / Borg server version.
.sp
If a local repository is given, the client code directly accesses the repository,
thus we show the client version also as the server version.
.sp
If a remote repository is given (e.g., ssh:), the remote Borg is queried and
its version is displayed as the server version.
.sp
Examples:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
# local repo (client uses 1.4.0 alpha version)
$ borg version /mnt/backup
1.4.0a / 1.4.0a
# remote repo (client uses 1.4.0 alpha, server uses 1.2.7 release)
$ borg version ssh://borg@borgbackup:repo
1.4.0a / 1.2.7
.EE
.UNINDENT
.UNINDENT
.sp
Due to the version tuple format used in Borg client/server negotiation, only
a simplified version is displayed (as provided by \fBborg.version.format_version\fP).
.sp
There is also \fBborg \-\-version\fP to display a potentially more precise client version.
.SH OPTIONS
.sp
See \fIborg\-common(1)\fP for common options of Borg commands.
.SS arguments
.INDENT 0.0
.TP
.B REPOSITORY
repository (used to determine client/server situation)
.UNINDENT
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH Author
The Borg Collective
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,27 +28,35 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG-WITH-LOCK" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg-with-lock" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg-with-lock \- run a user specified command with the repository lock held
.SH SYNOPSIS
.sp
borg [common options] with\-lock [options] REPOSITORY COMMAND [ARGS...]
.SH DESCRIPTION
.sp
This command runs a user\-specified command while the repository lock is held.
This command runs a user\-specified command while locking the repository. For example:
.INDENT 0.0
.INDENT 3.5
.sp
.EX
$ borg with\-lock /mnt/borgrepo rsync \-av /mnt/borgrepo /somewhere/else/borgrepo
.EE
.UNINDENT
.UNINDENT
.sp
It will first try to acquire the lock (make sure that no other operation is
running in the repo), then execute the given command as a subprocess and wait
for its termination, release the lock and return the user command\(aqs return
code as borg\(aqs return code.
running in the repository), then execute the given command as a subprocess and wait
for its termination, release the lock, and return the user command\(aqs return
code as Borg\(aqs return code.
.sp
\fBNOTE:\fP
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
If you copy a repository with the lock held, the lock will be present in
the copy. Thus, before using borg on the copy from a different host,
you need to use "borg break\-lock" on the copied repository, because
the copy. Thus, before using Borg on the copy from a different host,
you need to use \(dqborg break\-lock\(dq on the copied repository, because
Borg is cautious and does not automatically remove stale locks made by a different host.
.UNINDENT
.UNINDENT
@ -69,7 +78,6 @@ command arguments
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,6 @@
.\" Man page generated from reStructuredText.
'\" t
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,8 +29,8 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORG" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borg" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borg \- deduplicating and encrypting backup tool
.SH SYNOPSIS
.sp
@ -58,11 +60,9 @@ Before a backup can be made a repository has to be initialized:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg init \-\-encryption=repokey /path/to/repo
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.IP 2. 3
@ -71,11 +71,9 @@ Backup the \fB~/src\fP and \fB~/Documents\fP directories into an archive called
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg create /path/to/repo::Monday ~/src ~/Documents
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.IP 3. 3
@ -83,11 +81,9 @@ The next day create a new archive called \fITuesday\fP:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg create \-\-stats /path/to/repo::Tuesday ~/src ~/Documents
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -98,8 +94,7 @@ data (not shared with other archives):
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
Archive name: Tuesday
Archive fingerprint: bd31004d58f51ea06ff735d2e5ac49376901b21d58035f8fb05dbf866566e3c2
@ -116,8 +111,7 @@ All archives: 8.33 MB 8.34 MB 4.19 MB
Unique chunks Total chunks
Chunk index: 132 261
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.IP 4. 3
@ -125,13 +119,11 @@ List all archives in the repository:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg list /path/to/repo
Monday Mon, 2016\-02\-15 19:14:44
Tuesday Tue, 2016\-02\-16 19:15:11
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.IP 5. 3
@ -139,14 +131,12 @@ List the contents of the \fIMonday\fP archive:
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg list /path/to/repo::Monday
drwxr\-xr\-x user group 0 Mon, 2016\-02\-15 18:22:30 home/user/Documents
\-rw\-r\-\-r\-\- user group 7961 Mon, 2016\-02\-15 18:22:30 home/user/Documents/Important.doc
\&...
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.IP 6. 3
@ -154,28 +144,34 @@ Restore the \fIMonday\fP archive by extracting the files relative to the current
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg extract /path/to/repo::Monday
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.IP 7. 3
Recover disk space by manually deleting the \fIMonday\fP archive:
Delete the \fIMonday\fP archive (please note that this does \fBnot\fP free repo disk space):
.INDENT 3.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
$ borg delete /path/to/repo::Monday
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.IP 8. 3
Recover disk space by compacting the segment files in the repo:
.INDENT 3.0
.INDENT 3.5
.sp
.EX
$ borg compact /path/to/repo
.EE
.UNINDENT
.UNINDENT
.UNINDENT
.sp
\fBNOTE:\fP
\fBNote:\fP
.INDENT 0.0
.INDENT 3.5
Borg is quiet by default (it works on WARNING log level).
@ -194,18 +190,16 @@ in the example), but not in between them:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
borg create \-s \-\-progress repo::archive path # good and preferred
borg create repo::archive path \-s \-\-progress # also works
borg create \-s repo::archive path \-\-progress # works, but ugly
borg create repo::archive \-s \-\-progress path # BAD
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
This is due to a problem in the argparse module: \fI\%https://bugs.python.org/issue15112\fP
This is due to a problem in the argparse module: \%<https://\:bugs\:.python\:.org/\:issue15112>
.SS Repository URLs
.sp
\fBLocal filesystem\fP (or locally mounted network filesystem):
@ -219,13 +213,21 @@ expanded by your shell).
.sp
Note: you may also prepend a \fBfile://\fP to a filesystem path to get URL style.
.sp
\fBRemote repositories\fP accessed via ssh \fI\%user@host\fP:
\fBRemote repositories\fP accessed via ssh \%<user@\:host>:
.sp
\fBuser@host:/path/to/repo\fP \- remote repo, absolute path
\fBssh://user@host:port/path/to/repo\fP \- remote repo, absolute path, port can be given
.sp
\fBssh://user@host:port/path/to/repo\fP \- same, alternative syntax, port can be given
\fBuser@host:/path/to/repo\fP \- remote repo, absolute path, deprecated syntax
.sp
\fBRemote repositories with relative paths\fP can be given using this syntax:
\fBRemote repositories with relative paths, URL style syntax with port\fP:
.sp
\fBssh://user@host:port/./path/to/repo\fP \- path relative to current directory
.sp
\fBssh://user@host:port/~/path/to/repo\fP \- path relative to user\(aqs home directory
.sp
\fBssh://user@host:port/~other/path/to/repo\fP \- path relative to other\(aqs home directory (deprecated)
.sp
\fBRemote repositories with relative paths, deprecated SCP style syntax\fP:
.sp
\fBuser@host:path/to/repo\fP \- path relative to current directory
.sp
@ -236,24 +238,14 @@ Note: you may also prepend a \fBfile://\fP to a filesystem path to get URL style
Note: giving \fBuser@host:/./path/to/repo\fP or \fBuser@host:/~/path/to/repo\fP or
\fBuser@host:/~other/path/to/repo\fP is also supported, but not required here.
.sp
\fBRemote repositories with relative paths, alternative syntax with port\fP:
.sp
\fBssh://user@host:port/./path/to/repo\fP \- path relative to current directory
.sp
\fBssh://user@host:port/~/path/to/repo\fP \- path relative to user\(aqs home directory
.sp
\fBssh://user@host:port/~other/path/to/repo\fP \- path relative to other\(aqs home directory
.sp
If you frequently need the same repo URL, it is a good idea to set the
\fBBORG_REPO\fP environment variable to set a default for the repo URL:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
export BORG_REPO=\(aqssh://user@host:port/path/to/repo\(aq
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -285,11 +277,9 @@ If you want to capture the log output to a file, just redirect it:
.INDENT 0.0
.INDENT 3.5
.sp
.nf
.ft C
.EX
borg create repo::archive myfiles 2>> logfile
.ft P
.fi
.EE
.UNINDENT
.UNINDENT
.sp
@ -320,7 +310,7 @@ to get critical level output.
While you can set misc. log levels, do not expect that every command will
give different output on different log levels \- it\(aqs just a possibility.
.sp
\fBWARNING:\fP
\fBWarning:\fP
.INDENT 0.0
.INDENT 3.5
Options \fB\-\-critical\fP and \fB\-\-error\fP are provided for completeness,
@ -331,9 +321,8 @@ their usage is not recommended as you might miss important information.
.sp
Borg can exit with the following return codes (rc):
.TS
center;
|l|l|.
_
box center;
l|l.
T{
Return code
T} T{
@ -349,27 +338,40 @@ _
T{
1
T} T{
warning (operation reached its normal end, but there were warnings \-\-
generic warning (operation reached its normal end, but there were warnings \-\-
you should check the log, logged as WARNING)
T}
_
T{
2
T} T{
error (like a fatal error, a local or remote exception, the operation
generic error (like a fatal error, a local or remote exception, the operation
did not reach its normal end, logged as ERROR)
T}
_
T{
3..99
T} T{
specific error (enabled by BORG_EXIT_CODES=modern)
T}
_
T{
100..127
T} T{
specific warning (enabled by BORG_EXIT_CODES=modern)
T}
_
T{
128+N
T} T{
killed by signal N (e.g. 137 == kill \-9)
T}
_
.TE
.sp
If you use \fB\-\-show\-rc\fP, the return code is also logged at the indicated
level as the last log entry.
.sp
The modern exit codes (return codes, \(dqrc\(dq) are documented there: \fImsgid\fP
.SS Environment Variables
.sp
Borg uses some environment variables for automation:
@ -413,7 +415,11 @@ be checked.
Main usecase for this is to fully automate \fBborg change\-passphrase\fP\&.
.TP
.B BORG_DISPLAY_PASSPHRASE
When set, use the value to answer the "display the passphrase for verification" question when defining a new passphrase for encrypted repositories.
When set, use the value to answer the \(dqdisplay the passphrase for verification\(dq question when defining a new passphrase for encrypted repositories.
.TP
.B BORG_EXIT_CODES
When set to \(dqmodern\(dq, the borg process will return more specific exit codes (rc).
Default is \(dqlegacy\(dq and returns rc 2 for all errors, 1 for all warnings, 0 for success.
.TP
.B BORG_HOST_ID
Borg usually computes a host id from the FQDN plus the results of \fBuuid.getnode()\fP (which usually returns
@ -421,10 +427,10 @@ a unique id based on the MAC address of the network interface. Except if that MA
that case it returns a random value, which is not what we want (because it kills automatic stale lock removal).
So, if you have a all\-zero MAC address or other reasons to better externally control the host id, just set this
environment variable to a unique value. If all your FQDNs are unique, you can just use the FQDN. If not,
use \fI\%fqdn@uniqueid\fP\&.
use \%<fqdn@\:uniqueid>\&.
.TP
.B BORG_LOGGING_CONF
When set, use the given filename as \fI\%INI\fP\-style logging configuration.
When set, use the given filename as INI \%<https://\:docs\:.python\:.org/\:3/\:library/\:logging\:.config\:.html#\:configuration-file-format>\-style logging configuration.
A basic example conf can be found at \fBdocs/misc/logging.conf\fP\&.
.TP
.B BORG_RSH
@ -433,7 +439,7 @@ a custom identity file \fBssh \-i /path/to/private/key\fP\&. See \fBman ssh\fP f
the \fB\-\-rsh CMD\fP commandline option overrides the environment variable.
.TP
.B BORG_REMOTE_PATH
When set, use the given path as borg executable on the remote (defaults to "borg" if unset).
When set, use the given path as borg executable on the remote (defaults to \(dqborg\(dq if unset).
Using \fB\-\-remote\-path PATH\fP commandline option overrides the environment variable.
.TP
.B BORG_FILES_CACHE_SUFFIX
@ -442,15 +448,25 @@ When set to a value at least one character long, instructs borg to use a specifi
cache entries for backup sources other than the current sources.
.TP
.B BORG_FILES_CACHE_TTL
When set to a numeric value, this determines the maximum "time to live" for the files cache
When set to a numeric value, this determines the maximum \(dqtime to live\(dq for the files cache
entries (default: 20). The files cache is used to quickly determine whether a file is unchanged.
The FAQ explains this more detailed in: \fIalways_chunking\fP
.TP
.B BORG_USE_CHUNKS_ARCHIVE
When set to no (default: yes), the \fBchunks.archive.d\fP folder will not be used. This reduces
disk space usage but slows down cache resyncs.
.TP
.B BORG_SHOW_SYSINFO
When set to no (default: yes), system information (like OS, Python version, ...) in
exceptions is not shown.
Please only use for good reasons as it makes issues harder to analyze.
.TP
.B BORG_MSGPACK_VERSION_CHECK
Controls whether Borg checks the \fBmsgpack\fP version.
The default is \fByes\fP (strict check). Set to \fBno\fP to disable the version check and
allow any installed \fBmsgpack\fP version. Use this at your own risk; malfunctioning or
incompatible \fBmsgpack\fP versions may cause subtle bugs or repository data corruption.
.TP
.B BORG_FUSE_IMPL
Choose the lowlevel FUSE implementation borg shall use for \fBborg mount\fP\&.
This is a comma\-separated list of implementation names, they are tried in the
@ -495,23 +511,59 @@ write to disk behaviour (not continuously streaming to disk).
Retry opening a file without O_NOATIME if opening a file with O_NOATIME
caused EROFS. You will need this to make archives from volume shadow copies
in WSL1 (Windows Subsystem for Linux 1).
.TP
.B authenticated_no_key
Work around a lost passphrase or key for an \fBauthenticated\fP mode repository
(these are only authenticated, but not encrypted).
If the key is missing in the repository config, add \fBkey = anything\fP there.
.sp
This workaround is \fBonly\fP for emergencies and \fBonly\fP to extract data
from an affected repository (read\-only access):
.INDENT 7.0
.INDENT 3.5
.sp
.EX
BORG_WORKAROUNDS=authenticated_no_key borg extract repo::archive
.EE
.UNINDENT
.UNINDENT
.sp
After you have extracted all data you need, you MUST delete the repository:
.INDENT 7.0
.INDENT 3.5
.sp
.EX
BORG_WORKAROUNDS=authenticated_no_key borg delete repo
.EE
.UNINDENT
.UNINDENT
.sp
Now you can init a fresh repo. Make sure you do not use the workaround any more.
.TP
.B ignore_invalid_archive_tam
Work around invalid archive TAMs created by borg < 1.2.5, see #7791\&.
.sp
This workaround likely needs to get used only once when following the upgrade
instructions for CVE\-2023\-36811, see \fIarchives_tam_vuln\fP\&.
.sp
In normal production operations, this workaround should never be used.
.UNINDENT
.UNINDENT
.TP
.B Some automatic "answerers" (if set, they automatically answer confirmation questions):
.B Some automatic \(dqanswerers\(dq (if set, they automatically answer confirmation questions):
.INDENT 7.0
.TP
.B BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=no (or =yes)
For "Warning: Attempting to access a previously unknown unencrypted repository"
For \(dqWarning: Attempting to access a previously unknown unencrypted repository\(dq
.TP
.B BORG_RELOCATED_REPO_ACCESS_IS_OK=no (or =yes)
For "Warning: The repository at location ... was previously located at ..."
For \(dqWarning: The repository at location ... was previously located at ...\(dq
.TP
.B BORG_CHECK_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
For "This is a potentially dangerous function..." (check \-\-repair)
For \(dqThis is a potentially dangerous function...\(dq (check \-\-repair)
.TP
.B BORG_DELETE_I_KNOW_WHAT_I_AM_DOING=NO (or =YES)
For "You requested to completely DELETE the repository \fIincluding\fP all archives it contains:"
For \(dqYou requested to completely DELETE the repository \fIincluding\fP all archives it contains:\(dq
.UNINDENT
.sp
Note: answers are case sensitive. setting an invalid answer value might either give the default
@ -531,32 +583,49 @@ to modify \fBBORG_BASE_DIR\fP: the other paths for cache, config etc. will adapt
.TP
.B BORG_CACHE_DIR
Defaults to \fB$BORG_BASE_DIR/.cache/borg\fP\&. If \fBBORG_BASE_DIR\fP is not explicitly set while
\fI\%XDG env var\fP \fBXDG_CACHE_HOME\fP is set, then \fB$XDG_CACHE_HOME/borg\fP is being used instead.
XDG env var \%<https://\:specifications\:.freedesktop\:.org/\:basedir-spec/\:0\:.6/\:ar01s03\:.html> \fBXDG_CACHE_HOME\fP is set, then \fB$XDG_CACHE_HOME/borg\fP is being used instead.
This directory contains the local cache and might need a lot
of space for dealing with big repositories. Make sure you\(aqre aware of the associated
security aspects of the cache location: \fIcache_security\fP
.TP
.B BORG_CONFIG_DIR
Defaults to \fB$BORG_BASE_DIR/.config/borg\fP\&. If \fBBORG_BASE_DIR\fP is not explicitly set while
\fI\%XDG env var\fP \fBXDG_CONFIG_HOME\fP is set, then \fB$XDG_CONFIG_HOME/borg\fP is being used instead.
XDG env var \%<https://\:specifications\:.freedesktop\:.org/\:basedir-spec/\:0\:.6/\:ar01s03\:.html> \fBXDG_CONFIG_HOME\fP is set, then \fB$XDG_CONFIG_HOME/borg\fP is being used instead.
This directory contains all borg configuration directories, see the FAQ
for a security advisory about the data in this directory: \fIhome_config_borg\fP
.TP
.B BORG_SECURITY_DIR
Defaults to \fB$BORG_CONFIG_DIR/security\fP\&.
This directory contains information borg uses to track its usage of NONCES ("numbers used
once" \- usually in encryption context) and other security relevant data.
This directory contains information borg uses to track its usage of NONCES (\(dqnumbers used
once\(dq \- usually in encryption context) and other security relevant data.
.TP
.B BORG_KEYS_DIR
Defaults to \fB$BORG_CONFIG_DIR/keys\fP\&.
This directory contains keys for encrypted repositories.
.TP
.B BORG_KEY_FILE
When set, use the given filename as repository key file.
When set, use the given path as repository key file. Please note that this is only
for rather special applications that externally fully manage the key files:
.INDENT 7.0
.IP \(bu 2
this setting only applies to the keyfile modes (not to the repokey modes).
.IP \(bu 2
using a full, absolute path to the key file is recommended.
.IP \(bu 2
all directories in the given path must exist.
.IP \(bu 2
this setting forces borg to use the key file at the given location.
.IP \(bu 2
the key file must either exist (for most commands) or will be created (\fBborg init\fP).
.IP \(bu 2
you need to give a different path for different repositories.
.IP \(bu 2
you need to point to the correct key file matching the repository the command will operate on.
.UNINDENT
.TP
.B TMPDIR
This is where temporary files are stored (might need a lot of temporary space for some
operations), see \fI\%tempfile\fP for details.
operations), see tempfile \%<https://\:docs\:.python\:.org/\:3/\:library/\:tempfile\:.html#\:tempfile\:.gettempdir> for details.
.UNINDENT
.TP
.B Building:
@ -569,10 +638,6 @@ Adds given OpenSSL header file directory to the default locations (setup.py).
Adds given prefix directory to the default locations. If a \(aqinclude/lz4.h\(aq is found Borg
will be linked against the system liblz4 instead of a bundled implementation. (setup.py)
.TP
.B BORG_LIBB2_PREFIX
Adds given prefix directory to the default locations. If a \(aqinclude/blake2.h\(aq is found Borg
will be linked against the system libb2 instead of a bundled implementation. (setup.py)
.TP
.B BORG_LIBZSTD_PREFIX
Adds given prefix directory to the default locations. If a \(aqinclude/zstd.h\(aq is found Borg
will be linked against the system libzstd instead of a bundled implementation. (setup.py)
@ -582,7 +647,7 @@ will be linked against the system libzstd instead of a bundled implementation. (
Please note:
.INDENT 0.0
.IP \(bu 2
Be very careful when using the "yes" sayers, the warnings with prompt exist for your / your data\(aqs security/safety.
Be very careful when using the \(dqyes\(dq sayers, the warnings with prompt exist for your / your data\(aqs security/safety.
.IP \(bu 2
Also be very careful when putting your passphrase into a script, make sure it has appropriate file permissions (e.g.
mode 600, root:root).
@ -612,7 +677,7 @@ At least three directory levels with short names
Typically, file sizes up to a few hundred MB.
Large repositories may require large files (>2 GB).
.IP \(bu 2
Up to 1000 files per directory (10000 for repositories initialized with Borg 1.0)
Up to 1000 files per directory.
.IP \(bu 2
rename(2) / MoveFile(Ex) should work as specified, i.e. on the same file system
it should be a move (not a copy) operation, and in case of a directory
@ -626,16 +691,16 @@ config file), but the code tries to work also if hardlinks are not supported.
.SS Units
.sp
To display quantities, Borg takes care of respecting the
usual conventions of scale. Disk sizes are displayed in \fI\%decimal\fP, using powers of ten (so
\fBkB\fP means 1000 bytes). For memory usage, \fI\%binary prefixes\fP are used, and are
indicated using the \fI\%IEC binary prefixes\fP,
usual conventions of scale. Disk sizes are displayed in decimal \%<https://\:en\:.wikipedia\:.org/\:wiki/\:Decimal>, using powers of ten (so
\fBkB\fP means 1000 bytes). For memory usage, binary prefixes \%<https://\:en\:.wikipedia\:.org/\:wiki/\:Binary_prefix> are used, and are
indicated using the IEC binary prefixes \%<https://\:en\:.wikipedia\:.org/\:wiki/\:IEC_80000-13#\:Prefixes_for_binary_multiples>,
using powers of two (so \fBKiB\fP means 1024 bytes).
.SS Date and Time
.sp
We format date and time conforming to ISO\-8601, that is: YYYY\-MM\-DD and
HH:MM:SS (24h clock).
.sp
For more information about that, see: \fI\%https://xkcd.com/1179/\fP
For more information about that, see: \%<https://\:xkcd\:.com/\:1179/>
.sp
Unless otherwise noted, we display local date and time.
Internally, we store and process date and time as UTC.
@ -682,7 +747,7 @@ borg check: the repository check computes the checksums of all chunks
borg delete repo: low CPU usage
.TP
.B CPU (only for client/server operation):
When using borg in a client/server way with a \fI\%ssh:\-type\fP repo, the ssh
When using borg in a client/server way with a \%<ssh:-type> repo, the ssh
processes used for the transport layer will need some CPU on the client and
on the server due to the crypto they are doing \- esp. if you are pumping
big amounts of data.
@ -727,7 +792,7 @@ process.
For some OSes, this can be done just by setting the correct value in the
\&.bashrc (or equivalent login config file for other shells), however in
other cases it may be necessary to first enable \fBPermitUserEnvironment yes\fP
in your \fBsshd_config\fP file, then add \fBenvironment="TMPDIR=/my/big/tmpdir"\fP
in your \fBsshd_config\fP file, then add \fBenvironment=\(dqTMPDIR=/my/big/tmpdir\(dq\fP
at the start of the public key to be used in the \fBauthorized_hosts\fP file.
.TP
.B Cache files (client only):
@ -755,7 +820,7 @@ special files:
.IP \(bu 2
character and block device files (restored via mknod)
.IP \(bu 2
FIFOs ("named pipes")
FIFOs (\(dqnamed pipes\(dq)
.IP \(bu 2
special file \fIcontents\fP can be backed up in \fB\-\-read\-special\fP mode.
By default the metadata to create them with mknod(2), mkfifo(2) etc. is stored.
@ -782,9 +847,8 @@ On some platforms additional features are supported:
.\" Yes/No's are grouped by reason/mechanism/reference.
.
.TS
center;
|l|l|l|l|.
_
box center;
l|l|l|l.
T{
Platform
T} T{
@ -809,7 +873,7 @@ Yes [1]
T}
_
T{
Mac OS X
macOS
T} T{
Yes
T} T{
@ -867,7 +931,6 @@ No
T} T{
No
T}
_
.TE
.sp
Other Unix\-like operating systems may work as well, but have not been tested at all.
@ -875,7 +938,7 @@ Other Unix\-like operating systems may work as well, but have not been tested at
Note that most of the platform\-dependent features also depend on the file system.
For example, ntfs\-3g on Linux isn\(aqt able to convey NTFS ACLs.
.IP [1] 5
Only "nodump", "immutable", "compressed" and "append" are supported.
Only \(dqnodump\(dq, \(dqimmutable\(dq, \(dqcompressed\(dq and \(dqappend\(dq are supported.
Feature request #618 for more flags.
.IP [2] 5
Feature request #1332
@ -894,7 +957,7 @@ aka \fIBSD flags\fP\&. The Linux set of flags [1] is portable across platforms.
The BSDs define additional flags.
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP for common command line options
\fIborg\-common(1)\fP for common command\-line options
.sp
\fIborg\-init(1)\fP,
\fIborg\-create(1)\fP, \fIborg\-mount(1)\fP, \fIborg\-extract(1)\fP,
@ -905,19 +968,18 @@ The BSDs define additional flags.
\fIborg\-compression(1)\fP, \fIborg\-patterns(1)\fP, \fIborg\-placeholders(1)\fP
.INDENT 0.0
.IP \(bu 2
Main web site \fI\%https://www.borgbackup.org/\fP
Main web site \%<https://\:www\:.borgbackup\:.org/>
.IP \(bu 2
Releases \fI\%https://github.com/borgbackup/borg/releases\fP
Releases \%<https://\:github\:.com/\:borgbackup/\:borg/\:releases>
.IP \(bu 2
Changelog \fI\%https://github.com/borgbackup/borg/blob/master/docs/changes.rst\fP
Changelog \%<https://\:github\:.com/\:borgbackup/\:borg/\:blob/\:master/\:docs/\:changes\:.rst>
.IP \(bu 2
GitHub \fI\%https://github.com/borgbackup/borg\fP
GitHub \%<https://\:github\:.com/\:borgbackup/\:borg>
.IP \(bu 2
Security contact \fI\%https://borgbackup.readthedocs.io/en/latest/support.html#security\-contact\fP
Security contact \%<https://\:borgbackup\:.readthedocs\:.io/\:en/\:latest/\:support\:.html#\:security-contact>
.UNINDENT
.SH AUTHOR
.SH Author
The Borg Collective
orphan:
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -1,4 +1,5 @@
.\" Man page generated from reStructuredText.
.\" Man page generated from reStructuredText
.\" by the Docutils 0.22.4 manpage writer.
.
.
.nr rst2man-indent-level 0
@ -27,18 +28,42 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]]
.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]]
.in \\n[rst2man-indent\\n[rst2man-indent-level]]u
..
.TH "BORGFS" 1 "2022-02-19" "" "borg backup tool"
.SH NAME
.TH "borgfs" "1" "2026-03-18" "" "borg backup tool"
.SH Name
borgfs \- Mount archive or an entire repository as a FUSE filesystem
.SH SYNOPSIS
.sp
borgfs [options] REPOSITORY_OR_ARCHIVE MOUNTPOINT [PATH...]
.SH DESCRIPTION
.sp
This command mounts an archive as a FUSE filesystem. This can be useful for
browsing an archive or restoring individual files. Unless the \fB\-\-foreground\fP
option is given the command will run in the background until the filesystem
is \fBumounted\fP\&.
This command mounts a repository or an archive as a FUSE filesystem.
This can be useful for browsing or restoring individual files.
.sp
When restoring, take into account that the current FUSE implementation does
not support special filesystem flags and ACLs.
.sp
When mounting a repository, the top directories will be named like the
archives and the directory structure below these will be loaded on\-demand from
the repository when entering these directories, so expect some delay.
.sp
Care should be taken, as Borg backs up symlinks as\-is. When an archive
or repository is mounted, it is possible to “jump” outside the mount point
by following a symlink. If this happens, files or directories (or versions of them)
that are not part of the archive or repository may appear to be within the mount point.
.sp
Unless the \fB\-\-foreground\fP option is given the command will run in the
background until the filesystem is \fBunmounted\fP\&.
.sp
Performance tips:
.INDENT 0.0
.IP \(bu 2
when doing a \(dqwhole repository\(dq mount:
do not enter archive directories if not needed; this avoids on\-demand loading.
.IP \(bu 2
only mount a specific archive, not the whole repository.
.IP \(bu 2
only mount specific paths in a specific archive, not the complete archive.
.UNINDENT
.sp
The command \fBborgfs\fP provides a wrapper for \fBborg mount\fP\&. This can also be
used in fstab entries:
@ -59,7 +84,7 @@ override the user and group ids of all files (i.e., \fBborg mount \-o
uid=1000,gid=1000\fP).
.sp
The man page references \fBuser_id\fP and \fBgroup_id\fP mount options
(implemented by fuse) which specify the user and group id of the mount owner
(implemented by FUSE) which specify the user and group id of the mount owner
(aka, the user who does the mounting). It is set automatically by libfuse (or
the filesystem if libfuse is not used). However, you should not specify these
manually. Unlike the \fBuid\fP and \fBgid\fP mount options which affect all files,
@ -69,16 +94,16 @@ manually. Unlike the \fBuid\fP and \fBgid\fP mount options which affect all file
Additional mount options supported by borg:
.INDENT 0.0
.IP \(bu 2
versions: when used with a repository mount, this gives a merged, versioned
view of the files in the archives. EXPERIMENTAL, layout may change in future.
\fBversions\fP: when used with a repository mount, this gives a merged, versioned
view of the files in the archives. EXPERIMENTAL; the layout may change in the future.
.IP \(bu 2
allow_damaged_files: by default damaged files (where missing chunks were
replaced with runs of zeros by borg check \fB\-\-repair\fP) are not readable and
\fBallow_damaged_files\fP: by default damaged files (where missing chunks were
replaced with runs of zeros by \fBborg check \-\-repair\fP) are not readable and
return EIO (I/O error). Set this option to read such files.
.IP \(bu 2
ignore_permissions: for security reasons the "default_permissions" mount
option is internally enforced by borg. "ignore_permissions" can be given to
not enforce "default_permissions".
\fBignore_permissions\fP: for security reasons the \fBdefault_permissions\fP mount
option is internally enforced by borg. \fBignore_permissions\fP can be given to
not enforce \fBdefault_permissions\fP\&.
.UNINDENT
.sp
The BORG_MOUNT_DATA_CACHE_ENTRIES environment variable is meant for advanced users
@ -90,8 +115,15 @@ When the daemonized process receives a signal or crashes, it does not unmount.
Unmounting in these cases could cause an active rsync or similar process
to unintentionally delete data.
.sp
When running in the foreground ^C/SIGINT unmounts cleanly, but other
signals or crashes do not.
When running in the foreground, ^C/SIGINT cleanly unmounts the filesystem,
but other signals or crashes do not.
.sp
Debugging:
.sp
\fBborg mount\fP usually daemonizes and the daemon process sends stdout/stderr
to /dev/null. Thus, you need to either use \fB\-f / \-\-foreground\fP to make it stay
in the foreground and not daemonize, or use \fBBORG_LOGGING_CONF\fP to reconfigure
the logger to output to a file.
.SH OPTIONS
.sp
See \fIborg\-common(1)\fP for common options of Borg commands.
@ -107,7 +139,7 @@ where to mount filesystem
.B PATH
paths to extract; patterns are supported
.UNINDENT
.SS optional arguments
.SS options
.INDENT 0.0
.TP
.B \-V\fP,\fB \-\-version
@ -132,13 +164,13 @@ use numeric user and group identifiers from archive(s)
.INDENT 0.0
.TP
.BI \-P \ PREFIX\fR,\fB \ \-\-prefix \ PREFIX
only consider archive names starting with this prefix.
only consider archive names starting with this prefix. (deprecated)
.TP
.BI \-a \ GLOB\fR,\fB \ \-\-glob\-archives \ GLOB
only consider archive names matching the glob. sh: rules apply, see "borg help patterns". \fB\-\-prefix\fP and \fB\-\-glob\-archives\fP are mutually exclusive.
only consider archive names matching the glob. sh: rules apply (without actually using the sh: prefix), see \(dqborg help patterns\(dq.
.TP
.BI \-\-sort\-by \ KEYS
Comma\-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp
Comma\-separated list of sorting keys; valid keys are: timestamp, archive, name, id; default is: timestamp
.TP
.BI \-\-first \ N
consider first N archives after other filters were applied
@ -146,7 +178,7 @@ consider first N archives after other filters were applied
.BI \-\-last \ N
consider last N archives after other filters were applied
.UNINDENT
.SS Exclusion options
.SS Include/Exclude options
.INDENT 0.0
.TP
.BI \-e \ PATTERN\fR,\fB \ \-\-exclude \ PATTERN
@ -167,7 +199,6 @@ Remove the specified number of leading path elements. Paths with fewer elements
.SH SEE ALSO
.sp
\fIborg\-common(1)\fP
.SH AUTHOR
.SH Author
The Borg Collective
.\" Generated by docutils manpage writer.
.
.\" End of generated man page.

View file

@ -40,7 +40,7 @@ NOTES
SEE ALSO
--------
`borg-common(1)` for common command line options
`borg-common(1)` for common command-line options
`borg-init(1)`,
`borg-create(1)`, `borg-mount(1)`, `borg-extract(1)`,

View file

@ -0,0 +1,4 @@
Do NOT run the examples without isolation (e.g Vagrant) or
this code may make undesirable changes to your host.
Running `vagrant up` in this directory will update the screencasts.

75
docs/misc/asciinema/Vagrantfile vendored Normal file
View file

@ -0,0 +1,75 @@
Vagrant.configure("2") do |config|
config.vm.box = "debian/bullseye64"
config.vm.provision "install dependencies", type: "shell", inline: <<-SHELL
apt-get update
apt-get install -y wget expect gpg asciinema ssh adduser fuse
mkdir -p /wallpaper
wget \
--user-agent="borgbackup demo screencast" \
--input-file=/vagrant/sample-wallpapers.txt \
--directory-prefix=/wallpaper
SHELL
config.vm.provision "record install", type: "shell", inline: <<-SHELL
gpg --recv-keys "6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393"
asciinema rec -c 'expect /vagrant/install.tcl' --overwrite /vagrant/install.json < /dev/null
SHELL
config.vm.provision "record basic usage", type: "shell", inline: <<-SHELL
# `rm` below allows quick re-exec via:
# vagrant vagrant provision --provision-with "record basic usage"
# this is useful when testing changes
rm -r /media/backup/borgdemo || true
rm -r ~/.ssh/ || true
rm -r Wallpaper || true
deluser --remove-home borgdemo || true
# In case we have skipped "record install"
if [ ! -e /usr/local/bin/borg ] ; then
wget https://github.com/borgbackup/borg/releases/download/1.2.1/borg-linux64
install --owner root --group root --mode 755 borg-linux64 /usr/local/bin/borg
fi
mkdir -p /media/backup/borgdemo
mkdir Wallpaper
cp -r /wallpaper Wallpaper/bigcollection
cp /wallpaper/Trapper_cabin.jpg Wallpaper/deer.jpg
adduser --disabled-password borgdemo
echo '127.0.0.1 remoteserver.example' >> /etc/hosts
ssh-keygen -f ~/.ssh/id_rsa -N ''
ssh-keyscan remoteserver.example > ~/.ssh/known_hosts
runuser -u borgdemo mkdir ~borgdemo/.ssh
runuser -u borgdemo tee ~borgdemo/.ssh/authorized_keys < ~/.ssh/id_rsa.pub
asciinema rec -c 'expect /vagrant/basic.tcl' --overwrite /vagrant/basic.json < /dev/null
SHELL
config.vm.provision "record advanced usage", type: "shell", inline: <<-SHELL
rm -r /media/backup/borgdemo || true
rm -r Wallpaper || true
# In case we have skipped "record install"
if [ ! -e /usr/local/bin/borg ] ; then
wget https://github.com/borgbackup/borg/releases/download/1.2.1/borg-linux64
install --owner root --group root --mode 755 borg-linux64 /usr/local/bin/borg
fi
mkdir -p /media/backup/borgdemo
mkdir Wallpaper
cp -r /wallpaper Wallpaper/bigcollection
cp /wallpaper/Trapper_cabin.jpg Wallpaper/deer.jpg
mkdir -p ~/Downloads/big
dd if=/dev/zero of=loopbackfile.img bs=100M count=4
losetup /dev/loop0 loopbackfile.img
# Make it look as if the adv. usage screencast was recorded after basic usage
export BORG_PASSPHRASE='1234'
borg init --encryption=repokey /media/backup/borgdemo
borg create --compression lz4 /media/backup/borgdemo::backup1 Wallpaper
echo "new nice file" > Wallpaper/newfile.txt
borg create --compression lz4 /media/backup/borgdemo::backup2 Wallpaper
mv Wallpaper/bigcollection Wallpaper/bigcollection_NEW
borg create --compression lz4 /media/backup/borgdemo::backup3 Wallpaper
unset BORG_PASSPHRASE
asciinema rec -c 'expect /vagrant/advanced.tcl' --overwrite /vagrant/advanced.json < /dev/null
SHELL
end

File diff suppressed because it is too large Load diff

View file

@ -1,5 +1,11 @@
# Configuration for send -h
# Tries to emulate a human typing
# Tweak this if typing is too fast or too slow
set send_human {.05 .1 1 .01 .2}
set script {
# For the pro users, here are some advanced features of borg, so you can impress your friends. ;)
# Note: This screencast was made with borg version 1.1.0 older or newer borg versions may behave differently.
# Note: This screencast was made with __BORG_VERSION__ older or newer borg versions may behave differently.
# First of all, we can use several environment variables for borg.
# E.g. we do not want to type in our repo path and password again and again
@ -27,13 +33,12 @@ borg info :: --last 1
# So let's rename our last archive:
borg rename ::specialbackup backup-block-device
<up>
borg info :: --last 1
# A very important step if you choose keyfile mode (where the keyfile is only saved locally) is to export your keyfile and possibly print it, etc.
borg key export :: --qr-code file.html # this creates a nice HTML, but when you want something simpler
< remove comment >
< let there: borg check > --paper # this is a "manual input"-only backup (but it is also included in the --qr-code option)
borg key export --qr-html :: file.html # this creates a nice HTML, but when you want something simpler
borg key export --paper :: # this is a "manual input"-only backup (but it is also included in the --qr-code option)
## MAINTENANCE ##
# Sometimes backups get broken or we want a regular "checkup" that everything is okay
@ -63,3 +68,18 @@ ls -la /tmp/mount
borg umount /tmp/mount
# That's it, but of course there is more to explore, so have a look at the docs.
}
set script [string trim $script]
set script [string map [list __BORG_VERSION__ [exec borg -V]] $script]
set script [split $script \n]
set ::env(PS1) "$ "
set stty_init -echo
spawn -noecho /bin/sh
foreach line $script {
expect "$ "
send_user -h $line\n
send $line\n
}
expect "$ "

File diff suppressed because it is too large Load diff

View file

@ -1,5 +1,11 @@
# Configuration for send -h
# Tries to emulate a human typing
# Tweak this if typing is too fast or too slow
set send_human {.05 .1 1 .01 .2}
set script {
# Here you'll see some basic commands to start working with borg.
# Note: This teaser screencast was made with borg version 1.1.0 older or newer borg versions may behave differently.
# Note: This teaser screencast was made with __BORG_VERSION__ older or newer borg versions may behave differently.
# But let's start.
# First of all, you can always get help:
@ -16,7 +22,6 @@ borg create --stats --progress --compression lz4 /media/backup/borgdemo::backup1
# So let's add a new file
echo "new nice file" > Wallpaper/newfile.txt
<up>
borg create --stats --progress --compression lz4 /media/backup/borgdemo::backup2 Wallpaper
# Wow, this was a lot faster!
@ -24,7 +29,7 @@ borg create --stats --progress --compression lz4 /media/backup/borgdemo::backup2
# Borg recognized that most files did not change and deduplicated them.
# But what happens, when we move a dir and create a new backup?
mv
mv Wallpaper/bigcollection Wallpaper/bigcollection_NEW
borg create --stats --progress --compression lz4 /media/backup/borgdemo::backup3 Wallpaper
@ -40,7 +45,8 @@ borg list /media/backup/borgdemo::backup3 | grep 'deer.jpg'
# Oh, we found our picture. Now extract it
mv Wallpaper Wallpaper.orig
borg extract /media/backup/borgdemo::backup3 <copy>
borg extract /media/backup/borgdemo::backup3 Wallpaper/deer.jpg
# And check that it's the same:
diff -s Wallpaper/deer.jpg Wallpaper.orig/deer.jpg
@ -51,3 +57,33 @@ borg init --encryption=repokey borgdemo@remoteserver.example:./demo
# Easy, isn't it? That's all you need to know for basic usage.
# If you want to see more, have a look at the screencast showing the "advanced usage".
# In any case, enjoy using borg!
}
set script [string trim $script]
set script [string map [list __BORG_VERSION__ [exec borg -V]] $script]
set script [split $script \n]
foreach line $script {
send_user "$ "
send_user -h $line\n
spawn -noecho /bin/sh -c $line
expect {
"Enter new passphrase: " {
send -h "correct horse battery staple\n"
exp_continue
}
"Enter same passphrase again: " {
send -h "correct horse battery staple\n"
exp_continue
}
"Enter passphrase for key /media/backup/borgdemo: " {
send -h "correct horse battery staple\n"
exp_continue
}
-ex {Do you want your passphrase to be displayed for verification? [yN]: } {
send \n
exp_continue
}
eof
}
}

File diff suppressed because it is too large Load diff

View file

@ -1,9 +1,15 @@
# Configuration for send -h
# Tries to emulate a human typing
# Tweak this if typing is too fast or too slow
set send_human {.05 .1 1 .01 .2}
set script [string trim {
# This asciinema will show you the installation of borg as a standalone binary. Usually you only need this if you want to have an up-to-date version of borg or no package is available for your distro/OS.
# First, we need to download the version, we'd like to install
wget -q --show-progress https://github.com/borgbackup/borg/releases/download/1.1.0b6/borg-linux64
wget -q --show-progress https://github.com/borgbackup/borg/releases/download/1.2.1/borg-linux64
# and do not forget the GPG signature!
wget -q --show-progress https://github.com/borgbackup/borg/releases/download/1.1.0b6/borg-linux64.asc
wget -q --show-progress https://github.com/borgbackup/borg/releases/download/1.2.1/borg-linux64.asc
# In this case, we have already imported the public key of a borg developer. So we only need to verify it:
gpg --verify borg-linux64.asc
@ -19,3 +25,14 @@ sudo chmod 755 /usr/local/bin/borg
borg -V
# That's it! Check out the other screencasts to see how to actually use borgbackup.
}]
# wget may be slow
set timeout -1
foreach line [split $script \n] {
send_user "$ "
send_user -h $line\n
spawn -noecho /bin/sh -c $line
expect eof
}

View file

@ -0,0 +1,30 @@
https://upload.wikimedia.org/wikipedia/commons/2/22/Pseudo_kleinian_001_OpenCL_45154214_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/d/da/Mengerschwamm_Iteration_5_x_Mandelbulb_OpenCL_528814521414_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/e/eb/Blixos_logon_screen.jpg
https://upload.wikimedia.org/wikipedia/commons/9/90/Great_smokey_mountains_national_park_with_woman_sitting_under_tree_in_foreground.jpg
https://upload.wikimedia.org/wikipedia/commons/d/d2/Mengerschwamm_x_Generalized_Fold_Box_OpenCL_18915424_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/3/3d/Red_interesting_background.jpg
https://upload.wikimedia.org/wikipedia/commons/4/43/KIFS_OpenCL_54815_5K.jpg
https://upload.wikimedia.org/wikipedia/commons/a/a1/ProjectStealth.png
https://upload.wikimedia.org/wikipedia/commons/8/8d/KIFS_OpenCL_5434735835_5K.jpg
https://upload.wikimedia.org/wikipedia/commons/d/db/Harvett_Fox_-_Wallpaper_%2816x9_ratio%2C_without_character_logo%2C_transparent_variant%29_%28vector_version%29.svg
https://upload.wikimedia.org/wikipedia/commons/7/7f/Generalized_Fold_Box_OpenCL_4258952414_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/5/58/Mandelbox_Vary_Scale_4D_OpenCL_9648145412_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/6/62/Trapper_cabin.jpg
https://upload.wikimedia.org/wikipedia/commons/f/fd/Openarch.png
https://upload.wikimedia.org/wikipedia/commons/a/a5/Mandelbox_-_Variable_8K_6595424.jpg
https://upload.wikimedia.org/wikipedia/commons/d/d6/Mengerschwamm_Iteration_6_x_Generalized_Fold_Box_OpenCL_14048152404910_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/c/cf/Sierp_Oktaeder_x_Menger_4D_OpenCL_51241841541_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/5/59/Airbus_Wing_01798_changed.jpg
https://upload.wikimedia.org/wikipedia/commons/8/8a/Holytrinfruitlandpark1b.jpg
https://upload.wikimedia.org/wikipedia/commons/5/5c/Abox_-_Mod_12_OpenCL_45184521485_5K.jpg
https://upload.wikimedia.org/wikipedia/commons/d/d2/Menger_4D_x_Quaternion_OpenCL_644289452_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/3/3e/Gabrielsond.jpg
https://upload.wikimedia.org/wikipedia/commons/8/80/Mix_Pinski_4D_x_Mengerschwamm_OpenCL_461481542_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/8/84/Belinda_Vixen_-_Wallpaper_%28without_character_wordmark_and_hair_variant%29_%2816x9_ratio%29.jpg
https://upload.wikimedia.org/wikipedia/commons/0/00/Sierp_Oktaeder_Iteration_7_x_Menger_4D_OpenCL_2154188450481_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/2/24/Abox_4D_OpenCL_545185481_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/0/08/Sierpinski_4D_OpenCL_485274854_5K.jpg
https://upload.wikimedia.org/wikipedia/commons/0/09/Vereinigung_Sierpinski_4D_und_Mengerschwamm_OpenCl_6184524.jpg
https://upload.wikimedia.org/wikipedia/commons/a/ae/Mengerschwamm_OpenCL_955141845_8K.jpg
https://upload.wikimedia.org/wikipedia/commons/6/64/Free_high-resolution_pictures_you_can_use_on_your_personal_and_commercial_projects._%2814168975789%29.jpg

View file

@ -18,7 +18,7 @@ determined by the windows contents rather than the min/max. chunk size).
Default: 21 (statistically, chunks will be about 2^21 == 2MiB in size)
HASH_WINDOW_SIZE: the size of the window used for the rolling hash computation.
Default: 4095B
Must be an odd number. Default: 4095B
Trying it out
@ -114,4 +114,3 @@ $ ls -l /extra/repo-xl/index*
$ du -sk /extra/repo-xl/
14253464 /extra/repo-xl/

View file

@ -100,3 +100,27 @@ example simple. They all work in basically the same way.
The weekly rule is easy to understand roughly, but hard to understand in all
details. If interested, read "ISO 8601:2000 standard week-based year".
The 13weekly and 3monthly rules are two different strategies for keeping one
every quarter of a year. There are `multiple ways` to define a quarter-year;
borg prune recognizes two:
* --keep-13weekly keeps one backup every 13 weeks using ISO 8601:2000's
definition of the week-based year. January 4th is always included in the
first week of a year, and January 1st to 3rd may be in week 52 or 53 of the
previous year. Week 53 is also in the fourth quarter of the year.
* --keep-3monthly keeps one backup every 3 months. January 1st to
March 31, April 1st to June 30th, July 1st to September 30th, and October 1st
to December 31st form the quarters.
If the subtleties of the definition of a quarter year don't matter to you, a
short summary of behavior is:
* --keep-13weekly favors keeping backups at the beginning of Jan, Apr, July,
and Oct.
* --keep-3monthly favors keeping backups at the end of Dec, Mar, Jun, and Sept.
* Both strategies will have some overlap in which backups are kept.
* The differences are negligible unless backups considered for deletion were
created weekly or more frequently.
.. _multiple ways: https://en.wikipedia.org/wiki/Calendar_year#Quarter_year

View file

@ -8,7 +8,7 @@ Quick Start
This chapter will get you started with Borg and covers
various use cases.
A step by step example
A step-by-step example
----------------------
.. include:: quickstart_example.rst.inc
@ -20,12 +20,14 @@ A *Borg archive* is the result of a single backup (``borg create``). An archive
stores a snapshot of the data of the files "inside" it. One can later extract or
mount an archive to restore from a backup.
*Repositories* are filesystem directories acting as self-contained stores of archives.
Repositories can be accessed locally via path or remotely via ssh. Under the hood,
*Repositories* are file system directories acting as self-contained stores of archives.
Repositories can be accessed locally via path or remotely via SSH. Under the hood,
repositories contain data blocks and a manifest tracking which blocks are in each
archive. If some data hasn't changed from one backup to another, Borg can simply
reference an already uploaded data chunk (deduplication).
.. _about_free_space:
Important note about free space
-------------------------------
@ -35,7 +37,7 @@ a good amount of free space on the filesystem that has your backup repository
repositories. See also :ref:`cache-memory-usage`.
Borg doesn't use space reserved for root on repository disks (even when run as root),
on file systems which do not support this mechanism (e.g. XFS) we recommend to reserve
on file systems which do not support this mechanism (e.g., XFS) we recommend reserving
some space in Borg itself just to be safe by adjusting the ``additional_free_space``
setting (a good starting point is ``2G``)::
@ -47,7 +49,7 @@ by deleting/pruning archives. This mechanism is not bullet-proof in some
circumstances [1]_.
If you *really* run out of disk space, it can be hard or impossible to free space,
because Borg needs free space to operate - even to delete backup
because Borg needs free space to operate even to delete backup
archives.
You can use some monitoring process or just include the free space information
@ -56,38 +58,38 @@ in your backup log files (you check them regularly anyway, right?).
Also helpful:
- create a big file as a "space reserve", that you can delete to free space
- if you use LVM: use a LV + a filesystem that you can resize later and have
- if you use LVM: use an LV + a file system that you can resize later and have
some unallocated PEs you can add to the LV.
- consider using quotas
- use `prune` and `compact` regularly
.. [1] This failsafe can fail in these circumstances:
.. [1] This fail-safe can fail in these circumstances:
- The underlying file system doesn't support statvfs(2), or returns incorrect
data, or the repository doesn't reside on a single file system
- The underlying file system does not support statvfs(2), or returns incorrect
data, or the repository does not reside on a single file system
- Other tasks fill the disk simultaneously
- Hard quotas (which may not be reflected in statvfs(2))
Important note about permissions
--------------------------------
To avoid permissions issues (in your borg repository or borg cache), **always
To avoid permissions issues (in your Borg repository or Borg cache), **always
access the repository using the same user account**.
If you want to backup files of other users or the operating system, running
borg as root likely will be required (otherwise you'ld get `Permission denied`
If you want to back up files of other users or the operating system, running
Borg as root likely will be required (otherwise you'd get `Permission denied`
errors).
If you only back up your own files, you neither need nor want to run borg as
If you only back up your own files, you neither need nor want to run Borg as
root, just run it as your normal user.
For a local repository just always use the same user to invoke borg.
For a local repository just always use the same user to invoke Borg.
For a remote repository: always use e.g. borg@remote_host. You can use this
from different local users, the remote user running borg and accessing the
from different local users; the remote user running Borg and accessing the
repo will always be `borg`.
If you need to access a local repository from different users, you can use the
same method by using ssh to borg@localhost.
same method by using SSH to borg@localhost.
Important note about files changing during the backup process
-------------------------------------------------------------
@ -182,31 +184,31 @@ backed up and that the ``prune`` command is keeping and deleting the correct bac
--show-rc \
--compression lz4 \
--exclude-caches \
--exclude '/home/*/.cache/*' \
--exclude '/var/tmp/*' \
--exclude 'home/*/.cache/*' \
--exclude 'var/tmp/*' \
\
::'{hostname}-{now}' \
/etc \
/home \
/root \
/var \
/var
backup_exit=$?
info "Pruning repository"
# Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly
# archives of THIS machine. The '{hostname}-' prefix is very important to
# archives of THIS machine. The '{hostname}-*' matching is very important to
# limit prune's operation to this machine's archives and not apply to
# other machines' archives also:
borg prune \
--list \
--prefix '{hostname}-' \
--glob-archives '{hostname}-*' \
--show-rc \
--keep-daily 7 \
--keep-weekly 4 \
--keep-monthly 6 \
--keep-monthly 6
prune_exit=$?
@ -309,36 +311,30 @@ Backup compression
------------------
The default is lz4 (very fast, but low compression ratio), but other methods are
supported for different situations.
supported for different situations. Compression not only helps you save disk space,
but will especially speed up remote backups since less data needs to be transferred.
You can use zstd for a wide range from high speed (and relatively low
compression) using N=1 to high compression (and lower speed) using N=22.
zstd is a modern compression algorithm and might be preferable over zlib and
lzma, except if you need compatibility to older borg versions (< 1.1.4) that
did not yet offer zstd.::
zstd is a modern compression algorithm which can be parametrized to anything between
N=1 for highest speed (and relatively low compression) to N=22 for highest compression
(and lower speed)::
$ borg create --compression zstd,N /path/to/repo::arch ~
Other options are:
If you have a fast repo storage and you want minimum CPU usage, no compression::
If you have a fast repo storage and you want minimum CPU usage you can disable
compression::
$ borg create --compression none /path/to/repo::arch ~
If you have a less fast repo storage and you want a bit more compression (N=0..9,
0 means no compression, 9 means high compression):
You can also use zlib and lzma instead of zstd, although zstd usually provides the
the best compression for a given resource consumption. You may want to use these
algorithms if you need compatibility to older borg versions (< 1.1.4) that
did not yet offer zstd. Please see :ref:`borg_compression` for all options.
::
An interesting alternative is ``auto``, which first checks with lz4 whether a chunk is
compressible (that check is very fast), and only if it is, compresses it with the
specified algorithm::
$ borg create --compression zlib,N /path/to/repo::arch ~
If you have a very slow repo storage and you want high compression (N=0..9, 0 means
low compression, 9 means high compression):
::
$ borg create --compression lzma,N /path/to/repo::arch ~
$ borg create --compression auto,zstd,7 /path/to/repo::arch ~
You'll need to experiment a bit to find the best compression for your use case.
Keep an eye on CPU load and throughput.
@ -374,18 +370,8 @@ For automated backups the passphrase can be specified using the
.. warning:: The repository data is totally inaccessible without the key
and the key passphrase.
Make a backup copy of the key file (``keyfile`` mode) or repo config
file (``repokey`` mode) and keep it at a safe place, so you still have
the key in case it gets corrupted or lost. Also keep your passphrase
at a safe place. You can make backups using :ref:`borg_key_export`
subcommand.
If you want to print a backup of your key to paper use the ``--paper``
option of this command and print the result, or print this `template`_
if you need a version with QR-Code.
A backup inside of the backup that is encrypted with that key/passphrase
won't help you with that, of course.
In any case, make a backup of the borg key, see :ref:`borg_key_export` for
more details.
In case you lose your repository and the security information, but have an
older copy of it to restore from, don't use that later for creating new
@ -393,8 +379,6 @@ For automated backups the passphrase can be specified using the
values). It is better to initialize a new Borg repository. See also:
:ref:`faq_corrupt_repo`
.. _template: paperkey.html
.. _remote_repos:
Remote repositories
@ -406,7 +390,8 @@ is installed on the remote host, in which case the following syntax is used::
$ borg init user@hostname:/path/to/repo
Note: please see the usage chapter for a full documentation of repo URLs.
Note: Please see the usage chapter for a full documentation of repo URLs. Also
see :ref:`ssh_configuration` for recommended settings to avoid disconnects and hangs.
Remote operations over SSH can be automated with SSH keys. You can restrict the
use of the SSH keypair by prepending a forced command to the SSH public key in

View file

@ -15,7 +15,7 @@ Security
--------
In case you discover a security issue, please use this contact for reporting it
privately and please, if possible, use encrypted E-Mail:
privately and please, if possible, use encrypted email:
Thomas Waldmann <tw@waldmann-edv.de>
@ -28,7 +28,7 @@ Verifying signed releases
-------------------------
`Releases <https://github.com/borgbackup/borg/releases>`_ are signed with the
same GPG key and a .asc file is provided for each binary.
same GPG key and an .asc file is provided for each binary.
To verify a signature, the public key needs to be known to GPG. It can be
imported into the local keystore from a keyserver with the fingerprint::
@ -41,7 +41,7 @@ If GPG successfully imported the key, the output should include (among other thi
gpg: Total number processed: 1
...
To verify for example the signature of the borg-linux64 binary::
To verify, for example, the signature of the borg-linux64 binary::
gpg --verify borg-linux64.asc

View file

@ -22,7 +22,7 @@ Usage
window.location.pathname = replaced;
}
}
// Fixup anchored links from when usage.html contained all the commands
// Fix up anchored links from when usage.html contained all the commands
else if(hash.startsWith("borg-key") || hash == "borg-change-passphrase") {
window.location.hash = "";
window.location.pathname = window.location.pathname.replace("usage.html", "usage/key.html");
@ -47,6 +47,7 @@ Usage
usage/prune
usage/compact
usage/info
usage/version
usage/mount
usage/key
usage/upgrade

View file

@ -17,7 +17,7 @@ borg benchmark crud
+-------------------------------------------------------+----------------+----------------------------------------------+
| | ``REPOSITORY`` | repository to use for benchmark (must exist) |
+-------------------------------------------------------+----------------+----------------------------------------------+
| | ``PATH`` | path were to create benchmark input data |
| | ``PATH`` | path where to create benchmark input data |
+-------------------------------------------------------+----------------+----------------------------------------------+
| .. class:: borg-common-opt-ref |
| |
@ -37,7 +37,7 @@ borg benchmark crud
REPOSITORY
repository to use for benchmark (must exist)
PATH
path were to create benchmark input data
path where to create benchmark input data
:ref:`common_options`
@ -48,7 +48,7 @@ Description
This command benchmarks borg CRUD (create, read, update, delete) operations.
It creates input data below the given PATH and backups this data into the given REPO.
It creates input data below the given PATH and backs up this data into the given REPO.
The REPO must already exist (it could be a fresh empty repo or an existing repo, the
command will create / read / update / delete some archives named borg-benchmark-crud\* there.
@ -64,17 +64,17 @@ thus the measurement results do not necessarily reflect performance with real da
Also, due to the kind of content used, no compression is used in these benchmarks.
C- == borg create (1st archive creation, no compression, do not use files cache)
C-Z- == all-zero files. full dedup, this is primarily measuring reader/chunker/hasher.
C-R- == random files. no dedup, measuring throughput through all processing stages.
C-Z- == all-zero files. full deduplication; this primarily measures reader/chunker/hasher.
C-R- == random files. no deduplication, measuring throughput through all processing stages.
R- == borg extract (extract archive, dry-run, do everything, but do not write files to disk)
R-Z- == all zero files. Measuring heavily duplicated files.
R-Z- == all-zero files. Measuring heavily duplicated files.
R-R- == random files. No duplication here, measuring throughput through all processing
stages, except writing to disk.
U- == borg create (2nd archive creation of unchanged input files, measure files cache speed)
The throughput value is kind of virtual here, it does not actually read the file.
U-Z- == needs to check the 2 all-zero chunks' existence in the repo.
U-Z- == needs to check the two all-zero chunks' existence in the repo.
U-R- == needs to check existence of a lot of different chunks in the repo.
D- == borg delete archive (delete last remaining archive, measure deletion + compaction)
@ -82,4 +82,4 @@ D- == borg delete archive (delete last remaining archive, measure deletion + com
D-R- == many chunks to delete / many segments to compact/remove.
Please note that there might be quite some variance in these measurements.
Try multiple measurements and having a otherwise idle machine (and network, if you use it).
Try multiple measurements and have an otherwise idle machine (and network, if you use it).

View file

@ -43,5 +43,5 @@ Description
~~~~~~~~~~~
This command breaks the repository and cache locks.
Please use carefully and only while no borg process (on any machine) is
trying to access the Cache or the Repository.
Please use with care and only when no borg process (on any machine) is
trying to access the cache or the repository.

Some files were not shown because too many files have changed in this diff Show more