Commit graph

1360 commits

Author SHA1 Message Date
Björn Ketelaars
e2ee7fd24b Support msgpack-0.6.2.
This addresses
https://github.com/borgbackup/borg/issues/4360#issuecomment-541107988
2019-10-15 05:22:20 +02:00
Julien Lamy
cfa76e066b Add missing attribute to FUSE entries 2019-09-22 17:12:01 +02:00
TW
53f8882d31
Merge pull request #4751 from ThomasWaldmann/noatime-default
create: make --noatime the default, deprecate --noatime, fixes #4673
2019-09-06 23:08:38 +02:00
Thomas Waldmann
3f63234974 create: make --noatime the default, deprecate --noatime, fixes #4673
also: add --atime option to enable storing files' atime.
2019-09-06 21:48:54 +02:00
TW
aa7df50a2d
Merge pull request #4635 from ThomasWaldmann/ctrlc-checkpoint
first ctrl-c: checkpoint and abort, fixes #4606
2019-09-06 21:44:07 +02:00
TW
975f5627b1
Merge pull request #4736 from saurvs/forward-port-4729
new BORG_WORKAROUNDS mechanism, basesyncfile, fixes #4710
2019-08-27 21:55:58 +02:00
Thomas Waldmann
b43d3bb7fa new BORG_WORKAROUNDS mechanism, basesyncfile, fixes #4710
remove WSL autodetection. if WSL still has this problem, you need to
set BORG_WORKAROUNDS=basesyncfile in the borg process environment to
work around it.

(cherry picked from commit beb948fc71)
2019-08-27 19:37:40 +05:30
Thomas Waldmann
cb2d31ed98 fix partial extract for hardlinked contentless file types, fixes #4725
if the file is not a regular file, but a hardlink slave with a not
extracted hardlink master, chunks will be None and we must not call
preload(chunks).

(cherry picked from commit 291d58efa1)
2019-08-27 19:20:20 +05:30
Thomas Waldmann
9732fe4965 special behaviour on first ctrl-c, fixes #4606
like:
 - try saving a checkpoint if borg create is ctrl-c-ed
2019-08-25 22:49:09 +02:00
TW
373bd8abd3
Merge pull request #4696 from jrast/win10
WIP jrast/borg:win10, PR for better review and testing
2019-08-25 22:41:05 +02:00
Jürg Rast
bff97a99e1 Windows specific directory handling
On windows os.open does not work for directories.
If borg tries to open an directory on windows, None is returned
as file descriptor. The archive and archiver where adjusted to
handle the case if a file descriptor is None.
2019-08-24 10:17:18 +02:00
Jürg Rast
6b426d08d7 Initial work to build and run borg under windows
- Created a batch file to build borg on windows
- Adjusted setup.py to be runnable on windows and build the windows
extension
- Extracted the free space check to a function in the platform module
- Created the minimal needed (dummy) functions for the windows platform
module
2019-08-24 10:17:18 +02:00
ntova
627ca1376b Fix various code blocks in the docs (#4708)
Fix various code blocks in the docs

- rst markup: put codeblock markup on separate line to make it better visible / separate it from normal text colons.

- borg help texts in archiver.py: put codeblock markup instead of colon - that way it looks like a single colon when using the cli help and also works as code block markup.
2019-08-09 23:13:11 +02:00
TW
97eca3d287
Merge pull request #4713 from saurvs/forward-port-4698
Forward port #4698
2019-08-09 16:02:29 +02:00
TW
8a401d3c0b
Merge pull request #4712 from saurvs/forward-port-4646
Forward port #4646
2019-08-09 15:58:11 +02:00
Thomas Waldmann
8b49c4d2df Repository.check_can_create_repository: use stat() to check
similar issue as #4695.

(cherry picked from commit 4911720faf)
2019-08-09 15:10:15 +05:30
Thomas Waldmann
bb7a9e6c20 Repository.open: use stat() to check for repo dir, fixes #4695
(cherry picked from commit ec3fad0f85)
2019-08-09 15:09:48 +05:30
Thomas Waldmann
9c3a572638 SecurityManager.known(): check all files, fixes #4614
Before this fix, because known() only checked the key-type file, the
location file could go missing without being automatically regenerated.

Now a save() is triggered if not all files are present.

(cherry picked from commit c68cad9727)
2019-08-09 15:05:30 +05:30
Thomas Waldmann
d4d4b7e4a9 after double-force delete, warn about necessary repair, fixes #4704
the borg check --repair is needed to clean up all the orphaned chunks.

if the message is emitted on INFO log level, people likely do not see
it if they use default WARNING log level (they did not give -v).

(cherry picked from commit 389afcae9d)
2019-08-09 14:59:39 +05:30
TW
76655b18d0
Merge pull request #4669 from ThomasWaldmann/forward-ports-master
forward ports to master
2019-07-07 14:39:31 +02:00
Thomas Waldmann
097677b14f fix preloading for old remote servers, fixes #4652
if the remote server does not support the dict-based argument passing
yet (e.g. when accidentally using borg 0.29 on rsync.net), the GET rpc
call argument list failed to generate if the command currently being
processed was not a GET also.

note: rsync.net has a more recent borg as "borg1".
2019-07-02 23:13:30 +02:00
Thomas Waldmann
c0a617c6a9 cope with ANY error when importing pytest into borg.testsuite, #4652
this happened because the user had pytest5 installed somehow,
although it is incompatible with python 3.4:

# python3 /usr/local/bin/borg --version
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/borg/archiver.py", line 81, in <module>
    from .selftest import selftest
  File "/usr/local/lib/python3.4/dist-packages/borg/selftest.py", line 21, in <module>
    from .testsuite.hashindex import HashIndexDataTestCase, HashIndexRefcountingTestCase, HashIndexTestCase
  File "/usr/local/lib/python3.4/dist-packages/borg/testsuite/__init__.py", line 29, in <module>
    from pytest import raises
  File "/usr/local/lib/python3.4/dist-packages/pytest.py", line 6, in <module>
    from _pytest.assertion import register_assert_rewrite
  File "/usr/local/lib/python3.4/dist-packages/_pytest/assertion/__init__.py", line 6, in <module>
    from _pytest.assertion import rewrite
  File "/usr/local/lib/python3.4/dist-packages/_pytest/assertion/rewrite.py", line 443, in <module>
    ast.MatMult: "@",
AttributeError: 'module' object has no attribute 'MatMult'
2019-07-02 23:12:43 +02:00
Ellis Michael
b6a7a0806e
Fix invalid archive error message 2019-06-30 18:04:54 -07:00
Thiago Macieira
575a348ce4 Fix broken test that relied on improper zlib assumptions
This test assumed that zlib would always produce the same output for a
given input, and that assumption has proven incorrect. It rested on
zlib never improving. The zlib shipped in Clear Linux has algorithm
improvements, which caused this test to break.

This patch instead decompresses the expected value and compares that.

Signed-off-by: Thiago Macieira <thiago.macieira@intel.com>
2019-06-25 16:00:11 -07:00
Elmar Hoffmann
443958a79b support xxh64 checksum in addition to the hashlib hashes in borg list
also: point out it is not a cryptographic hash
2019-06-24 08:46:58 +02:00
Elmar Hoffmann
a26065064d enable placeholder usage in all extra archive arguments 2019-06-23 02:28:26 +02:00
Thomas Waldmann
71c7efd17c extract: fix KeyError for "partial" extraction, fixes #4607
note that "partial" even applied to giving an always matching condition.

"full" is only assumed if no conditions are given.
2019-06-10 20:18:44 +02:00
TW
8a6463d7bc
Merge pull request #4605 from ThomasWaldmann/use-borg-repo
Use BORG_REPO
2019-06-04 23:37:27 +02:00
Felix Schwarz
2ff06c58f0 ability to use a system-provided version of "xxhash"
The build process can be controlled via environment variables
similar to other bundled libraries in borgbackup. The main difference
is probably that upstream does not provide a pkgconfig file for
xxhash.

Therefore borg will probably fail to detect the system-provided
version by default (tested on Fedora, seems like Debian and Ubuntu
do not ship a pkgconfig file either). I kept the pkgconfig lookup
code anyway to keep the code as similar as possible to
"setup_compress.py"/"setup_crypto.py".

Setting BORG_LIBXXHASH_PREFIX=/usr helps borg to detect xxhash
on my system (Fedora). You can force the use of the bundled
version of xxhash by setting BORG_USE_BUNDLED_XXHASH=1.
2019-06-04 21:59:26 +02:00
Felix Schwarz
ce146c8b55 reference struct "XXH64_state_t" only via an opaque pointer
Upstream recommends this whenever xxhash is linked dynamically:
https://github.com/Cyan4973/xxHash/issues/63#issuecomment-218376237

This change is required to unbundle xxhash in the next commit.
2019-06-04 21:59:26 +02:00
Thomas Waldmann
42de5b9a79 location arg - should it be optional? fixes #4541.
compact / config: yes, should be optional (can get repo from BORG_REPO
environment variable)

some debug commands: rather not, have non-optional addtl. posargs
2019-06-04 19:36:20 +02:00
Thomas Waldmann
c0804ff5a2 argparser: always use REPOSITORY in metavar
also: use more consistent help phrasing.
2019-06-04 19:36:10 +02:00
Martin Stone
883906939f
Add example of exclude-norecurse rule in help patterns 2019-06-02 18:46:03 +01:00
Thomas Waldmann
11bb4eb8fc filter out selinux xattrs, fixes #4574 2019-05-21 00:00:07 +02:00
TW
57c0103c32
Merge pull request #4570 from ThomasWaldmann/rename-fuse-based-test-master
rename test_mount_hardlinks to test_fuse_mount_hardlinks (master)
2019-05-16 05:30:06 +02:00
TW
5625b36c42
Merge pull request #4569 from ThomasWaldmann/filter-msgpack-warning-master
pure-py msgpack warning shall not make a lot of tests fail, fixes #4558
2019-05-16 02:10:32 +02:00
Thomas Waldmann
5e75c9ed86 rename test_mount_hardlinks to test_fuse_mount_hardlinks
so both can be excluded by "not test_fuse"

(fedora does this for testing)
2019-05-16 00:57:12 +02:00
Thomas Waldmann
cb4e602b34 pure-py msgpack warning shall not make a lot of tests fail, fixes #4558
if the tests use a pure-python msgpack, 1 test which is specifically
made for that, will fail. esp. for linux distribution packages, this
will still point to the problem (that the msgpack package is not built
or installed correctly).
2019-05-16 00:51:07 +02:00
Thomas Waldmann
eb16233a83 enable placeholder usage in --comment, fixes #4559 2019-05-16 00:40:31 +02:00
TW
320b935c97
Merge pull request #4528 from shawnl/back-port
2 forward ports
2019-05-12 19:09:44 +02:00
TW
2bc0b1e973
Merge pull request #4531 from ThomasWaldmann/whitelist-client-options
use whitelist approach for borg serve, fixes #4097
2019-05-11 23:42:47 +02:00
TW
839e92fc4b
Merge pull request #4539 from ThomasWaldmann/fix-extract-hardlinks
slave hardlinks extraction issue, see #4350 (master)
2019-05-11 22:14:40 +02:00
Thomas Waldmann
cc41e36405 enable placeholder usage in --glob-archives, fixes #4495
they are supported for --prefix already, so it totally makes sense to
also support them for the more powerful/generic --glob-archives.
2019-05-09 20:28:41 +02:00
Thomas Waldmann
3082b1f1eb compact: move --cleanup-commits description from examples to docs 2019-05-06 23:10:07 +02:00
Thomas Waldmann
4b1cbc8319 compact: improve docs
- say when it is useful
- point to --verbose and --progress
2019-05-06 23:00:45 +02:00
Thomas Waldmann
8b75dde0fa compact: log freed space at INFO level
note: correctness of value depends on correctness/completeness of
repository.compact datastructure.
2019-05-06 22:47:25 +02:00
Thomas Waldmann
f33f318d81 preload chunks for hardlink slaves w/o preloaded master, fixes #4350
also split the hardlink extraction test into 2 tests.
2019-05-06 02:06:58 +02:00
Thomas Waldmann
633bd1152e use whitelist approach for borg serve, fixes #4097 2019-04-30 20:35:18 +02:00
Thomas Waldmann
d9a1761d0e do not check python/libc for borg serve, fixes #4483
This is a forward port of 24118459c6

the check checks whether follow_symlinks=False is supported, which
requires that the glibc is recent enough / python was compiled
for a recent enough glibc. follow_symlinks=False is only used for borg
create and extract, but not needed for borg serve.

thus, this makes it possible to run "borg serve" even on a bit older
servers. be careful, due to the presence of this check on the server
side until now, older server systems are not really much tested.
2019-04-27 09:22:26 -05:00
Thomas Waldmann
61b9283567 fix "all archives" stats considering part files, fixes #4329
archives with borg 1.2 metadata: quick, we have precomputed size data
in archive metadata and just need to sum up for all archives.

archives without borg 1.2 metadata: expensive, need to read all
archives' items metadata streams and sum up for all items.
2019-04-20 22:22:26 +02:00