upstream/borgbackup
1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2026-03-26 04:14:45 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
4363 commits 9 branches 151 tags 72 MiB
Commit graph

748 commits

Author SHA1 Message Date
Marian Beermann
dfd748689c test_attic013_acl_bug: use def instead of lambda 2016-11-13 17:30:42 +01:00
Marian Beermann
c2eb2539b9
Document maintenance merge command 2016-11-13 16:07:06 +01:00
Marian Beermann
0d2b76fa7d Merge branch '1.0-maint' into merge/1.0-maint 2016-11-13 15:58:42 +01:00
Marian Beermann
00fae3f3e3
repository: check_free_space, special case small repos 2016-11-12 14:42:13 +01:00
enkore
80b86709ef Merge pull request #1833 from enkore/issue/1821 
borg init: fix free space check crashing if disk is full
 2016-11-12 01:23:34 +01:00
enkore
e26348a055 Merge pull request #1834 from enkore/issue/1813 
Clarify cache/repository README file
 2016-11-11 21:25:30 +01:00
Marian Beermann
bd4a0fe23b
Clarify cache/repository README file 2016-11-11 21:24:17 +01:00
Marian Beermann
ac5f9b61ec
debug, key: enable --help 
A small merge mishap in 2a864be8; 1.0-maint has it correctly.
 2016-11-11 19:54:11 +01:00
Marian Beermann
6c955579a8 repository: test_additional_free_space, assert it still exists 2016-11-11 16:39:11 +01:00
Marian Beermann
5b31a2cd61
borg init: fix free space check crashing if disk is full 2016-11-11 16:21:05 +01:00
enkore
223f7890fe Merge pull request #1819 from enkore/issue/1800 
Add BLAKE2b key types
 2016-11-11 03:11:38 +01:00
Marian Beermann
a31ca4f022
crypto: link against system libb2, if possible 2016-11-11 03:08:11 +01:00
Marian Beermann
76c93bb80b crypto blake2: don't release the GIL during key hashing 2016-11-11 03:07:05 +01:00
Marian Beermann
00ac7b14be Add BLAKE2b key types 2016-11-11 03:07:05 +01:00
TW
dc4325be25 Merge pull request #1826 from textshell/feature/repository-cleanup-rollback-api 
Repository: rollback's cleanup parameter is internal.
 2016-11-10 17:24:01 +01:00
Martin Hostettler
79c77e7a68 Repository: rollback's cleanup parameter is internal. 2016-11-10 13:44:58 +01:00
Marian Beermann
09f470bd85 Add crypto.blake2b_256 2016-11-10 00:06:50 +01:00
Marian Beermann
05a52ec8ec
Merge branch '1.0-maint' into merge/1.0-maint 2016-11-09 23:56:54 +01:00
Marian Beermann
676e69cac4 Parse & pass BORG_HOSTNAME_IS_UNIQUE env var to enable stale lock killing 2016-11-07 21:56:40 +01:00
Marian Beermann
cc14975f2d Add tests for stale lock killing and platform.process_alive 2016-11-07 21:56:07 +01:00
Marian Beermann
2bd8ac7762 platform: bump API version (and check consistency) 2016-11-07 21:54:16 +01:00
Marian Beermann
7930d055ec import platform module instead of functions 
(testability)
 2016-11-07 21:54:16 +01:00
Marian Beermann
8e1df7a364 Use logging instead of prints 2016-11-07 21:54:16 +01:00
Marian Beermann
c562f7750c Move platform-dependent code to platform package 2016-11-07 21:54:16 +01:00
Oleg Drokin
d490292be3 Detect and delete stale locks when it's safe 
If BORG_UNIQUE_HOSTNAME shell variable is set, stale locks
in both cache and repository are deleted.

Stale lock is defined as a lock that's originating from the same
hostname as us, and correspond to a pid that no longer exists.

This fixes #562
 2016-11-07 21:54:16 +01:00
Thomas Waldmann
8a15916284 Merge branch '1.0-maint' into merge-1.0-maint 
# Conflicts:
#	docs/changes.rst
#	docs/usage/help.rst.inc
#	src/borg/cache.py
#	src/borg/remote.py
#	src/borg/testsuite/__init__.py
#	src/borg/testsuite/archiver.py
 2016-10-31 18:24:13 +01:00
Thomas Waldmann
09e74af7bf Merge branch '1.0-maint' 2016-10-24 21:41:20 +02:00
TW
0ff2352691 Merge pull request #1748 from rciorba/shortcut_hashindex_set 
shortcut hashindex_set by having hashindex_lookup hint about address
 2016-10-23 16:48:43 +02:00
Radu Ciorba
766a43afc3 shortcut hashindex_set by having hashindex_lookup hint about address 2016-10-22 12:33:15 +03:00
TW
1a272ddd48 Merge pull request #1739 from ThomasWaldmann/borg-check-refactor 
borg check --first / --last / --sort / --prefix, fixes #1663
 2016-10-20 17:21:27 +02:00
Thomas Waldmann
4c884fd075 borg check --first / --last / --sort / --prefix, fixes #1663 2016-10-20 16:51:26 +02:00
Thomas Waldmann
e02d1a66b8 fix byte range error in test, fixes #1740 2016-10-20 03:04:19 +02:00
Thomas Waldmann
5a12c6377a fuse mount options: add tests 2016-10-15 01:24:05 +02:00
Thomas Waldmann
b5f9858055 move first/last/sort_by-multiple functionality into Manifest.list 
also: rename list_filtered to list_considering
 2016-10-15 01:04:56 +02:00
Thomas Waldmann
694c3978a1 refactor Archiver._get_filtered_archives -> Archives.list_filtered 
it did not belong into Archiver class (did not use "self"), but in into Archives.
 2016-10-15 01:03:16 +02:00
Thomas Waldmann
26e6ac4fea borg mount --first / --last / --sort / --prefix, fixes #1542 
also: use consider_part_files when dealing with multiple archives / whole repo mount
 2016-10-15 01:03:16 +02:00
Thomas Waldmann
775440fdcc Merge branch '1.0-maint' into merge-1.0-maint 2016-10-15 00:25:41 +02:00
enkore
34bd7cb4e2 Merge pull request #1716 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0-maint
 2016-10-14 21:18:30 +02:00
Thomas Waldmann
4e4847ccce point XDG_*_HOME to temp dirs for tests, fixes #1714 
otherwise it spoils the user's nonces and cache dirs with lots of files.

also: remove all BORG_* env vars from the outer environment
 2016-10-13 23:59:28 +02:00
Thomas Waldmann
b0cc4fd74a Merge branch '1.0-maint' into merge-1.0-maint 2016-10-13 19:10:04 +02:00
TW
a2dfe68609 Merge pull request #1706 from ThomasWaldmann/remove-debug-xxx 
remove debug-xxx commands, fixes #1627
 2016-10-12 23:25:43 +02:00
Thomas Waldmann
9ad9ae8ff0 Repository.scan(): avoid iterating same repo segments/objects repeatedly 
segments: avoid some listdir()
objects: avoid repeatedly re-reading object headers / seeking in segment

Fixes #1610.
 2016-10-10 05:39:38 +02:00
Thomas Waldmann
b88e82d99d remove debug-xxx commands, fixes #1627 
we use "debug xxx" subcommands now. docs updated.

also makes "borg help" shorter as not all debug-xxx commands
show up, but just 1 main "debug" command.
 2016-10-10 00:22:01 +02:00
enkore
8567fcfe58 Merge pull request #1418 from textshell/fix/preload 
Cleanup prefetch code in RemoteRepository
 2016-10-07 09:49:16 +02:00
Thomas Waldmann
f31112b975 clarify borg diff help, fixes #980 2016-10-07 04:09:05 +02:00
Martin Hostettler
02557f16b0 RemoteRepository: Rename cache to chunkid_to_msgids. Also fetch_from_cache to pop_preload_msgid. 2016-10-06 20:48:26 +02:00
Martin Hostettler
2608a5620a RemoteRepository: Always store chunk ids in cache instead of rpc argument encoding of get request. 2016-10-06 20:38:46 +02:00
Martin Hostettler
e9605d67ad RemoteReposity: prefetch can only be 'get'. 2016-10-06 20:38:46 +02:00
Thomas Waldmann
cdb8d64fe2 check for index vs. segment files object count mismatch 2016-10-05 17:36:36 +02:00
Thomas Waldmann
6624ca9cdb verify_data: do a linear scan in disk-order 2016-10-05 17:36:36 +02:00
Thomas Waldmann
90111363ba repo.list() yielding IDs in on-disk order 2016-10-05 17:36:36 +02:00
enkore
27bc73c23e borg info <repo>: print general repo information (#1680) 2016-10-03 14:53:16 +02:00
Thomas Waldmann
8df6cb8156 hashindex: bump api_version 
note:
merging the respective changeset from 1.0-maint was not effective
as we already had version 3, so there was no increase.
 2016-09-30 23:59:41 +02:00
Frank Sachsenheim
089995ef73 Changes on filters after feedback 2016-09-30 20:29:16 +02:00
Frank Sachsenheim
bd7cc38d6e Changes arg processor names to camelcase 2016-09-30 20:14:20 +02:00
Frank Sachsenheim
e0e9edfb42 Adds archives filters for list 2016-09-30 20:14:20 +02:00
Frank Sachsenheim
369d0a0881 Adds archives filters for info 2016-09-30 20:14:20 +02:00
Frank Sachsenheim
f2d4d36cea Allows delete to be used with archive filters 2016-09-30 20:14:20 +02:00
Frank Sachsenheim
17f2363935 Adds --prefix to the archives filters arguments 
- adds prefix argument to helpers.Archives.list
- also renames function PrefixSpec to prefix_spec
 2016-09-30 19:48:17 +02:00
Frank Sachsenheim
f6b9276de9 Adds arguments to filter archives 
These are: --sort-by, --first and --last

Includes a method to obtain a list of archive infos filtered by these

Archives.list:
- ensure reverse is always applied
- always return a list
 2016-09-30 19:48:17 +02:00
Thomas Waldmann
19eb75984e borg check --verify-data tuning 2016-09-29 18:40:02 +02:00
Thomas Waldmann
ba30098079 Merge branch '1.0-maint' into merge-1.0-maint 2016-09-29 12:57:29 +02:00
Thomas Waldmann
2a864be84f Merge branch '1.0-maint' into merge-1.0-maint 
# Conflicts:
#	setup.py
#	src/borg/archiver.py
#	src/borg/helpers.py
 2016-09-27 21:31:41 +02:00
enkore
ea9e859026 Merge pull request #1635 from enkore/issue/1624 
Fix broken --progress for double-cell paths
 2016-09-27 13:14:16 +02:00
Marian Beermann
9cef0a9ed8 Fix broken --progress ellipsis for double-cell paths 2016-09-27 11:35:45 +02:00
TW
233ae43551 Merge pull request #1640 from ThomasWaldmann/improve-borg-help 
Re-Indent borg help helptexts, again.
 2016-09-25 23:26:01 +02:00
TW
43432f9a14 Merge pull request #1615 from enkore/issue/1598 
Fix #1598
 2016-09-25 23:09:47 +02:00
Thomas Waldmann
bf681e98ce Re-Indent borg help helptexts, again. 2016-09-25 22:50:08 +02:00
TW
f36d551092 Merge pull request #1634 from ThomasWaldmann/improve-compression-docs 
borg help compression, fixes #1582
 2016-09-25 22:42:56 +02:00
Marian Beermann
8164524d99 Fix broken --progress for double-cell paths 2016-09-25 22:18:37 +02:00
Thomas Waldmann
28c57f98c9 Merge branch '1.0-maint' 
# Conflicts:
#	src/borg/archive.py
#	src/borg/archiver.py
#	src/borg/helpers.py
#	src/borg/testsuite/archiver.py
 2016-09-25 17:30:55 +02:00
Marian Beermann
66316e10b9 Fix indentation in borg help compression 2016-09-25 11:53:54 +02:00
Marian Beermann
7b1f10347a Repository: compact: fix incorrect preservation of delete tags 2016-09-25 10:13:31 +02:00
Marian Beermann
abace16945 Repository.check: log transaction IDs 2016-09-25 10:12:42 +02:00
Thomas Waldmann
d6bfdafdef borg help compression, fixes #1582 2016-09-25 02:32:02 +02:00
TW
d3e79c4e49 Merge pull request #1625 from bauerj/master 
Change {utcnow} and {now} to ISO-8601 format
 2016-09-23 18:24:10 +02:00
Johann Bauer
a56dc44e1f Change {utcnow} and {now} to ISO-8601 format 2016-09-23 17:58:04 +02:00
Martin Hostettler
84b3295a0d Archiver,RemoteRepository: Add --remote-ratelimit 
The --remote-ratelimit option adds a very simple rate limit for the
sending data to the remote.

Currently implemented by sleeping if the transmission speed is greater
than the limit.
 2016-09-18 19:00:26 +02:00
Thomas Waldmann
260ef31728 fuse: refactor for less encoding dance 2016-09-16 16:38:08 +02:00
Thomas Waldmann
39170479aa fuse: fix hardlinks in versions view, fixes #1599 2016-09-16 16:38:08 +02:00
Thomas Waldmann
c021cf466a fuse: add test for hardlinks in versions view 2016-09-16 16:38:08 +02:00
Thomas Waldmann
001500ab99 fuse: refactor file versioning code 2016-09-16 16:38:08 +02:00
Thomas Waldmann
5d22078f35 fuse: add parameter check to ItemCache.get to make potential failures more clear 2016-09-16 16:38:08 +02:00
Marian Beermann
ae5b4980f2 Repository.check: improve object count mismatch diagnostic 2016-09-15 11:23:58 +02:00
TW
783d01f621 Merge pull request #1586 from ThomasWaldmann/borg-check-delete-defect-chunks 
borg check: delete chunks with integrity errors, fixes #1575
 2016-09-14 22:55:50 +02:00
Thomas Waldmann
b4c7cce67d borg check: delete chunks with integrity errors, fixes #1575 
so they can be "repaired" immediately and maybe healed later.
 2016-09-14 21:07:48 +02:00
Thomas Waldmann
57a3adb6b3 borg recreate: also catch SIGHUP 2016-09-14 02:55:13 +02:00
Thomas Waldmann
1287d1ae92 Merge branch '1.0-maint' into merge-1.0-maint 
# Conflicts:
#	docs/development.rst
#	src/borg/archive.py
#	src/borg/archiver.py
#	src/borg/hashindex.pyx
#	src/borg/testsuite/hashindex.py
 2016-09-14 02:53:41 +02:00
Thomas Waldmann
dd89181062 fuse: refactor / optimize process_inner 2016-09-01 06:26:33 +02:00
Thomas Waldmann
6f50cc3413 fuse: simplify path segments loop 2016-09-01 05:18:10 +02:00
Thomas Waldmann
d64ccff1c4 fuse: remove unneeded function definition 
code only had 1 caller, so just inline it.
 2016-09-01 04:57:11 +02:00
Thomas Waldmann
34eb475463 fuse: remove unneeded safe_encode 
fsencode will happen in _find_inode()
 2016-09-01 04:33:42 +02:00
Thomas Waldmann
8b5372b54c fuse: slightly refactor shadowing detection 
if it is not a directory, the old code was a NOP, so we can just check that first.
 2016-09-01 04:30:55 +02:00
Thomas Waldmann
1f04820d9d fuse: implement versions view 
all archives, all items are read to build a unified view.
files are represented by a same-name directory with the versions of the file.
A filename suffix computed by adler32(chunkids) is used to disambiguate the versions.

also: refactor code a little, create methods for leaves, inner nodes.
 2016-08-27 23:17:57 +02:00
enkore
69c51f268d Merge pull request #1374 from textshell/issue/1039-aes-iv 
Fix AES-CTR nonce reusage
 2016-08-27 15:52:21 +02:00
Martin Hostettler
620f505a14 Reserve nonce space for AES-CTR before using it. 
Reusing the nonce totally breaks AES-CTR confidentiality.

This code uses a reservation of nonce space and stores the next nonce available for
a future reservation on the client and in the repository.

Local storage is needed to protect against evil repositories that try to gain access to
encrypted data by not saving nonce reservations and aborting the connection or otherwise
forcing a rollback.

Storage in the repository is needed to protect against another client writing to the
repository after a transaction was aborted and thus not seeing the last used nonce from
the manifest.

With a real counter mode cipher protection for the multiple client case with an actively
evil repository is not possible. But this still protects against cases where the attacker can
not arbitrarily change the repository but can read everything stored and abort connections
or crash the server.

Fixes #22
 2016-08-27 15:31:50 +02:00
Marian Beermann
e4e6787906 recreate: fix crash if recompress and deduplication beyond autocommit_threshold 
ie. it means that if all the recompressed chunks were already in the repo
no data would be written, so there would be no active txn, so failure
ensues.
 2016-08-27 14:31:57 +02:00
enkore
e60ed80433 Merge pull request #1539 from enkore/f/recreate-cp 
recreate: fix crash if archive doesn't have chunker_params
 2016-08-27 14:06:32 +02:00
Marian Beermann
e289fb3539 recreate: fix crash if archive doesn't have chunker_params 2016-08-27 14:01:28 +02:00
enkore
1d750e802e Merge pull request #1536 from enkore/f/prip2 
ProgressIndicatorPercent / extract minor cosmetic enhancement
 2016-08-27 00:36:43 +02:00
TW
168ce1dc86 Merge pull request #1535 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0 maint
 2016-08-27 00:32:40 +02:00
Marian Beermann
f0a32575a5 extract: --progress: no extra line for 'Calculating size' 2016-08-26 22:15:44 +02:00
Marian Beermann
517ccc2d58 ProgressIndicatorPercent: output(message) to override output 2016-08-26 22:15:29 +02:00
Thomas Waldmann
25fa443d2b repo tests: convert some more byte literals to H(x) 2016-08-26 20:59:22 +02:00
Thomas Waldmann
1974824595 Merge branch '1.0-maint' 2016-08-26 20:47:23 +02:00
Marian Beermann
cd39ccb821 fix overeager storing of hardlink masters 
n.b. we only need to store them for items that we wouldn't extract.

this also fixes an intersting edge case in extracting hard links
with --strip-components
 2016-08-25 21:16:20 +02:00
Marian Beermann
b5d7f1df26 extract: fix incorrect progress output for hard links 
this produces correct output if any (non proper) subset of hardlinks are
extracted.
 2016-08-25 01:12:30 +02:00
Martin Hostettler
b845a074cb tests: TestBuildFilter: Adjust from item_is_hardlink_master to peek_and_store_hardlink_masters. 2016-08-22 23:36:43 +02:00
Martin Hostettler
a026febdb0 Archiver.build_filter: strip_components is no longer a optional parameter. 2016-08-22 23:07:38 +02:00
Martin Hostettler
972392e290 extract: When doing a partial restore don't leak prefetched chunks. 
The filter function passed to iter_items (with preload=True) may never return True for items that
are not really extracted later because that would leak prefetched items.

For restoring hard linked files the item containing the actual chunks might not be matched
or implicitly removed from the restore by strip_components. For this reason the chunk list or all
items that can potentially be used as hardlink target needs to be stored.

To achive both requirements at the same time the filter function needs to store the needed information
for the hardlinks while not returning True just because it could be a hardlink target.

Known problems: When using progress indication the calculated extracted_size now can be smaller
than the actual extracted size in presence of hard links (master is not restored) instead of
bigger (potential master not used in restore).
 2016-08-22 22:58:58 +02:00
Thomas Waldmann
d1d2738381 re-enable fuse tests for RemoteArchiver 
at some time they had deadlock issues, but it worked for me now.
 2016-08-22 00:14:10 +02:00
Marian Beermann
53d0140bd5 Repository: add compact_segments progress 2016-08-21 21:55:53 +02:00
TW
2f65f7bdec Merge pull request #1449 from enkore/f/extractprogress 
extract --progress
 2016-08-21 20:24:44 +02:00
Marian Beermann
ab31ffaa85 ProgressIndicatorPercent: remove same_line 2016-08-21 19:57:36 +02:00
Marian Beermann
e7d44cec39 extract: --progress: Calculating size 2016-08-21 19:57:36 +02:00
TW
ab96c6496c Merge pull request #1513 from ThomasWaldmann/improve-error-logging 
improve error logging, fixes #1440
 2016-08-21 18:29:18 +02:00
Marian Beermann
4d214e2503 Simplify and test Item.file_size 2016-08-21 01:09:21 +02:00
Marian Beermann
5924915d35 Flip ProgressIndicatorPercent same_line default to True 
Every production use of this uses same_line=True
 2016-08-21 01:09:20 +02:00
Marian Beermann
8709cec57c borg-extract --progress 2016-08-21 01:07:55 +02:00
enkore
81dd381701 Merge pull request #1507 from ThomasWaldmann/use-modified-pyinstaller 
use patched LDLP-preserving pyinstaller
 2016-08-21 00:16:24 +02:00
enkore
9baa024f4a Merge pull request #1508 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0 maint
 2016-08-21 00:12:00 +02:00
Thomas Waldmann
41e4707d6a fixup: meant "tb", not "traceback" 2016-08-20 23:09:42 +02:00
TW
357f4ef72e Merge pull request #1511 from c4rlo/lru-cache 
helpers.py: replace memoize with functools.lru_cache
 2016-08-20 21:19:17 +02:00
Thomas Waldmann
a0c8c40f27 improve error logging, fixes #1440 
archiver: split traceback and msg, have separate log level for traceback, log LockTimeout at debug level,
for "Error" exceptions: always log the traceback, either at ERROR or DEBUG level.

remote: if we have an "Error" typed exception instance, we can use its traceback flag and .get_message()
as we do locally.
 2016-08-20 17:23:02 +02:00
Carlo Teubner
61af307ad4 helpers.py: replace memoize usages with lru_cache 2016-08-20 13:06:16 +01:00
Thomas Waldmann
e7fccaccb2 restore original LDLP, if possible, fixes #1498 
see https://github.com/pyinstaller/pyinstaller/pull/2148
 2016-08-20 12:31:23 +02:00
Carlo Teubner
28076ee588 helpers.py: replace memoize with functools.lru_cache 2016-08-20 10:01:59 +01:00
Carlo Teubner
13761c4ec2 daemonize(): use os.devnull instead of hardcoded /dev/null 2016-08-20 10:00:03 +01:00
Thomas Waldmann
c355f3617a Merge branch '1.0-maint' 2016-08-20 00:04:55 +02:00
Thomas Waldmann
53d14e96de borg list: test for --list-format deprecation 2016-08-19 14:55:49 +02:00
TW
32e84f33e9 Merge pull request #1484 from enkore/issue/1396 
Track shadowing of log entries
 2016-08-19 14:09:34 +02:00
Marian Beermann
c39e395ecf Exclude incompatible tests for ArchiverTestCaseBinary 2016-08-17 23:01:20 +02:00
Marian Beermann
26e8ff2cbc Repository: don't use defaultdict for shadow index 
avoids errors by accidentally inserting an empty list and makes it more clear.
 2016-08-17 01:17:49 +02:00
Marian Beermann
833f8d1373 Track entire session 
Note how this enables the much stricter check in test_moved_deletes_are_tracked
 2016-08-17 01:17:49 +02:00
Marian Beermann
ac41ebcf78 Track shadowing of log entries 
Fixes (benign) index vs. log inconsistencies when segments are skipped
during compaction.
 2016-08-17 01:17:49 +02:00
Marian Beermann
21c3fb3b93 compact_segments: add segment-level logging 2016-08-17 01:16:25 +02:00
TW
f7c1c71ff1 Merge pull request #1489 from enkore/issue/1486 
Fix borg-list --list-format
 2016-08-16 21:35:25 +02:00
Marian Beermann
f3defb02de Fix borg-list empty format 
Should produce empty output, not default output.
 2016-08-16 20:46:58 +02:00
Marian Beermann
146d531d9e Fix borg-list --list-format 2016-08-16 20:46:54 +02:00
Thomas Waldmann
30cd7f3f21 fix timestamp key name 2016-08-16 20:36:29 +02:00
Thomas Waldmann
2f70e3f69a src cleanup: do not use XXX in strings if just any other string works also 
XXX is usually used in comments to mark questionable places in the source.
 2016-08-15 16:01:23 +02:00
Thomas Waldmann
3c8d86354b src cleanup: get rid of text-as-bytes in borg.remote 2016-08-15 15:52:19 +02:00
Thomas Waldmann
1f056d9e8a more safe interface for manifest.archives 2016-08-15 15:03:05 +02:00
Thomas Waldmann
b6d0eb99a5 add and use ManifestItem 2016-08-15 02:01:13 +02:00
Thomas Waldmann
c8922c8b3d use ArchiveItem 2016-08-15 01:11:33 +02:00
Thomas Waldmann
585407f4f4 introduce ArchiveItem, see #1157 2016-08-14 22:04:01 +02:00
Marian Beermann
f0e9a55ebf recreate: document that absolute patterns won't match 2016-08-14 17:46:59 +02:00
Marian Beermann
93b1cf3453 recreate: --target 2016-08-14 17:46:59 +02:00
Marian Beermann
88798ae949 recreate: --always-recompress, --compression-from 
what a mess
 2016-08-14 17:46:59 +02:00
Marian Beermann
5433b1a1e4 Add Compressor.detect(data) -> CompressBase impl 2016-08-14 17:41:34 +02:00
Marian Beermann
c2c90645ad Add Key.assert_id function 2016-08-14 17:41:33 +02:00
Marian Beermann
a80b371d09 Add decompress arg to Key.decrypt 2016-08-14 17:41:33 +02:00
Marian Beermann
ac80332ce7 Merge branch '1.0-maint' into master 
# Conflicts:
#	src/borg/repository.py
#	src/borg/testsuite/repository.py
 2016-08-14 15:51:49 +02:00
Thomas Waldmann
42b6a838da fix cyclic import issue, fix tests 
needed to increase ChunkBuffer size due to increased items stream chunk size
to get the test working.
 2016-08-14 15:26:56 +02:00
Thomas Waldmann
8be6761c26 Merge commit 'feb7e2517ef7ec07cc638953a86c726aada7d37e' 2016-08-14 15:05:11 +02:00
enkore
f8b2ffe999 Merge pull request #1458 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0 maint
 2016-08-11 14:18:03 +02:00
enkore
006d1df517 Merge pull request #1450 from enkore/f/recreatefactors 
minor recreate refactor
 2016-08-09 23:28:58 +02:00
Marian Beermann
7d9fba5a41 ArchiveRecreater.copy_items: progress from target instead of source 
(it produces the same output, just less confusing)
 2016-08-09 21:20:13 +02:00
TW
ff5d9a3f12 Merge pull request #1447 from enkore/f/debug-topic 
Debugging topics and some other minor stuff
 2016-08-09 20:53:07 +02:00
Thomas Waldmann
c955d59273 Merge branch '1.0-maint' 2016-08-09 20:09:13 +02:00
Marian Beermann
0a65e83df3 check --verify-data: remove incorrect help paragraph 
(since d0ec7e76bb - doesn't mean this probably useful feature may not
 be reimplemented, on the contrary)
 2016-08-08 16:38:31 +02:00
Marian Beermann
bec5051ea5 helpers.format_timedelta: use timedelta.total_seconds() 2016-08-08 16:38:31 +02:00
Marian Beermann
389ca94407 Silence repeated "Processing files..." log entries when --no-files-cache 2016-08-08 16:38:31 +02:00
Marian Beermann
4d7a52d8e0 --debug-topic for granular debug logging 
Especially: disables file-compression logging by default
 2016-08-08 16:38:31 +02:00
Marian Beermann
7b70b74ad1 ArchiveRecreater.incref_partial_chunks try to use __contains__ 2016-08-08 15:54:47 +02:00
Marian Beermann
5fe6c09c34 Refactor ArchiveRecreater.try_resume 2016-08-08 15:54:41 +02:00
Thomas Waldmann
dbe862f3d8 Merge branch '1.0-maint' into merge-1.0-maint 2016-08-06 01:24:22 +02:00
TW
9b0d8df0f8 Merge pull request #1373 from PlasmaPower/filesystem-test-skipping 
Filesystem feature detection and test skipping
 2016-08-03 17:46:34 +02:00
TW
29e5e558bc Merge pull request #1412 from ThomasWaldmann/info-repo 
borg info REPO
 2016-08-02 23:29:39 +02:00
Thomas Waldmann
770a892b2d implement borg info REPO 
currently it is just the same global stats also shown in "borg info ARCHIVE",
just without the archive-specific stats.

also: add separate test for "borg info".
 2016-08-02 20:06:24 +02:00
enkore
b7a563885d Merge pull request #1413 from ThomasWaldmann/store-mountpoint-dirs 
save mountpoint dirs, fixes #1033
 2016-08-01 15:51:53 +02:00
Thomas Waldmann
2b454fc54b save mountpoint dirs, fixes #1033 2016-08-01 15:25:54 +02:00
Lee Bousfield
b86b5d952a
Filesystem feature detection and test skipping 2016-07-31 01:19:31 -04:00
Thomas Waldmann
9865509105 docs: remove borg list example, be more specific about compression heuristics 2016-07-30 21:21:45 +02:00
TW
02e229b255 Merge pull request #1377 from enkore/issue/1060 
Use atomic file writing and updating for configuration and key files
 2016-07-30 16:06:01 +02:00
Marian Beermann
e9a73b808f Check for sufficient free space before committing 2016-07-30 00:04:27 +02:00
Marian Beermann
389503db60 Repository.compact_segments: always save_space 2016-07-30 00:03:31 +02:00
Thomas Waldmann
999f0ae187 fix chunk refcounts considering part files 2016-07-28 19:39:29 +02:00
Thomas Waldmann
04ad1d1b0b use "part file", "part", etc. consistently 
use .borg_part_N as filename to avoid collisions
 2016-07-28 19:39:29 +02:00
Thomas Waldmann
a7fca52f07 create part items rather than checkpoint items 
checkpoint items: chunks abc, abcdef, abcdefghi, ....
part items: chunks abc, def, ghi

solves the "restore a big file over unstable connection issue" (not very comfortably, but
one can just extract one part after the other and concatenate them to get the full file)
 2016-07-28 19:39:29 +02:00
Thomas Waldmann
e5bd6cef20 add --consider-checkpoint-files option, update FAQ 2016-07-28 19:39:29 +02:00
Thomas Waldmann
49233be25d filter out checkpoint files 
do not:
- list them
- extract them
- diff them
- include them for recreate
 2016-07-28 19:39:29 +02:00
Thomas Waldmann
dd5f957e6f in-file checkpoints, fixes #1198, fixes #1093 
also: unify code for stdin and on-disk file processing
 2016-07-28 19:39:29 +02:00
Thomas Waldmann
75d91c4bd1 file chunking: refactor code a little 
so it directly works with item.chunks list instead of a temporary list.
 2016-07-28 19:39:28 +02:00
Thomas Waldmann
bda50b5958 stdin chunking: get Item into usable state early 2016-07-28 19:39:28 +02:00
Thomas Waldmann
9226fc6f6f split stat_attrs into cheap and expensive part 
we already have stat results in st, so computing stat_simple_attrs is
rather cheap (except the username/groupname lookup maybe) and gets the
most important stuff right in the Item, so it is brought early into a
good state.

after chunking, stat_ext_attrs is called to add the more expensive-to-get
attributes, like bsdflags, xattrs and ACLs.
 2016-07-28 19:39:28 +02:00
Marian Beermann
0ae48dafbb
ObjectNotFound: give ID as hex-string 2016-07-28 18:41:50 +02:00
Marian Beermann
d0ec7e76bb Fix borg-check --verify-data failing with rebuilt objects 
There are some instances where --repair would do something. In these
instances --verify-data would fail if --repair was not given also, since
the changes without --repair are only in the index, but not in the
repository.
 2016-07-28 18:41:08 +02:00
Marian Beermann
1b6b0cfae6 Fix borg-check --verify-data tripping over ObjectNotFounds 2016-07-28 18:40:20 +02:00
Lee Bousfield
b652f4039c
Remove trailing whitespace 2016-07-28 10:35:41 -04:00
Marian Beermann
1bad3fc552
Merge branch '1.0-maint' into master 2016-07-28 09:48:58 +02:00
Marian Beermann
ac8c8119a7
Merge branch '1.0-maint' into master 2016-07-28 00:29:43 +02:00
Marian Beermann
2e3fc9ddfc SyncFile/SaveFile: default binary=False, just like open() 2016-07-26 22:49:25 +02:00
Marian Beermann
863ab66908 SaveFile: unlink(tmppath): only ignore FileNotFoundError 2016-07-26 22:40:23 +02:00
Marian Beermann
dec671d8ff SaveFile: os.replace instead of rename 2016-07-26 22:39:45 +02:00
Lee Bousfield
ff108ef24b Catch a ValueError when checking if a user exists 2016-07-23 15:03:22 -04:00
Lee Bousfield
868586dd37 Fixed noatime detection 2016-07-23 13:05:35 -04:00
Lee Bousfield
88bffab935
Allow running tests outside of Vagrant 
Fixes #1361
 2016-07-23 10:31:16 -04:00
Marian Beermann
52a72cf73a Fix some item subscripts that came from 1.0-maint 2016-07-17 19:13:09 +02:00
TW
963bdd340c Merge pull request #1330 from enkore/issue/972 
recreate: fix hogged signal handlers
 2016-07-14 14:53:21 +02:00
Marian Beermann
fcb6930c53
recreate: fix hogged signal handlers 2016-07-14 02:11:13 +02:00
Marian Beermann
2e6eb63174
Merge branch '1.0-maint' into master 2016-07-14 01:28:26 +02:00
Thomas Waldmann
3baa8a3728 Merge branch '1.0-maint' 
# Conflicts:
#	docs/changes.rst
#	docs/usage/mount.rst.inc
#	src/borg/archive.py
#	src/borg/archiver.py
#	src/borg/fuse.py
#	src/borg/testsuite/archiver.py
 2016-07-11 01:23:27 +02:00
Marian Beermann
f4be2b3523 Use platform.SaveFile for repository, cache and key files 
Fixes #1060
 2016-07-09 21:10:46 +02:00
Marian Beermann
36ebc82748 Add platform.SaveFile 2016-07-09 21:10:26 +02:00
TW
4f7a211b02 Merge pull request #1290 from enkore/issue/1287-bp 
sync_dir: silence fsync() failing with EINVAL
 2016-07-08 18:31:26 +02:00
Marian Beermann
567617ebbe
sync_dir: silence fsync() failing with EINVAL 2016-07-08 18:05:46 +02:00
TW
65475ea1aa Merge pull request #1286 from enkore/issue/1269 
Remove InternalOSError
 2016-07-08 14:00:07 +02:00
Marian Beermann
6c3ca63456
Remove InternalOSError 
obsoleted by 5b453856ec and e96905c6b
 2016-07-08 13:35:14 +02:00
Marian Beermann
11e02461d2
1.0-maint merge fixup (hexlify..decode -> bin_to_hex) 2016-07-08 13:29:51 +02:00
Marian Beermann
aade62c9f9
wrap get_flags with backup_io 2016-07-08 13:26:10 +02:00
Thomas Waldmann
fe627246e4 Merge branch '1.0-maint' 2016-07-08 12:13:52 +02:00
Thomas Waldmann
12f55f4e9f Merge branch '1.0-maint' 2016-07-06 01:33:53 +02:00
Thomas Waldmann
f363ddd7ca Merge branch '1.0-maint' 2016-07-04 20:11:21 +02:00
enkore
69f73b1752 Merge pull request #1227 from ThomasWaldmann/checkpoint-interval 
increase checkpoint-interval default from 300 to 1800s, fixes #896
 2016-06-30 17:09:56 +02:00
Thomas Waldmann
87d6755108 Merge branch '1.0-maint' 2016-06-29 18:28:33 +02:00
Thomas Waldmann
c201654f45 increase checkpoint-interval default from 300 to 1800s, fixes #896 
doing a checkpoint can be expensive, if the repo connection is slow and/or
accessed via a network filesystem and/or the repo is large.

thus: don't do it too often.
 2016-06-27 20:05:20 +02:00
Thomas Waldmann
5791b89c04 use bytes type for ACL attributes, fixes #1205 
this saves some encoding / decoding dance compared to having a str interface.
 2016-06-26 23:04:04 +02:00
Thomas Waldmann
323894d195 Merge branch '1.0-maint' 2016-06-26 20:46:12 +02:00
Thomas Waldmann
ed128e517b Merge branch '1.0-maint' 2016-06-25 17:18:14 +02:00
TW
3ff6ac09a6 Merge pull request #1179 from Abogical/list-format 
Apply --format when listing repository archives, fixes #1166
 2016-06-24 18:29:03 +02:00
Abogical
6658112f8c Add documentation for listing repository archives 2016-06-21 20:01:01 +03:00
Abogical
87173d25de Added testsuite for listing repository archives 2016-06-20 19:23:22 +03:00
Abogical
c839eb63f7 Apply --format when listing repository archives 2016-06-20 19:23:15 +03:00
Abogical
0942bcd1b4 Move code block for the write() function at do_list() 2016-06-20 16:14:52 +03:00
James Clarke
d4f5172771 Correctly handle multiple LD_PRELOAD entries; fixes #1111 2016-06-19 22:04:03 +01:00
TW
d9a14e3658 Merge pull request #1184 from c4rlo/linux-syncfile-errno-fix 
platform.linux: small SyncFile error handling fix
 2016-06-18 14:49:18 +02:00
Carlo Teubner
58d236c619 platform.linux: small SyncFile error handling fix 2016-06-18 13:39:22 +01:00
Carlo Teubner
ff470f49f3 typo fixes in comment+docstring 2016-06-18 12:57:54 +01:00
Thomas Waldmann
9a64835b4d Merge branch '1.0-maint' 
Also: add missing keys to ARCHIVE_KEYS set.
 2016-06-13 00:14:08 +02:00
TW
8c26945097 Merge pull request #1158 from ThomasWaldmann/pretty-key 
add Key/EncryptedKey PropDict, see #1157
 2016-06-12 14:51:46 +02:00
Thomas Waldmann
d2468d37df add Key/EncryptedKey PropDict 2016-06-12 04:28:40 +02:00
Michael Herold
d45a13babf Changes archive format from uuid to uuid4 
This allows for future support of other uuid versions
 2016-06-11 22:57:03 +02:00
Michael Herold
5aa97e283c Adds uuid to archive format tags 2016-06-11 12:03:55 +02:00
jenkins
b86775ab4f
Fix test_sparse_file for solaris 2016-06-10 12:21:03 +02:00
Thomas Waldmann
d813775065 Merge branch '1.0-maint' 2016-06-09 20:29:44 +02:00
Marian Beermann
e41f3ecc5b
SyncFile: open with O_EXCL (or equivalent) 2016-06-04 22:47:27 +02:00
Thomas Waldmann
75244ea6d1 fuse: simplify timestamp processing 2016-06-04 17:34:03 +02:00
Thomas Waldmann
60da32123a refactor to use Item class, fixes #1071 2016-06-04 17:24:55 +02:00
Marian Beermann
0ca1a18b33
Make borg.platform a package 2016-05-31 15:38:56 +02:00
Marian Beermann
0333f91eb2 platform: rearrange imports 2016-05-31 15:38:16 +02:00
Marian Beermann
2d90bb2c55 Compose platform_base from the platform-specific functionality 
Noticed while working on the windows branch.
 2016-05-31 15:38:16 +02:00
Thomas Waldmann
d3deec9fc9 cleanup / reorder imports 
import stdlib
from stdlib import x

import 3rd party

import borg stuff
 2016-05-31 01:18:03 +02:00
Thomas Waldmann
49484d6d67 split helpers import into multiple ones 2016-05-31 00:33:13 +02:00
Thomas Waldmann
7e0ea1635a Merge branch 'master' into move-to-src 2016-05-30 19:41:30 +02:00
Thomas Waldmann
045e5a1203 Merge branch 'master' into move-to-src 2016-05-30 19:38:16 +02:00
Thomas Waldmann
3ce35f6843 Merge branch 'master' into move-to-src 2016-05-21 19:06:01 +02:00
Thomas Waldmann
d1ea925a5b move borg package to src/ 2016-05-05 20:19:50 +02:00