upstream/borgbackup
1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2026-03-26 12:26:08 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
4363 commits 9 branches 151 tags 72 MiB
Commit graph

748 commits

Author SHA1 Message Date
TW
168ce1dc86 Merge pull request #1535 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0 maint
 2016-08-27 00:32:40 +02:00
Marian Beermann
f0a32575a5 extract: --progress: no extra line for 'Calculating size' 2016-08-26 22:15:44 +02:00
Marian Beermann
517ccc2d58 ProgressIndicatorPercent: output(message) to override output 2016-08-26 22:15:29 +02:00
Thomas Waldmann
25fa443d2b repo tests: convert some more byte literals to H(x) 2016-08-26 20:59:22 +02:00
Thomas Waldmann
1974824595 Merge branch '1.0-maint' 2016-08-26 20:47:23 +02:00
Marian Beermann
cd39ccb821 fix overeager storing of hardlink masters 
n.b. we only need to store them for items that we wouldn't extract.

this also fixes an intersting edge case in extracting hard links
with --strip-components
 2016-08-25 21:16:20 +02:00
Marian Beermann
b5d7f1df26 extract: fix incorrect progress output for hard links 
this produces correct output if any (non proper) subset of hardlinks are
extracted.
 2016-08-25 01:12:30 +02:00
Martin Hostettler
b845a074cb tests: TestBuildFilter: Adjust from item_is_hardlink_master to peek_and_store_hardlink_masters. 2016-08-22 23:36:43 +02:00
Martin Hostettler
a026febdb0 Archiver.build_filter: strip_components is no longer a optional parameter. 2016-08-22 23:07:38 +02:00
Martin Hostettler
972392e290 extract: When doing a partial restore don't leak prefetched chunks. 
The filter function passed to iter_items (with preload=True) may never return True for items that
are not really extracted later because that would leak prefetched items.

For restoring hard linked files the item containing the actual chunks might not be matched
or implicitly removed from the restore by strip_components. For this reason the chunk list or all
items that can potentially be used as hardlink target needs to be stored.

To achive both requirements at the same time the filter function needs to store the needed information
for the hardlinks while not returning True just because it could be a hardlink target.

Known problems: When using progress indication the calculated extracted_size now can be smaller
than the actual extracted size in presence of hard links (master is not restored) instead of
bigger (potential master not used in restore).
 2016-08-22 22:58:58 +02:00
Thomas Waldmann
d1d2738381 re-enable fuse tests for RemoteArchiver 
at some time they had deadlock issues, but it worked for me now.
 2016-08-22 00:14:10 +02:00
Marian Beermann
53d0140bd5 Repository: add compact_segments progress 2016-08-21 21:55:53 +02:00
TW
2f65f7bdec Merge pull request #1449 from enkore/f/extractprogress 
extract --progress
 2016-08-21 20:24:44 +02:00
Marian Beermann
ab31ffaa85 ProgressIndicatorPercent: remove same_line 2016-08-21 19:57:36 +02:00
Marian Beermann
e7d44cec39 extract: --progress: Calculating size 2016-08-21 19:57:36 +02:00
TW
ab96c6496c Merge pull request #1513 from ThomasWaldmann/improve-error-logging 
improve error logging, fixes #1440
 2016-08-21 18:29:18 +02:00
Marian Beermann
4d214e2503 Simplify and test Item.file_size 2016-08-21 01:09:21 +02:00
Marian Beermann
5924915d35 Flip ProgressIndicatorPercent same_line default to True 
Every production use of this uses same_line=True
 2016-08-21 01:09:20 +02:00
Marian Beermann
8709cec57c borg-extract --progress 2016-08-21 01:07:55 +02:00
enkore
81dd381701 Merge pull request #1507 from ThomasWaldmann/use-modified-pyinstaller 
use patched LDLP-preserving pyinstaller
 2016-08-21 00:16:24 +02:00
enkore
9baa024f4a Merge pull request #1508 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0 maint
 2016-08-21 00:12:00 +02:00
Thomas Waldmann
41e4707d6a fixup: meant "tb", not "traceback" 2016-08-20 23:09:42 +02:00
TW
357f4ef72e Merge pull request #1511 from c4rlo/lru-cache 
helpers.py: replace memoize with functools.lru_cache
 2016-08-20 21:19:17 +02:00
Thomas Waldmann
a0c8c40f27 improve error logging, fixes #1440 
archiver: split traceback and msg, have separate log level for traceback, log LockTimeout at debug level,
for "Error" exceptions: always log the traceback, either at ERROR or DEBUG level.

remote: if we have an "Error" typed exception instance, we can use its traceback flag and .get_message()
as we do locally.
 2016-08-20 17:23:02 +02:00
Carlo Teubner
61af307ad4 helpers.py: replace memoize usages with lru_cache 2016-08-20 13:06:16 +01:00
Thomas Waldmann
e7fccaccb2 restore original LDLP, if possible, fixes #1498 
see https://github.com/pyinstaller/pyinstaller/pull/2148
 2016-08-20 12:31:23 +02:00
Carlo Teubner
28076ee588 helpers.py: replace memoize with functools.lru_cache 2016-08-20 10:01:59 +01:00
Carlo Teubner
13761c4ec2 daemonize(): use os.devnull instead of hardcoded /dev/null 2016-08-20 10:00:03 +01:00
Thomas Waldmann
c355f3617a Merge branch '1.0-maint' 2016-08-20 00:04:55 +02:00
Thomas Waldmann
53d14e96de borg list: test for --list-format deprecation 2016-08-19 14:55:49 +02:00
TW
32e84f33e9 Merge pull request #1484 from enkore/issue/1396 
Track shadowing of log entries
 2016-08-19 14:09:34 +02:00
Marian Beermann
c39e395ecf Exclude incompatible tests for ArchiverTestCaseBinary 2016-08-17 23:01:20 +02:00
Marian Beermann
26e8ff2cbc Repository: don't use defaultdict for shadow index 
avoids errors by accidentally inserting an empty list and makes it more clear.
 2016-08-17 01:17:49 +02:00
Marian Beermann
833f8d1373 Track entire session 
Note how this enables the much stricter check in test_moved_deletes_are_tracked
 2016-08-17 01:17:49 +02:00
Marian Beermann
ac41ebcf78 Track shadowing of log entries 
Fixes (benign) index vs. log inconsistencies when segments are skipped
during compaction.
 2016-08-17 01:17:49 +02:00
Marian Beermann
21c3fb3b93 compact_segments: add segment-level logging 2016-08-17 01:16:25 +02:00
TW
f7c1c71ff1 Merge pull request #1489 from enkore/issue/1486 
Fix borg-list --list-format
 2016-08-16 21:35:25 +02:00
Marian Beermann
f3defb02de Fix borg-list empty format 
Should produce empty output, not default output.
 2016-08-16 20:46:58 +02:00
Marian Beermann
146d531d9e Fix borg-list --list-format 2016-08-16 20:46:54 +02:00
Thomas Waldmann
30cd7f3f21 fix timestamp key name 2016-08-16 20:36:29 +02:00
Thomas Waldmann
2f70e3f69a src cleanup: do not use XXX in strings if just any other string works also 
XXX is usually used in comments to mark questionable places in the source.
 2016-08-15 16:01:23 +02:00
Thomas Waldmann
3c8d86354b src cleanup: get rid of text-as-bytes in borg.remote 2016-08-15 15:52:19 +02:00
Thomas Waldmann
1f056d9e8a more safe interface for manifest.archives 2016-08-15 15:03:05 +02:00
Thomas Waldmann
b6d0eb99a5 add and use ManifestItem 2016-08-15 02:01:13 +02:00
Thomas Waldmann
c8922c8b3d use ArchiveItem 2016-08-15 01:11:33 +02:00
Thomas Waldmann
585407f4f4 introduce ArchiveItem, see #1157 2016-08-14 22:04:01 +02:00
Marian Beermann
f0e9a55ebf recreate: document that absolute patterns won't match 2016-08-14 17:46:59 +02:00
Marian Beermann
93b1cf3453 recreate: --target 2016-08-14 17:46:59 +02:00
Marian Beermann
88798ae949 recreate: --always-recompress, --compression-from 
what a mess
 2016-08-14 17:46:59 +02:00
Marian Beermann
5433b1a1e4 Add Compressor.detect(data) -> CompressBase impl 2016-08-14 17:41:34 +02:00
Marian Beermann
c2c90645ad Add Key.assert_id function 2016-08-14 17:41:33 +02:00
Marian Beermann
a80b371d09 Add decompress arg to Key.decrypt 2016-08-14 17:41:33 +02:00
Marian Beermann
ac80332ce7 Merge branch '1.0-maint' into master 
# Conflicts:
#	src/borg/repository.py
#	src/borg/testsuite/repository.py
 2016-08-14 15:51:49 +02:00
Thomas Waldmann
42b6a838da fix cyclic import issue, fix tests 
needed to increase ChunkBuffer size due to increased items stream chunk size
to get the test working.
 2016-08-14 15:26:56 +02:00
Thomas Waldmann
8be6761c26 Merge commit 'feb7e2517ef7ec07cc638953a86c726aada7d37e' 2016-08-14 15:05:11 +02:00
enkore
f8b2ffe999 Merge pull request #1458 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0 maint
 2016-08-11 14:18:03 +02:00
enkore
006d1df517 Merge pull request #1450 from enkore/f/recreatefactors 
minor recreate refactor
 2016-08-09 23:28:58 +02:00
Marian Beermann
7d9fba5a41 ArchiveRecreater.copy_items: progress from target instead of source 
(it produces the same output, just less confusing)
 2016-08-09 21:20:13 +02:00
TW
ff5d9a3f12 Merge pull request #1447 from enkore/f/debug-topic 
Debugging topics and some other minor stuff
 2016-08-09 20:53:07 +02:00
Thomas Waldmann
c955d59273 Merge branch '1.0-maint' 2016-08-09 20:09:13 +02:00
Marian Beermann
0a65e83df3 check --verify-data: remove incorrect help paragraph 
(since d0ec7e76bb - doesn't mean this probably useful feature may not
 be reimplemented, on the contrary)
 2016-08-08 16:38:31 +02:00
Marian Beermann
bec5051ea5 helpers.format_timedelta: use timedelta.total_seconds() 2016-08-08 16:38:31 +02:00
Marian Beermann
389ca94407 Silence repeated "Processing files..." log entries when --no-files-cache 2016-08-08 16:38:31 +02:00
Marian Beermann
4d7a52d8e0 --debug-topic for granular debug logging 
Especially: disables file-compression logging by default
 2016-08-08 16:38:31 +02:00
Marian Beermann
7b70b74ad1 ArchiveRecreater.incref_partial_chunks try to use __contains__ 2016-08-08 15:54:47 +02:00
Marian Beermann
5fe6c09c34 Refactor ArchiveRecreater.try_resume 2016-08-08 15:54:41 +02:00
Thomas Waldmann
dbe862f3d8 Merge branch '1.0-maint' into merge-1.0-maint 2016-08-06 01:24:22 +02:00
TW
9b0d8df0f8 Merge pull request #1373 from PlasmaPower/filesystem-test-skipping 
Filesystem feature detection and test skipping
 2016-08-03 17:46:34 +02:00
TW
29e5e558bc Merge pull request #1412 from ThomasWaldmann/info-repo 
borg info REPO
 2016-08-02 23:29:39 +02:00
Thomas Waldmann
770a892b2d implement borg info REPO 
currently it is just the same global stats also shown in "borg info ARCHIVE",
just without the archive-specific stats.

also: add separate test for "borg info".
 2016-08-02 20:06:24 +02:00
enkore
b7a563885d Merge pull request #1413 from ThomasWaldmann/store-mountpoint-dirs 
save mountpoint dirs, fixes #1033
 2016-08-01 15:51:53 +02:00
Thomas Waldmann
2b454fc54b save mountpoint dirs, fixes #1033 2016-08-01 15:25:54 +02:00
Lee Bousfield
b86b5d952a
Filesystem feature detection and test skipping 2016-07-31 01:19:31 -04:00
Thomas Waldmann
9865509105 docs: remove borg list example, be more specific about compression heuristics 2016-07-30 21:21:45 +02:00
TW
02e229b255 Merge pull request #1377 from enkore/issue/1060 
Use atomic file writing and updating for configuration and key files
 2016-07-30 16:06:01 +02:00
Marian Beermann
e9a73b808f Check for sufficient free space before committing 2016-07-30 00:04:27 +02:00
Marian Beermann
389503db60 Repository.compact_segments: always save_space 2016-07-30 00:03:31 +02:00
Thomas Waldmann
999f0ae187 fix chunk refcounts considering part files 2016-07-28 19:39:29 +02:00
Thomas Waldmann
04ad1d1b0b use "part file", "part", etc. consistently 
use .borg_part_N as filename to avoid collisions
 2016-07-28 19:39:29 +02:00
Thomas Waldmann
a7fca52f07 create part items rather than checkpoint items 
checkpoint items: chunks abc, abcdef, abcdefghi, ....
part items: chunks abc, def, ghi

solves the "restore a big file over unstable connection issue" (not very comfortably, but
one can just extract one part after the other and concatenate them to get the full file)
 2016-07-28 19:39:29 +02:00
Thomas Waldmann
e5bd6cef20 add --consider-checkpoint-files option, update FAQ 2016-07-28 19:39:29 +02:00
Thomas Waldmann
49233be25d filter out checkpoint files 
do not:
- list them
- extract them
- diff them
- include them for recreate
 2016-07-28 19:39:29 +02:00
Thomas Waldmann
dd5f957e6f in-file checkpoints, fixes #1198, fixes #1093 
also: unify code for stdin and on-disk file processing
 2016-07-28 19:39:29 +02:00
Thomas Waldmann
75d91c4bd1 file chunking: refactor code a little 
so it directly works with item.chunks list instead of a temporary list.
 2016-07-28 19:39:28 +02:00
Thomas Waldmann
bda50b5958 stdin chunking: get Item into usable state early 2016-07-28 19:39:28 +02:00
Thomas Waldmann
9226fc6f6f split stat_attrs into cheap and expensive part 
we already have stat results in st, so computing stat_simple_attrs is
rather cheap (except the username/groupname lookup maybe) and gets the
most important stuff right in the Item, so it is brought early into a
good state.

after chunking, stat_ext_attrs is called to add the more expensive-to-get
attributes, like bsdflags, xattrs and ACLs.
 2016-07-28 19:39:28 +02:00
Marian Beermann
0ae48dafbb
ObjectNotFound: give ID as hex-string 2016-07-28 18:41:50 +02:00
Marian Beermann
d0ec7e76bb Fix borg-check --verify-data failing with rebuilt objects 
There are some instances where --repair would do something. In these
instances --verify-data would fail if --repair was not given also, since
the changes without --repair are only in the index, but not in the
repository.
 2016-07-28 18:41:08 +02:00
Marian Beermann
1b6b0cfae6 Fix borg-check --verify-data tripping over ObjectNotFounds 2016-07-28 18:40:20 +02:00
Lee Bousfield
b652f4039c
Remove trailing whitespace 2016-07-28 10:35:41 -04:00
Marian Beermann
1bad3fc552
Merge branch '1.0-maint' into master 2016-07-28 09:48:58 +02:00
Marian Beermann
ac8c8119a7
Merge branch '1.0-maint' into master 2016-07-28 00:29:43 +02:00
Marian Beermann
2e3fc9ddfc SyncFile/SaveFile: default binary=False, just like open() 2016-07-26 22:49:25 +02:00
Marian Beermann
863ab66908 SaveFile: unlink(tmppath): only ignore FileNotFoundError 2016-07-26 22:40:23 +02:00
Marian Beermann
dec671d8ff SaveFile: os.replace instead of rename 2016-07-26 22:39:45 +02:00
Lee Bousfield
ff108ef24b Catch a ValueError when checking if a user exists 2016-07-23 15:03:22 -04:00
Lee Bousfield
868586dd37 Fixed noatime detection 2016-07-23 13:05:35 -04:00
Lee Bousfield
88bffab935
Allow running tests outside of Vagrant 
Fixes #1361
 2016-07-23 10:31:16 -04:00
Marian Beermann
52a72cf73a Fix some item subscripts that came from 1.0-maint 2016-07-17 19:13:09 +02:00
TW
963bdd340c Merge pull request #1330 from enkore/issue/972 
recreate: fix hogged signal handlers
 2016-07-14 14:53:21 +02:00
Marian Beermann
fcb6930c53
recreate: fix hogged signal handlers 2016-07-14 02:11:13 +02:00
Marian Beermann
2e6eb63174
Merge branch '1.0-maint' into master 2016-07-14 01:28:26 +02:00
Thomas Waldmann
3baa8a3728 Merge branch '1.0-maint' 
# Conflicts:
#	docs/changes.rst
#	docs/usage/mount.rst.inc
#	src/borg/archive.py
#	src/borg/archiver.py
#	src/borg/fuse.py
#	src/borg/testsuite/archiver.py
 2016-07-11 01:23:27 +02:00
Marian Beermann
f4be2b3523 Use platform.SaveFile for repository, cache and key files 
Fixes #1060
 2016-07-09 21:10:46 +02:00
Marian Beermann
36ebc82748 Add platform.SaveFile 2016-07-09 21:10:26 +02:00
TW
4f7a211b02 Merge pull request #1290 from enkore/issue/1287-bp 
sync_dir: silence fsync() failing with EINVAL
 2016-07-08 18:31:26 +02:00
Marian Beermann
567617ebbe
sync_dir: silence fsync() failing with EINVAL 2016-07-08 18:05:46 +02:00
TW
65475ea1aa Merge pull request #1286 from enkore/issue/1269 
Remove InternalOSError
 2016-07-08 14:00:07 +02:00
Marian Beermann
6c3ca63456
Remove InternalOSError 
obsoleted by 5b453856ec and e96905c6b
 2016-07-08 13:35:14 +02:00
Marian Beermann
11e02461d2
1.0-maint merge fixup (hexlify..decode -> bin_to_hex) 2016-07-08 13:29:51 +02:00
Marian Beermann
aade62c9f9
wrap get_flags with backup_io 2016-07-08 13:26:10 +02:00
Thomas Waldmann
fe627246e4 Merge branch '1.0-maint' 2016-07-08 12:13:52 +02:00
Thomas Waldmann
12f55f4e9f Merge branch '1.0-maint' 2016-07-06 01:33:53 +02:00
Thomas Waldmann
f363ddd7ca Merge branch '1.0-maint' 2016-07-04 20:11:21 +02:00
enkore
69f73b1752 Merge pull request #1227 from ThomasWaldmann/checkpoint-interval 
increase checkpoint-interval default from 300 to 1800s, fixes #896
 2016-06-30 17:09:56 +02:00
Thomas Waldmann
87d6755108 Merge branch '1.0-maint' 2016-06-29 18:28:33 +02:00
Thomas Waldmann
c201654f45 increase checkpoint-interval default from 300 to 1800s, fixes #896 
doing a checkpoint can be expensive, if the repo connection is slow and/or
accessed via a network filesystem and/or the repo is large.

thus: don't do it too often.
 2016-06-27 20:05:20 +02:00
Thomas Waldmann
5791b89c04 use bytes type for ACL attributes, fixes #1205 
this saves some encoding / decoding dance compared to having a str interface.
 2016-06-26 23:04:04 +02:00
Thomas Waldmann
323894d195 Merge branch '1.0-maint' 2016-06-26 20:46:12 +02:00
Thomas Waldmann
ed128e517b Merge branch '1.0-maint' 2016-06-25 17:18:14 +02:00
TW
3ff6ac09a6 Merge pull request #1179 from Abogical/list-format 
Apply --format when listing repository archives, fixes #1166
 2016-06-24 18:29:03 +02:00
Abogical
6658112f8c Add documentation for listing repository archives 2016-06-21 20:01:01 +03:00
Abogical
87173d25de Added testsuite for listing repository archives 2016-06-20 19:23:22 +03:00
Abogical
c839eb63f7 Apply --format when listing repository archives 2016-06-20 19:23:15 +03:00
Abogical
0942bcd1b4 Move code block for the write() function at do_list() 2016-06-20 16:14:52 +03:00
James Clarke
d4f5172771 Correctly handle multiple LD_PRELOAD entries; fixes #1111 2016-06-19 22:04:03 +01:00
TW
d9a14e3658 Merge pull request #1184 from c4rlo/linux-syncfile-errno-fix 
platform.linux: small SyncFile error handling fix
 2016-06-18 14:49:18 +02:00
Carlo Teubner
58d236c619 platform.linux: small SyncFile error handling fix 2016-06-18 13:39:22 +01:00
Carlo Teubner
ff470f49f3 typo fixes in comment+docstring 2016-06-18 12:57:54 +01:00
Thomas Waldmann
9a64835b4d Merge branch '1.0-maint' 
Also: add missing keys to ARCHIVE_KEYS set.
 2016-06-13 00:14:08 +02:00
TW
8c26945097 Merge pull request #1158 from ThomasWaldmann/pretty-key 
add Key/EncryptedKey PropDict, see #1157
 2016-06-12 14:51:46 +02:00
Thomas Waldmann
d2468d37df add Key/EncryptedKey PropDict 2016-06-12 04:28:40 +02:00
Michael Herold
d45a13babf Changes archive format from uuid to uuid4 
This allows for future support of other uuid versions
 2016-06-11 22:57:03 +02:00
Michael Herold
5aa97e283c Adds uuid to archive format tags 2016-06-11 12:03:55 +02:00
jenkins
b86775ab4f
Fix test_sparse_file for solaris 2016-06-10 12:21:03 +02:00
Thomas Waldmann
d813775065 Merge branch '1.0-maint' 2016-06-09 20:29:44 +02:00
Marian Beermann
e41f3ecc5b
SyncFile: open with O_EXCL (or equivalent) 2016-06-04 22:47:27 +02:00
Thomas Waldmann
75244ea6d1 fuse: simplify timestamp processing 2016-06-04 17:34:03 +02:00
Thomas Waldmann
60da32123a refactor to use Item class, fixes #1071 2016-06-04 17:24:55 +02:00
Marian Beermann
0ca1a18b33
Make borg.platform a package 2016-05-31 15:38:56 +02:00
Marian Beermann
0333f91eb2 platform: rearrange imports 2016-05-31 15:38:16 +02:00
Marian Beermann
2d90bb2c55 Compose platform_base from the platform-specific functionality 
Noticed while working on the windows branch.
 2016-05-31 15:38:16 +02:00
Thomas Waldmann
d3deec9fc9 cleanup / reorder imports 
import stdlib
from stdlib import x

import 3rd party

import borg stuff
 2016-05-31 01:18:03 +02:00
Thomas Waldmann
49484d6d67 split helpers import into multiple ones 2016-05-31 00:33:13 +02:00
Thomas Waldmann
7e0ea1635a Merge branch 'master' into move-to-src 2016-05-30 19:41:30 +02:00
Thomas Waldmann
045e5a1203 Merge branch 'master' into move-to-src 2016-05-30 19:38:16 +02:00
Thomas Waldmann
3ce35f6843 Merge branch 'master' into move-to-src 2016-05-21 19:06:01 +02:00
Thomas Waldmann
d1ea925a5b move borg package to src/ 2016-05-05 20:19:50 +02:00