upstream/borgbackup
1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2026-03-25 03:45:09 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 2
4363 commits 9 branches 151 tags 72 MiB
Commit graph

748 commits

Author SHA1 Message Date
Thomas Waldmann
a842001385 fix error msg, it is --keep-within, not --within 2017-03-12 11:53:42 +01:00
Marian Beermann
cdb4df0885 --log-json: time property on most progress/log objects, remove is_prompt 2017-03-09 21:36:37 +01:00
Marian Beermann
e98b5b20df yes(): handle JSON output 2017-03-09 21:36:37 +01:00
Abdel-Rahman
63b5cbfc99 extract: warning RC for unextracted big extended attributes, followup (#2258) 
* Set warning exit code when xattr is too big

* Warnings for more extended attributes errors (ENOTSUP, EACCES)

* Add tests for all xattr warnings
 2017-03-08 17:13:42 +01:00
enkore
fc41c98a86 Redo key_creator, key_factory, centralise key knowledge (#2272) 
* key: put key metadata (name, storage) into key classses

* keymanager: use key-declared storage types
 2017-03-08 17:08:54 +01:00
Mark Edgington
e4391dec54 docs: improve --exclude-if-present and --keep-exclude-tags 2017-03-07 22:32:52 -05:00
Abdel-Rahman
4b33c3fe14 Add return code functions (#2199) 2017-03-05 13:33:42 +01:00
TW
89114d4885 Merge pull request #2198 from Abogical/too-big-xattr 
Handle big extended attributes. Fixes #2161
 2017-03-04 17:54:58 +01:00
Thomas Waldmann
fd0649767a hashindex: rebuild hashtable if we have too little empty buckets, fixes #2246 
if there are too many deleted buckets (tombstones), hashtable performance goes down the drain.
in the worst case of 0 empty buckets and lots of tombstones, this results in full table scans for
new / unknown keys.
thus we make sure we always have a good amount of empty buckets.
 2017-03-04 03:09:18 +01:00
TW
503e9a27e6 Fix compression exceptions (#2224) 
* trigger bug in --verify-data, see #2221

* raise decompression errors as DecompressionError, fixes #2221

this is a subclass of IntegrityError, so borg check --verify-data works correctly if
the decompressor stumbles over corrupted data before the plaintext gets verified
(in a unencrypted repository, otherwise the MAC check would fail first).

* fixup: fix exception docstring, add placeholder, change wording
 2017-03-04 00:01:02 +01:00
enkore
6468b8a0c2 Merge pull request #2229 from enkore/f/jsonp 
JSON progress indicators
 2017-03-03 14:39:28 +01:00
Marian Beermann
c50ffc21b0 list: only load cache if needed 2017-03-02 00:24:22 +01:00
Marian Beermann
abb0a20d4f list: files->items, clarifications 2017-03-01 16:58:06 +01:00
Thomas Waldmann
d5707929fd [docs] improve remote-path description 
(ported bebehei's 1.0-maint change to master)
 2017-03-01 01:53:23 +01:00
TW
0db058345b Merge pull request #2217 from ThomasWaldmann/bench-cmd 
borg benchmark crud command
 2017-02-28 21:49:04 +01:00
Thomas Waldmann
9f3a970cec borg benchmark crud command, fixes #1788 2017-02-28 20:50:25 +01:00
Alexander 'Leo' Bergolth
5cd424e4be --patterns-from was accessing args.roots instead of args.paths 
add a test case that parses a command containing --patterns-from
 2017-02-28 14:18:18 +01:00
Marian Beermann
6ee0585b33 extract: fix missing call to ProgressIndicator.finish 2017-02-28 02:04:46 +01:00
Marian Beermann
8e1edaf258 ArchiveFormatter: add "start" key for compatibility with "info" 2017-02-28 01:33:36 +01:00
Marian Beermann
d327109604 json progress: emit info (current $something) 2017-02-28 01:33:36 +01:00
Marian Beermann
fcad0ddab4 pass msgid for common errors 2017-02-28 01:19:20 +01:00
Marian Beermann
9f446aa6d4 pass --log-json to remote 
Obviously this means that --log-json with remote repos requires 1.1
on the remote end, but if you don't have that, then random "Remote:"
lines would break stderr anyway.
 2017-02-28 01:19:20 +01:00
Marian Beermann
d5515b6952 add msgid to progress output 2017-02-28 01:19:20 +01:00
Marian Beermann
6288c9f751 enhance JSON progress information 
separate output types, extra information
 2017-02-27 20:30:20 +01:00
enkore
7c9c4b61d7 Merge pull request #2157 from ThomasWaldmann/add-filesize 
archived file items: add size metadata
 2017-02-27 18:05:43 +01:00
Thomas Waldmann
0721cb1ede files cache: update inode number, fixes #2226 2017-02-27 15:30:55 +01:00
TW
1a67b82726 Merge pull request #2220 from ThomasWaldmann/compression-default-lz4 
use lz4 compression by default, fixes #2179
 2017-02-27 00:31:43 +01:00
enkore
46787a90a0 Merge pull request #2204 from ThomasWaldmann/test-delete-force 
Test delete --force
 2017-02-26 21:23:23 +01:00
kmq
757921dbdc
Document relative path usage #1868 2017-02-26 20:33:15 +02:00
Thomas Waldmann
2ad5f903fe add test for borg delete --force 2017-02-26 18:49:10 +01:00
TW
9bc825a27a Merge pull request #2184 from ThomasWaldmann/zap 
borg delete --force --force to delete severely corrupted archives, fixes #1975
 2017-02-26 18:44:31 +01:00
Thomas Waldmann
92e5db0c2e use lz4 compression by default, fixes #2179 
not for create_src_archive() though as this triggers a bug that has to get fixed outside this PR first.
 2017-02-26 18:38:29 +01:00
enkore
ecd13f11b0 Merge pull request #2218 from kmq/master 
document snapshot usage #2178
 2017-02-26 17:34:15 +01:00
Marian Beermann
70c11976bc Add --log-json option for structured logging output 2017-02-26 16:25:58 +01:00
kmq
f7d28e76a0
document snapshot usage #2178 2017-02-26 13:47:26 +02:00
enkore
0f9b7a270d Merge pull request #2203 from enkore/f/jsono 
JSON output for major commands
 2017-02-25 23:05:35 +01:00
enkore
5c46136433 Merge pull request #2210 from enkore/f/rxtx 
RemoteRepository: account rx/tx bytes
 2017-02-25 22:41:27 +01:00
Thomas Waldmann
c8ec698d73 Location: accept //servername/share/path 2017-02-24 04:22:12 +01:00
Thomas Waldmann
4d81b186ec borg delete --force --force to delete severely corrupted archives, fixes #1975 2017-02-24 01:28:42 +01:00
Abogical
e487b8404c Add testsuite to test handling of too big xattr 2017-02-23 23:42:56 +02:00
Abogical
4c9bc96fb7 Print a warning for too big extended attributes 2017-02-23 23:42:56 +02:00
Thomas Waldmann
7da0a9c982 borg extract: check file size consistency 2017-02-23 21:46:15 +01:00
Thomas Waldmann
adc4da280d borg check: check file size consistency 2017-02-23 21:46:15 +01:00
Marian Beermann
4f1db82f6d info <archive>: use Archive.info() for both JSON and human display 2017-02-23 21:39:56 +01:00
Thomas Waldmann
50068c596d rename Item.file_size -> get_size 
file_size is misleading here because one thinks of on-disk file size,
but for compressed=True, there is no such on-disk file.
 2017-02-23 21:27:05 +01:00
Thomas Waldmann
97bb1b7d9a deduplicate / refactor item (c)size code 2017-02-23 21:27:05 +01:00
Thomas Waldmann
0021052dbd reduce code duplication 2017-02-23 21:24:37 +01:00
Thomas Waldmann
ae6742fb34 fuse: use precomputed size from Item 2017-02-23 21:24:37 +01:00
Thomas Waldmann
fe8e14cb2c fuse: get rid of chunk accounting 
the chunk accounting code tried to reflect repo space usage via the st_blocks of the files.
so, a specific chunk that was shared between multiple files [inodes] was only accounted for one specific file.
thus, the overall "du" of everything in the fuse mounted repo was maybe correctly reflecting the repo space usage,
but the decision which file has the chunk (the space) was kind of arbitrary and not really useful.

otoh, a simple fuse getattr() was rather expensive due to this as it needed to iterate over the chunks list
to compute the st_blocks value. also it needed quite some memory for the accounting.

thus, st_blocks is now just ceil(size / blocksize).

also: fixed bug that st_blocks was a floating point value previously.

also: preparing for further optimization of size computation (see next cs)
 2017-02-23 21:24:37 +01:00
Thomas Waldmann
a52b54dc3c archived file items: add size metadata 
if an item has a chunk list, pre-compute the total size and store it into "size" metadata entry.

this speeds up access to item size (e.g. for regular files) and could also be used to verify the validity of the chunks list.

note about hardlinks: size is only stored for hardlink masters (only they have an own chunk list)
 2017-02-23 21:24:37 +01:00
Marian Beermann
f3c7e7cd36 RemoteRepository: account rx/tx bytes 2017-02-23 16:50:52 +01:00
Marian Beermann
8cdf192511 list: add "name" key for consistency with info cmd 2017-02-23 15:23:27 +01:00
Marian Beermann
56563a4392 move JSON generation and utilities to helpers 2017-02-23 15:23:27 +01:00
enkore
4862efe718 Merge pull request #2194 from jdchristensen/master 
In api decorator, pass wait argument to RemoteRepository.call; fixes …
 2017-02-23 14:50:33 +01:00
Marian Beermann
1f8c0929bf list: --json for archive contents listing 2017-02-23 14:28:30 +01:00
Marian Beermann
25781f53d4 list: --json for archive listing 2017-02-23 14:28:15 +01:00
Marian Beermann
6180f5055c info: --json for archives 2017-02-23 14:28:15 +01:00
Marian Beermann
2ab5d0f213 use custom JSON encoder for repr'ing Borg objects consistently 2017-02-23 14:27:52 +01:00
Marian Beermann
7cbade2f8c create: add --json option 2017-02-23 12:00:21 +01:00
Marian Beermann
cc26bdf810 info: add --json option 2017-02-23 11:55:14 +01:00
Marian Beermann
b0e4f13fba set MAX_DATA_SIZE = 20971479 bytes in solid stone 2017-02-23 00:34:40 +01:00
Marian Beermann
69f7810658 info: show utilization of maximum archive size 
See #1452

This is 100 % accurate.

Also increases maximum data size by ~41 bytes. Not 100 % side-effect free;
if you manage to exactly land in that area then older Borg would not read
it. OTOH it gives us a nice round number there.
 2017-02-22 23:47:21 +01:00
Dan Christensen
20a5282a4c In api decorator, pass wait argument to RemoteRepository.call; fixes #2185 2017-02-22 16:26:08 -05:00
TW
268d74bb43 Merge pull request #2181 from ThomasWaldmann/fix-2180 
archive check: detect and fix missing replacement chunks, fixes #2180
 2017-02-21 21:57:23 +01:00
enkore
cd1df96ef5 Merge pull request #2171 from ThomasWaldmann/limit-unpacker 
borg rpc: use limited msgpack.Unpacker
 2017-02-21 21:53:09 +01:00
Thomas Waldmann
8d7dfe739f fix ChunkIndex.__contains__ assertion for big-endian archs 
also: add some missing assertion messages

severity:

- no issue on little-endian platforms (== most, including x86/x64)
- harmless even on big-endian as long as refcount is below 0xfffbffff,
  which is very likely always the case in practice anyway.
 2017-02-20 07:38:55 +01:00
Thomas Waldmann
b82f648875 archive check: detect and fix missing all-zero replacement chunks, fixes #2180 2017-02-19 03:05:41 +01:00
enkore
63d4cf4c82 Merge pull request #2172 from Abogical/master 
Add dsize and dcsize keys, fixes #2164
 2017-02-18 18:24:52 +01:00
Abogical
38e4817b48 Correct calculation of unique chunks 2017-02-18 12:37:28 +02:00
Abogical
cd3cbee962 Refactor unique chunks summing 2017-02-18 12:36:08 +02:00
Abogical
6ed0746934 Count non-unique chunks deduplicated sizes 2017-02-17 17:33:04 +02:00
Abogical
59571115a1 Add tests for dsize and dcsize 2017-02-17 15:26:14 +02:00
Abogical
5aa74abedf Add dsize and dcsize keys 
These keys shows the amount of deduplicated size and compressed
size of each file in the archive.
 2017-02-17 14:28:39 +02:00
Thomas Waldmann
b05893e723 borg rpc: use limited msgpack.Unpacker, fixes #2139 
we do not trust the remote, so we are careful unpacking its responses.

the remote could return manipulated msgpack data that announces e.g.
a huge array or map or string. the local would then need to allocate huge
amounts of RAM in expectation of that data (no matter whether really
that much is coming or not).

by using limits in the Unpacker, a ValueError will be raised if unexpected
amounts of data shall get unpacked. memory DoS will be avoided.
 2017-02-17 05:44:48 +01:00
Marian Beermann
caeff71a6c init: mandatory --encryption arg 2017-02-16 19:03:23 +01:00
Marian Beermann
788b608aa9 setup.py build_usage/build_man fixes 2017-02-14 23:02:56 +01:00
Marian Beermann
73990b878f create: handle BackupOSError on a per-path level in one spot 2017-02-14 20:39:29 +01:00
Alexander 'Leo' Bergolth
30a5c5e44b add two new options --pattern and --patterns-from as discussed in #1406 
# Conflicts:
#	src/borg/archiver.py
#	src/borg/helpers.py
#	src/borg/testsuite/helpers.py

Original-Commit: 876b670d
 2017-02-13 21:47:51 +01:00
Thomas Waldmann
8d432b01e1 paperkey.html - decode as utf-8, fixes #2150 
hardcoded the encoding for reading it. while utf-8 is the default
encoding on many systems, it does not work everywhere.

and when it tries to decode with the ascii decoder, it fails.
 2017-02-13 21:24:34 +01:00
Marian Beermann
1fabb2df58 key export: center QR code on the page 2017-02-13 21:24:33 +01:00
Marian Beermann
04bd6fb013 add test for export key --qr-html 2017-02-13 21:24:33 +01:00
Benedikt Heine
2cdb583879 clearify doc for same filesystems 
# Conflicts:
#	src/borg/archiver.py

Original-Commit: d3a2f36b03
 2017-02-13 21:24:33 +01:00
Martin Hostettler
179f1bc147 Add qr html export mode to key export command 2017-02-13 21:24:33 +01:00
Martin Hostettler
32e73e8c7e Manifest: Make sure manifest timestamp is strictly monotonically increasing. 
Computer clocks are often not set very accurately set, but borg
assumes manifest timestamps are never going back in time.

Ensure that this is actually the case.

# Conflicts:
#	src/borg/helpers.py

Original-Commit: 6b8cf0a
 2017-02-13 20:43:57 +01:00
Marian Beermann
e4486cd370 fix rST warning in Repository.scan 2017-02-12 18:51:41 +01:00
Marian Beermann
15dfaae223 docs: create: move --exclude note to main doc 2017-02-12 15:28:59 +01:00
Marian Beermann
0710bbd40e docs: create: move item flags to main doc 2017-02-12 15:28:58 +01:00
Marian Beermann
497da8df04 docs: init: fix markup/typos 2017-02-12 15:27:06 +01:00
Martin Hostettler
e8335dba0f archiver: Add 'debug dump-manifest' and 'debug dump-archive' commands. 2017-02-10 22:56:54 +01:00
Marian Beermann
7e486074e8 docs: list: don't print key listings in fat (html + man) 2017-02-05 15:46:46 +01:00
Marian Beermann
c7106e756e create real nice man pages 2017-02-05 15:46:09 +01:00
Marian Beermann
5bc03cc042 migrate-to-repokey: ask using canonical_path() as we do everywhere else 2017-02-04 15:16:18 +01:00
Marian Beermann
4e0422cdf0 Move change-passphrase to the "key" group (but leave old name, too) 2017-02-04 15:10:45 +01:00
Marian Beermann
5fe32866e6 Move migrate-to-repokey to the "key" command group 2017-02-04 15:10:45 +01:00
TW
c6ea34be96 Merge pull request #2111 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0-maint
 2017-02-01 12:13:37 +01:00
enkore
f52bf883c3 Merge pull request #2010 from costela/master 
change dir_is_tagged to use os.path.exists()
 2017-02-01 00:53:36 +01:00
Radu Ciorba
a85cf75465 fix wrong skip_hint on hashindex_set when encountering tombstones 
hashindex_lookup would always hint at skipping whatever it's probe
length had been with no regard for tombstones it had encountered. This
meant new keys would not overwrite first tombstones, but would always
land on empty buckets.

The regression was introduced in #1748
 2017-01-30 23:29:08 +02:00
Thomas Waldmann
7f2a108c94 fixup: do not access os.POSIX_FADV_* early 
before we know posix_fadvise support exists on the platform.
 2017-01-30 03:11:42 +01:00
Leo Antunes
dd6b90fe6c change dir_is_tagged to use os.path.exists() 
Add --keep-exclude-tags option as alias to --keep-tag-files and
deprecate the later. Also make tagging accept directories as tags,
allowing things like `--exclude-if-present .git`.

fixes #1999
 2017-01-29 18:13:51 +01:00
Thomas Waldmann
c0dc644ef6 Merge branch '1.0-maint' into merge-1.0-maint 
# Conflicts:
#	MANIFEST.in
#	Vagrantfile
#	docs/changes.rst
#	docs/usage/mount.rst.inc
#	src/borg/archiver.py
#	src/borg/fuse.py
#	src/borg/repository.py
 2017-01-29 05:49:53 +01:00
Thomas Waldmann
e0dfb656ee fix crc32 compilation issues on wheezy 
moving the declaration / assignment from the middle of the function to the beginning.
 2017-01-15 15:09:40 +01:00
Thomas Waldmann
740312f725 Merge branch '1.0-maint' into merge-1.0-maint 
# Conflicts:
#	requirements.d/development.txt
#	tox.ini
 2017-01-15 01:15:04 +01:00
TW
abfcf9955b Merge pull request #2045 from ThomasWaldmann/fix-platform 
posix platform module: only build / import on non-win32 platforms
 2017-01-14 16:06:28 +01:00
Marian Beermann
ae0f1422bf crc: openbsd has no Intel intrinsics 2017-01-14 15:24:03 +01:00
Thomas Waldmann
e4c5db4efc posix platform module: only build / import on non-win32 platforms, fixes #2041 
rather use a inverted check like "not windows".
also: add a base implementation for this stuff, just raising NotImplementedError
 2017-01-14 14:48:49 +01:00
Thomas Waldmann
ededb6f2c8 fix crc32 compile error, fixes #2039 2017-01-14 05:06:15 +01:00
Thomas Waldmann
e431d60cc5 merge 1.0-maint into master 
# Conflicts:
#	src/borg/crypto.pyx
#	src/borg/hashindex.pyx
#	src/borg/helpers.py
#	src/borg/platform/__init__.py
#	src/borg/platform/darwin.pyx
#	src/borg/platform/freebsd.pyx
#	src/borg/platform/linux.pyx
#	src/borg/remote.py
 2017-01-14 03:07:11 +01:00
Marian Beermann
5cc292c52c fix performance regression in "borg info ::archive" 2017-01-13 15:33:38 +01:00
Marian Beermann
f482c32423 crc32: sb8: remove some unneeded stuff, use hashindex byteorder detection 2017-01-13 00:33:34 +01:00
Marian Beermann
292fb1e2a9 crc: disable nice CLMUL version due to clang bugs. 2017-01-12 21:30:37 +01:00
Marian Beermann
7923088ff9 check: pick better insufficent archives matched warning from TW's merge 2017-01-12 17:04:51 +01:00
Marian Beermann
1d40675ce4 merge fixup 2017-01-12 15:04:57 +01:00
Marian Beermann
ecad0ed53a Merge branch '1.0-maint' into merge/1.0-maint 
# Conflicts: ... everywhere ...
#	.travis.yml
#	Vagrantfile
#	borg/testsuite/key.py
#	docs/changes.rst
#	docs/quickstart.rst
#	docs/usage.rst
#	docs/usage/upgrade.rst.inc
#	src/borg/archive.py
#	src/borg/archiver.py
#	src/borg/crypto.pyx
#	src/borg/helpers.py
#	src/borg/key.py
#	src/borg/remote.py
#	src/borg/repository.py
#	src/borg/testsuite/archive.py
#	src/borg/testsuite/archiver.py
#	src/borg/testsuite/crypto.py
#	src/borg/testsuite/helpers.py
#	src/borg/testsuite/repository.py
#	src/borg/upgrader.py
#	tox.ini
 2017-01-12 15:01:41 +01:00
enkore
07049098ee Merge pull request #2014 from enkore/f/hostname->fqdn 
posix: use fully-qualified hostname + node ID
 2017-01-12 02:07:08 +01:00
enkore
ded22d3650 Merge pull request #1927 from ThomasWaldmann/blake2b-mac-key-length 
demonstrate mac key length issue
 2017-01-12 01:21:13 +01:00
Marian Beermann
81bd55eec3 key testsuite: update blake2 test data to include padded keys 2017-01-12 01:05:59 +01:00
Marian Beermann
b9770c348f posix: use fully-qualified hostname + node ID 
The node ID is usually the 48 bit MAC of the primary network
interface.
 2017-01-12 00:54:17 +01:00
enkore
902c423086 Merge pull request #1970 from enkore/f/dbg-speed 
fast crc
 2017-01-02 16:11:11 +01:00
Marian Beermann
56818d1db8 Credit the excellent work of these people in AUTHORS 2017-01-02 15:54:14 +01:00
Marian Beermann
9afebead84 two fast CRC implementations 
CRC slice by 8 for generic CPUs outperforms zlib CRC32 on ppc
and x86 (ARM untested but expected to as well).

PCLMULQDQ derived from Intel's zlib patches outperforms every other
CRC implementation by a huge margin.
 2017-01-02 15:54:14 +01:00
enkore
cf19db466d Merge pull request #1989 from Abogical/master 
Happy new year!
 2017-01-01 01:14:08 +01:00
Abogical
f043b966da It's 2017 2016-12-31 16:53:30 +02:00
Thomas Waldmann
7d48878cfe borg init --encryption - remove default 
use --encryption=repokey for all the tests except if they
have special needs.
 2016-12-24 02:53:44 +01:00
enkore
a4eef67e3b Merge pull request #1912 from Abogical/prune-progress 
Add prune progress display
 2016-12-22 14:57:26 +01:00
Marian Beermann
742bfa33c4 flake8 2016-12-21 00:36:31 +01:00
Marian Beermann
f9aefd6a68 Merge commit '343387b36ef' into merge/1.0-maint 
# Conflicts:
#	src/borg/archiver.py
 2016-12-20 23:00:28 +01:00
Marian Beermann
df40b3840c upgrade: --disable-tam 
# Conflicts:
#	src/borg/helpers.py
#	src/borg/testsuite/archiver.py
 2016-12-20 22:53:57 +01:00
Marian Beermann
c7c8c0fb57 init: explain manifest auth compatibility 
# Conflicts:
#	src/borg/archiver.py
 2016-12-20 22:53:55 +01:00
Marian Beermann
d15fb241bd check: handle duplicate archive items neatly 
# Conflicts:
#	src/borg/archive.py
 2016-12-20 22:53:55 +01:00
Marian Beermann
5e1cb9d899 Add tertiary authentication for metadata (TAM) 2016-12-20 22:53:53 +01:00
Marian Beermann
84d13751d5 Fix processing of remote ~/ and ~user/ paths 2016-12-17 14:33:40 +01:00
Marian Beermann
63ce627a35 fix in-file checkpoints when clock jumps 2016-12-17 13:59:37 +01:00
Marian Beermann
a9db2a2e55 Merge branch '1.0-maint' into master 
# Conflicts:
#	src/borg/archive.py
#	src/borg/archiver.py
#	src/borg/helpers.py
 2016-12-17 13:26:28 +01:00
Marian Beermann
87f8b2bb64 Merge branch 'merge-1.0-maint' of https://github.com/ThomasWaldmann/borg into ThomasWaldmann-merge-1.0-maint 2016-12-17 13:21:58 +01:00
Marian Beermann
34e19ccb6a mention failed operation in per-file warnings 
on the one hand one can say it's ugly global state, on the other it's
totally handy!

just have to keep that in mind for MT, but it's rather obvious.
 2016-12-14 15:20:08 +01:00
Thomas Waldmann
c855c687eb Merge branch '1.0-maint' 2016-12-14 03:33:53 +01:00
Marian Beermann
be18418b74 cache: no archive caches => work directly on master cache (no merges) 2016-12-08 22:45:59 +01:00
Marian Beermann
e169510116 cache: don't create Item in fetch_and_build_idx 2016-12-08 22:39:04 +01:00
Thomas Waldmann
58752c9de9 add assertion to demonstrate mac key length issue in test data 2016-12-04 17:56:23 +01:00
Abogical
0994a0a681 Let prune --list display archives deleted per total archives 2016-12-03 19:34:45 +02:00
Marian Beermann
b7eaeee266 clean imports, remove unused code 2016-12-03 17:50:50 +01:00
Marian Beermann
8b2e7ec680 don't do "bigint" conversion for nanosecond mtime 
2**63 nanoseconds are 292 years, so this change is good until 2262.
See also https://en.wikipedia.org/wiki/Time_formatting_and_storage_bugs#Year_2262

I expect that we will have plenty of time to revert this commit in time
for 2262.

timespec := time_t + long, so it's probably only 64 bits on some platforms
anyway.
 2016-12-03 11:52:49 +01:00
Marian Beermann
b3707f7175 Replace backup_io with a singleton 
This is some 15 times faster than @contextmanager, because no instance
creation is involved and no generator has to be maintained. Overall
difference is low, but still nice for a very simple change.
 2016-12-03 11:52:48 +01:00
Marian Beermann
b885841c39 make item native code 
This makes an surprisingly large difference. Test case: ~70000 empty files.
(Ie. little data shoveling, lots of metadata shoveling). Before: 9.1 seconds
+- 0.1 seconds. After: 8.4 seconds +- 0.1 seconds.). That's a huge
win for changing a few lines.

I'd expect that this improves performance in almost all areas that touch
the items (list, delete, prune).
 2016-12-03 00:11:36 +01:00
enkore
e9d7f928e2 Merge pull request #1921 from enkore/f/recreate-fixes 
recreate fixes
 2016-12-03 00:02:01 +01:00
Marian Beermann
a9395dd8b1 recreate: don't rechunkify unless explicitly told so 2016-12-02 20:19:59 +01:00
Marian Beermann
30df63c509 recreate: remove special-cased --dry-run 2016-12-02 18:15:11 +01:00
Marian Beermann
c6f0969352 recreate: update --help 2016-12-02 13:20:43 +01:00
Marian Beermann
c1ccad82c3 recreate: update/remove/rename outdated comments 2016-12-02 12:54:27 +01:00
Marian Beermann
eade10a0a8 recreate: fix crash on checkpoint 2016-12-02 11:39:10 +01:00
OEM Configuration (temporary user)
0f07b6acf4 borg info:fixed bug when called without arguments,issue #1914 2016-12-02 10:27:53 +00:00
Marian Beermann
eb940e6779 recreate: fix rechunking dropping all chunks on the floor 2016-12-02 11:20:26 +01:00
Marian Beermann
b410392899 recreate repo: fix only one archive being processed 2016-12-02 11:09:52 +01:00
TW
10d143925c Merge pull request #1904 from enkore/issue/1896 
blake2b key modes: use B2B as MAC; longer keys.
 2016-12-01 00:59:55 +01:00
Thomas Waldmann
a100fb67eb Merge branch '1.0-maint' into merge-1.0-maint 
# Conflicts:
#	AUTHORS
#	src/borg/archive.py
#	src/borg/key.py
 2016-11-30 05:38:04 +01:00
Abogical
029469d544 Let prune accept and pass the progress argument 2016-11-30 02:39:27 +02:00
Marian Beermann
8949f2c758
blake2b key modes: use B2B as MAC; longer keys. 2016-11-30 00:24:05 +01:00
enkore
f4c916e01f Merge pull request #1901 from ThomasWaldmann/show-broken 
implement "health" item formatter key, fixes #1749
 2016-11-29 23:48:55 +01:00
Thomas Waldmann
01ad1a5153 implement "health" item formatter key, fixes #1749 2016-11-29 23:02:24 +01:00
TW
99c38e8fc0 Merge pull request #1892 from Abogical/sync-progress 
Add cache.sync progress display
 2016-11-29 21:52:07 +01:00
Abogical
f3ce6be30b Add cache.sync progress display 2016-11-29 22:15:27 +02:00
Marian Beermann
bd96b43af9
borg info: print security directory 2016-11-29 17:52:34 +01:00
Marian Beermann
f62a22392e Implement security dir perks 
Key type, location and manifest timestamp checks now survive cache
deletion. This also means that you can now delete your cache and avoid
previous warnings, since Borg can still tell it's safe.
 2016-11-29 17:52:32 +01:00
Marian Beermann
c3a2dc5f55 Rename BORG_NONCES_DIR to BORG_SECURITY_DIR 2016-11-29 17:43:06 +01:00
Thomas Waldmann
c83a124e65 Merge branch '1.0-maint' (into master) 2016-11-28 02:23:32 +01:00
enkore
5917822735 Merge pull request #1881 from Abogical/cache-progress 
Add begin_txn progress message
 2016-11-26 19:11:08 +01:00
Abogical
be6341b956 Add begin_txn progress message 2016-11-26 19:38:00 +02:00
enkore
1250b14ce2 Merge pull request #1874 from Abogical/cache-progress 
Add progress display for cache commit
 2016-11-26 00:50:16 +01:00
enkore
50b97665c8 Merge pull request #1878 from ThomasWaldmann/borg-new-passphrase 
implement BORG_NEW_PASSPHRASE, fixes #1768
 2016-11-25 20:39:26 +01:00
Thomas Waldmann
d519ae6b6f remove some unneeded test overriding methods 
they are not there any more in the base class, so there is nothing to override.
 2016-11-25 01:31:28 +01:00
Thomas Waldmann
fa8ce2d977 add test for borg change-passphrase / BORG_NEW_PASSPHRASE 2016-11-25 00:06:35 +01:00
Thomas Waldmann
0a0b913739 implement BORG_NEW_PASSPHRASE, fixes #1768 2016-11-24 23:43:27 +01:00
Abogical
f9b3d28c19 Add progress messages for cache.commit 2016-11-24 15:44:00 +02:00
Abogical
e999f3ff51 Add ProgressIndicatorMessage and abstract class ProgressIndicatorBase 2016-11-24 15:43:51 +02:00
enkore
abe5923866 Merge pull request #1862 from enkore/f/simple-recreate 
The Defenestration of recreate's bloat
 2016-11-20 19:59:45 +01:00
Marian Beermann
15cefe8dde
recreate/archiver tests: add check_cache tool - lints refcounts 2016-11-20 18:13:53 +01:00
Marian Beermann
93b03ea231 recreate: re-use existing checkpoint functionality 2016-11-20 18:13:51 +01:00
Marian Beermann
44935aa8ea recreate: remove interruption blah, autocommit blah, resuming blah 2016-11-19 16:49:20 +01:00
Thomas Waldmann
afab1cc9b0 Merge branch '1.0-maint' 
# Conflicts:
#	borg/platform.py
#	src/borg/archiver.py
#	src/borg/helpers.py
#	src/borg/platform/darwin.pyx
#	src/borg/platform/freebsd.pyx
#	src/borg/platform/linux.pyx
#	src/borg/testsuite/__init__.py
 2016-11-17 23:56:34 +01:00
Martin Hostettler
f37109848f remote: Remove unused type variable. 2016-11-17 20:03:33 +01:00
Martin Hostettler
1edff44b3d Repository,remote: Rename argument of get to 'id'. 2016-11-17 20:03:32 +01:00
Martin Hostettler
8955d8bb2a remote: Test that the legacy free rpc bootstrap works. 2016-11-17 20:03:32 +01:00
Martin Hostettler
bd3a4a2f92 Add testing for exception transport. 2016-11-17 20:03:32 +01:00
Martin Hostettler
e14406fdbf remote: Redo exception handling 2016-11-17 20:03:32 +01:00
Martin Hostettler
4854fcef2e remote: Move open to a normal api stub. 2016-11-17 19:56:30 +01:00
Martin Hostettler
ba553ec628 remote: Introduce rpc protocol with named parameters. 2016-11-17 19:56:30 +01:00
Martin Hostettler
6c1b337ce2 remote: Replace broken exception argument restoration with code that uses a fixed value. 2016-11-17 19:56:29 +01:00
Martin Hostettler
d25e9aa4f0 remote: Use single quotes 2016-11-17 19:56:29 +01:00
Martin Hostettler
0da913f8ed remote: Decode method name as utf-8 instead of ascii for consistency. 2016-11-17 19:56:29 +01:00
Martin Hostettler
c580d9c173 version: Add version parsing and formating 2016-11-17 19:56:29 +01:00
Marian Beermann
c380d91805 fixes for flake 3.1.1 2016-11-14 21:50:20 +01:00
enkore
cf8f8fb746 Merge pull request #1846 from Abogical/master 
Improve extract progress display, for #1721
 2016-11-14 21:43:35 +01:00
Abogical
34f529c7df satisfy codecov 2016-11-14 17:39:42 +02:00
Abogical
3896f26ab2 use monkeypatch 2016-11-14 16:41:37 +02:00
Abogical
76638d0562 If there is a small space for ellipsis_truncate, show '...' only 2016-11-14 15:16:14 +02:00
Abogical
af925d2723 do not justify if we're not outputing to a terminal 2016-11-14 14:49:24 +02:00
Abogical
3232769ffc Respond to feedback 2016-11-14 11:43:46 +02:00
Abogical
1362d2e90f Set COLUMNS & LINES as if it was a terminal 2016-11-14 01:00:11 +02:00
Abogical
467fe38b15 Compatibility with python 3.4 2016-11-14 00:25:27 +02:00
Abogical
b737866905 Improve extract progress display and ProgressIndicatorPercent 2016-11-13 23:41:01 +02:00
Marian Beermann
dfd748689c test_attic013_acl_bug: use def instead of lambda 2016-11-13 17:30:42 +01:00
Marian Beermann
c2eb2539b9
Document maintenance merge command 2016-11-13 16:07:06 +01:00
Marian Beermann
0d2b76fa7d Merge branch '1.0-maint' into merge/1.0-maint 2016-11-13 15:58:42 +01:00
Marian Beermann
00fae3f3e3
repository: check_free_space, special case small repos 2016-11-12 14:42:13 +01:00
enkore
80b86709ef Merge pull request #1833 from enkore/issue/1821 
borg init: fix free space check crashing if disk is full
 2016-11-12 01:23:34 +01:00
enkore
e26348a055 Merge pull request #1834 from enkore/issue/1813 
Clarify cache/repository README file
 2016-11-11 21:25:30 +01:00
Marian Beermann
bd4a0fe23b
Clarify cache/repository README file 2016-11-11 21:24:17 +01:00
Marian Beermann
ac5f9b61ec
debug, key: enable --help 
A small merge mishap in 2a864be8; 1.0-maint has it correctly.
 2016-11-11 19:54:11 +01:00
Marian Beermann
6c955579a8 repository: test_additional_free_space, assert it still exists 2016-11-11 16:39:11 +01:00
Marian Beermann
5b31a2cd61
borg init: fix free space check crashing if disk is full 2016-11-11 16:21:05 +01:00
enkore
223f7890fe Merge pull request #1819 from enkore/issue/1800 
Add BLAKE2b key types
 2016-11-11 03:11:38 +01:00
Marian Beermann
a31ca4f022
crypto: link against system libb2, if possible 2016-11-11 03:08:11 +01:00
Marian Beermann
76c93bb80b crypto blake2: don't release the GIL during key hashing 2016-11-11 03:07:05 +01:00
Marian Beermann
00ac7b14be Add BLAKE2b key types 2016-11-11 03:07:05 +01:00
TW
dc4325be25 Merge pull request #1826 from textshell/feature/repository-cleanup-rollback-api 
Repository: rollback's cleanup parameter is internal.
 2016-11-10 17:24:01 +01:00
Martin Hostettler
79c77e7a68 Repository: rollback's cleanup parameter is internal. 2016-11-10 13:44:58 +01:00
Marian Beermann
09f470bd85 Add crypto.blake2b_256 2016-11-10 00:06:50 +01:00
Marian Beermann
05a52ec8ec
Merge branch '1.0-maint' into merge/1.0-maint 2016-11-09 23:56:54 +01:00
Marian Beermann
676e69cac4 Parse & pass BORG_HOSTNAME_IS_UNIQUE env var to enable stale lock killing 2016-11-07 21:56:40 +01:00
Marian Beermann
cc14975f2d Add tests for stale lock killing and platform.process_alive 2016-11-07 21:56:07 +01:00
Marian Beermann
2bd8ac7762 platform: bump API version (and check consistency) 2016-11-07 21:54:16 +01:00
Marian Beermann
7930d055ec import platform module instead of functions 
(testability)
 2016-11-07 21:54:16 +01:00
Marian Beermann
8e1df7a364 Use logging instead of prints 2016-11-07 21:54:16 +01:00
Marian Beermann
c562f7750c Move platform-dependent code to platform package 2016-11-07 21:54:16 +01:00
Oleg Drokin
d490292be3 Detect and delete stale locks when it's safe 
If BORG_UNIQUE_HOSTNAME shell variable is set, stale locks
in both cache and repository are deleted.

Stale lock is defined as a lock that's originating from the same
hostname as us, and correspond to a pid that no longer exists.

This fixes #562
 2016-11-07 21:54:16 +01:00
Thomas Waldmann
8a15916284 Merge branch '1.0-maint' into merge-1.0-maint 
# Conflicts:
#	docs/changes.rst
#	docs/usage/help.rst.inc
#	src/borg/cache.py
#	src/borg/remote.py
#	src/borg/testsuite/__init__.py
#	src/borg/testsuite/archiver.py
 2016-10-31 18:24:13 +01:00
Thomas Waldmann
09e74af7bf Merge branch '1.0-maint' 2016-10-24 21:41:20 +02:00
TW
0ff2352691 Merge pull request #1748 from rciorba/shortcut_hashindex_set 
shortcut hashindex_set by having hashindex_lookup hint about address
 2016-10-23 16:48:43 +02:00
Radu Ciorba
766a43afc3 shortcut hashindex_set by having hashindex_lookup hint about address 2016-10-22 12:33:15 +03:00
TW
1a272ddd48 Merge pull request #1739 from ThomasWaldmann/borg-check-refactor 
borg check --first / --last / --sort / --prefix, fixes #1663
 2016-10-20 17:21:27 +02:00
Thomas Waldmann
4c884fd075 borg check --first / --last / --sort / --prefix, fixes #1663 2016-10-20 16:51:26 +02:00
Thomas Waldmann
e02d1a66b8 fix byte range error in test, fixes #1740 2016-10-20 03:04:19 +02:00
Thomas Waldmann
5a12c6377a fuse mount options: add tests 2016-10-15 01:24:05 +02:00
Thomas Waldmann
b5f9858055 move first/last/sort_by-multiple functionality into Manifest.list 
also: rename list_filtered to list_considering
 2016-10-15 01:04:56 +02:00
Thomas Waldmann
694c3978a1 refactor Archiver._get_filtered_archives -> Archives.list_filtered 
it did not belong into Archiver class (did not use "self"), but in into Archives.
 2016-10-15 01:03:16 +02:00
Thomas Waldmann
26e6ac4fea borg mount --first / --last / --sort / --prefix, fixes #1542 
also: use consider_part_files when dealing with multiple archives / whole repo mount
 2016-10-15 01:03:16 +02:00
Thomas Waldmann
775440fdcc Merge branch '1.0-maint' into merge-1.0-maint 2016-10-15 00:25:41 +02:00
enkore
34bd7cb4e2 Merge pull request #1716 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0-maint
 2016-10-14 21:18:30 +02:00
Thomas Waldmann
4e4847ccce point XDG_*_HOME to temp dirs for tests, fixes #1714 
otherwise it spoils the user's nonces and cache dirs with lots of files.

also: remove all BORG_* env vars from the outer environment
 2016-10-13 23:59:28 +02:00
Thomas Waldmann
b0cc4fd74a Merge branch '1.0-maint' into merge-1.0-maint 2016-10-13 19:10:04 +02:00
TW
a2dfe68609 Merge pull request #1706 from ThomasWaldmann/remove-debug-xxx 
remove debug-xxx commands, fixes #1627
 2016-10-12 23:25:43 +02:00
Thomas Waldmann
9ad9ae8ff0 Repository.scan(): avoid iterating same repo segments/objects repeatedly 
segments: avoid some listdir()
objects: avoid repeatedly re-reading object headers / seeking in segment

Fixes #1610.
 2016-10-10 05:39:38 +02:00
Thomas Waldmann
b88e82d99d remove debug-xxx commands, fixes #1627 
we use "debug xxx" subcommands now. docs updated.

also makes "borg help" shorter as not all debug-xxx commands
show up, but just 1 main "debug" command.
 2016-10-10 00:22:01 +02:00
enkore
8567fcfe58 Merge pull request #1418 from textshell/fix/preload 
Cleanup prefetch code in RemoteRepository
 2016-10-07 09:49:16 +02:00
Thomas Waldmann
f31112b975 clarify borg diff help, fixes #980 2016-10-07 04:09:05 +02:00
Martin Hostettler
02557f16b0 RemoteRepository: Rename cache to chunkid_to_msgids. Also fetch_from_cache to pop_preload_msgid. 2016-10-06 20:48:26 +02:00
Martin Hostettler
2608a5620a RemoteRepository: Always store chunk ids in cache instead of rpc argument encoding of get request. 2016-10-06 20:38:46 +02:00
Martin Hostettler
e9605d67ad RemoteReposity: prefetch can only be 'get'. 2016-10-06 20:38:46 +02:00
Thomas Waldmann
cdb8d64fe2 check for index vs. segment files object count mismatch 2016-10-05 17:36:36 +02:00
Thomas Waldmann
6624ca9cdb verify_data: do a linear scan in disk-order 2016-10-05 17:36:36 +02:00
Thomas Waldmann
90111363ba repo.list() yielding IDs in on-disk order 2016-10-05 17:36:36 +02:00
enkore
27bc73c23e borg info <repo>: print general repo information (#1680) 2016-10-03 14:53:16 +02:00
Thomas Waldmann
8df6cb8156 hashindex: bump api_version 
note:
merging the respective changeset from 1.0-maint was not effective
as we already had version 3, so there was no increase.
 2016-09-30 23:59:41 +02:00
Frank Sachsenheim
089995ef73 Changes on filters after feedback 2016-09-30 20:29:16 +02:00
Frank Sachsenheim
bd7cc38d6e Changes arg processor names to camelcase 2016-09-30 20:14:20 +02:00
Frank Sachsenheim
e0e9edfb42 Adds archives filters for list 2016-09-30 20:14:20 +02:00
Frank Sachsenheim
369d0a0881 Adds archives filters for info 2016-09-30 20:14:20 +02:00
Frank Sachsenheim
f2d4d36cea Allows delete to be used with archive filters 2016-09-30 20:14:20 +02:00
Frank Sachsenheim
17f2363935 Adds --prefix to the archives filters arguments 
- adds prefix argument to helpers.Archives.list
- also renames function PrefixSpec to prefix_spec
 2016-09-30 19:48:17 +02:00
Frank Sachsenheim
f6b9276de9 Adds arguments to filter archives 
These are: --sort-by, --first and --last

Includes a method to obtain a list of archive infos filtered by these

Archives.list:
- ensure reverse is always applied
- always return a list
 2016-09-30 19:48:17 +02:00
Thomas Waldmann
19eb75984e borg check --verify-data tuning 2016-09-29 18:40:02 +02:00
Thomas Waldmann
ba30098079 Merge branch '1.0-maint' into merge-1.0-maint 2016-09-29 12:57:29 +02:00
Thomas Waldmann
2a864be84f Merge branch '1.0-maint' into merge-1.0-maint 
# Conflicts:
#	setup.py
#	src/borg/archiver.py
#	src/borg/helpers.py
 2016-09-27 21:31:41 +02:00
enkore
ea9e859026 Merge pull request #1635 from enkore/issue/1624 
Fix broken --progress for double-cell paths
 2016-09-27 13:14:16 +02:00
Marian Beermann
9cef0a9ed8 Fix broken --progress ellipsis for double-cell paths 2016-09-27 11:35:45 +02:00
TW
233ae43551 Merge pull request #1640 from ThomasWaldmann/improve-borg-help 
Re-Indent borg help helptexts, again.
 2016-09-25 23:26:01 +02:00
TW
43432f9a14 Merge pull request #1615 from enkore/issue/1598 
Fix #1598
 2016-09-25 23:09:47 +02:00
Thomas Waldmann
bf681e98ce Re-Indent borg help helptexts, again. 2016-09-25 22:50:08 +02:00
TW
f36d551092 Merge pull request #1634 from ThomasWaldmann/improve-compression-docs 
borg help compression, fixes #1582
 2016-09-25 22:42:56 +02:00
Marian Beermann
8164524d99 Fix broken --progress for double-cell paths 2016-09-25 22:18:37 +02:00
Thomas Waldmann
28c57f98c9 Merge branch '1.0-maint' 
# Conflicts:
#	src/borg/archive.py
#	src/borg/archiver.py
#	src/borg/helpers.py
#	src/borg/testsuite/archiver.py
 2016-09-25 17:30:55 +02:00
Marian Beermann
66316e10b9 Fix indentation in borg help compression 2016-09-25 11:53:54 +02:00
Marian Beermann
7b1f10347a Repository: compact: fix incorrect preservation of delete tags 2016-09-25 10:13:31 +02:00
Marian Beermann
abace16945 Repository.check: log transaction IDs 2016-09-25 10:12:42 +02:00
Thomas Waldmann
d6bfdafdef borg help compression, fixes #1582 2016-09-25 02:32:02 +02:00
TW
d3e79c4e49 Merge pull request #1625 from bauerj/master 
Change {utcnow} and {now} to ISO-8601 format
 2016-09-23 18:24:10 +02:00
Johann Bauer
a56dc44e1f Change {utcnow} and {now} to ISO-8601 format 2016-09-23 17:58:04 +02:00
Martin Hostettler
84b3295a0d Archiver,RemoteRepository: Add --remote-ratelimit 
The --remote-ratelimit option adds a very simple rate limit for the
sending data to the remote.

Currently implemented by sleeping if the transmission speed is greater
than the limit.
 2016-09-18 19:00:26 +02:00
Thomas Waldmann
260ef31728 fuse: refactor for less encoding dance 2016-09-16 16:38:08 +02:00
Thomas Waldmann
39170479aa fuse: fix hardlinks in versions view, fixes #1599 2016-09-16 16:38:08 +02:00
Thomas Waldmann
c021cf466a fuse: add test for hardlinks in versions view 2016-09-16 16:38:08 +02:00
Thomas Waldmann
001500ab99 fuse: refactor file versioning code 2016-09-16 16:38:08 +02:00
Thomas Waldmann
5d22078f35 fuse: add parameter check to ItemCache.get to make potential failures more clear 2016-09-16 16:38:08 +02:00
Marian Beermann
ae5b4980f2 Repository.check: improve object count mismatch diagnostic 2016-09-15 11:23:58 +02:00
TW
783d01f621 Merge pull request #1586 from ThomasWaldmann/borg-check-delete-defect-chunks 
borg check: delete chunks with integrity errors, fixes #1575
 2016-09-14 22:55:50 +02:00
Thomas Waldmann
b4c7cce67d borg check: delete chunks with integrity errors, fixes #1575 
so they can be "repaired" immediately and maybe healed later.
 2016-09-14 21:07:48 +02:00
Thomas Waldmann
57a3adb6b3 borg recreate: also catch SIGHUP 2016-09-14 02:55:13 +02:00
Thomas Waldmann
1287d1ae92 Merge branch '1.0-maint' into merge-1.0-maint 
# Conflicts:
#	docs/development.rst
#	src/borg/archive.py
#	src/borg/archiver.py
#	src/borg/hashindex.pyx
#	src/borg/testsuite/hashindex.py
 2016-09-14 02:53:41 +02:00
Thomas Waldmann
dd89181062 fuse: refactor / optimize process_inner 2016-09-01 06:26:33 +02:00
Thomas Waldmann
6f50cc3413 fuse: simplify path segments loop 2016-09-01 05:18:10 +02:00
Thomas Waldmann
d64ccff1c4 fuse: remove unneeded function definition 
code only had 1 caller, so just inline it.
 2016-09-01 04:57:11 +02:00
Thomas Waldmann
34eb475463 fuse: remove unneeded safe_encode 
fsencode will happen in _find_inode()
 2016-09-01 04:33:42 +02:00
Thomas Waldmann
8b5372b54c fuse: slightly refactor shadowing detection 
if it is not a directory, the old code was a NOP, so we can just check that first.
 2016-09-01 04:30:55 +02:00
Thomas Waldmann
1f04820d9d fuse: implement versions view 
all archives, all items are read to build a unified view.
files are represented by a same-name directory with the versions of the file.
A filename suffix computed by adler32(chunkids) is used to disambiguate the versions.

also: refactor code a little, create methods for leaves, inner nodes.
 2016-08-27 23:17:57 +02:00
enkore
69c51f268d Merge pull request #1374 from textshell/issue/1039-aes-iv 
Fix AES-CTR nonce reusage
 2016-08-27 15:52:21 +02:00
Martin Hostettler
620f505a14 Reserve nonce space for AES-CTR before using it. 
Reusing the nonce totally breaks AES-CTR confidentiality.

This code uses a reservation of nonce space and stores the next nonce available for
a future reservation on the client and in the repository.

Local storage is needed to protect against evil repositories that try to gain access to
encrypted data by not saving nonce reservations and aborting the connection or otherwise
forcing a rollback.

Storage in the repository is needed to protect against another client writing to the
repository after a transaction was aborted and thus not seeing the last used nonce from
the manifest.

With a real counter mode cipher protection for the multiple client case with an actively
evil repository is not possible. But this still protects against cases where the attacker can
not arbitrarily change the repository but can read everything stored and abort connections
or crash the server.

Fixes #22
 2016-08-27 15:31:50 +02:00
Marian Beermann
e4e6787906 recreate: fix crash if recompress and deduplication beyond autocommit_threshold 
ie. it means that if all the recompressed chunks were already in the repo
no data would be written, so there would be no active txn, so failure
ensues.
 2016-08-27 14:31:57 +02:00
enkore
e60ed80433 Merge pull request #1539 from enkore/f/recreate-cp 
recreate: fix crash if archive doesn't have chunker_params
 2016-08-27 14:06:32 +02:00
Marian Beermann
e289fb3539 recreate: fix crash if archive doesn't have chunker_params 2016-08-27 14:01:28 +02:00
enkore
1d750e802e Merge pull request #1536 from enkore/f/prip2 
ProgressIndicatorPercent / extract minor cosmetic enhancement
 2016-08-27 00:36:43 +02:00
TW
168ce1dc86 Merge pull request #1535 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0 maint
 2016-08-27 00:32:40 +02:00
Marian Beermann
f0a32575a5 extract: --progress: no extra line for 'Calculating size' 2016-08-26 22:15:44 +02:00
Marian Beermann
517ccc2d58 ProgressIndicatorPercent: output(message) to override output 2016-08-26 22:15:29 +02:00
Thomas Waldmann
25fa443d2b repo tests: convert some more byte literals to H(x) 2016-08-26 20:59:22 +02:00
Thomas Waldmann
1974824595 Merge branch '1.0-maint' 2016-08-26 20:47:23 +02:00
Marian Beermann
cd39ccb821 fix overeager storing of hardlink masters 
n.b. we only need to store them for items that we wouldn't extract.

this also fixes an intersting edge case in extracting hard links
with --strip-components
 2016-08-25 21:16:20 +02:00
Marian Beermann
b5d7f1df26 extract: fix incorrect progress output for hard links 
this produces correct output if any (non proper) subset of hardlinks are
extracted.
 2016-08-25 01:12:30 +02:00
Martin Hostettler
b845a074cb tests: TestBuildFilter: Adjust from item_is_hardlink_master to peek_and_store_hardlink_masters. 2016-08-22 23:36:43 +02:00
Martin Hostettler
a026febdb0 Archiver.build_filter: strip_components is no longer a optional parameter. 2016-08-22 23:07:38 +02:00
Martin Hostettler
972392e290 extract: When doing a partial restore don't leak prefetched chunks. 
The filter function passed to iter_items (with preload=True) may never return True for items that
are not really extracted later because that would leak prefetched items.

For restoring hard linked files the item containing the actual chunks might not be matched
or implicitly removed from the restore by strip_components. For this reason the chunk list or all
items that can potentially be used as hardlink target needs to be stored.

To achive both requirements at the same time the filter function needs to store the needed information
for the hardlinks while not returning True just because it could be a hardlink target.

Known problems: When using progress indication the calculated extracted_size now can be smaller
than the actual extracted size in presence of hard links (master is not restored) instead of
bigger (potential master not used in restore).
 2016-08-22 22:58:58 +02:00
Thomas Waldmann
d1d2738381 re-enable fuse tests for RemoteArchiver 
at some time they had deadlock issues, but it worked for me now.
 2016-08-22 00:14:10 +02:00
Marian Beermann
53d0140bd5 Repository: add compact_segments progress 2016-08-21 21:55:53 +02:00
TW
2f65f7bdec Merge pull request #1449 from enkore/f/extractprogress 
extract --progress
 2016-08-21 20:24:44 +02:00
Marian Beermann
ab31ffaa85 ProgressIndicatorPercent: remove same_line 2016-08-21 19:57:36 +02:00
Marian Beermann
e7d44cec39 extract: --progress: Calculating size 2016-08-21 19:57:36 +02:00
TW
ab96c6496c Merge pull request #1513 from ThomasWaldmann/improve-error-logging 
improve error logging, fixes #1440
 2016-08-21 18:29:18 +02:00
Marian Beermann
4d214e2503 Simplify and test Item.file_size 2016-08-21 01:09:21 +02:00
Marian Beermann
5924915d35 Flip ProgressIndicatorPercent same_line default to True 
Every production use of this uses same_line=True
 2016-08-21 01:09:20 +02:00
Marian Beermann
8709cec57c borg-extract --progress 2016-08-21 01:07:55 +02:00
enkore
81dd381701 Merge pull request #1507 from ThomasWaldmann/use-modified-pyinstaller 
use patched LDLP-preserving pyinstaller
 2016-08-21 00:16:24 +02:00
enkore
9baa024f4a Merge pull request #1508 from ThomasWaldmann/merge-1.0-maint 
Merge 1.0 maint
 2016-08-21 00:12:00 +02:00
Thomas Waldmann
41e4707d6a fixup: meant "tb", not "traceback" 2016-08-20 23:09:42 +02:00
TW
357f4ef72e Merge pull request #1511 from c4rlo/lru-cache 
helpers.py: replace memoize with functools.lru_cache
 2016-08-20 21:19:17 +02:00
Thomas Waldmann
a0c8c40f27 improve error logging, fixes #1440 
archiver: split traceback and msg, have separate log level for traceback, log LockTimeout at debug level,
for "Error" exceptions: always log the traceback, either at ERROR or DEBUG level.

remote: if we have an "Error" typed exception instance, we can use its traceback flag and .get_message()
as we do locally.
 2016-08-20 17:23:02 +02:00
Carlo Teubner
61af307ad4 helpers.py: replace memoize usages with lru_cache 2016-08-20 13:06:16 +01:00
Thomas Waldmann
e7fccaccb2 restore original LDLP, if possible, fixes #1498 
see https://github.com/pyinstaller/pyinstaller/pull/2148
 2016-08-20 12:31:23 +02:00
Carlo Teubner
28076ee588 helpers.py: replace memoize with functools.lru_cache 2016-08-20 10:01:59 +01:00
Carlo Teubner
13761c4ec2 daemonize(): use os.devnull instead of hardcoded /dev/null 2016-08-20 10:00:03 +01:00
Thomas Waldmann
c355f3617a Merge branch '1.0-maint' 2016-08-20 00:04:55 +02:00
Thomas Waldmann
53d14e96de borg list: test for --list-format deprecation 2016-08-19 14:55:49 +02:00
TW
32e84f33e9 Merge pull request #1484 from enkore/issue/1396 
Track shadowing of log entries
 2016-08-19 14:09:34 +02:00
Marian Beermann
c39e395ecf Exclude incompatible tests for ArchiverTestCaseBinary 2016-08-17 23:01:20 +02:00
Marian Beermann
26e8ff2cbc Repository: don't use defaultdict for shadow index 
avoids errors by accidentally inserting an empty list and makes it more clear.
 2016-08-17 01:17:49 +02:00
Marian Beermann
833f8d1373 Track entire session 
Note how this enables the much stricter check in test_moved_deletes_are_tracked
 2016-08-17 01:17:49 +02:00
Marian Beermann
ac41ebcf78 Track shadowing of log entries 
Fixes (benign) index vs. log inconsistencies when segments are skipped
during compaction.
 2016-08-17 01:17:49 +02:00
Marian Beermann
21c3fb3b93 compact_segments: add segment-level logging 2016-08-17 01:16:25 +02:00
TW
f7c1c71ff1 Merge pull request #1489 from enkore/issue/1486 
Fix borg-list --list-format
 2016-08-16 21:35:25 +02:00
Marian Beermann
f3defb02de Fix borg-list empty format 
Should produce empty output, not default output.
 2016-08-16 20:46:58 +02:00
Marian Beermann
146d531d9e Fix borg-list --list-format 2016-08-16 20:46:54 +02:00
Thomas Waldmann
30cd7f3f21 fix timestamp key name 2016-08-16 20:36:29 +02:00
Thomas Waldmann
2f70e3f69a src cleanup: do not use XXX in strings if just any other string works also 
XXX is usually used in comments to mark questionable places in the source.
 2016-08-15 16:01:23 +02:00
Thomas Waldmann
3c8d86354b src cleanup: get rid of text-as-bytes in borg.remote 2016-08-15 15:52:19 +02:00
Thomas Waldmann
1f056d9e8a more safe interface for manifest.archives 2016-08-15 15:03:05 +02:00
Thomas Waldmann
b6d0eb99a5 add and use ManifestItem 2016-08-15 02:01:13 +02:00
Thomas Waldmann
c8922c8b3d use ArchiveItem 2016-08-15 01:11:33 +02:00
Thomas Waldmann
585407f4f4 introduce ArchiveItem, see #1157 2016-08-14 22:04:01 +02:00
Marian Beermann
f0e9a55ebf recreate: document that absolute patterns won't match 2016-08-14 17:46:59 +02:00
Marian Beermann
93b1cf3453 recreate: --target 2016-08-14 17:46:59 +02:00
Marian Beermann
88798ae949 recreate: --always-recompress, --compression-from 
what a mess
 2016-08-14 17:46:59 +02:00
Marian Beermann
5433b1a1e4 Add Compressor.detect(data) -> CompressBase impl 2016-08-14 17:41:34 +02:00