upstream/borgbackup
1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2026-03-18 00:26:13 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 6
4363 commits 9 branches 151 tags 70 MiB
Commit graph

748 commits

Author SHA1 Message Date
Marian Beermann
944a4abd58 chunker: don't do uint32_t >> 32 2017-06-13 11:42:43 +02:00
Marian Beermann
4490a8bbc3 cache sync: don't do memcpy(..., 0, 0) 
!ctx->buf => ctx->tail - ctx->head == 0
 2017-06-13 11:16:04 +02:00
Thomas Waldmann
0b00c14c27 don't write to disk with --stdout, fixes #2645 
if we always give stdout to extract_item(), it gets into the stdout-
processing branch which only emits data from items that have chunks
and does nothing for items which don't.
 2017-06-13 01:12:14 +02:00
Marian Beermann
c9c227f2ca cache sync: check Operation.READ compatibility with manifest 2017-06-12 23:46:49 +02:00
Marian Beermann
78cbf695c4 cache sync: suppress GCC C90/C99 int literal warning 
warning: this decimal constant is unsigned only in ISO C90

Raised by GCC 4.9.2 on PowerPC.

The warning is bogus here due to the immediate explicit cast; newer
versions don't emit it.
 2017-06-12 10:53:55 +02:00
enkore
148a8a855b Merge pull request #2663 from enkore/f/fcs-changes 
cache sync changes
 2017-06-11 21:27:25 +02:00
Marian Beermann
783a5926d6 cache sync: introduce BORG_NO_PYTHON 
textshell edition
 2017-06-11 20:23:17 +02:00
Marian Beermann
3c6372f841 cache sync: convert incoming integers to uint64_t 2017-06-11 20:20:00 +02:00
Marian Beermann
f786211b12 RepositoryCache: truncate+unlink errored file 2017-06-11 20:20:00 +02:00
enkore
49d74e12fc Merge pull request #2661 from enkore/f/consistent-key-names 
normalize authenticated key modes
 2017-06-11 20:01:32 +02:00
Marian Beermann
221dc1c4c7 normalize authenticated key modes 
rename authenticated to authenticated-blake2, consistent with the other
blake2 key modes

add authenticated mode that fills the blank and is consistent with the
other "unqualified" key modes
 2017-06-11 18:07:09 +02:00
Marian Beermann
5f5371f0b1 implement --glob-archives/-a 2017-06-11 12:15:12 +02:00
Marian Beermann
49ca3dca33 cache sync: move assert() behind declarations 2017-06-11 11:42:39 +02:00
textshell
86363dcd4b Merge pull request #2648 from textshell/feature/mandatory-features-master 
Add minimal version of in repository mandatory feature flags. (master)
 2017-06-10 17:50:28 +02:00
enkore
33a7331bda Merge pull request #2643 from enkore/f/experimental-patterns 
mark --pattern, --patterns-from as experimental
 2017-06-10 17:15:22 +02:00
enkore
13f396d5ad Merge pull request #2638 from enkore/f/fastcachesync-minify 
Compact chunks.archive.d
 2017-06-10 17:13:25 +02:00
enkore
d06ee5648c Merge pull request #2572 from enkore/f/fastcachesync 
Improve cache sync speed
 2017-06-10 17:12:51 +02:00
Martin Hostettler
e63808a63f Add tests for cache compatibility code. 2017-06-10 11:43:33 +02:00
Martin Hostettler
b8ad8b84da Cache: Wipe cache if compatibility is not sure 
Add detection of possibly incompatible combinations
of the borg versions maintaining the cache and the featues used.
 2017-06-10 11:42:48 +02:00
Martin Hostettler
005068dd6d Improve robustness of monkey patching borg.constants.PBKDF2_ITERATIONS. And add lots of warnings. 2017-06-10 11:42:42 +02:00
Marian Beermann
5cab72035a mark --pattern, --patterns-from as experimental 2017-06-10 10:29:02 +02:00
Marian Beermann
827c478500 nanorst for "borg help TOPIC" 2017-06-10 10:29:02 +02:00
Marian Beermann
0e31f78dd6 cache sync: avoid "l" and such as a variable name, note vanilla changes 2017-06-10 10:17:28 +02:00
Marian Beermann
5eb43b8464 cache sync: give overview of the source's structure 2017-06-10 10:17:28 +02:00
Marian Beermann
310a71e4f0 cache sync: use ro_buffer to accept bytes, memoryview, ... 2017-06-10 10:17:28 +02:00
Martin Hostettler
1176a1c4e4 permit manifest version 2 as well 1 one. 2017-06-09 23:22:18 +02:00
Martin Hostettler
193e8bcaef Add tests for mandatory repository feature flags. 2017-06-09 23:20:07 +02:00
Martin Hostettler
fd0250d34a Add minimal version of in repository mandatory feature flags. 
This should allow us to make sure older borg versions can be cleanly
prevented from doing operations that are no longer safe because of
repository format evolution. This allows more fine grained control than
just incrementing the manifest version. So for example a change that
still allows new archives to be created but would corrupt the repository
when an old version tries to delete an archive or check the repository
would add the new feature to the check and delete set but leave it out
of the write set.

This is somewhat inspired by ext{2,3,4} which uses sets for
compat (everything except fsck), ro-compat (may only be accessed
read-only by older versions) and features (refuse all access).
 2017-06-09 23:13:31 +02:00
Andrea Gelmini
e4247cc0d2 Fix typos 2017-06-09 16:49:30 +02:00
Marian Beermann
1d5d50463c hashindex_compact: use memmove for possibly overlapping copy 2017-06-09 12:23:27 +02:00
Marian Beermann
a75bfae2cf testsuite: Corruption test_chunks_archive, adapt for .compact 2017-06-09 12:23:27 +02:00
Marian Beermann
92a01f9d6c cache sync: fix incorrect .integrity location for .compact 2017-06-09 12:23:27 +02:00
Marian Beermann
3789459a41 cache sync: extract read_archive_index function 2017-06-09 12:23:26 +02:00
Marian Beermann
09a9d892cf cache sync: convert existing archive chunks idx to compact 2017-06-09 12:23:26 +02:00
Marian Beermann
295ac86d71 testsuite: hashindex: test compact -> merge 2017-06-09 12:23:26 +02:00
Marian Beermann
1379af22de HashIndexCompactTestCase 2017-06-09 12:23:26 +02:00
Marian Beermann
6e011b9354 cache: compact hashindex before writing to chunks.archive.d 2017-06-09 12:23:26 +02:00
TW
eeaeaf9e12 Merge pull request #2634 from ThomasWaldmann/cygwin 
enable remote tests on cygwin
 2017-06-09 00:08:01 +02:00
Marian Beermann
b75c214af5 hashindex: Cython defines PY_SSIZE_T_CLEAN 2017-06-08 15:16:52 +02:00
Thomas Waldmann
4766d66875 enable remote tests on cygwin 
the cygwin issue that caused these tests to break was fixed in cygwin
at least since cygwin 2.8.0 (maybe even since 2.7.0).

also added a comment to our workaround (os_write wrapper, that is needed
still for people running older cygwin versions) that it can be removed
when cygwin 2.8.0 is considered ancient (and everybody has upgraded to
some fixed version).
 2017-06-08 02:23:57 +02:00
enkore
19c8adb109 Merge pull request #2632 from enkore/docs/nanorst 
nanorst for --help
 2017-06-08 02:00:55 +02:00
Marian Beermann
5b9c34f523 borg.algorithms definition 2017-06-07 23:53:19 +02:00
Marian Beermann
3f8a0221ee Revert "move chunker to borg.algorithms" 
This reverts commit 956b50b29c.

# Conflicts:
#	setup.py
#	src/borg/archive.py
#	src/borg/helpers.py
 2017-06-07 23:51:42 +02:00
Marian Beermann
1cf031045c nanorst for --help 2017-06-07 18:16:11 +02:00
TW
af979c2b2a Merge pull request #2617 from ThomasWaldmann/fix-parse-version 
fix parse_version, add tests, fixes #2556
 2017-06-07 02:04:25 +02:00
Thomas Waldmann
a7e8e8ccd9 fix parse_version, add tests, fixes #2556 2017-06-07 01:15:58 +02:00
Marian Beermann
1135114520 helpers: truncate_and_unlink doc 2017-06-06 19:52:08 +02:00
Marian Beermann
ed0a5c798f platform.SaveFile: truncate_and_unlink temporary 
SaveFile is typically used for small files where this is not
necessary. The sole exception is the files cache.
 2017-06-06 18:13:20 +02:00
Marian Beermann
95064cd241 repository: truncate segments before unlinking 2017-06-06 17:21:45 +02:00
enkore
68b61acffd Merge pull request #2619 from ThomasWaldmann/remove-attic-dependency 
remove attic dependency of the tests, fixes #2505
 2017-06-06 11:43:52 +02:00
Thomas Waldmann
f5d2e67129 whitespace changes by coala 2017-06-06 04:46:15 +02:00
Thomas Waldmann
fed5873e29 remove attic dependency of the tests, fixes #2505 
attic.tar.gz contains a repo + corresponding keyfile - all the
upgrader module tests need.

.tar.gz because the .tar was 20x bigger.
 2017-06-06 04:26:48 +02:00
TW
88669611b1 Merge pull request #2615 from ThomasWaldmann/suppress-timestamp-future-errmsg 
tests: suppress tar's future timestamp warning
 2017-06-05 16:59:12 +02:00
Thomas Waldmann
4e6c56538a test: suppress tar's future timestamp warning 
in this case, it is expected as we archived a file with such a ts.
 2017-06-05 15:43:13 +02:00
Marian Beermann
909f099b1a algorithms.checksums: work around GCC 4.4 bug by disabling CLMUL 
Also disabling this code path for 4.5; 4.6 was tested iirc.
 2017-06-05 15:07:49 +02:00
TW
50bcd7843d recreate: keep timestamps as in original archive, fixes #2384 (#2607) 
the timestamps of the recreated archive (in the archive metadata and
also in the manifest) are now as they were for the original archive.

they are important metadata about the archive contents and should
therefore be kept "as is".

note: when using -v --stats, the timestamps shown there for recreate
      are about the recreate start/end/duration.
 2017-06-05 09:59:17 +02:00
enkore
60811fc2ce Merge pull request #2593 from enkore/issue/2384.1 
recreate: if single archive is not processed, exit 2
 2017-06-04 23:01:20 +02:00
Thomas Waldmann
ffcf6b76b6 DEFAULT_SEGMENTS_PER_DIR = 1000 
prettier increments for the directory names.
 2017-06-03 21:54:41 +02:00
TW
a2d08e3186 Merge pull request #2575 from edgimar/master 
patterns: don't recurse with ! / --exclude for pf:, fixes #2509
 2017-06-03 21:47:11 +02:00
Mark Edgington
fd0215c3c2 patterns: don't recurse with !/--exclude for path-prefix (pf:) 
Fixes issue #2509
 2017-06-03 12:13:32 -04:00
Marian Beermann
8ad309ae2a recreate: if single archive is not processed, exit 2 2017-06-03 15:47:01 +02:00
Marian Beermann
8dfe2a8080 remote: show path in PathNotAllowed 
not 100 % sure whether "if old_server" is required, so let's play it safe.
1.0 -> 1.1 server is no problem.
 2017-06-03 15:07:08 +02:00
Marian Beermann
07fbba4ee9 serve: add --restrict-to-repository 2017-06-03 15:07:08 +02:00
Marian Beermann
5af66dbb12 cache sync: add more refcount tests 2017-06-03 15:02:27 +02:00
Marian Beermann
4faaa7d1fa RepositoryCache: abort on data corruption 2017-06-03 12:27:35 +02:00
Marian Beermann
b544af2af1 RepositoryCache: checksum decrypted cache 2017-06-03 12:23:23 +02:00
Marian Beermann
54e023c75a repository: add complementary index corruption test 2017-06-02 21:44:45 +02:00
Marian Beermann
2e067a7ae8 repository: add refcount corruption test 2017-06-02 21:44:45 +02:00
Marian Beermann
f61ee038d0 repository: checksum index and hints 2017-06-02 21:44:45 +02:00
Marian Beermann
5b3667b617 cache sync: macros in all-caps 2017-06-02 19:31:56 +02:00
Marian Beermann
795cdfc9ab cache sync: move stat initialization to main unpack 2017-06-02 19:30:53 +02:00
Marian Beermann
cb98cb838d fuse: fix read(2) caching data in metadata cache 
The OS page cache is responsible for handling this and is much more
empowered to do a good job at that than Borg.
 2017-06-02 17:43:15 +02:00
Marian Beermann
67b97f2223 cache sync: cleanup progress handling, unused parameters 2017-06-02 17:43:15 +02:00
Marian Beermann
7f04e00ba2 testsuite: add TestRepositoryCache 2017-06-02 17:43:15 +02:00
Marian Beermann
835b0e5ee0 cache sync/remote: compressed, decrypted cache 2017-06-02 17:43:15 +02:00
Marian Beermann
c786a5941e CacheSynchronizer: redo as quasi FSM on top of unpack.h 
This is a (relatively) simple state machine running in the
data callbacks invoked by the msgpack unpacking stack machine
(the same machine is used in msgpack-c and msgpack-python,
changes are minor and cosmetic, e.g. removal of msgpack_unpack_object,
removal of the C++ template thus porting to C and so on).

Compared to the previous solution this has multiple advantages
- msgpack-c dependency is removed
- this approach is faster and requires fewer and smaller
  memory allocations

Testability of the two solutions does not differ in my
professional opinion(tm).

Two other changes were rolled up; _hashindex.c can be compiled
without Python.h again (handy for fuzzing and testing);
a "small" bug in the cache sync was fixed which allocated too
large archive indices, leading to excessive archive.chunks.d
disk usage (that actually gave me an idea).
 2017-06-02 17:43:15 +02:00
Marian Beermann
bf895950ac RepositoryCache: limit cache size 
Unbounded cache size is inacceptable.

I don't see why a full-fledged repository needs to be used here, either,
since this cache requires none of the consistency or durability guarantees
made by it (and bought with a performance impact).

A notable issue is that posix_fadvise is slow (for some reason) on tmpfs,
which could eat 30-35 % of the total CPU time of a cache sync.
 2017-06-02 17:43:15 +02:00
Marian Beermann
167875b753 cache sync: fix n^2 behaviour in lookup_name 2017-06-02 17:43:14 +02:00
Marian Beermann
9f8b967a6f cache sync: initialize master index to known capacity 2017-06-02 17:43:14 +02:00
Marian Beermann
740898d83b CacheSynchronizer 2017-06-02 17:43:14 +02:00
Marian Beermann
6c91a750d1 algorithms: rename crc32 to checksums 2017-06-01 21:26:42 +02:00
Marian Beermann
0221e31058 file_integrity: use xxh64 2017-06-01 21:26:42 +02:00
enkore
23d591c1c3 Merge pull request #2565 from enkore/issue/2517 
Implement storage quotas
 2017-06-01 14:39:17 +02:00
Marian Beermann
bcf4b4492b testsuite: add test for parse_storage_quota 2017-05-31 20:28:17 +02:00
Marian Beermann
4e6a771ee7 BORG_PASSCOMMAND: use same cmd-string splitting as BORG_RSH 2017-05-31 19:41:17 +02:00
enkore
349a4ade7c Merge pull request #2568 from enkore/issue/1101.integration.cache 
1101.integration.cache
 2017-05-31 19:34:43 +02:00
TuXicc
578b76af3a Added BORG_PASSCOMMAND environment variable (#2573) 2017-05-31 19:25:21 +02:00
Marian Beermann
f8b48dc8d7 remote: propagate Error.traceback correctly 2017-05-31 18:48:48 +02:00
Marian Beermann
4edf77788d Implement storage quotas 2017-05-31 18:36:03 +02:00
Marian Beermann
9032aa062b testsuite: simplify ArchiverCorruptionTestCase 2017-05-31 18:08:20 +02:00
Marian Beermann
0a5d9b6f7c cache sync: close archive chunks file before renaming 2017-05-31 18:06:28 +02:00
Marian Beermann
d35d388d9c cache integrity: handle interference from old versions 2017-05-25 17:44:01 +02:00
Marian Beermann
50ac9d914d testsuite: add ArchiverCorruptionTestCase 2017-05-25 17:44:01 +02:00
Marian Beermann
83bca02a4e file_integrity: hash_part: mix length into state 2017-05-25 17:44:01 +02:00
Marian Beermann
d463dd89aa hashindex: read/write: use hash_part for HashHeader 2017-05-25 17:44:01 +02:00
Marian Beermann
f59affe585 cache: fix possible printf issue with archive names in sync 2017-05-25 17:44:01 +02:00
Marian Beermann
addd7addfe cache: chunks.archive.d: autofix corruption 2017-05-25 17:44:01 +02:00
Marian Beermann
1dfe693003 cache: integrity checking in archive.chunks.d 2017-05-25 16:28:46 +02:00
Marian Beermann
2b518b7188 cache: add integrity checking of chunks and files caches 2017-05-25 16:28:46 +02:00
Marian Beermann
06cf15cc6d hashindex: read/write: accept file-like objects for path 2017-05-25 14:04:41 +02:00