Commit graph

857 commits

Author SHA1 Message Date
Thomas Waldmann
7e4d6cac0f fix LDLP restoration for subprocesses, fixes #3077
(cherry picked from commit b5069770b7)
2017-10-01 20:58:18 +02:00
Thomas Waldmann
eab9f5a07b implement files cache mode control, fixes #911
You can now control the files cache mode using this option:

--files-cache={ctime,mtime,size,inode,rechunk,disabled}*

(only some combinations are supported)

Previously, only these modes were supported:
- mtime,size,inode (default of borg < 1.1.0rc4)
- mtime,size (by using --ignore-inode)
- disabled (by using --no-files-cache)

Now, you additionally get:
- ctime alternatively to mtime (more safe), e.g.:
  ctime,size,inode (this is the new default of borg >= 1.1.0rc4)
- rechunk (consider all files as changed, rechunk them)

Deprecated:
- --ignore-inodes (use modes without "inode")
- --no-files-cache (use "disabled" mode)

The tests needed some changes:
- previously, we use os.utime() to set a files mtime (atime) to specific
  values, but that does not work for ctime.
- now use time.sleep() to create the "latest file" that usually does
  not end up in the files cache (see FAQ)

(cherry picked from commit 5e2de8ba67)
2017-10-01 01:27:18 +02:00
Thomas Waldmann
f2b48cd5c8 remote: deal with partial lines, fixes #2637
due to block buffering (in borg, pipes, sshd, ssh) partial lines might
be received. for plain text, this causes cosmetic issues, for json it
causes tracebacks due to parse errors.

the code now makes sure handle_remote_line() only gets called with a
complete line (which is terminated by any universal newline char, a
pure \r seems to be needed for remote progress displays).

it also fixes a yet undiscovered partial utf-8-sequence decoding issue
that might occur for the same reason.

(cherry picked from commit 8646216a06)
2017-09-26 22:06:13 +02:00
Thomas Waldmann
37a5442bef use prepared env for xattr module's fakeroot version check
(cherry picked from commit a57e23fdb3)
2017-09-25 05:54:50 +02:00
Thomas Waldmann
d7a5ca2e6a use prepared env for borg export-tar --tar-filter subprocess
(cherry picked from commit cf59f653e5)
2017-09-25 05:54:30 +02:00
Thomas Waldmann
d6f810560c use prepared env for borg umount
(cherry picked from commit b88da10641)
2017-09-25 05:53:56 +02:00
Thomas Waldmann
3b20c18c3b use prepared env for borg with-lock
(cherry picked from commit 6da5bf4b85)
2017-09-25 05:51:53 +02:00
Thomas Waldmann
2f7d7bdbb8 use prepared env for calling BORG_PASSCOMMAND, fixes #3050
(cherry picked from commit 6a6fd31804)
2017-09-25 05:51:02 +02:00
Thomas Waldmann
5dd16672c0 refactor/fix subprocess env preparation
refactor: make a generally usable function

fix: remove support code for ancient pyinstaller

the "else" branch was needed for pyinstaller < 20160820 because it did
not have the LD_LIBRARY_PATH_ORIG env var, so we just killed LDLP
because we had no better way.

but with borg tests running under fakeroot, this is troublesome as
fakeroot uses this also and can't find its library without it.

so, just remove it, we do not need to support old pyinstaller.

(cherry picked from commit ba941b0801)
2017-09-25 05:50:19 +02:00
Thomas Waldmann
f48f86444d remove client_supports_log_v3 flag, fixes #3033
the client_supports_log_v3 flag was added to differentiate 1.1.0 beta3
to beta5 clients (which did not support parsing json log format from
server) from >= 1.1.0beta6 clients (which support it).
for clients older than 1.1.0b3, no json log format will be negotiated
anyway.

by removing the client_supports_log_v3 flag support, we drop support for
clients using 1.1.0beta3..5.

thus, a client is now expected to either support old log format (like
borg 1.0.x) or new json format (like borg 1.1.0 >= beta6).

client     server comment
===========================================
any        0.29+  uses $LOG plain remote log format
any        1.0.x  uses $LOG plain remote log format
1.0.x      1.1.0  uses $LOG plain remote log format
1.1.0b1/b2 1.1.0  (uses $LOG plain remote log format)
1.1.0b3-b5 1.1.0  (malfunction)
1.1.0b6    1.1.0  (uses json remote log format)
1.1.0rc    1.1.0  uses json remote log format
1.1.x      1.1.0  uses json remote log format

(beta testing is over and betas are unsupported now)

Note: client_supports_log_v3 flag was added in changeset
      18a2902c9c

(cherry picked from commit 54c5049fb9)
2017-09-25 02:41:15 +02:00
Thomas Waldmann
b6abee4d68 docs: change-passphrase only changes the passphrase, fixes #2990
(cherry picked from commit 713be765d1)
2017-09-23 21:30:18 +02:00
Thomas Waldmann
f44fd6ba19 flush json mode progress stderr output
if borg stderr is not connected to a tty, but to ssh (when using
borg client/server), sys.stderr is block buffered (tty: line buffered).

thus we better flush explicitly after emitting a line as the receiving
side can not handle partial json at the end of the block.

also, it might solve some delays, when output didn't arrive at
receiving side in time.

(cherry picked from commit 2b75b278da)
2017-09-21 19:18:40 +02:00
enkore
ecc5ceec07 delete: support naming multiple archives (#3017)
(cherry picked from commit 7c5a9d89b2)
2017-09-09 23:28:47 +02:00
TW
319606e1cf Merge pull request #3008 from ThomasWaldmann/fix-2994-1.1
fix .isoformat() issues (1.1-maint)
2017-09-07 16:00:59 +02:00
TW
af407e3e7f Merge pull request #3007 from ThomasWaldmann/fd-cache-invalidation-1.1
FD cache invalidation (1.1-maint)
2017-09-07 15:56:26 +02:00
Thomas Waldmann
da2f8dbe81 get rid of datetime.isoformat to avoid bugs like #2994
(cherry picked from commit 928bde8676)
2017-09-07 15:11:53 +02:00
Thomas Waldmann
a9aa3c5f34 use safe parse_timestamp to parse timestamps, fixes #2994
also: refactor so it is possible to get tz-unaware datetime
objects from parse_timestamp.

(cherry picked from commit 7996a87357)
2017-09-07 15:06:23 +02:00
Thomas Waldmann
457f5ceb30 use ISO_FORMAT* constants
(cherry picked from commit eebb117349)
2017-09-07 15:02:29 +02:00
Thomas Waldmann
764324304d move ISO_FORMAT to constants module
also: add ISO_FORMAT_NO_USECS
(cherry picked from commit 1cb158a4b5)
2017-09-07 14:56:27 +02:00
Thomas Waldmann
88e1340445 repo cleanup/write: invalidate cached FDs
(cherry picked from commit 7122913825)
2017-09-07 14:31:07 +02:00
Thomas Waldmann
2a58fe4266 repo: add test case for uncommitted garbage segment files
(cherry picked from commit 4a4c8884ee)
2017-09-07 14:29:37 +02:00
Thomas Waldmann
eeaa43b668 with-lock: close segment file before invoking subprocess
(cherry picked from commit b9dce0ebdc)
2017-09-07 14:25:22 +02:00
TW
93cedbb761 Merge pull request #2993 from ThomasWaldmann/issue/2975-1.1
test_detect_attic_repo: don't test mount
2017-09-02 19:17:29 +02:00
TW
c77dd42302 Merge pull request #2992 from ThomasWaldmann/cleanup-log-1.1
add debug logging for repository cleanup
2017-09-02 18:52:05 +02:00
Marian Beermann
f85494d574 test_detect_attic_repo: don't test mount
since mount is not always available and if it works for all the other
commands, then it is likely it works for mount as well.

(cherry picked from commit a6be34f8f7)
2017-09-02 17:57:42 +02:00
Thomas Waldmann
9ca490ad5e add debug logging for repository cleanup
so we can know whether it did a cleanup and if so,
which and how many segments were cleaned up.

(cherry picked from commit 57f808e4bb)
2017-09-02 17:55:16 +02:00
Thomas Waldmann
9a362aecc4 recover_segment: use mmap(), fixes #2982
(cherry picked from commit 9fc4d00bf6)
2017-09-02 17:50:50 +02:00
Marian Beermann
35c8975a5c keymanager: don't depend on optional readline module
(cherry picked from commit b8793d9577)
2017-08-30 11:29:57 +02:00
TW
84d13f92b3 Merge pull request #2968 from ThomasWaldmann/fix-timestamp-option-1.1
borg create --timestamp: set start time, fixes #2957
2017-08-27 15:19:26 +02:00
TW
1b7b58e712 Merge pull request #2964 from ThomasWaldmann/detect-attic-repos-1.1
Detect non-upgraded Attic repositories
2017-08-27 14:52:53 +02:00
TW
143d82c640 Merge pull request #2965 from ThomasWaldmann/issue/2376-1.1
Issue/2376 PR backported to 1.1
2017-08-27 14:51:23 +02:00
Thomas Waldmann
ac404e3a74 borg create --timestamp: set start time, fixes #2957
(cherry picked from commit 8a299ae24c)
2017-08-25 04:38:24 +02:00
Marian Beermann
987a99dffe create: document exclusion through nodump
(cherry picked from commit 495f838d88)
2017-08-25 04:25:18 +02:00
Marian Beermann
d5697fb4a8 always use microseconds for ISO 8601 output
(cherry picked from commit ab4981eff6)
2017-08-25 04:20:06 +02:00
Marian Beermann
008571228f one datetime formatter to rule them all
(cherry picked from commit a836f451ab)
2017-08-25 04:00:26 +02:00
Marian Beermann
fa65c9b143 list: fix weird mixup of mtime/isomtime
(cherry picked from commit 2ff29891f1)
2017-08-25 03:38:30 +02:00
Lukas Fleischer
ff93b6b972 Detect non-upgraded Attic repositories
When opening a repository, always try to read the magic number of the
latest segment and compare it to the Attic segment magic (unless the
repository is opened for upgrading). If an Attic segment is detected,
raise a dedicated exception, telling the user to upgrade the repository
first.

Fixes #1933.

(cherry picked from commit 0943b322e3)
2017-08-25 03:30:31 +02:00
enkore
818e5c8e01 Merge pull request #2952 from enkore/f/killthreads
delete various nogil and threading related lines
2017-08-21 12:19:03 +02:00
Marian Beermann
dc6049cb73 delete various nogil and threading related lines 2017-08-20 21:30:41 +02:00
Thomas Waldmann
4eadb59c10 ignore corrupt files cache, fixes #2939
ignore the files cache when corrupt and emit a warning message
so the users notices that there is a problem.

(cherry picked from commit 5beaa5bd02)
2017-08-19 01:09:26 +02:00
Thomas Waldmann
09e3a02fbc migrate locks to child PID when daemonize is used
also:

increase platform api version due to change in get_process_id behaviour.

(cherry picked from commit 6f94949a36)
(cherry picked from commit 5bad764637)
2017-08-13 21:40:56 +02:00
Thomas Waldmann
eb7d473dc7 test json timestamps for iso format
(cherry picked from commit c3b0214e89)
2017-08-07 21:56:39 +02:00
Thomas Waldmann
74c9277d76 repo last_modified: use iso8601 timestamp format with --json
like yyyy-mm-ddThh:mm:ss - no tz yet, this likely needs more refactoring
to tz aware and utc datetime objects everywhere, currently there are
naive datetime objects and also localtime at quite some places.

(cherry picked from commit 32174dd9c8)
2017-08-07 21:56:28 +02:00
Thomas Waldmann
65940be21a archives list: use iso8601 timestamp format with --json
like yyyy-mm-ddThh:mm:ss - no tz yet, this likely needs more refactoring
to tz aware and utc datetime objects everywhere, currently there are
naive datetime objects and also localtime at quite some places.

(cherry picked from commit b64561fe6f)
2017-08-07 21:56:15 +02:00
Thomas Waldmann
d17553add6 archive listing: use iso8601 timestamp format with --json-lines
like yyyy-mm-ddThh:mm:ss - no tz yet, this likely needs more refactoring
to tz aware and utc datetime objects everywhere, currently there are
naive datetime objects and also localtime at quite some places.

(cherry picked from commit 043d794b91)
2017-08-07 21:56:02 +02:00
enkore
51bcba332b Merge pull request #2921 from ThomasWaldmann/fix-serve-exitcode-1.1
fix exitcode of borg serve, fixes #2910
2017-08-07 12:21:58 +02:00
Thomas Waldmann
68d505c974 fix exitcode of borg serve, fixes #2910
(cherry picked from commit 583de3eeb1)
2017-08-06 17:34:10 +02:00
Simon Frei
a1bffc193b Only compare contents when chunker params match (fixes #2899) 2017-08-06 01:24:53 +02:00
enkore
daa88e07f2 Merge pull request #2877 from Alexander-N/pylint-rules
Activate more linting rules in .coafile
2017-07-25 09:56:25 +02:00
enkore
c1d7cd9b90 Merge pull request #2873 from enkore/issue/2869
with-lock, info docs
2017-07-24 23:50:51 +02:00