Commit graph

205 commits

Author SHA1 Message Date
Narendra Vardi
60d2b9c193 add BORG_CONFIG_DIR env (#3083)
(cherry picked from commit 56fed4f964)
2017-11-03 18:37:56 +01:00
Thomas Waldmann
a68d463041 don't crash in first part of truncate_and_unlink, fixes #3117
(cherry picked from commit 7a689b1295)
2017-10-11 00:16:51 +02:00
Thomas Waldmann
7e4d6cac0f fix LDLP restoration for subprocesses, fixes #3077
(cherry picked from commit b5069770b7)
2017-10-01 20:58:18 +02:00
Thomas Waldmann
eab9f5a07b implement files cache mode control, fixes #911
You can now control the files cache mode using this option:

--files-cache={ctime,mtime,size,inode,rechunk,disabled}*

(only some combinations are supported)

Previously, only these modes were supported:
- mtime,size,inode (default of borg < 1.1.0rc4)
- mtime,size (by using --ignore-inode)
- disabled (by using --no-files-cache)

Now, you additionally get:
- ctime alternatively to mtime (more safe), e.g.:
  ctime,size,inode (this is the new default of borg >= 1.1.0rc4)
- rechunk (consider all files as changed, rechunk them)

Deprecated:
- --ignore-inodes (use modes without "inode")
- --no-files-cache (use "disabled" mode)

The tests needed some changes:
- previously, we use os.utime() to set a files mtime (atime) to specific
  values, but that does not work for ctime.
- now use time.sleep() to create the "latest file" that usually does
  not end up in the files cache (see FAQ)

(cherry picked from commit 5e2de8ba67)
2017-10-01 01:27:18 +02:00
Thomas Waldmann
d6f810560c use prepared env for borg umount
(cherry picked from commit b88da10641)
2017-09-25 05:53:56 +02:00
Thomas Waldmann
5dd16672c0 refactor/fix subprocess env preparation
refactor: make a generally usable function

fix: remove support code for ancient pyinstaller

the "else" branch was needed for pyinstaller < 20160820 because it did
not have the LD_LIBRARY_PATH_ORIG env var, so we just killed LDLP
because we had no better way.

but with borg tests running under fakeroot, this is troublesome as
fakeroot uses this also and can't find its library without it.

so, just remove it, we do not need to support old pyinstaller.

(cherry picked from commit ba941b0801)
2017-09-25 05:50:19 +02:00
Thomas Waldmann
f44fd6ba19 flush json mode progress stderr output
if borg stderr is not connected to a tty, but to ssh (when using
borg client/server), sys.stderr is block buffered (tty: line buffered).

thus we better flush explicitly after emitting a line as the receiving
side can not handle partial json at the end of the block.

also, it might solve some delays, when output didn't arrive at
receiving side in time.

(cherry picked from commit 2b75b278da)
2017-09-21 19:18:40 +02:00
Thomas Waldmann
da2f8dbe81 get rid of datetime.isoformat to avoid bugs like #2994
(cherry picked from commit 928bde8676)
2017-09-07 15:11:53 +02:00
Thomas Waldmann
a9aa3c5f34 use safe parse_timestamp to parse timestamps, fixes #2994
also: refactor so it is possible to get tz-unaware datetime
objects from parse_timestamp.

(cherry picked from commit 7996a87357)
2017-09-07 15:06:23 +02:00
Thomas Waldmann
457f5ceb30 use ISO_FORMAT* constants
(cherry picked from commit eebb117349)
2017-09-07 15:02:29 +02:00
Marian Beermann
d5697fb4a8 always use microseconds for ISO 8601 output
(cherry picked from commit ab4981eff6)
2017-08-25 04:20:06 +02:00
Marian Beermann
008571228f one datetime formatter to rule them all
(cherry picked from commit a836f451ab)
2017-08-25 04:00:26 +02:00
Marian Beermann
dc6049cb73 delete various nogil and threading related lines 2017-08-20 21:30:41 +02:00
Thomas Waldmann
09e3a02fbc migrate locks to child PID when daemonize is used
also:

increase platform api version due to change in get_process_id behaviour.

(cherry picked from commit 6f94949a36)
(cherry picked from commit 5bad764637)
2017-08-13 21:40:56 +02:00
Thomas Waldmann
74c9277d76 repo last_modified: use iso8601 timestamp format with --json
like yyyy-mm-ddThh:mm:ss - no tz yet, this likely needs more refactoring
to tz aware and utc datetime objects everywhere, currently there are
naive datetime objects and also localtime at quite some places.

(cherry picked from commit 32174dd9c8)
2017-08-07 21:56:28 +02:00
Thomas Waldmann
65940be21a archives list: use iso8601 timestamp format with --json
like yyyy-mm-ddThh:mm:ss - no tz yet, this likely needs more refactoring
to tz aware and utc datetime objects everywhere, currently there are
naive datetime objects and also localtime at quite some places.

(cherry picked from commit b64561fe6f)
2017-08-07 21:56:15 +02:00
Thomas Waldmann
d17553add6 archive listing: use iso8601 timestamp format with --json-lines
like yyyy-mm-ddThh:mm:ss - no tz yet, this likely needs more refactoring
to tz aware and utc datetime objects everywhere, currently there are
naive datetime objects and also localtime at quite some places.

(cherry picked from commit 043d794b91)
2017-08-07 21:56:02 +02:00
Marian Beermann
2fe37dba7f umount: try fusermount, then try umount 2017-07-24 13:55:32 +02:00
TW
e36be956f1 Merge pull request #2792 from ThomasWaldmann/comment-placeholder
support borg list repo --format {comment}, fixes #2081
2017-07-06 03:12:01 +02:00
Thomas Waldmann
852505fb69 ItemFormatter: code defensively, always use new dict for returned object 2017-07-05 23:37:42 +02:00
Thomas Waldmann
75c602d294 support borg list repo --format {comment}, fixes #2081
Also supported: {bcomment} for binary and {end} for backup end time.

Refactor so ArchiveFormatter works similar to ItemFormatter.
2017-07-05 23:37:42 +02:00
enkore
6ffa797bde Merge pull request #2791 from ThomasWaldmann/dash_open
WIP dash_open
2017-07-05 10:02:34 +02:00
Thomas Waldmann
a96dede632 dash_open: generalized and renamed open_file_or_stdin 2017-07-05 01:58:31 +02:00
TW
9ca666907b use limited unpacker for outer key, fixes #2174 (#2790)
* use limited unpacker for outer key, fixes #2174

* fixup: higher inner key data size limit
2017-07-04 22:51:31 +02:00
TW
9bd522819e Merge pull request #2778 from enkore/f/fuse-versions-numbering
fuse: versions view, linear numbering by archive time
2017-07-03 22:43:07 +02:00
Thomas Waldmann
ac98fe3fbe prune: fix checkpoints processing with --glob-archives
before the fix, the archives_checkpoints list could not always contain
checkpoints as the glob regex matched the end of the name, so there
could be no additional ".checkpoint" after the match.
2017-07-03 19:34:48 +02:00
Marian Beermann
9a856533ba fuse: versions view, linear numbering by archive time 2017-07-03 12:38:10 +02:00
Marian Beermann
41b52ff0f5 Archives.list: reverse result after applying all other filters 2017-07-03 00:02:26 +02:00
Marian Beermann
2db377d6fb fix --last, --first accepting negative values
also don't accept zero, because every use of these doesn't cover that case,
and it arguably doesn't make a lot of sense.
2017-07-03 00:02:26 +02:00
Marian Beermann
51c1c22c1f fix reversed archive ordering with --last 2017-07-02 22:48:54 +02:00
Marian Beermann
29646d5b5e key import: allow reading from stdin 2017-06-27 16:18:21 +02:00
Thomas Waldmann
89f3cab6cd move get_limited_unpacker to helpers
also: move some constants to borg.constants
2017-06-25 23:36:28 +02:00
Thomas Waldmann
6c2c51939d Manifest: use limited unpacker 2017-06-25 23:36:28 +02:00
Ed Blackman
9c5425dda8 Split up interval parsing from filtering for --keep-within
Fixes #2610

Parse --keep-within argument early, via new validator method interval
passed to argparse type=, so that better error messages can be given.

Also swallows ValueError stacktrace per the comment in the old code that
including it wasn't desirable.
2017-06-25 12:10:13 -04:00
TW
7df6597fa1 Merge pull request #2654 from enkore/f/no-cache-sync
create: --no-cache-sync
2017-06-18 03:22:06 +02:00
Marian Beermann
125b02c5c8 docs: list: fix unintended block quote 2017-06-18 02:10:10 +02:00
Marian Beermann
8aa745ddbd create: --no-cache-sync 2017-06-18 02:01:26 +02:00
Thomas Waldmann
72ef24cbc0 hashindex: implement KeyError 2017-06-17 23:25:32 +02:00
Marian Beermann
e189a4d302 info: use CacheSynchronizer & HashIndex.stats_against 2017-06-13 14:34:10 +02:00
Marian Beermann
5f5371f0b1 implement --glob-archives/-a 2017-06-11 12:15:12 +02:00
textshell
86363dcd4b Merge pull request #2648 from textshell/feature/mandatory-features-master
Add minimal version of in repository mandatory feature flags. (master)
2017-06-10 17:50:28 +02:00
enkore
13f396d5ad Merge pull request #2638 from enkore/f/fastcachesync-minify
Compact chunks.archive.d
2017-06-10 17:13:25 +02:00
enkore
d06ee5648c Merge pull request #2572 from enkore/f/fastcachesync
Improve cache sync speed
2017-06-10 17:12:51 +02:00
Martin Hostettler
b8ad8b84da Cache: Wipe cache if compatibility is not sure
Add detection of possibly incompatible combinations
of the borg versions maintaining the cache and the featues used.
2017-06-10 11:42:48 +02:00
Martin Hostettler
1176a1c4e4 permit manifest version 2 as well 1 one. 2017-06-09 23:22:18 +02:00
Martin Hostettler
fd0250d34a Add minimal version of in repository mandatory feature flags.
This should allow us to make sure older borg versions can be cleanly
prevented from doing operations that are no longer safe because of
repository format evolution. This allows more fine grained control than
just incrementing the manifest version. So for example a change that
still allows new archives to be created but would corrupt the repository
when an old version tries to delete an archive or check the repository
would add the new feature to the check and delete set but leave it out
of the write set.

This is somewhat inspired by ext{2,3,4} which uses sets for
compat (everything except fsck), ro-compat (may only be accessed
read-only by older versions) and features (refuse all access).
2017-06-09 23:13:31 +02:00
Andrea Gelmini
e4247cc0d2 Fix typos 2017-06-09 16:49:30 +02:00
Marian Beermann
6e011b9354 cache: compact hashindex before writing to chunks.archive.d 2017-06-09 12:23:26 +02:00
Marian Beermann
3f8a0221ee Revert "move chunker to borg.algorithms"
This reverts commit 956b50b29c.

# Conflicts:
#	setup.py
#	src/borg/archive.py
#	src/borg/helpers.py
2017-06-07 23:51:42 +02:00
Marian Beermann
1135114520 helpers: truncate_and_unlink doc 2017-06-06 19:52:08 +02:00