Note: although I hopefully fixed all the conflicts,
some tests are quite broken.
Conflicts:
borg/_chunker.c
borg/archive.py
borg/archiver.py
borg/cache.py
borg/helpers.py
borg/testsuite/archiver.py
refactorings:
- introduced concept of default answer:
if the answer string is in the defaultish sequence, the return value of yes() will be the default.
e.g. if just pressing <enter> when asked on the console or if an empty string or "default" is
in the environment variable for overriding.
if an environment var has an invalid value and no retries are enabled: return default
if retries are enabled, next retry won't use the env var again, but either ask via input().
- simplify:
only one default - this should be a SAFE default as it is used in some special conditions
like EOF or invalid input with retries disallowed.
no isatty() magic, the "yes" shell command exists, so we could receive input even if it is not from a tty.
- clean:
separate retry flag from retry_msg
The fnmatch module in Python's standard library implements a pattern
format for paths which is similar to shell patterns. However, “*”
matches any character including path separators. This newly introduced
pattern syntax with the selector “sh” no longer matches the path
separator with “*”. Instead “**/” can be used to match zero or more
directory levels.
The “extract” command supports extracting all files underneath a given
set of prefix paths. The forthcoming support for extracting files using
a pattern (i.e. only files ending in “.zip”) requires the introduction
of path prefixes as a third pattern style, making it also available for
exclusions.
A function to parse pattern specifications was introduced in commit
2bafece. Since then it had a hardcoded default style of “fm”, meaning
fnmatch. With the forthcoming support for extracting files using
patterns this default style must be more flexible.
The utility functions “adjust_patterns” and “exclude_path” produce
respectively use a standard list object containing pattern objects.
With the forthcoming introduction of patterns for filtering files
to be extracted it's better to move the logic of these classes into
a single class.
The wrapper allows adding any number of patterns to an internal list
together with a value to be returned if a match function finds that
one of the patterns matches. A fallback value is returned otherwise.
The class names “IncludePattern” and “ExcludePattern” may have been
appropriate when they were the only styles. With the recent addition of
regular expression support and with at least one more style being added
in forthcoming changes these classes should be renamed to be more
descriptive. “ExcludeRegex” is also renamed to match the new names.
Patterns to exclude files can be loaded from a text file using the
“--exclude-from” option. Whitespace at the beginning or end of lines was
not stripped. Indented comments would be interpreted as a pattern and
a misplaced space at the end of a line--some text editors don't strip
them--could cause an exclusion pattern to not match as desired. With the
recent addition of regular expression support for exclusions the spaces
can be matched if necessary (“^\s” or “\s$”), though it's highly
unlikely that there are many paths deliberately starting or ending with
whitespace.
The existing option to exclude files and directories, “--exclude”, is
implemented using fnmatch[1]. fnmatch matches the slash (“/”) with “*”
and thus makes it impossible to write patterns where a directory with
a given name should be excluded at a specific depth in the directory
hierarchy, but not anywhere else. Consider this structure:
home/
home/aaa
home/aaa/.thumbnails
home/user
home/user/img
home/user/img/.thumbnails
fnmatch incorrectly excludes “home/user/img/.thumbnails” with a pattern
of “home/*/.thumbnails” when the intention is to exclude “.thumbnails”
in all home directories while retaining directories with the same name
in all other locations.
With this change regular expressions are introduced as an additional
pattern syntax. The syntax is selected using a prefix on “--exclude”'s
value. “re:” is for regular expression and “fm:”, the default, selects
fnmatch. Selecting the syntax is necessary when regular expressions are
desired or when the desired fnmatch pattern starts with two alphanumeric
characters followed by a colon (i.e. “aa:something/*”). The exclusion
described above can be implemented as follows:
--exclude 're:^home/[^/]+/\.thumbnails$'
The “--exclude-from” option permits loading exclusions from a text file
where the same prefixes can now be used, e.g. “re:\.tmp$”.
The documentation has been extended and now not only describes the two
pattern styles, but also the file format supported by “--exclude-from”.
This change has been discussed in issue #43 and in change request #497.
[1] https://docs.python.org/3/library/fnmatch.html
Signed-off-by: Michael Hanselmann <public@hansmi.ch>
The two classes for applying inclusion and exclusion patterns contained
unnecessarily duplicated logic. The introduction of a shared base class
allows for easier reuse, especially considering that two more classes
are going to be added in forthcoming changes (regular expressions and
shell-style patterns).
prune and create now both require --verbose --stats to show stats.
it was implemented in this way (and not with print) so you can feed the stats data
into the logging system, too.
delete now says "Archive deleted" in verbose mode (for consistency,
it already said "Repository deleted" when deleting a repo).
also: add helpers.log_multi to comfortably and prettily output a block of log lines
The parsing code for exclude files (given via `--exclude-from`) was not
tested. Its core is factorized into a separate function to facilitate an
easier test. The observable behaviour is unchanged.
The read_msgpack and write_msgpack functions were only used in one place
each. Since msgpack is read and written in lots of places, having
functions with these generic names is confusing. Also, the helpers
module is quite a mess, so reducing its size seems to be a good idea.
this was making us require mock, which is really a test component and
shouldn't be part of the runtime dependencies. furthermore, it was
making the imports and the code more brittle: it may have been
possible that, through an environment variable, backups could be
corrupted because mock libraries would be configured instead of real
once, which is a risk we shouldn't be taking.
finally, this was used only to build docs, which we will build and
commit to git by hand with a fully working borg when relevant.
see #384.
We also add --keep-tag-files to keep in the archive the root directory and the
tag/exclusion file in the archive.
This is taken from a attic PR (and adapted for borg):
commit f61e22cacc90e76e6c8f4b23677eee62c09e97ac
Author: Yuri D'Elia <yuri.delia@eurac.edu>
Date: Mon Dec 15 12:27:43 2014 +0100
Add a new --exclude-if-present command-line flag to ``borg create``. If
specified, directories containing the specified tag file will be excluded from
the backup. The flag can be repeated to ignore more than a single tag file,
irregardless of the contents.
This is taken from a attic PR (and adapted for borg):
commit 3462a9ca90388dc5d8b4fa4218a32769676b3623
Author: Yuri D'Elia <yuri.delia@eurac.edu>
Date: Sun Dec 7 19:15:17 2014 +0100
subclasses of "Error": do not show traceback
(this is used when a failure is expected and has rather trivial reasons and usually
does not need debugging)
subclasses of "ErrorWithTraceback": show a traceback
(this is for severe and rather unexpected stuff, like consistency / corruption issues
or stuff that might need debugging)
I reviewed all the Error subclasses whether they fit into the one or other class.
Also: fixed docstring typo, docstring formatting
the reason for a slow msgpack can be:
- pip install: missing compiler (gcc)
- pip install: missing compiler parts (e.g. gcc-c++)
- pip install: cached wheel package that was built while the compiler wasn't present
- distribution package: badly built msgpack package
