the statically allocated COMPR_BUFFER was right size for chunks,
but not for the archive item which could get larger if you have
many millions of files/dirs.
every chunk has the encryption key type as first byte and we do not want to rewrite the whole repo
to change the passphrase type to repokey type. thus we simply dispatch this type to repokey
handler.
if there is a repokey that contains the same secrets as they were derived from the passphrase, it will just work.
if there is none yet, one needs to run migrate-to-repokey command to create it.
this is available in python 3.4+.
note:
before removing the pbkdf tests, i ran them with the pbkdf from stdlib to make sure it gives same result.
long term testing of this now belongs into stdlib tests, not into borg.
this was making us require mock, which is really a test component and
shouldn't be part of the runtime dependencies. furthermore, it was
making the imports and the code more brittle: it may have been
possible that, through an environment variable, backups could be
corrupted because mock libraries would be configured instead of real
once, which is a risk we shouldn't be taking.
finally, this was used only to build docs, which we will build and
commit to git by hand with a fully working borg when relevant.
see #384.
subclasses of "Error": do not show traceback
(this is used when a failure is expected and has rather trivial reasons and usually
does not need debugging)
subclasses of "ErrorWithTraceback": show a traceback
(this is for severe and rather unexpected stuff, like consistency / corruption issues
or stuff that might need debugging)
I reviewed all the Error subclasses whether they fit into the one or other class.
Also: fixed docstring typo, docstring formatting
instead of applying this only to usage generation, use it as a generic
mechanism to disable loading of Cython code.
it may be incomplete: there may be other places where Cython code is
loaded that is not checked, but that is sufficient to build the usage
docs. the environment variable used is documented as such in the
docs/usage.rst.
we also move the check to a helper function and document it
better. this has the unfortunate side effect of moving includes
around, but I can't think of a better way.
this is such a crude hack it is totally embarrassing....
the proper solution would probably be to move the `build_parser()`
function out of `Archiver` completely, but this is such an undertaking
that i doubt it is worth doing since we're looking at switching to
click anyways.
the main problem in moving build_parser() out is that it references
`self` all the time, so it *needs* an archiver context that it can
reuse. we could make the function static and pass self in there by
hand, but it seems like almost a worse hack... and besides, we would
need to load the archiver in order to do that, which would break usage
all over again...
the heuristics i used are the following:
1. if we are prompting the use, use print on stderr (input() may
produce some stuff on stdout, but it's outside the scope of this
patch). we do not want those prompts to end up on the standard
output in case we are piping stuff around
2. if the command is primarily producing output for the user on the
console (`list`, `info`, `help`), we simply print on the default
file descriptor.
3. everywhere else, we use the logging module with varying levels of
verbosity, as appropriate.
the logging level varies: most is logging.info(), in some place
logging.warning() or logging.error() are used when the condition is
clearly an error or warning. in other cases, we keep using print, but
force writing to sys.stderr, unless we interact with the user.
there were 77 calls to print before this commit, now there are 7, most
of which in the archiver module, which interacts directly with the
user. in one case there, we still use print() only because logging is
not setup properly yet during argument parsing.
it could be argued that commands like info or list should use print
directly, but we have converted them anyways, without ill effects on
the unit tests
unit tests still use print() in some places
this switches all informational output to stderr, which should help
with, if not fixjborg/attic#312 directly
New null and lz4 compression.
Giving -C 0 now uses null compression, not zlib level 0 any more
(null has almost zero overhead while zlib-level0 still had to package everything into zlib frames).
Giving -C 10 uses new lz4 compression, super fast compression and even faster decompression.
See borg create --help (and --compression argument).
fix some issues, clean up, optimize:
CNULL: always return bytes
LZ4: deal with getting memoryviews
Compressor: give bytes to detect(), avoid memoryviews
for lz4, always use same COMPR_BUFFER, avoid memory management costs.
check --chunker-params CHUNK_MAX_EXP upper limit
