Commit graph

633 commits

Author SHA1 Message Date
Oleg Drokin
ce176b52bf Workround a bug in Linux fadvise FADV_DONTNEED, fixes #907
Despite what the man page says, Linux does not discard the initial
partial page only. The ending page would be truncated no matter if
it is partial or not.
Page-align the fadvise size to take care of this.
Also while we are at it, roll back initial fadvise offset to the
previous page boundary to actually throw away that page as we
no longer need it having read the second part now and the first
time in the previous call.

This patch has a noticeable impact in my Linux testing when the file
is on the rotating media. The total test runtime decreased by a bit
over 10%, but since over half of that time was actually cpu time,
the actual iowait time decreased around 20%.
2016-04-15 20:16:47 -04:00
Marian Beermann
29ebdbadae
refcounting: use uint32_t, protect against overflows, fix merging for BE 2016-04-14 23:38:56 +02:00
Marian Beermann
c90745cdbb Port hashindex_summarize into ChunkIndex.summarize 2016-04-14 11:46:12 +02:00
Danny Edel
60ca65a438 reproducible doc for ProgressIndicator*
building the API docs captured sys.stderr's IO charset, making the build
unreproducible.
2016-04-13 14:54:34 +02:00
Marian Beermann
90a9fbd21d
hashindex_summarize: fix missing byte-order conversion
Fixes #886
2016-04-11 22:22:24 +02:00
Marian Beermann
d12c1deece
borg delete: fix --cache-only for broken caches
This also makes --cache-only idempotent: it won't fail if the cache
is already deleted.

Fixes #874
2016-04-11 12:50:42 +02:00
Björn Ketelaars
fd10d1386c fix issue #873 2016-04-10 16:24:58 +02:00
Thomas Waldmann
7861de3930 reorder log level options, so --info shows up in help 2016-04-08 22:55:07 +02:00
Marian Beermann
29bc396559
borg create: add --ignore-inode option
This is mainly meant for use with network file systems like sshfs and
possibly CIFS, which don't convey proper inode numbers.
2016-04-06 23:23:14 +02:00
Marian Beermann
b59f92a533 Test suite: Reset exit code of persistent archiver
#844
2016-04-06 10:04:35 +02:00
TW
515b7454a3 Merge pull request #839 from enkore/issue/791
If BORG_PASSPHRASE is present but wrong, don't prompt for password, f…
2016-04-04 21:47:21 +02:00
TW
9a7bad295c Merge pull request #838 from enkore/issue/792
Remote: don't print tracebacks for Error exceptions handled downstream
2016-04-04 21:37:04 +02:00
Marian Beermann
890e44ef60 RemoteRepository: clean up pipe if remote open() fails 2016-04-04 17:10:14 +02:00
Marian Beermann
425a4d819d Remote: don't print tracebacks for Error exceptions handled downstream
(What we'd really want is to have proper exception transporting over
 the RPC)

Fixes #792
2016-04-04 17:09:52 +02:00
Marian Beermann
8f716d8ca1 If BORG_PASSPHRASE is present but wrong, don't prompt for password, fail instead.
Leaves PassphraseKey alone, since I cannot test it.

Fixes #791
2016-04-03 21:37:15 +02:00
TW
a519333376 add --warning, --error, --critical for completeness, fixes #826
it's not recommended to suppress warnings or errors,
but the user may decide this on his own.

note: --warning is not given to borg serve so a <= 1.0.0 borg
will still work as server. it is not needed as it is the default.
2016-04-03 20:20:48 +02:00
TW
7b6ae60df5 Merge pull request #834 from enkore/issue/826
ArchiveChecker: move "orphaned objects check skipped" to INFO log level
2016-04-03 19:15:30 +02:00
Marian Beermann
d7299c9230 ArchiveChecker: move "orphaned objects check skipped" to INFO log level
Fixes #826
2016-04-03 18:23:40 +02:00
TW
fc10ac0ef9 Merge pull request #830 from enkore/feature/1.0ctxmng
Make Repository a context manager, use decorators for wrapping withs
2016-04-03 17:36:38 +02:00
Marian Beermann
83586ae09c Make Repository a context manager, use decorators for wrapping withs
(Remote)Repository.close() is not a public API anymore, but a private
API. It shall not be used from within other classes than Repository
or it's tests. The proper way is to use a context manager now. However,
for RPC/Remote compatibility with Borg 1.0 it is kept and unchanged.

Repositories are not opened by __init__ now anymore, it is done
by binding it to a context manager. (This SHOULD be compatible both ways
with remote, since opening the repo is handled by a RepositoryServer method)

Decorators @with_repository() and @with_archive simplify
context manager handling and remove unnecessary indentation.

Backported to 1.0-maint
2016-04-03 17:14:43 +02:00
TW
8d9dbc6901 Merge pull request #820 from ipha/master
fix incorrect _ns variables
2016-03-31 23:53:19 +02:00
TW
ad57178ad7 Merge pull request #819 from enkore/fix/chunker-io2
Chunker: fix wrong EOF assumption, check for return type
2016-03-31 23:45:47 +02:00
Mitch Bigelow
f406bf7879 fix incorrect _ns variables 2016-03-31 17:31:53 -04:00
Marian Beermann
061bf59d5d Chunker: fix wrong EOF assumption[1], check for return type[2]
[1]
This worked incidentally because OSes tend to return at least one page
worth of data when EOF is not reached. Increasing WINDOW_SIZE beyond
the page size might have lead to data loss.

[2]
If read() of the passed Python object returned something not-bytes,
PyBytes_Size returns -1 (ssize_t) which becomes a very larger number for
memcpy()s size_t.
2016-03-31 22:03:17 +02:00
Alexander Pyhalov
e494d24d6a failing hashindex tests on netbsd, fixes #804 2016-03-31 21:32:27 +02:00
Marian Beermann
effa992d22 append-only mode: use single file for log, disallow check --repair, docs 2016-03-31 18:23:08 +02:00
Marian Beermann
ffc8cf6c97 Append-only mode for repositories
New repository config item, repository.append_only, causes Borg to never
delete or append to existing data files. Hints and indices are handled
as before, old ones are deleted, because they can get quite large, but
are automatically reconstructed: no need to keep them.

When append_only is activated a file /path/to/repo/transactions/<NUMBER>
will be created for every commit.
Deleting all segments <NUMBER+1> and higher will rollback to that commit.

Note that this only influences Borg behaviour. Since repository config
can't be altered remotely (except for repository.key) this can't be
disabled when accessed remotely over SSH with "borg serve" as the
forced command.

This is only a feature to support the use case, and does not replace
appropriate file system permissions or monitoring.

Resolves #809
2016-03-31 17:29:17 +02:00
Thomas Waldmann
3e120378e0 fix capitalization, add ellipses, change log level to debug for 2 messages, fixes #798 2016-03-26 14:31:54 +01:00
TW
42beb2dc92 Merge pull request #785 from pyhalov/illumos-build-1.0
Fix build on illumos
2016-03-21 16:07:52 +01:00
Alexander Pyhalov
f63be63347 Fix build on illumos 2016-03-21 17:39:55 +03:00
Alexander Pyhalov
e98f823f34 Don't try to backup doors or event ports 2016-03-21 17:35:51 +03:00
Thomas Waldmann
f486a6772c do not sleep for >60s while waiting for lock, fixes #773 2016-03-19 21:19:30 +01:00
Marian Beermann
489eb78cbb Possible fix to silently skipping EIO and the like? #748 2016-03-16 01:32:48 +01:00
Thomas Waldmann
3115bb4d1b remove useless/misleading libc version display, fixes #738
platform.libc_ver() is broken, it uses string comparison to compare version numbers.
2016-03-12 01:29:14 +01:00
Thomas Waldmann
7abcaa0d9d fix reproducible build of api docs 2016-03-11 23:37:37 +01:00
Mitch Bigelow
353774973e file stats are packed with 'int_to_bigint' in the archive. Unpack these before passing to llfuse 2016-03-10 15:50:45 -05:00
Thomas Waldmann
95c4cf221d implement password roundtrip, fixes #695 2016-03-05 16:21:44 +01:00
Thomas Waldmann
3dc2fc03f0 added retry counter for passwords, fixes #703 2016-03-05 00:54:19 +01:00
Thomas Waldmann
b461091be1 fix compiler warnings, fixes #697 2016-03-02 14:45:02 +01:00
Teemu Toivanen
9c9ac2fd58 Remote end does not need cache nor keys directories, do not make them.
Move keys and cache creation to get_keys_dir and get_cache_dir helper functions.
And remove keys_dir and cache_dir from run(). cache and keys will only be created
by code which uses them. Remote end should not call get_keys_dir and get_cache_dir
2016-03-01 12:04:59 +02:00
Teemu Toivanen
79f42571ae unit test for archive format
tests formatting with {pid} and not equal results from same
time string. (adds import time to tests)
2016-02-25 00:41:24 +02:00
Teemu Toivanen
f7c1632aee Add format options to location.
Add support for python format options for location:
tags:
 pid
 fqdn
 hostname
 now
 utcnow
 user
2016-02-25 00:41:24 +02:00
Thomas Waldmann
dc5467e9bc simplify processing for borg list --short
just use the respective format string.
2016-02-21 21:35:00 +01:00
Thomas Waldmann
d14e6f720c --list-format: make it compatible with old archives
they do not have atime and ctime, use mtime instead.
2016-02-21 21:11:56 +01:00
Thomas Waldmann
f7666807cc refactored --list-format code a bit
note: LF as line ending is enough, python will translate that to CRLF on the platforms (windows) needing CRLF.
2016-02-21 21:05:30 +01:00
Thomas Waldmann
b28f4d886b fix formatkeys so it always has all available formatting keys
- always: no matter whether we use a user-defined format or not
- all: fix bug: it only had the basic keys, not the advanced keys
2016-02-21 19:47:25 +01:00
Thomas Waldmann
08a7ce5cc3 Merge branch 'feature/list-formatting' of https://github.com/manwegit/borg into feature/list-formatting 2016-02-21 19:15:31 +01:00
TW
fb83f619a0 Merge pull request #682 from ThomasWaldmann/validate-newname
add a simple archivename validator, fixes #680
2016-02-20 13:33:43 +01:00
TW
6ff78cfb71 Merge pull request #679 from ThomasWaldmann/prune--list
borg prune -v --list enables the keep/prune list output, fixes #658
2016-02-20 13:30:45 +01:00
Thomas Waldmann
4a60d777d8 add a simple archivename validator, fixes #680 2016-02-19 23:01:20 +01:00