748 commits

Author	SHA1	Message	Date
Marian Beermann	f3c7e7cd36	RemoteRepository: account rx/tx bytes	2017-02-23 16:50:52 +01:00
Marian Beermann	8cdf192511	list: add "name" key for consistency with info cmd	2017-02-23 15:23:27 +01:00
Marian Beermann	56563a4392	move JSON generation and utilities to helpers	2017-02-23 15:23:27 +01:00
enkore	4862efe718	Merge pull request #2194 from jdchristensen/master ... In api decorator, pass wait argument to RemoteRepository.call; fixes …	2017-02-23 14:50:33 +01:00
Marian Beermann	1f8c0929bf	list: --json for archive contents listing	2017-02-23 14:28:30 +01:00
Marian Beermann	25781f53d4	list: --json for archive listing	2017-02-23 14:28:15 +01:00
Marian Beermann	6180f5055c	info: --json for archives	2017-02-23 14:28:15 +01:00
Marian Beermann	2ab5d0f213	use custom JSON encoder for repr'ing Borg objects consistently	2017-02-23 14:27:52 +01:00
Marian Beermann	7cbade2f8c	create: add --json option	2017-02-23 12:00:21 +01:00
Marian Beermann	cc26bdf810	info: add --json option	2017-02-23 11:55:14 +01:00
Marian Beermann	b0e4f13fba	set MAX_DATA_SIZE = 20971479 bytes in solid stone	2017-02-23 00:34:40 +01:00
Marian Beermann	69f7810658	info: show utilization of maximum archive size ... See #1452 This is 100 % accurate. Also increases maximum data size by ~41 bytes. Not 100 % side-effect free; if you manage to exactly land in that area then older Borg would not read it. OTOH it gives us a nice round number there.	2017-02-22 23:47:21 +01:00
Dan Christensen	20a5282a4c	In api decorator, pass wait argument to RemoteRepository.call; fixes #2185	2017-02-22 16:26:08 -05:00
TW	268d74bb43	Merge pull request #2181 from ThomasWaldmann/fix-2180 ... archive check: detect and fix missing replacement chunks, fixes #2180	2017-02-21 21:57:23 +01:00
enkore	cd1df96ef5	Merge pull request #2171 from ThomasWaldmann/limit-unpacker ... borg rpc: use limited msgpack.Unpacker	2017-02-21 21:53:09 +01:00
Thomas Waldmann	8d7dfe739f	fix ChunkIndex.__contains__ assertion for big-endian archs ... also: add some missing assertion messages severity: - no issue on little-endian platforms (== most, including x86/x64) - harmless even on big-endian as long as refcount is below 0xfffbffff, which is very likely always the case in practice anyway.	2017-02-20 07:38:55 +01:00
Thomas Waldmann	b82f648875	archive check: detect and fix missing all-zero replacement chunks, fixes #2180	2017-02-19 03:05:41 +01:00
enkore	63d4cf4c82	Merge pull request #2172 from Abogical/master ... Add dsize and dcsize keys, fixes #2164	2017-02-18 18:24:52 +01:00
Abogical	38e4817b48	Correct calculation of unique chunks	2017-02-18 12:37:28 +02:00
Abogical	cd3cbee962	Refactor unique chunks summing	2017-02-18 12:36:08 +02:00
Abogical	6ed0746934	Count non-unique chunks deduplicated sizes	2017-02-17 17:33:04 +02:00
Abogical	59571115a1	Add tests for dsize and dcsize	2017-02-17 15:26:14 +02:00
Abogical	5aa74abedf	Add dsize and dcsize keys ... These keys shows the amount of deduplicated size and compressed size of each file in the archive.	2017-02-17 14:28:39 +02:00
Thomas Waldmann	b05893e723	borg rpc: use limited msgpack.Unpacker, fixes #2139 ... we do not trust the remote, so we are careful unpacking its responses. the remote could return manipulated msgpack data that announces e.g. a huge array or map or string. the local would then need to allocate huge amounts of RAM in expectation of that data (no matter whether really that much is coming or not). by using limits in the Unpacker, a ValueError will be raised if unexpected amounts of data shall get unpacked. memory DoS will be avoided.	2017-02-17 05:44:48 +01:00
Marian Beermann	caeff71a6c	init: mandatory --encryption arg	2017-02-16 19:03:23 +01:00
Marian Beermann	788b608aa9	setup.py build_usage/build_man fixes	2017-02-14 23:02:56 +01:00
Marian Beermann	73990b878f	create: handle BackupOSError on a per-path level in one spot	2017-02-14 20:39:29 +01:00
Alexander 'Leo' Bergolth	30a5c5e44b	add two new options --pattern and --patterns-from as discussed in #1406 ... # Conflicts: # src/borg/archiver.py # src/borg/helpers.py # src/borg/testsuite/helpers.py Original-Commit: 876b670d	2017-02-13 21:47:51 +01:00
Thomas Waldmann	8d432b01e1	paperkey.html - decode as utf-8, fixes #2150 ... hardcoded the encoding for reading it. while utf-8 is the default encoding on many systems, it does not work everywhere. and when it tries to decode with the ascii decoder, it fails.	2017-02-13 21:24:34 +01:00
Marian Beermann	1fabb2df58	key export: center QR code on the page	2017-02-13 21:24:33 +01:00
Marian Beermann	04bd6fb013	add test for export key --qr-html	2017-02-13 21:24:33 +01:00
Benedikt Heine	2cdb583879	clearify doc for same filesystems ... # Conflicts: # src/borg/archiver.py Original-Commit: d3a2f36b03	2017-02-13 21:24:33 +01:00
Martin Hostettler	179f1bc147	Add qr html export mode to key export command	2017-02-13 21:24:33 +01:00
Martin Hostettler	32e73e8c7e	Manifest: Make sure manifest timestamp is strictly monotonically increasing. ... Computer clocks are often not set very accurately set, but borg assumes manifest timestamps are never going back in time. Ensure that this is actually the case. # Conflicts: # src/borg/helpers.py Original-Commit: 6b8cf0a	2017-02-13 20:43:57 +01:00
Marian Beermann	e4486cd370	fix rST warning in Repository.scan	2017-02-12 18:51:41 +01:00
Marian Beermann	15dfaae223	docs: create: move --exclude note to main doc	2017-02-12 15:28:59 +01:00
Marian Beermann	0710bbd40e	docs: create: move item flags to main doc	2017-02-12 15:28:58 +01:00
Marian Beermann	497da8df04	docs: init: fix markup/typos	2017-02-12 15:27:06 +01:00
Martin Hostettler	e8335dba0f	archiver: Add 'debug dump-manifest' and 'debug dump-archive' commands.	2017-02-10 22:56:54 +01:00
Marian Beermann	7e486074e8	docs: list: don't print key listings in fat (html + man)	2017-02-05 15:46:46 +01:00
Marian Beermann	c7106e756e	create real nice man pages	2017-02-05 15:46:09 +01:00
Marian Beermann	5bc03cc042	migrate-to-repokey: ask using canonical_path() as we do everywhere else	2017-02-04 15:16:18 +01:00
Marian Beermann	4e0422cdf0	Move change-passphrase to the "key" group (but leave old name, too)	2017-02-04 15:10:45 +01:00
Marian Beermann	5fe32866e6	Move migrate-to-repokey to the "key" command group	2017-02-04 15:10:45 +01:00
TW	c6ea34be96	Merge pull request #2111 from ThomasWaldmann/merge-1.0-maint ... Merge 1.0-maint	2017-02-01 12:13:37 +01:00
enkore	f52bf883c3	Merge pull request #2010 from costela/master ... change dir_is_tagged to use os.path.exists()	2017-02-01 00:53:36 +01:00
Radu Ciorba	a85cf75465	fix wrong skip_hint on hashindex_set when encountering tombstones ... hashindex_lookup would always hint at skipping whatever it's probe length had been with no regard for tombstones it had encountered. This meant new keys would not overwrite first tombstones, but would always land on empty buckets. The regression was introduced in #1748	2017-01-30 23:29:08 +02:00
Thomas Waldmann	7f2a108c94	fixup: do not access os.POSIX_FADV_* early ... before we know posix_fadvise support exists on the platform.	2017-01-30 03:11:42 +01:00
Leo Antunes	dd6b90fe6c	change dir_is_tagged to use os.path.exists() ... Add --keep-exclude-tags option as alias to --keep-tag-files and deprecate the later. Also make tagging accept directories as tags, allowing things like `--exclude-if-present .git`. fixes #1999	2017-01-29 18:13:51 +01:00
Thomas Waldmann	c0dc644ef6	Merge branch '1.0-maint' into merge-1.0-maint ... # Conflicts: # MANIFEST.in # Vagrantfile # docs/changes.rst # docs/usage/mount.rst.inc # src/borg/archiver.py # src/borg/fuse.py # src/borg/repository.py	2017-01-29 05:49:53 +01:00
Thomas Waldmann	e0dfb656ee	fix crc32 compilation issues on wheezy ... moving the declaration / assignment from the middle of the function to the beginning.	2017-01-15 15:09:40 +01:00
Thomas Waldmann	740312f725	Merge branch '1.0-maint' into merge-1.0-maint ... # Conflicts: # requirements.d/development.txt # tox.ini	2017-01-15 01:15:04 +01:00
TW	abfcf9955b	Merge pull request #2045 from ThomasWaldmann/fix-platform ... posix platform module: only build / import on non-win32 platforms	2017-01-14 16:06:28 +01:00
Marian Beermann	ae0f1422bf	crc: openbsd has no Intel intrinsics	2017-01-14 15:24:03 +01:00
Thomas Waldmann	e4c5db4efc	posix platform module: only build / import on non-win32 platforms, fixes #2041 ... rather use a inverted check like "not windows". also: add a base implementation for this stuff, just raising NotImplementedError	2017-01-14 14:48:49 +01:00
Thomas Waldmann	ededb6f2c8	fix crc32 compile error, fixes #2039	2017-01-14 05:06:15 +01:00
Thomas Waldmann	e431d60cc5	merge 1.0-maint into master ... # Conflicts: # src/borg/crypto.pyx # src/borg/hashindex.pyx # src/borg/helpers.py # src/borg/platform/__init__.py # src/borg/platform/darwin.pyx # src/borg/platform/freebsd.pyx # src/borg/platform/linux.pyx # src/borg/remote.py	2017-01-14 03:07:11 +01:00
Marian Beermann	5cc292c52c	fix performance regression in "borg info ::archive"	2017-01-13 15:33:38 +01:00
Marian Beermann	f482c32423	crc32: sb8: remove some unneeded stuff, use hashindex byteorder detection	2017-01-13 00:33:34 +01:00
Marian Beermann	292fb1e2a9	crc: disable nice CLMUL version due to clang bugs.	2017-01-12 21:30:37 +01:00
Marian Beermann	7923088ff9	check: pick better insufficent archives matched warning from TW's merge	2017-01-12 17:04:51 +01:00
Marian Beermann	1d40675ce4	merge fixup	2017-01-12 15:04:57 +01:00
Marian Beermann	ecad0ed53a	Merge branch '1.0-maint' into merge/1.0-maint ... # Conflicts: ... everywhere ... # .travis.yml # Vagrantfile # borg/testsuite/key.py # docs/changes.rst # docs/quickstart.rst # docs/usage.rst # docs/usage/upgrade.rst.inc # src/borg/archive.py # src/borg/archiver.py # src/borg/crypto.pyx # src/borg/helpers.py # src/borg/key.py # src/borg/remote.py # src/borg/repository.py # src/borg/testsuite/archive.py # src/borg/testsuite/archiver.py # src/borg/testsuite/crypto.py # src/borg/testsuite/helpers.py # src/borg/testsuite/repository.py # src/borg/upgrader.py # tox.ini	2017-01-12 15:01:41 +01:00
enkore	07049098ee	Merge pull request #2014 from enkore/f/hostname->fqdn ... posix: use fully-qualified hostname + node ID	2017-01-12 02:07:08 +01:00
enkore	ded22d3650	Merge pull request #1927 from ThomasWaldmann/blake2b-mac-key-length ... demonstrate mac key length issue	2017-01-12 01:21:13 +01:00
Marian Beermann	81bd55eec3	key testsuite: update blake2 test data to include padded keys	2017-01-12 01:05:59 +01:00
Marian Beermann	b9770c348f	posix: use fully-qualified hostname + node ID ... The node ID is usually the 48 bit MAC of the primary network interface.	2017-01-12 00:54:17 +01:00
enkore	902c423086	Merge pull request #1970 from enkore/f/dbg-speed ... fast crc	2017-01-02 16:11:11 +01:00
Marian Beermann	56818d1db8	Credit the excellent work of these people in AUTHORS	2017-01-02 15:54:14 +01:00
Marian Beermann	9afebead84	two fast CRC implementations ... CRC slice by 8 for generic CPUs outperforms zlib CRC32 on ppc and x86 (ARM untested but expected to as well). PCLMULQDQ derived from Intel's zlib patches outperforms every other CRC implementation by a huge margin.	2017-01-02 15:54:14 +01:00
enkore	cf19db466d	Merge pull request #1989 from Abogical/master ... Happy new year!	2017-01-01 01:14:08 +01:00
Abogical	f043b966da	It's 2017	2016-12-31 16:53:30 +02:00
Thomas Waldmann	7d48878cfe	borg init --encryption - remove default ... use --encryption=repokey for all the tests except if they have special needs.	2016-12-24 02:53:44 +01:00
enkore	a4eef67e3b	Merge pull request #1912 from Abogical/prune-progress ... Add prune progress display	2016-12-22 14:57:26 +01:00
Marian Beermann	742bfa33c4	flake8	2016-12-21 00:36:31 +01:00
Marian Beermann	f9aefd6a68	Merge commit '343387b36ef' into merge/1.0-maint ... # Conflicts: # src/borg/archiver.py	2016-12-20 23:00:28 +01:00
Marian Beermann	df40b3840c	upgrade: --disable-tam ... # Conflicts: # src/borg/helpers.py # src/borg/testsuite/archiver.py	2016-12-20 22:53:57 +01:00
Marian Beermann	c7c8c0fb57	init: explain manifest auth compatibility ... # Conflicts: # src/borg/archiver.py	2016-12-20 22:53:55 +01:00
Marian Beermann	d15fb241bd	check: handle duplicate archive items neatly ... # Conflicts: # src/borg/archive.py	2016-12-20 22:53:55 +01:00
Marian Beermann	5e1cb9d899	Add tertiary authentication for metadata (TAM)	2016-12-20 22:53:53 +01:00
Marian Beermann	84d13751d5	Fix processing of remote ~/ and ~user/ paths	2016-12-17 14:33:40 +01:00
Marian Beermann	63ce627a35	fix in-file checkpoints when clock jumps	2016-12-17 13:59:37 +01:00
Marian Beermann	a9db2a2e55	Merge branch '1.0-maint' into master ... # Conflicts: # src/borg/archive.py # src/borg/archiver.py # src/borg/helpers.py	2016-12-17 13:26:28 +01:00
Marian Beermann	87f8b2bb64	Merge branch 'merge-1.0-maint' of https://github.com/ThomasWaldmann/borg into ThomasWaldmann-merge-1.0-maint	2016-12-17 13:21:58 +01:00
Marian Beermann	34e19ccb6a	mention failed operation in per-file warnings ... on the one hand one can say it's ugly global state, on the other it's totally handy! just have to keep that in mind for MT, but it's rather obvious.	2016-12-14 15:20:08 +01:00
Thomas Waldmann	c855c687eb	Merge branch '1.0-maint'	2016-12-14 03:33:53 +01:00
Marian Beermann	be18418b74	cache: no archive caches => work directly on master cache (no merges)	2016-12-08 22:45:59 +01:00
Marian Beermann	e169510116	cache: don't create Item in fetch_and_build_idx	2016-12-08 22:39:04 +01:00
Thomas Waldmann	58752c9de9	add assertion to demonstrate mac key length issue in test data	2016-12-04 17:56:23 +01:00
Abogical	0994a0a681	Let prune --list display archives deleted per total archives	2016-12-03 19:34:45 +02:00
Marian Beermann	b7eaeee266	clean imports, remove unused code	2016-12-03 17:50:50 +01:00
Marian Beermann	8b2e7ec680	don't do "bigint" conversion for nanosecond mtime ... 2**63 nanoseconds are 292 years, so this change is good until 2262. See also https://en.wikipedia.org/wiki/Time_formatting_and_storage_bugs#Year_2262 I expect that we will have plenty of time to revert this commit in time for 2262. timespec := time_t + long, so it's probably only 64 bits on some platforms anyway.	2016-12-03 11:52:49 +01:00
Marian Beermann	b3707f7175	Replace backup_io with a singleton ... This is some 15 times faster than @contextmanager, because no instance creation is involved and no generator has to be maintained. Overall difference is low, but still nice for a very simple change.	2016-12-03 11:52:48 +01:00
Marian Beermann	b885841c39	make item native code ... This makes an surprisingly large difference. Test case: ~70000 empty files. (Ie. little data shoveling, lots of metadata shoveling). Before: 9.1 seconds +- 0.1 seconds. After: 8.4 seconds +- 0.1 seconds.). That's a huge win for changing a few lines. I'd expect that this improves performance in almost all areas that touch the items (list, delete, prune).	2016-12-03 00:11:36 +01:00
enkore	e9d7f928e2	Merge pull request #1921 from enkore/f/recreate-fixes ... recreate fixes	2016-12-03 00:02:01 +01:00
Marian Beermann	a9395dd8b1	recreate: don't rechunkify unless explicitly told so	2016-12-02 20:19:59 +01:00
Marian Beermann	30df63c509	recreate: remove special-cased --dry-run	2016-12-02 18:15:11 +01:00
Marian Beermann	c6f0969352	recreate: update --help	2016-12-02 13:20:43 +01:00
Marian Beermann	c1ccad82c3	recreate: update/remove/rename outdated comments	2016-12-02 12:54:27 +01:00
Marian Beermann	eade10a0a8	recreate: fix crash on checkpoint	2016-12-02 11:39:10 +01:00
OEM Configuration (temporary user)	0f07b6acf4	borg info:fixed bug when called without arguments,issue #1914	2016-12-02 10:27:53 +00:00
Marian Beermann	eb940e6779	recreate: fix rechunking dropping all chunks on the floor	2016-12-02 11:20:26 +01:00
Marian Beermann	b410392899	recreate repo: fix only one archive being processed	2016-12-02 11:09:52 +01:00
TW	10d143925c	Merge pull request #1904 from enkore/issue/1896 ... blake2b key modes: use B2B as MAC; longer keys.	2016-12-01 00:59:55 +01:00
Thomas Waldmann	a100fb67eb	Merge branch '1.0-maint' into merge-1.0-maint ... # Conflicts: # AUTHORS # src/borg/archive.py # src/borg/key.py	2016-11-30 05:38:04 +01:00
Abogical	029469d544	Let prune accept and pass the progress argument	2016-11-30 02:39:27 +02:00
Marian Beermann	8949f2c758	blake2b key modes: use B2B as MAC; longer keys.	2016-11-30 00:24:05 +01:00
enkore	f4c916e01f	Merge pull request #1901 from ThomasWaldmann/show-broken ... implement "health" item formatter key, fixes #1749	2016-11-29 23:48:55 +01:00
Thomas Waldmann	01ad1a5153	implement "health" item formatter key, fixes #1749	2016-11-29 23:02:24 +01:00
TW	99c38e8fc0	Merge pull request #1892 from Abogical/sync-progress ... Add cache.sync progress display	2016-11-29 21:52:07 +01:00
Abogical	f3ce6be30b	Add cache.sync progress display	2016-11-29 22:15:27 +02:00
Marian Beermann	bd96b43af9	borg info: print security directory	2016-11-29 17:52:34 +01:00
Marian Beermann	f62a22392e	Implement security dir perks ... Key type, location and manifest timestamp checks now survive cache deletion. This also means that you can now delete your cache and avoid previous warnings, since Borg can still tell it's safe.	2016-11-29 17:52:32 +01:00
Marian Beermann	c3a2dc5f55	Rename BORG_NONCES_DIR to BORG_SECURITY_DIR	2016-11-29 17:43:06 +01:00
Thomas Waldmann	c83a124e65	Merge branch '1.0-maint' (into master)	2016-11-28 02:23:32 +01:00
enkore	5917822735	Merge pull request #1881 from Abogical/cache-progress ... Add begin_txn progress message	2016-11-26 19:11:08 +01:00
Abogical	be6341b956	Add begin_txn progress message	2016-11-26 19:38:00 +02:00
enkore	1250b14ce2	Merge pull request #1874 from Abogical/cache-progress ... Add progress display for cache commit	2016-11-26 00:50:16 +01:00
enkore	50b97665c8	Merge pull request #1878 from ThomasWaldmann/borg-new-passphrase ... implement BORG_NEW_PASSPHRASE, fixes #1768	2016-11-25 20:39:26 +01:00
Thomas Waldmann	d519ae6b6f	remove some unneeded test overriding methods ... they are not there any more in the base class, so there is nothing to override.	2016-11-25 01:31:28 +01:00
Thomas Waldmann	fa8ce2d977	add test for borg change-passphrase / BORG_NEW_PASSPHRASE	2016-11-25 00:06:35 +01:00
Thomas Waldmann	0a0b913739	implement BORG_NEW_PASSPHRASE, fixes #1768	2016-11-24 23:43:27 +01:00
Abogical	f9b3d28c19	Add progress messages for cache.commit	2016-11-24 15:44:00 +02:00
Abogical	e999f3ff51	Add ProgressIndicatorMessage and abstract class ProgressIndicatorBase	2016-11-24 15:43:51 +02:00
enkore	abe5923866	Merge pull request #1862 from enkore/f/simple-recreate ... The Defenestration of recreate's bloat	2016-11-20 19:59:45 +01:00
Marian Beermann	15cefe8dde	recreate/archiver tests: add check_cache tool - lints refcounts	2016-11-20 18:13:53 +01:00
Marian Beermann	93b03ea231	recreate: re-use existing checkpoint functionality	2016-11-20 18:13:51 +01:00
Marian Beermann	44935aa8ea	recreate: remove interruption blah, autocommit blah, resuming blah	2016-11-19 16:49:20 +01:00
Thomas Waldmann	afab1cc9b0	Merge branch '1.0-maint' ... # Conflicts: # borg/platform.py # src/borg/archiver.py # src/borg/helpers.py # src/borg/platform/darwin.pyx # src/borg/platform/freebsd.pyx # src/borg/platform/linux.pyx # src/borg/testsuite/__init__.py	2016-11-17 23:56:34 +01:00
Martin Hostettler	f37109848f	remote: Remove unused type variable.	2016-11-17 20:03:33 +01:00
Martin Hostettler	1edff44b3d	Repository,remote: Rename argument of get to 'id'.	2016-11-17 20:03:32 +01:00
Martin Hostettler	8955d8bb2a	remote: Test that the legacy free rpc bootstrap works.	2016-11-17 20:03:32 +01:00
Martin Hostettler	bd3a4a2f92	Add testing for exception transport.	2016-11-17 20:03:32 +01:00
Martin Hostettler	e14406fdbf	remote: Redo exception handling	2016-11-17 20:03:32 +01:00
Martin Hostettler	4854fcef2e	remote: Move open to a normal api stub.	2016-11-17 19:56:30 +01:00
Martin Hostettler	ba553ec628	remote: Introduce rpc protocol with named parameters.	2016-11-17 19:56:30 +01:00
Martin Hostettler	6c1b337ce2	remote: Replace broken exception argument restoration with code that uses a fixed value.	2016-11-17 19:56:29 +01:00
Martin Hostettler	d25e9aa4f0	remote: Use single quotes	2016-11-17 19:56:29 +01:00
Martin Hostettler	0da913f8ed	remote: Decode method name as utf-8 instead of ascii for consistency.	2016-11-17 19:56:29 +01:00
Martin Hostettler	c580d9c173	version: Add version parsing and formating	2016-11-17 19:56:29 +01:00
Marian Beermann	c380d91805	fixes for flake 3.1.1	2016-11-14 21:50:20 +01:00
enkore	cf8f8fb746	Merge pull request #1846 from Abogical/master ... Improve extract progress display, for #1721	2016-11-14 21:43:35 +01:00
Abogical	34f529c7df	satisfy codecov	2016-11-14 17:39:42 +02:00
Abogical	3896f26ab2	use monkeypatch	2016-11-14 16:41:37 +02:00
Abogical	76638d0562	If there is a small space for ellipsis_truncate, show '...' only	2016-11-14 15:16:14 +02:00
Abogical	af925d2723	do not justify if we're not outputing to a terminal	2016-11-14 14:49:24 +02:00
Abogical	3232769ffc	Respond to feedback	2016-11-14 11:43:46 +02:00
Abogical	1362d2e90f	Set COLUMNS & LINES as if it was a terminal	2016-11-14 01:00:11 +02:00
Abogical	467fe38b15	Compatibility with python 3.4	2016-11-14 00:25:27 +02:00
Abogical	b737866905	Improve extract progress display and ProgressIndicatorPercent	2016-11-13 23:41:01 +02:00
Marian Beermann	dfd748689c	test_attic013_acl_bug: use def instead of lambda	2016-11-13 17:30:42 +01:00
Marian Beermann	c2eb2539b9	Document maintenance merge command	2016-11-13 16:07:06 +01:00
Marian Beermann	0d2b76fa7d	Merge branch '1.0-maint' into merge/1.0-maint	2016-11-13 15:58:42 +01:00
Marian Beermann	00fae3f3e3	repository: check_free_space, special case small repos	2016-11-12 14:42:13 +01:00
enkore	80b86709ef	Merge pull request #1833 from enkore/issue/1821 ... borg init: fix free space check crashing if disk is full	2016-11-12 01:23:34 +01:00
enkore	e26348a055	Merge pull request #1834 from enkore/issue/1813 ... Clarify cache/repository README file	2016-11-11 21:25:30 +01:00
Marian Beermann	bd4a0fe23b	Clarify cache/repository README file	2016-11-11 21:24:17 +01:00
Marian Beermann	ac5f9b61ec	debug, key: enable --help ... A small merge mishap in 2a864be8; 1.0-maint has it correctly.	2016-11-11 19:54:11 +01:00
Marian Beermann	6c955579a8	repository: test_additional_free_space, assert it still exists	2016-11-11 16:39:11 +01:00
Marian Beermann	5b31a2cd61	borg init: fix free space check crashing if disk is full	2016-11-11 16:21:05 +01:00
enkore	223f7890fe	Merge pull request #1819 from enkore/issue/1800 ... Add BLAKE2b key types	2016-11-11 03:11:38 +01:00
Marian Beermann	a31ca4f022	crypto: link against system libb2, if possible	2016-11-11 03:08:11 +01:00
Marian Beermann	76c93bb80b	crypto blake2: don't release the GIL during key hashing	2016-11-11 03:07:05 +01:00
Marian Beermann	00ac7b14be	Add BLAKE2b key types	2016-11-11 03:07:05 +01:00
TW	dc4325be25	Merge pull request #1826 from textshell/feature/repository-cleanup-rollback-api ... Repository: rollback's cleanup parameter is internal.	2016-11-10 17:24:01 +01:00
Martin Hostettler	79c77e7a68	Repository: rollback's cleanup parameter is internal.	2016-11-10 13:44:58 +01:00
Marian Beermann	09f470bd85	Add crypto.blake2b_256	2016-11-10 00:06:50 +01:00
Marian Beermann	05a52ec8ec	Merge branch '1.0-maint' into merge/1.0-maint	2016-11-09 23:56:54 +01:00
Marian Beermann	676e69cac4	Parse & pass BORG_HOSTNAME_IS_UNIQUE env var to enable stale lock killing	2016-11-07 21:56:40 +01:00
Marian Beermann	cc14975f2d	Add tests for stale lock killing and platform.process_alive	2016-11-07 21:56:07 +01:00
Marian Beermann	2bd8ac7762	platform: bump API version (and check consistency)	2016-11-07 21:54:16 +01:00
Marian Beermann	7930d055ec	import platform module instead of functions ... (testability)	2016-11-07 21:54:16 +01:00
Marian Beermann	8e1df7a364	Use logging instead of prints	2016-11-07 21:54:16 +01:00
Marian Beermann	c562f7750c	Move platform-dependent code to platform package	2016-11-07 21:54:16 +01:00
Oleg Drokin	d490292be3	Detect and delete stale locks when it's safe ... If BORG_UNIQUE_HOSTNAME shell variable is set, stale locks in both cache and repository are deleted. Stale lock is defined as a lock that's originating from the same hostname as us, and correspond to a pid that no longer exists. This fixes #562	2016-11-07 21:54:16 +01:00
Thomas Waldmann	8a15916284	Merge branch '1.0-maint' into merge-1.0-maint ... # Conflicts: # docs/changes.rst # docs/usage/help.rst.inc # src/borg/cache.py # src/borg/remote.py # src/borg/testsuite/__init__.py # src/borg/testsuite/archiver.py	2016-10-31 18:24:13 +01:00
Thomas Waldmann	09e74af7bf	Merge branch '1.0-maint'	2016-10-24 21:41:20 +02:00
TW	0ff2352691	Merge pull request #1748 from rciorba/shortcut_hashindex_set ... shortcut hashindex_set by having hashindex_lookup hint about address	2016-10-23 16:48:43 +02:00
Radu Ciorba	766a43afc3	shortcut hashindex_set by having hashindex_lookup hint about address	2016-10-22 12:33:15 +03:00
TW	1a272ddd48	Merge pull request #1739 from ThomasWaldmann/borg-check-refactor ... borg check --first / --last / --sort / --prefix, fixes #1663	2016-10-20 17:21:27 +02:00
Thomas Waldmann	4c884fd075	borg check --first / --last / --sort / --prefix, fixes #1663	2016-10-20 16:51:26 +02:00
Thomas Waldmann	e02d1a66b8	fix byte range error in test, fixes #1740	2016-10-20 03:04:19 +02:00
Thomas Waldmann	5a12c6377a	fuse mount options: add tests	2016-10-15 01:24:05 +02:00
Thomas Waldmann	b5f9858055	move first/last/sort_by-multiple functionality into Manifest.list ... also: rename list_filtered to list_considering	2016-10-15 01:04:56 +02:00
Thomas Waldmann	694c3978a1	refactor Archiver._get_filtered_archives -> Archives.list_filtered ... it did not belong into Archiver class (did not use "self"), but in into Archives.	2016-10-15 01:03:16 +02:00
Thomas Waldmann	26e6ac4fea	borg mount --first / --last / --sort / --prefix, fixes #1542 ... also: use consider_part_files when dealing with multiple archives / whole repo mount	2016-10-15 01:03:16 +02:00
Thomas Waldmann	775440fdcc	Merge branch '1.0-maint' into merge-1.0-maint	2016-10-15 00:25:41 +02:00
enkore	34bd7cb4e2	Merge pull request #1716 from ThomasWaldmann/merge-1.0-maint ... Merge 1.0-maint	2016-10-14 21:18:30 +02:00
Thomas Waldmann	4e4847ccce	point XDG_*_HOME to temp dirs for tests, fixes #1714 ... otherwise it spoils the user's nonces and cache dirs with lots of files. also: remove all BORG_* env vars from the outer environment	2016-10-13 23:59:28 +02:00
Thomas Waldmann	b0cc4fd74a	Merge branch '1.0-maint' into merge-1.0-maint	2016-10-13 19:10:04 +02:00
TW	a2dfe68609	Merge pull request #1706 from ThomasWaldmann/remove-debug-xxx ... remove debug-xxx commands, fixes #1627	2016-10-12 23:25:43 +02:00
Thomas Waldmann	9ad9ae8ff0	Repository.scan(): avoid iterating same repo segments/objects repeatedly ... segments: avoid some listdir() objects: avoid repeatedly re-reading object headers / seeking in segment Fixes #1610.	2016-10-10 05:39:38 +02:00
Thomas Waldmann	b88e82d99d	remove debug-xxx commands, fixes #1627 ... we use "debug xxx" subcommands now. docs updated. also makes "borg help" shorter as not all debug-xxx commands show up, but just 1 main "debug" command.	2016-10-10 00:22:01 +02:00
enkore	8567fcfe58	Merge pull request #1418 from textshell/fix/preload ... Cleanup prefetch code in RemoteRepository	2016-10-07 09:49:16 +02:00
Thomas Waldmann	f31112b975	clarify borg diff help, fixes #980	2016-10-07 04:09:05 +02:00
Martin Hostettler	02557f16b0	RemoteRepository: Rename cache to chunkid_to_msgids. Also fetch_from_cache to pop_preload_msgid.	2016-10-06 20:48:26 +02:00
Martin Hostettler	2608a5620a	RemoteRepository: Always store chunk ids in cache instead of rpc argument encoding of get request.	2016-10-06 20:38:46 +02:00
Martin Hostettler	e9605d67ad	RemoteReposity: prefetch can only be 'get'.	2016-10-06 20:38:46 +02:00
Thomas Waldmann	cdb8d64fe2	check for index vs. segment files object count mismatch	2016-10-05 17:36:36 +02:00
Thomas Waldmann	6624ca9cdb	verify_data: do a linear scan in disk-order	2016-10-05 17:36:36 +02:00
Thomas Waldmann	90111363ba	repo.list() yielding IDs in on-disk order	2016-10-05 17:36:36 +02:00
enkore	27bc73c23e	borg info <repo>: print general repo information (#1680)	2016-10-03 14:53:16 +02:00
Thomas Waldmann	8df6cb8156	hashindex: bump api_version ... note: merging the respective changeset from 1.0-maint was not effective as we already had version 3, so there was no increase.	2016-09-30 23:59:41 +02:00
Frank Sachsenheim	089995ef73	Changes on filters after feedback	2016-09-30 20:29:16 +02:00
Frank Sachsenheim	bd7cc38d6e	Changes arg processor names to camelcase	2016-09-30 20:14:20 +02:00
Frank Sachsenheim	e0e9edfb42	Adds archives filters for list	2016-09-30 20:14:20 +02:00
Frank Sachsenheim	369d0a0881	Adds archives filters for info	2016-09-30 20:14:20 +02:00
Frank Sachsenheim	f2d4d36cea	Allows delete to be used with archive filters	2016-09-30 20:14:20 +02:00
Frank Sachsenheim	17f2363935	Adds --prefix to the archives filters arguments ... - adds prefix argument to helpers.Archives.list - also renames function PrefixSpec to prefix_spec	2016-09-30 19:48:17 +02:00
Frank Sachsenheim	f6b9276de9	Adds arguments to filter archives ... These are: --sort-by, --first and --last Includes a method to obtain a list of archive infos filtered by these Archives.list: - ensure reverse is always applied - always return a list	2016-09-30 19:48:17 +02:00
Thomas Waldmann	19eb75984e	borg check --verify-data tuning	2016-09-29 18:40:02 +02:00
Thomas Waldmann	ba30098079	Merge branch '1.0-maint' into merge-1.0-maint	2016-09-29 12:57:29 +02:00
Thomas Waldmann	2a864be84f	Merge branch '1.0-maint' into merge-1.0-maint ... # Conflicts: # setup.py # src/borg/archiver.py # src/borg/helpers.py	2016-09-27 21:31:41 +02:00
enkore	ea9e859026	Merge pull request #1635 from enkore/issue/1624 ... Fix broken --progress for double-cell paths	2016-09-27 13:14:16 +02:00
Marian Beermann	9cef0a9ed8	Fix broken --progress ellipsis for double-cell paths	2016-09-27 11:35:45 +02:00
TW	233ae43551	Merge pull request #1640 from ThomasWaldmann/improve-borg-help ... Re-Indent borg help helptexts, again.	2016-09-25 23:26:01 +02:00
TW	43432f9a14	Merge pull request #1615 from enkore/issue/1598 ... Fix #1598	2016-09-25 23:09:47 +02:00
Thomas Waldmann	bf681e98ce	Re-Indent borg help helptexts, again.	2016-09-25 22:50:08 +02:00
TW	f36d551092	Merge pull request #1634 from ThomasWaldmann/improve-compression-docs ... borg help compression, fixes #1582	2016-09-25 22:42:56 +02:00
Marian Beermann	8164524d99	Fix broken --progress for double-cell paths	2016-09-25 22:18:37 +02:00
Thomas Waldmann	28c57f98c9	Merge branch '1.0-maint' ... # Conflicts: # src/borg/archive.py # src/borg/archiver.py # src/borg/helpers.py # src/borg/testsuite/archiver.py	2016-09-25 17:30:55 +02:00
Marian Beermann	66316e10b9	Fix indentation in borg help compression	2016-09-25 11:53:54 +02:00
Marian Beermann	7b1f10347a	Repository: compact: fix incorrect preservation of delete tags	2016-09-25 10:13:31 +02:00
Marian Beermann	abace16945	Repository.check: log transaction IDs	2016-09-25 10:12:42 +02:00
Thomas Waldmann	d6bfdafdef	borg help compression, fixes #1582	2016-09-25 02:32:02 +02:00
TW	d3e79c4e49	Merge pull request #1625 from bauerj/master ... Change {utcnow} and {now} to ISO-8601 format	2016-09-23 18:24:10 +02:00
Johann Bauer	a56dc44e1f	Change {utcnow} and {now} to ISO-8601 format	2016-09-23 17:58:04 +02:00
Martin Hostettler	84b3295a0d	Archiver,RemoteRepository: Add --remote-ratelimit ... The --remote-ratelimit option adds a very simple rate limit for the sending data to the remote. Currently implemented by sleeping if the transmission speed is greater than the limit.	2016-09-18 19:00:26 +02:00
Thomas Waldmann	260ef31728	fuse: refactor for less encoding dance	2016-09-16 16:38:08 +02:00
Thomas Waldmann	39170479aa	fuse: fix hardlinks in versions view, fixes #1599	2016-09-16 16:38:08 +02:00
Thomas Waldmann	c021cf466a	fuse: add test for hardlinks in versions view	2016-09-16 16:38:08 +02:00
Thomas Waldmann	001500ab99	fuse: refactor file versioning code	2016-09-16 16:38:08 +02:00
Thomas Waldmann	5d22078f35	fuse: add parameter check to ItemCache.get to make potential failures more clear	2016-09-16 16:38:08 +02:00
Marian Beermann	ae5b4980f2	Repository.check: improve object count mismatch diagnostic	2016-09-15 11:23:58 +02:00
TW	783d01f621	Merge pull request #1586 from ThomasWaldmann/borg-check-delete-defect-chunks ... borg check: delete chunks with integrity errors, fixes #1575	2016-09-14 22:55:50 +02:00
Thomas Waldmann	b4c7cce67d	borg check: delete chunks with integrity errors, fixes #1575 ... so they can be "repaired" immediately and maybe healed later.	2016-09-14 21:07:48 +02:00
Thomas Waldmann	57a3adb6b3	borg recreate: also catch SIGHUP	2016-09-14 02:55:13 +02:00
Thomas Waldmann	1287d1ae92	Merge branch '1.0-maint' into merge-1.0-maint ... # Conflicts: # docs/development.rst # src/borg/archive.py # src/borg/archiver.py # src/borg/hashindex.pyx # src/borg/testsuite/hashindex.py	2016-09-14 02:53:41 +02:00
Thomas Waldmann	dd89181062	fuse: refactor / optimize process_inner	2016-09-01 06:26:33 +02:00
Thomas Waldmann	6f50cc3413	fuse: simplify path segments loop	2016-09-01 05:18:10 +02:00
Thomas Waldmann	d64ccff1c4	fuse: remove unneeded function definition ... code only had 1 caller, so just inline it.	2016-09-01 04:57:11 +02:00
Thomas Waldmann	34eb475463	fuse: remove unneeded safe_encode ... fsencode will happen in _find_inode()	2016-09-01 04:33:42 +02:00
Thomas Waldmann	8b5372b54c	fuse: slightly refactor shadowing detection ... if it is not a directory, the old code was a NOP, so we can just check that first.	2016-09-01 04:30:55 +02:00
Thomas Waldmann	1f04820d9d	fuse: implement versions view ... all archives, all items are read to build a unified view. files are represented by a same-name directory with the versions of the file. A filename suffix computed by adler32(chunkids) is used to disambiguate the versions. also: refactor code a little, create methods for leaves, inner nodes.	2016-08-27 23:17:57 +02:00
enkore	69c51f268d	Merge pull request #1374 from textshell/issue/1039-aes-iv ... Fix AES-CTR nonce reusage	2016-08-27 15:52:21 +02:00
Martin Hostettler	620f505a14	Reserve nonce space for AES-CTR before using it. ... Reusing the nonce totally breaks AES-CTR confidentiality. This code uses a reservation of nonce space and stores the next nonce available for a future reservation on the client and in the repository. Local storage is needed to protect against evil repositories that try to gain access to encrypted data by not saving nonce reservations and aborting the connection or otherwise forcing a rollback. Storage in the repository is needed to protect against another client writing to the repository after a transaction was aborted and thus not seeing the last used nonce from the manifest. With a real counter mode cipher protection for the multiple client case with an actively evil repository is not possible. But this still protects against cases where the attacker can not arbitrarily change the repository but can read everything stored and abort connections or crash the server. Fixes #22	2016-08-27 15:31:50 +02:00
Marian Beermann	e4e6787906	recreate: fix crash if recompress and deduplication beyond autocommit_threshold ... ie. it means that if all the recompressed chunks were already in the repo no data would be written, so there would be no active txn, so failure ensues.	2016-08-27 14:31:57 +02:00
enkore	e60ed80433	Merge pull request #1539 from enkore/f/recreate-cp ... recreate: fix crash if archive doesn't have chunker_params	2016-08-27 14:06:32 +02:00
Marian Beermann	e289fb3539	recreate: fix crash if archive doesn't have chunker_params	2016-08-27 14:01:28 +02:00
enkore	1d750e802e	Merge pull request #1536 from enkore/f/prip2 ... ProgressIndicatorPercent / extract minor cosmetic enhancement	2016-08-27 00:36:43 +02:00
TW	168ce1dc86	Merge pull request #1535 from ThomasWaldmann/merge-1.0-maint ... Merge 1.0 maint	2016-08-27 00:32:40 +02:00
Marian Beermann	f0a32575a5	extract: --progress: no extra line for 'Calculating size'	2016-08-26 22:15:44 +02:00
Marian Beermann	517ccc2d58	ProgressIndicatorPercent: output(message) to override output	2016-08-26 22:15:29 +02:00
Thomas Waldmann	25fa443d2b	repo tests: convert some more byte literals to H(x)	2016-08-26 20:59:22 +02:00
Thomas Waldmann	1974824595	Merge branch '1.0-maint'	2016-08-26 20:47:23 +02:00
Marian Beermann	cd39ccb821	fix overeager storing of hardlink masters ... n.b. we only need to store them for items that we wouldn't extract. this also fixes an intersting edge case in extracting hard links with --strip-components	2016-08-25 21:16:20 +02:00
Marian Beermann	b5d7f1df26	extract: fix incorrect progress output for hard links ... this produces correct output if any (non proper) subset of hardlinks are extracted.	2016-08-25 01:12:30 +02:00
Martin Hostettler	b845a074cb	tests: TestBuildFilter: Adjust from item_is_hardlink_master to peek_and_store_hardlink_masters.	2016-08-22 23:36:43 +02:00
Martin Hostettler	a026febdb0	Archiver.build_filter: strip_components is no longer a optional parameter.	2016-08-22 23:07:38 +02:00
Martin Hostettler	972392e290	extract: When doing a partial restore don't leak prefetched chunks. ... The filter function passed to iter_items (with preload=True) may never return True for items that are not really extracted later because that would leak prefetched items. For restoring hard linked files the item containing the actual chunks might not be matched or implicitly removed from the restore by strip_components. For this reason the chunk list or all items that can potentially be used as hardlink target needs to be stored. To achive both requirements at the same time the filter function needs to store the needed information for the hardlinks while not returning True just because it could be a hardlink target. Known problems: When using progress indication the calculated extracted_size now can be smaller than the actual extracted size in presence of hard links (master is not restored) instead of bigger (potential master not used in restore).	2016-08-22 22:58:58 +02:00
Thomas Waldmann	d1d2738381	re-enable fuse tests for RemoteArchiver ... at some time they had deadlock issues, but it worked for me now.	2016-08-22 00:14:10 +02:00
Marian Beermann	53d0140bd5	Repository: add compact_segments progress	2016-08-21 21:55:53 +02:00
TW	2f65f7bdec	Merge pull request #1449 from enkore/f/extractprogress ... extract --progress	2016-08-21 20:24:44 +02:00
Marian Beermann	ab31ffaa85	ProgressIndicatorPercent: remove same_line	2016-08-21 19:57:36 +02:00
Marian Beermann	e7d44cec39	extract: --progress: Calculating size	2016-08-21 19:57:36 +02:00
TW	ab96c6496c	Merge pull request #1513 from ThomasWaldmann/improve-error-logging ... improve error logging, fixes #1440	2016-08-21 18:29:18 +02:00
Marian Beermann	4d214e2503	Simplify and test Item.file_size	2016-08-21 01:09:21 +02:00
Marian Beermann	5924915d35	Flip ProgressIndicatorPercent same_line default to True ... Every production use of this uses same_line=True	2016-08-21 01:09:20 +02:00
Marian Beermann	8709cec57c	borg-extract --progress	2016-08-21 01:07:55 +02:00
enkore	81dd381701	Merge pull request #1507 from ThomasWaldmann/use-modified-pyinstaller ... use patched LDLP-preserving pyinstaller	2016-08-21 00:16:24 +02:00
enkore	9baa024f4a	Merge pull request #1508 from ThomasWaldmann/merge-1.0-maint ... Merge 1.0 maint	2016-08-21 00:12:00 +02:00
Thomas Waldmann	41e4707d6a	fixup: meant "tb", not "traceback"	2016-08-20 23:09:42 +02:00
TW	357f4ef72e	Merge pull request #1511 from c4rlo/lru-cache ... helpers.py: replace memoize with functools.lru_cache	2016-08-20 21:19:17 +02:00
Thomas Waldmann	a0c8c40f27	improve error logging, fixes #1440 ... archiver: split traceback and msg, have separate log level for traceback, log LockTimeout at debug level, for "Error" exceptions: always log the traceback, either at ERROR or DEBUG level. remote: if we have an "Error" typed exception instance, we can use its traceback flag and .get_message() as we do locally.	2016-08-20 17:23:02 +02:00
Carlo Teubner	61af307ad4	helpers.py: replace memoize usages with lru_cache	2016-08-20 13:06:16 +01:00
Thomas Waldmann	e7fccaccb2	restore original LDLP, if possible, fixes #1498 ... see https://github.com/pyinstaller/pyinstaller/pull/2148	2016-08-20 12:31:23 +02:00
Carlo Teubner	28076ee588	helpers.py: replace memoize with functools.lru_cache	2016-08-20 10:01:59 +01:00
Carlo Teubner	13761c4ec2	daemonize(): use os.devnull instead of hardcoded /dev/null	2016-08-20 10:00:03 +01:00
Thomas Waldmann	c355f3617a	Merge branch '1.0-maint'	2016-08-20 00:04:55 +02:00
Thomas Waldmann	53d14e96de	borg list: test for --list-format deprecation	2016-08-19 14:55:49 +02:00
TW	32e84f33e9	Merge pull request #1484 from enkore/issue/1396 ... Track shadowing of log entries	2016-08-19 14:09:34 +02:00
Marian Beermann	c39e395ecf	Exclude incompatible tests for ArchiverTestCaseBinary	2016-08-17 23:01:20 +02:00
Marian Beermann	26e8ff2cbc	Repository: don't use defaultdict for shadow index ... avoids errors by accidentally inserting an empty list and makes it more clear.	2016-08-17 01:17:49 +02:00
Marian Beermann	833f8d1373	Track entire session ... Note how this enables the much stricter check in test_moved_deletes_are_tracked	2016-08-17 01:17:49 +02:00
Marian Beermann	ac41ebcf78	Track shadowing of log entries ... Fixes (benign) index vs. log inconsistencies when segments are skipped during compaction.	2016-08-17 01:17:49 +02:00
Marian Beermann	21c3fb3b93	compact_segments: add segment-level logging	2016-08-17 01:16:25 +02:00
TW	f7c1c71ff1	Merge pull request #1489 from enkore/issue/1486 ... Fix borg-list --list-format	2016-08-16 21:35:25 +02:00
Marian Beermann	f3defb02de	Fix borg-list empty format ... Should produce empty output, not default output.	2016-08-16 20:46:58 +02:00
Marian Beermann	146d531d9e	Fix borg-list --list-format	2016-08-16 20:46:54 +02:00
Thomas Waldmann	30cd7f3f21	fix timestamp key name	2016-08-16 20:36:29 +02:00
Thomas Waldmann	2f70e3f69a	src cleanup: do not use XXX in strings if just any other string works also ... XXX is usually used in comments to mark questionable places in the source.	2016-08-15 16:01:23 +02:00
Thomas Waldmann	3c8d86354b	src cleanup: get rid of text-as-bytes in borg.remote	2016-08-15 15:52:19 +02:00
Thomas Waldmann	1f056d9e8a	more safe interface for manifest.archives	2016-08-15 15:03:05 +02:00
Thomas Waldmann	b6d0eb99a5	add and use ManifestItem	2016-08-15 02:01:13 +02:00
Thomas Waldmann	c8922c8b3d	use ArchiveItem	2016-08-15 01:11:33 +02:00
Thomas Waldmann	585407f4f4	introduce ArchiveItem, see #1157	2016-08-14 22:04:01 +02:00
Marian Beermann	f0e9a55ebf	recreate: document that absolute patterns won't match	2016-08-14 17:46:59 +02:00
Marian Beermann	93b1cf3453	recreate: --target	2016-08-14 17:46:59 +02:00
Marian Beermann	88798ae949	recreate: --always-recompress, --compression-from ... what a mess	2016-08-14 17:46:59 +02:00
Marian Beermann	5433b1a1e4	Add Compressor.detect(data) -> CompressBase impl	2016-08-14 17:41:34 +02:00
Marian Beermann	c2c90645ad	Add Key.assert_id function	2016-08-14 17:41:33 +02:00
Marian Beermann	a80b371d09	Add decompress arg to Key.decrypt	2016-08-14 17:41:33 +02:00
Marian Beermann	ac80332ce7	Merge branch '1.0-maint' into master ... # Conflicts: # src/borg/repository.py # src/borg/testsuite/repository.py	2016-08-14 15:51:49 +02:00
Thomas Waldmann	42b6a838da	fix cyclic import issue, fix tests ... needed to increase ChunkBuffer size due to increased items stream chunk size to get the test working.	2016-08-14 15:26:56 +02:00
Thomas Waldmann	8be6761c26	Merge commit 'feb7e2517ef7ec07cc638953a86c726aada7d37e'	2016-08-14 15:05:11 +02:00
enkore	f8b2ffe999	Merge pull request #1458 from ThomasWaldmann/merge-1.0-maint ... Merge 1.0 maint	2016-08-11 14:18:03 +02:00
enkore	006d1df517	Merge pull request #1450 from enkore/f/recreatefactors ... minor recreate refactor	2016-08-09 23:28:58 +02:00
Marian Beermann	7d9fba5a41	ArchiveRecreater.copy_items: progress from target instead of source ... (it produces the same output, just less confusing)	2016-08-09 21:20:13 +02:00
TW	ff5d9a3f12	Merge pull request #1447 from enkore/f/debug-topic ... Debugging topics and some other minor stuff	2016-08-09 20:53:07 +02:00
Thomas Waldmann	c955d59273	Merge branch '1.0-maint'	2016-08-09 20:09:13 +02:00
Marian Beermann	0a65e83df3	check --verify-data: remove incorrect help paragraph ... (since d0ec7e76bb - doesn't mean this probably useful feature may not be reimplemented, on the contrary)	2016-08-08 16:38:31 +02:00
Marian Beermann	bec5051ea5	helpers.format_timedelta: use timedelta.total_seconds()	2016-08-08 16:38:31 +02:00
Marian Beermann	389ca94407	Silence repeated "Processing files..." log entries when --no-files-cache	2016-08-08 16:38:31 +02:00
Marian Beermann	4d7a52d8e0	--debug-topic for granular debug logging ... Especially: disables file-compression logging by default	2016-08-08 16:38:31 +02:00
Marian Beermann	7b70b74ad1	ArchiveRecreater.incref_partial_chunks try to use __contains__	2016-08-08 15:54:47 +02:00
Marian Beermann	5fe6c09c34	Refactor ArchiveRecreater.try_resume	2016-08-08 15:54:41 +02:00
Thomas Waldmann	dbe862f3d8	Merge branch '1.0-maint' into merge-1.0-maint	2016-08-06 01:24:22 +02:00
TW	9b0d8df0f8	Merge pull request #1373 from PlasmaPower/filesystem-test-skipping ... Filesystem feature detection and test skipping	2016-08-03 17:46:34 +02:00
TW	29e5e558bc	Merge pull request #1412 from ThomasWaldmann/info-repo ... borg info REPO	2016-08-02 23:29:39 +02:00
Thomas Waldmann	770a892b2d	implement borg info REPO ... currently it is just the same global stats also shown in "borg info ARCHIVE", just without the archive-specific stats. also: add separate test for "borg info".	2016-08-02 20:06:24 +02:00
enkore	b7a563885d	Merge pull request #1413 from ThomasWaldmann/store-mountpoint-dirs ... save mountpoint dirs, fixes #1033	2016-08-01 15:51:53 +02:00
Thomas Waldmann	2b454fc54b	save mountpoint dirs, fixes #1033	2016-08-01 15:25:54 +02:00
Lee Bousfield	b86b5d952a	Filesystem feature detection and test skipping	2016-07-31 01:19:31 -04:00
Thomas Waldmann	9865509105	docs: remove borg list example, be more specific about compression heuristics	2016-07-30 21:21:45 +02:00
TW	02e229b255	Merge pull request #1377 from enkore/issue/1060 ... Use atomic file writing and updating for configuration and key files	2016-07-30 16:06:01 +02:00
Marian Beermann	e9a73b808f	Check for sufficient free space before committing	2016-07-30 00:04:27 +02:00
Marian Beermann	389503db60	Repository.compact_segments: always save_space	2016-07-30 00:03:31 +02:00
Thomas Waldmann	999f0ae187	fix chunk refcounts considering part files	2016-07-28 19:39:29 +02:00
Thomas Waldmann	04ad1d1b0b	use "part file", "part", etc. consistently ... use .borg_part_N as filename to avoid collisions	2016-07-28 19:39:29 +02:00
Thomas Waldmann	a7fca52f07	create part items rather than checkpoint items ... checkpoint items: chunks abc, abcdef, abcdefghi, .... part items: chunks abc, def, ghi solves the "restore a big file over unstable connection issue" (not very comfortably, but one can just extract one part after the other and concatenate them to get the full file)	2016-07-28 19:39:29 +02:00
Thomas Waldmann	e5bd6cef20	add --consider-checkpoint-files option, update FAQ	2016-07-28 19:39:29 +02:00
Thomas Waldmann	49233be25d	filter out checkpoint files ... do not: - list them - extract them - diff them - include them for recreate	2016-07-28 19:39:29 +02:00
Thomas Waldmann	dd5f957e6f	in-file checkpoints, fixes #1198, fixes #1093 ... also: unify code for stdin and on-disk file processing	2016-07-28 19:39:29 +02:00
Thomas Waldmann	75d91c4bd1	file chunking: refactor code a little ... so it directly works with item.chunks list instead of a temporary list.	2016-07-28 19:39:28 +02:00
Thomas Waldmann	bda50b5958	stdin chunking: get Item into usable state early	2016-07-28 19:39:28 +02:00
Thomas Waldmann	9226fc6f6f	split stat_attrs into cheap and expensive part ... we already have stat results in st, so computing stat_simple_attrs is rather cheap (except the username/groupname lookup maybe) and gets the most important stuff right in the Item, so it is brought early into a good state. after chunking, stat_ext_attrs is called to add the more expensive-to-get attributes, like bsdflags, xattrs and ACLs.	2016-07-28 19:39:28 +02:00
Marian Beermann	0ae48dafbb	ObjectNotFound: give ID as hex-string	2016-07-28 18:41:50 +02:00
Marian Beermann	d0ec7e76bb	Fix borg-check --verify-data failing with rebuilt objects ... There are some instances where --repair would do something. In these instances --verify-data would fail if --repair was not given also, since the changes without --repair are only in the index, but not in the repository.	2016-07-28 18:41:08 +02:00
Marian Beermann	1b6b0cfae6	Fix borg-check --verify-data tripping over ObjectNotFounds	2016-07-28 18:40:20 +02:00
Lee Bousfield	b652f4039c	Remove trailing whitespace	2016-07-28 10:35:41 -04:00
Marian Beermann	1bad3fc552	Merge branch '1.0-maint' into master	2016-07-28 09:48:58 +02:00
Marian Beermann	ac8c8119a7	Merge branch '1.0-maint' into master	2016-07-28 00:29:43 +02:00
Marian Beermann	2e3fc9ddfc	SyncFile/SaveFile: default binary=False, just like open()	2016-07-26 22:49:25 +02:00
Marian Beermann	863ab66908	SaveFile: unlink(tmppath): only ignore FileNotFoundError	2016-07-26 22:40:23 +02:00
Marian Beermann	dec671d8ff	SaveFile: os.replace instead of rename	2016-07-26 22:39:45 +02:00
Lee Bousfield	ff108ef24b	Catch a ValueError when checking if a user exists	2016-07-23 15:03:22 -04:00
Lee Bousfield	868586dd37	Fixed noatime detection	2016-07-23 13:05:35 -04:00
Lee Bousfield	88bffab935	Allow running tests outside of Vagrant ... Fixes #1361	2016-07-23 10:31:16 -04:00
Marian Beermann	52a72cf73a	Fix some item subscripts that came from 1.0-maint	2016-07-17 19:13:09 +02:00
TW	963bdd340c	Merge pull request #1330 from enkore/issue/972 ... recreate: fix hogged signal handlers	2016-07-14 14:53:21 +02:00
Marian Beermann	fcb6930c53	recreate: fix hogged signal handlers	2016-07-14 02:11:13 +02:00
Marian Beermann	2e6eb63174	Merge branch '1.0-maint' into master	2016-07-14 01:28:26 +02:00
Thomas Waldmann	3baa8a3728	Merge branch '1.0-maint' ... # Conflicts: # docs/changes.rst # docs/usage/mount.rst.inc # src/borg/archive.py # src/borg/archiver.py # src/borg/fuse.py # src/borg/testsuite/archiver.py	2016-07-11 01:23:27 +02:00
Marian Beermann	f4be2b3523	Use platform.SaveFile for repository, cache and key files ... Fixes #1060	2016-07-09 21:10:46 +02:00
Marian Beermann	36ebc82748	Add platform.SaveFile	2016-07-09 21:10:26 +02:00
TW	4f7a211b02	Merge pull request #1290 from enkore/issue/1287-bp ... sync_dir: silence fsync() failing with EINVAL	2016-07-08 18:31:26 +02:00
Marian Beermann	567617ebbe	sync_dir: silence fsync() failing with EINVAL	2016-07-08 18:05:46 +02:00
TW	65475ea1aa	Merge pull request #1286 from enkore/issue/1269 ... Remove InternalOSError	2016-07-08 14:00:07 +02:00
Marian Beermann	6c3ca63456	Remove InternalOSError ... obsoleted by 5b453856ec and e96905c6b	2016-07-08 13:35:14 +02:00
Marian Beermann	11e02461d2	1.0-maint merge fixup (hexlify..decode -> bin_to_hex)	2016-07-08 13:29:51 +02:00
Marian Beermann	aade62c9f9	wrap get_flags with backup_io	2016-07-08 13:26:10 +02:00
Thomas Waldmann	fe627246e4	Merge branch '1.0-maint'	2016-07-08 12:13:52 +02:00
Thomas Waldmann	12f55f4e9f	Merge branch '1.0-maint'	2016-07-06 01:33:53 +02:00
Thomas Waldmann	f363ddd7ca	Merge branch '1.0-maint'	2016-07-04 20:11:21 +02:00
enkore	69f73b1752	Merge pull request #1227 from ThomasWaldmann/checkpoint-interval ... increase checkpoint-interval default from 300 to 1800s, fixes #896	2016-06-30 17:09:56 +02:00
Thomas Waldmann	87d6755108	Merge branch '1.0-maint'	2016-06-29 18:28:33 +02:00
Thomas Waldmann	c201654f45	increase checkpoint-interval default from 300 to 1800s, fixes #896 ... doing a checkpoint can be expensive, if the repo connection is slow and/or accessed via a network filesystem and/or the repo is large. thus: don't do it too often.	2016-06-27 20:05:20 +02:00
Thomas Waldmann	5791b89c04	use bytes type for ACL attributes, fixes #1205 ... this saves some encoding / decoding dance compared to having a str interface.	2016-06-26 23:04:04 +02:00
Thomas Waldmann	323894d195	Merge branch '1.0-maint'	2016-06-26 20:46:12 +02:00
Thomas Waldmann	ed128e517b	Merge branch '1.0-maint'	2016-06-25 17:18:14 +02:00
TW	3ff6ac09a6	Merge pull request #1179 from Abogical/list-format ... Apply --format when listing repository archives, fixes #1166	2016-06-24 18:29:03 +02:00
Abogical	6658112f8c	Add documentation for listing repository archives	2016-06-21 20:01:01 +03:00
Abogical	87173d25de	Added testsuite for listing repository archives	2016-06-20 19:23:22 +03:00
Abogical	c839eb63f7	Apply --format when listing repository archives	2016-06-20 19:23:15 +03:00
Abogical	0942bcd1b4	Move code block for the write() function at do_list()	2016-06-20 16:14:52 +03:00
James Clarke	d4f5172771	Correctly handle multiple LD_PRELOAD entries; fixes #1111	2016-06-19 22:04:03 +01:00
TW	d9a14e3658	Merge pull request #1184 from c4rlo/linux-syncfile-errno-fix ... platform.linux: small SyncFile error handling fix	2016-06-18 14:49:18 +02:00
Carlo Teubner	58d236c619	platform.linux: small SyncFile error handling fix	2016-06-18 13:39:22 +01:00
Carlo Teubner	ff470f49f3	typo fixes in comment+docstring	2016-06-18 12:57:54 +01:00
Thomas Waldmann	9a64835b4d	Merge branch '1.0-maint' ... Also: add missing keys to ARCHIVE_KEYS set.	2016-06-13 00:14:08 +02:00
TW	8c26945097	Merge pull request #1158 from ThomasWaldmann/pretty-key ... add Key/EncryptedKey PropDict, see #1157	2016-06-12 14:51:46 +02:00
Thomas Waldmann	d2468d37df	add Key/EncryptedKey PropDict	2016-06-12 04:28:40 +02:00
Michael Herold	d45a13babf	Changes archive format from uuid to uuid4 ... This allows for future support of other uuid versions	2016-06-11 22:57:03 +02:00
Michael Herold	5aa97e283c	Adds uuid to archive format tags	2016-06-11 12:03:55 +02:00
jenkins	b86775ab4f	Fix test_sparse_file for solaris	2016-06-10 12:21:03 +02:00
Thomas Waldmann	d813775065	Merge branch '1.0-maint'	2016-06-09 20:29:44 +02:00
Marian Beermann	e41f3ecc5b	SyncFile: open with O_EXCL (or equivalent)	2016-06-04 22:47:27 +02:00
Thomas Waldmann	75244ea6d1	fuse: simplify timestamp processing	2016-06-04 17:34:03 +02:00
Thomas Waldmann	60da32123a	refactor to use Item class, fixes #1071	2016-06-04 17:24:55 +02:00
Marian Beermann	0ca1a18b33	Make borg.platform a package	2016-05-31 15:38:56 +02:00
Marian Beermann	0333f91eb2	platform: rearrange imports	2016-05-31 15:38:16 +02:00
Marian Beermann	2d90bb2c55	Compose platform_base from the platform-specific functionality ... Noticed while working on the windows branch.	2016-05-31 15:38:16 +02:00
Thomas Waldmann	d3deec9fc9	cleanup / reorder imports ... import stdlib from stdlib import x import 3rd party import borg stuff	2016-05-31 01:18:03 +02:00
Thomas Waldmann	49484d6d67	split helpers import into multiple ones	2016-05-31 00:33:13 +02:00
Thomas Waldmann	7e0ea1635a	Merge branch 'master' into move-to-src	2016-05-30 19:41:30 +02:00
Thomas Waldmann	045e5a1203	Merge branch 'master' into move-to-src	2016-05-30 19:38:16 +02:00
Thomas Waldmann	3ce35f6843	Merge branch 'master' into move-to-src	2016-05-21 19:06:01 +02:00
Thomas Waldmann	d1ea925a5b	move borg package to src/	2016-05-05 20:19:50 +02:00