upstream/borgbackup
1
0
Fork 0
mirror of https://github.com/borgbackup/borg.git synced 2026-04-22 14:49:03 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge branch 'master' into multithreading

Browse source 
Some tests are failing, some block, one new test needed skipping.
This commit is contained in:
Thomas Waldmann 2015-10-31 19:36:33 +01:00
commit f4b0f57618
52 changed files with 7900 additions and 1472 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
.gitignore vendored
View file

@ -2,7 +2,7 @@ MANIFEST
docs/_build
build
dist
env
borg-env
.tox
hashindex.c
chunker.c
@ -16,6 +16,7 @@ platform_linux.c
*.pyo
*.so
docs/usage/*.inc
docs/api.rst
.idea/
.cache/
borg/_version.py

7
.travis.yml
View file

@ -17,6 +17,9 @@ matrix:
- python: 3.4
os: linux
env: TOXENV=py34
- python: 3.5
os: linux
env: TOXENV=py35
- language: generic
os: osx
osx_image: xcode6.4
@ -29,6 +32,10 @@ matrix:
os: osx
osx_image: xcode6.4
env: TOXENV=py34
- language: generic
os: osx
osx_image: xcode6.4
env: TOXENV=py35
install:
- ./.travis/install.sh

4
.travis/install.sh
View file

@ -30,6 +30,10 @@ if [[ "$(uname -s)" == 'Darwin' ]]; then
pyenv install 3.4.3
pyenv global 3.4.3
;;
py35)
pyenv install 3.5.0
pyenv global 3.5.0
;;
esac
pyenv rehash
python -m pip install --user virtualenv

4
.travis/run.sh
View file

@ -17,7 +17,7 @@ source ~/.venv/bin/activate
if [[ "$(uname -s)" == "Darwin" ]]; then
# no fakeroot on OS X
sudo tox -e $TOXENV
sudo tox -e $TOXENV -r
else
fakeroot -u tox
fakeroot -u tox -r
fi

10
AUTHORS
View file

@ -1,10 +1,14 @@
Borg Developers / Contributors ("The Borg Collective")
``````````````````````````````````````````````````````
Contributors ("The Borg Collective")
====================================
- Thomas Waldmann <tw@waldmann-edv.de>
- Antoine Beaupré
- Antoine Beaupré <anarcat@debian.org>
- Radek Podgorny <radek@podgorny.cz>
- Yuri D'Elia
Attic authors
-------------
Borg is a fork of Attic. Attic is written and maintained
by Jonas Borgström and various contributors:

510
CHANGES.rst
View file

@ -1,510 +0,0 @@
Borg Changelog
==============
Version 0.27.0
--------------
New features:
- "borg upgrade" command - attic -> borg one time converter / migration, #21
- temporary hack to avoid using lots of disk space for chunks.archive.d, #235:
To use it: rm -rf chunks.archive.d ; touch chunks.archive.d
- respect XDG_CACHE_HOME, attic #181
- add support for arbitrary SSH commands, attic #99
- borg delete --cache-only REPO (only delete cache, not REPO), attic #123
Bug fixes:
- use Debian 7 (wheezy) to build pyinstaller borgbackup binaries, fixes slow
down observed when running the Centos6-built binary on Ubuntu, #222
- do not crash on empty lock.roster, fixes #232
- fix multiple issues with the cache config version check, #234
- fix segment entry header size check, attic #352
plus other error handling improvements / code deduplication there.
- always give segment and offset in repo IntegrityErrors
Other changes:
- stop producing binary wheels, remove docs about it, #147
- docs:
- add warning about prune
- generate usage include files only as needed
- development docs: add Vagrant section
- update / improve / reformat FAQ
- hint to single-file pyinstaller binaries from README
Version 0.26.1
--------------
This is a minor update, just docs and new pyinstaller binaries.
- docs update about python and binary requirements
- better docs for --read-special, fix #220
- re-built the binaries, fix #218 and #213 (glibc version issue)
- update web site about single-file pyinstaller binaries
Note: if you did a python-based installation, there is no need to upgrade.
Version 0.26.0
--------------
New features:
- Faster cache sync (do all in one pass, remove tar/compression stuff), #163
- BORG_REPO env var to specify the default repo, #168
- read special files as if they were regular files, #79
- implement borg create --dry-run, attic issue #267
- Normalize paths before pattern matching on OS X, #143
- support OpenBSD and NetBSD (except xattrs/ACLs)
- support / run tests on Python 3.5
Bug fixes:
- borg mount repo: use absolute path, attic #200, attic #137
- chunker: use off_t to get 64bit on 32bit platform, #178
- initialize chunker fd to -1, so it's not equal to STDIN_FILENO (0)
- fix reaction to "no" answer at delete repo prompt, #182
- setup.py: detect lz4.h header file location
- to support python < 3.2.4, add less buggy argparse lib from 3.2.6 (#194)
- fix for obtaining 'char *' from temporary Python value (old code causes
a compile error on Mint 17.2)
- llfuse 0.41 install troubles on some platforms, require < 0.41
(UnicodeDecodeError exception due to non-ascii llfuse setup.py)
- cython code: add some int types to get rid of unspecific python add /
subtract operations (avoid undefined symbol FPE_... error on some platforms)
- fix verbose mode display of stdin backup
- extract: warn if a include pattern never matched, fixes #209,
implement counters for Include/ExcludePatterns
- archive names with slashes are invalid, attic issue #180
- chunker: add a check whether the POSIX_FADV_DONTNEED constant is defined -
fixes building on OpenBSD.
Other changes:
- detect inconsistency / corruption / hash collision, #170
- replace versioneer with setuptools_scm, #106
- docs:
- pkg-config is needed for llfuse installation
- be more clear about pruning, attic issue #132
- unit tests:
- xattr: ignore security.selinux attribute showing up
- ext3 seems to need a bit more space for a sparse file
- do not test lzma level 9 compression (avoid MemoryError)
- work around strange mtime granularity issue on netbsd, fixes #204
- ignore st_rdev if file is not a block/char device, fixes #203
- stay away from the setgid and sticky mode bits
- use Vagrant to do easy cross-platform testing (#196), currently:
- Debian 7 "wheezy" 32bit, Debian 8 "jessie" 64bit
- Ubuntu 12.04 32bit, Ubuntu 14.04 64bit
- Centos 7 64bit
- FreeBSD 10.2 64bit
- OpenBSD 5.7 64bit
- NetBSD 6.1.5 64bit
- Darwin (OS X Yosemite)
Version 0.25.0
--------------
Compatibility notes:
- lz4 compression library (liblz4) is a new requirement (#156)
- the new compression code is very compatible: as long as you stay with zlib
compression, older borg releases will still be able to read data from a
repo/archive made with the new code (note: this is not the case for the
default "none" compression, use "zlib,0" if you want a "no compression" mode
that can be read by older borg). Also the new code is able to read repos and
archives made with older borg versions (for all zlib levels 0..9).
Deprecations:
- --compression N (with N being a number, as in 0.24) is deprecated.
We keep the --compression 0..9 for now to not break scripts, but it is
deprecated and will be removed later, so better fix your scripts now:
--compression 0 (as in 0.24) is the same as --compression zlib,0 (now).
BUT: if you do not want compression, you rather want --compression none
(which is the default).
--compression 1 (in 0.24) is the same as --compression zlib,1 (now)
--compression 9 (in 0.24) is the same as --compression zlib,9 (now)
New features:
- create --compression none (default, means: do not compress, just pass through
data "as is". this is more efficient than zlib level 0 as used in borg 0.24)
- create --compression lz4 (super-fast, but not very high compression)
- create --compression zlib,N (slower, higher compression, default for N is 6)
- create --compression lzma,N (slowest, highest compression, default N is 6)
- honor the nodump flag (UF_NODUMP) and do not backup such items
- list --short just outputs a simple list of the files/directories in an archive
Bug fixes:
- fixed --chunker-params parameter order confusion / malfunction, fixes #154
- close fds of segments we delete (during compaction)
- close files which fell out the lrucache
- fadvise DONTNEED now is only called for the byte range actually read, not for
the whole file, fixes #158.
- fix issue with negative "all archives" size, fixes #165
- restore_xattrs: ignore if setxattr fails with EACCES, fixes #162
Other changes:
- remove fakeroot requirement for tests, tests run faster without fakeroot
(test setup does not fail any more without fakeroot, so you can run with or
without fakeroot), fixes #151 and #91.
- more tests for archiver
- recover_segment(): don't assume we have an fd for segment
- lrucache refactoring / cleanup, add dispose function, py.test tests
- generalize hashindex code for any key length (less hardcoding)
- lock roster: catch file not found in remove() method and ignore it
- travis CI: use requirements file
- improved docs:
- replace hack for llfuse with proper solution (install libfuse-dev)
- update docs about compression
- update development docs about fakeroot
- internals: add some words about lock files / locking system
- support: mention BountySource and for what it can be used
- theme: use a lighter green
- add pypi, wheel, dist package based install docs
- split install docs into system-specific preparations and generic instructions
Version 0.24.0
--------------
Incompatible changes (compared to 0.23):
- borg now always issues --umask NNN option when invoking another borg via ssh
on the repository server. By that, it's making sure it uses the same umask
for remote repos as for local ones. Because of this, you must upgrade both
server and client(s) to 0.24.
- the default umask is 077 now (if you do not specify via --umask) which might
be a different one as you used previously. The default umask avoids that
you accidentally give access permissions for group and/or others to files
created by borg (e.g. the repository).
Deprecations:
- "--encryption passphrase" mode is deprecated, see #85 and #97.
See the new "--encryption repokey" mode for a replacement.
New features:
- borg create --chunker-params ... to configure the chunker, fixes #16
(attic #302, attic #300, and somehow also #41).
This can be used to reduce memory usage caused by chunk management overhead,
so borg does not create a huge chunks index/repo index and eats all your RAM
if you back up lots of data in huge files (like VM disk images).
See docs/misc/create_chunker-params.txt for more information.
- borg info now reports chunk counts in the chunk index.
- borg create --compression 0..9 to select zlib compression level, fixes #66
(attic #295).
- borg init --encryption repokey (to store the encryption key into the repo),
fixes #85
- improve at-end error logging, always log exceptions and set exit_code=1
- LoggedIO: better error checks / exceptions / exception handling
- implement --remote-path to allow non-default-path borg locations, #125
- implement --umask M and use 077 as default umask for better security, #117
- borg check: give a named single archive to it, fixes #139
- cache sync: show progress indication
- cache sync: reimplement the chunk index merging in C
Bug fixes:
- fix segfault that happened for unreadable files (chunker: n needs to be a
signed size_t), #116
- fix the repair mode, #144
- repo delete: add destroy to allowed rpc methods, fixes issue #114
- more compatible repository locking code (based on mkdir), maybe fixes #92
(attic #317, attic #201).
- better Exception msg if no Borg is installed on the remote repo server, #56
- create a RepositoryCache implementation that can cope with >2GiB,
fixes attic #326.
- fix Traceback when running check --repair, attic #232
- clarify help text, fixes #73.
- add help string for --no-files-cache, fixes #140
Other changes:
- improved docs:
- added docs/misc directory for misc. writeups that won't be included
"as is" into the html docs.
- document environment variables and return codes (attic #324, attic #52)
- web site: add related projects, fix web site url, IRC #borgbackup
- Fedora/Fedora-based install instructions added to docs
- Cygwin-based install instructions added to docs
- updated AUTHORS
- add FAQ entries about redundancy / integrity
- clarify that borg extract uses the cwd as extraction target
- update internals doc about chunker params, memory usage and compression
- added docs about development
- add some words about resource usage in general
- document how to backup a raw disk
- add note about how to run borg from virtual env
- add solutions for (ll)fuse installation problems
- document what borg check does, fixes #138
- reorganize borgbackup.github.io sidebar, prev/next at top
- deduplicate and refactor the docs / README.rst
- use borg-tmp as prefix for temporary files / directories
- short prune options without "keep-" are deprecated, do not suggest them
- improved tox configuration
- remove usage of unittest.mock, always use mock from pypi
- use entrypoints instead of scripts, for better use of the wheel format and
modern installs
- add requirements.d/development.txt and modify tox.ini
- use travis-ci for testing based on Linux and (new) OS X
- use coverage.py, pytest-cov and codecov.io for test coverage support
I forgot to list some stuff already implemented in 0.23.0, here they are:
New features:
- efficient archive list from manifest, meaning a big speedup for slow
repo connections and "list <repo>", "delete <repo>", "prune" (attic #242,
attic #167)
- big speedup for chunks cache sync (esp. for slow repo connections), fixes #18
- hashindex: improve error messages
Other changes:
- explicitly specify binary mode to open binary files
- some easy micro optimizations
Version 0.23.0
--------------
Incompatible changes (compared to attic, fork related):
- changed sw name and cli command to "borg", updated docs
- package name (and name in urls) uses "borgbackup" to have less collisions
- changed repo / cache internal magic strings from ATTIC* to BORG*,
changed cache location to .cache/borg/ - this means that it currently won't
accept attic repos (see issue #21 about improving that)
Bug fixes:
- avoid defect python-msgpack releases, fixes attic #171, fixes attic #185
- fix traceback when trying to do unsupported passphrase change, fixes attic #189
- datetime does not like the year 10.000, fixes attic #139
- fix "info" all archives stats, fixes attic #183
- fix parsing with missing microseconds, fixes attic #282
- fix misleading hint the fuse ImportError handler gave, fixes attic #237
- check unpacked data from RPC for tuple type and correct length, fixes attic #127
- fix Repository._active_txn state when lock upgrade fails
- give specific path to xattr.is_enabled(), disable symlink setattr call that
always fails
- fix test setup for 32bit platforms, partial fix for attic #196
- upgraded versioneer, PEP440 compliance, fixes attic #257
New features:
- less memory usage: add global option --no-cache-files
- check --last N (only check the last N archives)
- check: sort archives in reverse time order
- rename repo::oldname newname (rename repository)
- create -v output more informative
- create --progress (backup progress indicator)
- create --timestamp (utc string or reference file/dir)
- create: if "-" is given as path, read binary from stdin
- extract: if --stdout is given, write all extracted binary data to stdout
- extract --sparse (simple sparse file support)
- extra debug information for 'fread failed'
- delete <repo> (deletes whole repo + local cache)
- FUSE: reflect deduplication in allocated blocks
- only allow whitelisted RPC calls in server mode
- normalize source/exclude paths before matching
- use posix_fadvise to not spoil the OS cache, fixes attic #252
- toplevel error handler: show tracebacks for better error analysis
- sigusr1 / sigint handler to print current file infos - attic PR #286
- RPCError: include the exception args we get from remote
Other changes:
- source: misc. cleanups, pep8, style
- docs and faq improvements, fixes, updates
- cleanup crypto.pyx, make it easier to adapt to other AES modes
- do os.fsync like recommended in the python docs
- source: Let chunker optionally work with os-level file descriptor.
- source: Linux: remove duplicate os.fsencode calls
- source: refactor _open_rb code a bit, so it is more consistent / regular
- source: refactor indicator (status) and item processing
- source: use py.test for better testing, flake8 for code style checks
- source: fix tox >=2.0 compatibility (test runner)
- pypi package: add python version classifiers, add FreeBSD to platforms
Attic Changelog
===============
Here you can see the full list of changes between each Attic release until Borg
forked from Attic:
Version 0.17
------------
(bugfix release, released on X)
- Fix hashindex ARM memory alignment issue (#309)
- Improve hashindex error messages (#298)
Version 0.16
------------
(bugfix release, released on May 16, 2015)
- Fix typo preventing the security confirmation prompt from working (#303)
- Improve handling of systems with improperly configured file system encoding (#289)
- Fix "All archives" output for attic info. (#183)
- More user friendly error message when repository key file is not found (#236)
- Fix parsing of iso 8601 timestamps with zero microseconds (#282)
Version 0.15
------------
(bugfix release, released on Apr 15, 2015)
- xattr: Be less strict about unknown/unsupported platforms (#239)
- Reduce repository listing memory usage (#163).
- Fix BrokenPipeError for remote repositories (#233)
- Fix incorrect behavior with two character directory names (#265, #268)
- Require approval before accessing relocated/moved repository (#271)
- Require approval before accessing previously unknown unencrypted repositories (#271)
- Fix issue with hash index files larger than 2GB.
- Fix Python 3.2 compatibility issue with noatime open() (#164)
- Include missing pyx files in dist files (#168)
Version 0.14
------------
(feature release, released on Dec 17, 2014)
- Added support for stripping leading path segments (#95)
"attic extract --strip-segments X"
- Add workaround for old Linux systems without acl_extended_file_no_follow (#96)
- Add MacPorts' path to the default openssl search path (#101)
- HashIndex improvements, eliminates unnecessary IO on low memory systems.
- Fix "Number of files" output for attic info. (#124)
- limit create file permissions so files aren't read while restoring
- Fix issue with empty xattr values (#106)
Version 0.13
------------
(feature release, released on Jun 29, 2014)
- Fix sporadic "Resource temporarily unavailable" when using remote repositories
- Reduce file cache memory usage (#90)
- Faster AES encryption (utilizing AES-NI when available)
- Experimental Linux, OS X and FreeBSD ACL support (#66)
- Added support for backup and restore of BSDFlags (OSX, FreeBSD) (#56)
- Fix bug where xattrs on symlinks were not correctly restored
- Added cachedir support. CACHEDIR.TAG compatible cache directories
can now be excluded using ``--exclude-caches`` (#74)
- Fix crash on extreme mtime timestamps (year 2400+) (#81)
- Fix Python 3.2 specific lockf issue (EDEADLK)
Version 0.12
------------
(feature release, released on April 7, 2014)
- Python 3.4 support (#62)
- Various documentation improvements a new style
- ``attic mount`` now supports mounting an entire repository not only
individual archives (#59)
- Added option to restrict remote repository access to specific path(s):
``attic serve --restrict-to-path X`` (#51)
- Include "all archives" size information in "--stats" output. (#54)
- Added ``--stats`` option to ``attic delete`` and ``attic prune``
- Fixed bug where ``attic prune`` used UTC instead of the local time zone
when determining which archives to keep.
- Switch to SI units (Power of 1000 instead 1024) when printing file sizes
Version 0.11
------------
(feature release, released on March 7, 2014)
- New "check" command for repository consistency checking (#24)
- Documentation improvements
- Fix exception during "attic create" with repeated files (#39)
- New "--exclude-from" option for attic create/extract/verify.
- Improved archive metadata deduplication.
- "attic verify" has been deprecated. Use "attic extract --dry-run" instead.
- "attic prune --hourly|daily|..." has been deprecated.
Use "attic prune --keep-hourly|daily|..." instead.
- Ignore xattr errors during "extract" if not supported by the filesystem. (#46)
Version 0.10
------------
(bugfix release, released on Jan 30, 2014)
- Fix deadlock when extracting 0 sized files from remote repositories
- "--exclude" wildcard patterns are now properly applied to the full path
not just the file name part (#5).
- Make source code endianness agnostic (#1)
Version 0.9
-----------
(feature release, released on Jan 23, 2014)
- Remote repository speed and reliability improvements.
- Fix sorting of segment names to ignore NFS left over files. (#17)
- Fix incorrect display of time (#13)
- Improved error handling / reporting. (#12)
- Use fcntl() instead of flock() when locking repository/cache. (#15)
- Let ssh figure out port/user if not specified so we don't override .ssh/config (#9)
- Improved libcrypto path detection (#23).
Version 0.8.1
-------------
(bugfix release, released on Oct 4, 2013)
- Fix segmentation fault issue.
Version 0.8
-----------
(feature release, released on Oct 3, 2013)
- Fix xattr issue when backing up sshfs filesystems (#4)
- Fix issue with excessive index file size (#6)
- Support access of read only repositories.
- New syntax to enable repository encryption:
attic init --encryption="none|passphrase|keyfile".
- Detect and abort if repository is older than the cache.
Version 0.7
-----------
(feature release, released on Aug 5, 2013)
- Ported to FreeBSD
- Improved documentation
- Experimental: Archives mountable as fuse filesystems.
- The "user." prefix is no longer stripped from xattrs on Linux
Version 0.6.1
-------------
(bugfix release, released on July 19, 2013)
- Fixed an issue where mtime was not always correctly restored.
Version 0.6
-----------
First public release on July 9, 2013

1
CHANGES.rst Symbolic link
View file

@ -0,0 +1 @@
docs/changes.rst

96
README.rst
View file

@ -1,5 +1,12 @@
|screencast|
.. |screencast| image:: https://asciinema.org/a/28691.png
:alt: BorgBackup Installation and Basic Usage
:target: https://asciinema.org/a/28691?autoplay=1&speed=2
What is BorgBackup?
-------------------
===================
BorgBackup (short: Borg) is a deduplicating backup program.
Optionally, it supports compression and authenticated encryption.
@ -9,11 +16,13 @@ since only changes are stored.
The authenticated encryption technique makes it suitable for backups to not
fully trusted targets.
`Borg Installation docs <http://borgbackup.github.io/borgbackup/installation.html>`_
See the `installation manual`_ or, if you have already
downloaded Borg, ``docs/installation.rst`` to get started with Borg.
.. _installation manual: https://borgbackup.readthedocs.org/installation.html
Main features
~~~~~~~~~~~~~
-------------
**Space efficient storage**
Deduplication based on content-defined chunking is used to reduce the number
of bytes stored: each file is split into a number of variable length chunks
@ -63,16 +72,16 @@ Main features
Backup archives are mountable as userspace filesystems for easy interactive
backup examination and restores (e.g. by using a regular file manager).
**Easy installation**
For Linux, Mac OS X and FreeBSD, we offer a single-file pyinstaller binary
that does not require installing anything - you can just run it.
**Easy installation on multiple platforms**
We offer single-file binaries
that does not require installing anything - you can just run it on
the supported platforms:
**Platforms Borg works on**
* Linux
* Mac OS X
* FreeBSD
* OpenBSD and NetBSD (for both: no xattrs/ACLs support yet)
* Cygwin (unsupported)
* Linux
* Mac OS X
* FreeBSD
* OpenBSD and NetBSD (no xattrs/ACLs support or binaries yet)
* Cygwin (not supported, no binaries yet)
**Free and Open Source Software**
* security and functionality can be audited independently
@ -80,7 +89,7 @@ Main features
Easy to use
~~~~~~~~~~~
-----------
Initialize a new backup repository and create a backup archive::
$ borg init /mnt/backup
@ -88,7 +97,7 @@ Initialize a new backup repository and create a backup archive::
Now doing another backup, just to show off the great deduplication::
$ borg create --stats /mnt/backup::Tuesday ~/Documents
$ borg create --stats -C zlib,6 /mnt/backup::Tuesday ~/Documents
Archive name: Tuesday
Archive fingerprint: 387a5e3f9b0e792e91c...
@ -100,29 +109,68 @@ Now doing another backup, just to show off the great deduplication::
This archive: 57.16 MB 46.78 MB 151.67 kB <--- !
All archives: 114.02 MB 93.46 MB 44.81 MB
For a graphical frontend refer to our complementary project
`BorgWeb <https://github.com/borgbackup/borgweb>`_.
For a graphical frontend refer to our complementary project `BorgWeb`_.
Links
=====
* `Main Web Site <https://borgbackup.readthedocs.org/>`_
* `Releases <https://github.com/borgbackup/borg/releases>`_
* `PyPI packages <https://pypi.python.org/pypi/borgbackup>`_
* `ChangeLog <https://github.com/borgbackup/borg/blob/master/CHANGES.rst>`_
* `GitHub <https://github.com/borgbackup/borg>`_
* `Issue Tracker <https://github.com/borgbackup/borg/issues>`_
* `Bounties & Fundraisers <https://www.bountysource.com/teams/borgbackup>`_
* `Mailing List <http://librelist.com/browser/borgbackup/>`_
* `License <https://borgbackup.github.io/borgbackup/authors.html#license>`_
Related Projects
----------------
* `BorgWeb <https://borgbackup.github.io/borgweb/>`_
* `Atticmatic <https://github.com/witten/atticmatic/>`_
* `Attic <https://github.com/jborg/attic>`_
Notes
-----
Borg is a fork of `Attic <https://github.com/jborg/attic>`_ and maintained by
"`The Borg Collective <https://github.com/borgbackup/borg/blob/master/AUTHORS>`_".
Borg is a fork of `Attic`_ and maintained by "`The Borg collective`_".
Read `issue #1 <https://github.com/borgbackup/borg/issues/1>`_ about the initial
considerations regarding project goals and policy of the Borg project.
.. _The Borg collective: https://borgbackup.readthedocs.org/authors.html
Differences between Attic and Borg
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here's a (incomplete) list of some major changes:
* more open, faster paced development (see `issue #1 <https://github.com/borgbackup/borg/issues/1>`_)
* lots of attic issues fixed (see `issue #5 <https://github.com/borgbackup/borg/issues/5>`_)
* less chunk management overhead via --chunker-params option (less memory and disk usage)
* faster remote cache resync (useful when backing up multiple machines into same repo)
* compression: no, lz4, zlib or lzma compression, adjustable compression levels
* repokey replaces problematic passphrase mode (you can't change the passphrase nor the pbkdf2 iteration count in "passphrase" mode)
* simple sparse file support, great for virtual machine disk files
* can read special files (e.g. block devices) or from stdin, write to stdout
* mkdir-based locking is more compatible than attic's posix locking
* uses fadvise to not spoil / blow up the fs cache
* better error messages / exception handling
* better output for verbose mode, progress indication
* tested on misc. Linux systems, 32 and 64bit, FreeBSD, OpenBSD, NetBSD, Mac OS X
Please read the `ChangeLog`_ (or ``CHANGES.rst`` in the source distribution) for more
information.
BORG IS NOT COMPATIBLE WITH ORIGINAL ATTIC (but there is a one-way conversion).
BORG IS NOT COMPATIBLE WITH ORIGINAL ATTIC.
EXPECT THAT WE WILL BREAK COMPATIBILITY REPEATEDLY WHEN MAJOR RELEASE NUMBER
CHANGES (like when going from 0.x.y to 1.0.0). Please read CHANGES document.
CHANGES (like when going from 0.x.y to 1.0.0).
NOT RELEASED DEVELOPMENT VERSIONS HAVE UNKNOWN COMPATIBILITY PROPERTIES.
THIS IS SOFTWARE IN DEVELOPMENT, DECIDE YOURSELF WHETHER IT FITS YOUR NEEDS.
For more information, please also see the
`LICENSE <https://github.com/borgbackup/borg/blob/master/LICENSE>`_.
Borg is distributed under a 3-clause BSD license, see `License`_
for the complete license.
|build| |coverage|

7
Vagrantfile vendored
View file

@ -22,6 +22,7 @@ def packages_debianoid
apt-get update
# for building borgbackup and dependencies:
apt-get install -y libssl-dev libacl1-dev liblz4-dev libfuse-dev fuse pkg-config
usermod -a -G fuse vagrant
apt-get install -y fakeroot build-essential git
apt-get install -y python3-dev python3-setuptools
# for building python:
@ -137,7 +138,7 @@ end
def install_pyenv(boxname)
return <<-EOF
curl -s -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash
echo 'export PATH="$HOME/.pyenv/bin:$PATH"' >> ~/.bash_profile
echo 'export PATH="$HOME/.pyenv/bin:/vagrant/borg:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile
echo 'export PYTHON_CONFIGURE_OPTS="--enable-shared"' >> ~/.bash_profile
@ -232,7 +233,7 @@ def build_binary_with_pyinstaller(boxname)
cd /vagrant/borg
. borg-env/bin/activate
cd borg
pyinstaller -F -n borg --hidden-import=logging.config borg/__main__.py
pyinstaller -F -n borg.exe --distpath=/vagrant/borg --clean --hidden-import=logging.config borg/__main__.py
EOF
end
@ -247,8 +248,10 @@ def run_tests(boxname)
fi
# otherwise: just use the system python
if which fakeroot > /dev/null; then
echo "Running tox WITH fakeroot -u"
fakeroot -u tox --skip-missing-interpreters
else
echo "Running tox WITHOUT fakeroot -u"
tox --skip-missing-interpreters
fi
EOF

74
borg/archive.py
View file

@ -1,11 +1,16 @@
from binascii import hexlify
from datetime import datetime
from getpass import getuser
from itertools import groupby
import errno
import threading
import logging
from .logger import create_logger
logger = create_logger()
from .key import key_factory
from .remote import cache_if_remote
import msgpack
from multiprocessing import cpu_count
import os
import socket
@ -14,12 +19,17 @@ import sys
import time
from io import BytesIO
from . import xattr
from .platform import acl_get, acl_set
from .chunker import Chunker
from .hashindex import ChunkIndex
from .helpers import parse_timestamp, Error, uid2user, user2uid, gid2group, group2gid, \
Manifest, Statistics, decode_dict, st_mtime_ns, make_path_safe, StableDict, int_to_bigint, bigint_to_int, \
make_queue, TerminatedQueue
from .helpers import parse_timestamp, Error, uid2user, user2uid, gid2group, group2gid, format_timedelta, \
Manifest, Statistics, decode_dict, make_path_safe, StableDict, int_to_bigint, bigint_to_int, have_cython, \
st_atime_ns, st_ctime_ns, st_mtime_ns, make_queue, TerminatedQueue
if have_cython():
from .platform import acl_get, acl_set
from .chunker import Chunker
from .hashindex import ChunkIndex
import msgpack
else:
import mock
msgpack = mock.Mock()
ITEMS_BUFFER = 1024 * 1024
@ -317,7 +327,8 @@ class Archive:
def __init__(self, repository, key, manifest, name, cache=None, create=False,
checkpoint_interval=300, numeric_owner=False, progress=False,
chunker_params=CHUNKER_PARAMS):
chunker_params=CHUNKER_PARAMS,
start=datetime.now(), end=datetime.now()):
self.cwd = os.getcwd()
self.key = key
self.repository = repository
@ -330,6 +341,8 @@ class Archive:
self.name = name
self.checkpoint_interval = checkpoint_interval
self.numeric_owner = numeric_owner
self.start = start
self.end = end
self.pipeline = DownloadPipeline(self.repository, self.key)
if create:
self.pp = ParallelProcessor(self)
@ -375,6 +388,22 @@ class Archive:
"""Timestamp of archive creation in UTC"""
return parse_timestamp(self.metadata[b'time'])
@property
def fpr(self):
return hexlify(self.id).decode('ascii')
@property
def duration(self):
return format_timedelta(self.end-self.start)
def __str__(self):
return '''Archive name: {0.name}
Archive fingerprint: {0.fpr}
Start time: {0.start:%c}
End time: {0.end:%c}
Duration: {0.duration}
Number of files: {0.stats.nfiles}'''.format(self)
def __repr__(self):
return 'Archive(%r)' % self.name
@ -565,12 +594,17 @@ class Archive:
elif has_lchmod: # Not available on Linux
os.lchmod(path, item[b'mode'])
mtime = bigint_to_int(item[b'mtime'])
if b'atime' in item:
atime = bigint_to_int(item[b'atime'])
else:
# old archives only had mtime in item metadata
atime = mtime
if fd and utime_supports_fd: # Python >= 3.3
os.utime(fd, None, ns=(mtime, mtime))
os.utime(fd, None, ns=(atime, mtime))
elif utime_supports_follow_symlinks: # Python >= 3.3
os.utime(path, None, ns=(mtime, mtime), follow_symlinks=False)
os.utime(path, None, ns=(atime, mtime), follow_symlinks=False)
elif not symlink:
os.utime(path, (mtime / 1e9, mtime / 1e9))
os.utime(path, (atime / 1e9, mtime / 1e9))
acl_set(path, item, self.numeric_owner)
# Only available on OS X and FreeBSD
if has_lchflags and b'bsdflags' in item:
@ -609,7 +643,9 @@ class Archive:
b'mode': st.st_mode,
b'uid': st.st_uid, b'user': uid2user(st.st_uid),
b'gid': st.st_gid, b'group': gid2group(st.st_gid),
b'mtime': int_to_bigint(st_mtime_ns(st))
b'atime': int_to_bigint(st_atime_ns(st)),
b'ctime': int_to_bigint(st_ctime_ns(st)),
b'mtime': int_to_bigint(st_mtime_ns(st)),
}
if self.numeric_owner:
item[b'user'] = item[b'group'] = None
@ -677,7 +713,10 @@ class Archive:
else:
self.hard_links[st.st_ino, st.st_dev] = safe_path
path_hash = self.key.id_hash(os.path.join(self.cwd, path).encode('utf-8', 'surrogateescape'))
first_run = not cache.files
ids = cache.file_known_and_unchanged(path_hash, st)
if first_run:
logger.info('processing files')
chunks = None
if ids is not None:
# Make sure all ids are available
@ -713,7 +752,7 @@ class Archive:
@staticmethod
def _open_rb(path, st):
flags_normal = os.O_RDONLY | getattr(os, 'O_BINARY', 0)
flags_noatime = flags_normal | getattr(os, 'NO_ATIME', 0)
flags_noatime = flags_normal | getattr(os, 'O_NOATIME', 0)
euid = None
def open_simple(p, s):
@ -833,7 +872,7 @@ class ArchiveChecker:
self.orphan_chunks_check()
self.finish()
if not self.error_found:
self.report_progress('Archive consistency check complete, no problems found.')
logger.info('Archive consistency check complete, no problems found.')
return self.repair or not self.error_found
def init_chunks(self):
@ -855,7 +894,7 @@ class ArchiveChecker:
def report_progress(self, msg, error=False):
if error:
self.error_found = True
print(msg, file=sys.stderr if error else sys.stdout)
logger.log(logging.ERROR if error else logging.WARNING, msg)
def identify_key(self, repository):
cdata = repository.get(next(self.chunks.iteritems())[0])
@ -982,7 +1021,7 @@ class ArchiveChecker:
num_archives = 1
end = 1
for i, (name, info) in enumerate(archive_items[:end]):
self.report_progress('Analyzing archive {} ({}/{})'.format(name, num_archives - i, num_archives))
logger.info('Analyzing archive {} ({}/{})'.format(name, num_archives - i, num_archives))
archive_id = info[b'id']
if archive_id not in self.chunks:
self.report_progress('Archive metadata block is missing', error=True)
@ -994,7 +1033,8 @@ class ArchiveChecker:
archive = StableDict(msgpack.unpackb(data))
if archive[b'version'] != 1:
raise Exception('Unknown archive metadata version')
decode_dict(archive, (b'name', b'hostname', b'username', b'time')) # fixme: argv
decode_dict(archive, (b'name', b'hostname', b'username', b'time'))
archive[b'cmdline'] = [arg.decode('utf-8', 'surrogateescape') for arg in archive[b'cmdline']]
items_buffer = ChunkBuffer(self.key)
items_buffer.write_chunk = add_callback
for item in robust_iterator(archive):

298
borg/archiver.py
View file

@ -15,17 +15,21 @@ import textwrap
import traceback
from . import __version__
from .archive import Archive, ArchiveChecker, CHUNKER_PARAMS
from .compress import Compressor, COMPR_BUFFER
from .upgrader import AtticRepositoryUpgrader
from .repository import Repository
from .cache import Cache
from .key import key_creator
from .helpers import Error, location_validator, format_time, format_file_size, \
format_file_mode, ExcludePattern, IncludePattern, exclude_path, adjust_patterns, to_localtime, timestamp, \
get_cache_dir, get_keys_dir, format_timedelta, prune_within, prune_split, \
Manifest, remove_surrogates, update_excludes, format_archive, check_extension_modules, Statistics, \
is_cachedir, bigint_to_int, ChunkerParams, CompressionSpec
is_cachedir, bigint_to_int, ChunkerParams, CompressionSpec, have_cython, \
EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR
from .logger import create_logger, setup_logging
logger = create_logger()
if have_cython():
from .compress import Compressor, COMPR_BUFFER
from .upgrader import AtticRepositoryUpgrader
from .repository import Repository
from .cache import Cache
from .key import key_creator
from .archive import Archive, ArchiveChecker, CHUNKER_PARAMS
from .remote import RepositoryServer, RemoteRepository
has_lchflags = hasattr(os, 'lchflags')
@ -33,8 +37,9 @@ has_lchflags = hasattr(os, 'lchflags')
class Archiver:
def __init__(self):
self.exit_code = 0
def __init__(self, verbose=False):
self.exit_code = EXIT_SUCCESS
self.verbose = verbose
def open_repository(self, location, create=False, exclusive=False):
if location.proto == 'ssh':
@ -46,16 +51,21 @@ class Archiver:
def print_error(self, msg, *args):
msg = args and msg % args or msg
self.exit_code = 1
print('borg: ' + msg, file=sys.stderr)
self.exit_code = EXIT_ERROR
logger.error(msg)
def print_verbose(self, msg, *args, **kw):
def print_warning(self, msg, *args):
msg = args and msg % args or msg
self.exit_code = EXIT_WARNING # we do not terminate here, so it is a warning
logger.warning(msg)
def print_info(self, msg, *args):
if self.verbose:
msg = args and msg % args or msg
if kw.get('newline', True):
print(msg)
else:
print(msg, end=' ')
logger.info(msg)
def print_status(self, status, path):
self.print_info("%1s %s", status, remove_surrogates(path))
def do_serve(self, args):
"""Start in server mode. This command is usually not used manually.
@ -64,7 +74,7 @@ class Archiver:
def do_init(self, args):
"""Initialize an empty repository"""
print('Initializing repository at "%s"' % args.repository.orig)
logger.info('Initializing repository at "%s"' % args.repository.orig)
repository = self.open_repository(args.repository, create=True, exclusive=True)
key = key_creator(repository, args)
manifest = Manifest(key, repository)
@ -79,29 +89,29 @@ class Archiver:
repository = self.open_repository(args.repository, exclusive=args.repair)
if args.repair:
while not os.environ.get('BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'):
self.print_error("""Warning: 'check --repair' is an experimental feature that might result
self.print_warning("""'check --repair' is an experimental feature that might result
in data loss.
Type "Yes I am sure" if you understand this and want to continue.\n""")
if input('Do you want to continue? ') == 'Yes I am sure':
break
if not args.archives_only:
print('Starting repository check...')
logger.info('Starting repository check...')
if repository.check(repair=args.repair):
print('Repository check complete, no problems found.')
logger.info('Repository check complete, no problems found.')
else:
return 1
return EXIT_WARNING
if not args.repo_only and not ArchiveChecker().check(
repository, repair=args.repair, archive=args.repository.archive, last=args.last):
return 1
return 0
return EXIT_WARNING
return EXIT_SUCCESS
def do_change_passphrase(self, args):
"""Change repository key file passphrase"""
repository = self.open_repository(args.repository)
manifest, key = Manifest.load(repository)
key.change_passphrase()
return 0
return EXIT_SUCCESS
def do_create(self, args):
"""Create new archive"""
@ -117,7 +127,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
archive = Archive(repository, key, manifest, args.archive.archive, cache=cache,
create=True, checkpoint_interval=args.checkpoint_interval,
numeric_owner=args.numeric_owner, progress=args.progress,
chunker_params=args.chunker_params)
chunker_params=args.chunker_params, start=t0)
else:
archive = cache = None
try:
@ -142,17 +152,18 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
try:
status = archive.process_stdin(path, cache)
except IOError as e:
self.print_error('%s: %s', path, e)
status = 'E'
self.print_warning('%s: %s', path, e)
else:
status = '-'
self.print_verbose("%1s %s", status, path)
self.print_status(status, path)
continue
path = os.path.normpath(path)
if args.dontcross:
if args.one_file_system:
try:
restrict_dev = os.lstat(path).st_dev
except OSError as e:
self.print_error('%s: %s', path, e)
self.print_warning('%s: %s', path, e)
continue
else:
restrict_dev = None
@ -163,16 +174,12 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
if args.progress:
archive.stats.show_progress(final=True)
if args.stats:
t = datetime.now()
diff = t - t0
archive.end = datetime.now()
print('-' * 78)
print('Archive name: %s' % args.archive.archive)
print('Archive fingerprint: %s' % hexlify(archive.id).decode('ascii'))
print('Start time: %s' % t0.strftime('%c'))
print('End time: %s' % t.strftime('%c'))
print('Duration: %s' % format_timedelta(diff))
print('Number of files: %d' % archive.stats.nfiles)
archive.stats.print_('This archive:', cache)
print(str(archive))
print()
print(str(archive.stats))
print(str(cache))
print('-' * 78)
finally:
if not dry_run:
@ -186,7 +193,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
try:
st = os.lstat(path)
except OSError as e:
self.print_error('%s: %s', path, e)
self.print_warning('%s: %s', path, e)
return
if (st.st_ino, st.st_dev) in skip_inodes:
return
@ -203,7 +210,8 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
try:
status = archive.process_file(path, st, cache)
except IOError as e:
self.print_error('%s: %s', path, e)
status = 'E'
self.print_warning('%s: %s', path, e)
elif stat.S_ISDIR(st.st_mode):
if exclude_caches and is_cachedir(path):
return
@ -212,7 +220,8 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
try:
entries = os.listdir(path)
except OSError as e:
self.print_error('%s: %s', path, e)
status = 'E'
self.print_warning('%s: %s', path, e)
else:
for filename in sorted(entries):
entry_path = os.path.normpath(os.path.join(path, filename))
@ -232,7 +241,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
# Ignore unix sockets
return
else:
self.print_error('Unknown file type: %s', path)
self.print_warning('Unknown file type: %s', path)
return
# Status output
# A lowercase character means a file type other than a regular file,
@ -249,13 +258,13 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
status = '-' # dry run, item was not backed up
# output ALL the stuff - it can be easily filtered using grep.
# even stuff considered unchanged might be interesting.
self.print_verbose("%1s %s", status, remove_surrogates(path))
self.print_status(status, path)
def do_extract(self, args):
"""Extract archive contents"""
# be restrictive when restoring files, restore permissions later
if sys.getfilesystemencoding() == 'ascii':
print('Warning: File system encoding is "ascii", extracting non-ascii filenames will not be supported.')
logger.warning('Warning: File system encoding is "ascii", extracting non-ascii filenames will not be supported.')
repository = self.open_repository(args.archive)
manifest, key = Manifest.load(repository)
archive = Archive(repository, key, manifest, args.archive.archive,
@ -272,7 +281,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
item[b'path'] = os.sep.join(orig_path.split(os.sep)[strip_components:])
if not item[b'path']:
continue
self.print_verbose(remove_surrogates(orig_path))
self.print_info(remove_surrogates(orig_path))
try:
if dry_run:
archive.extract_item(item, dry_run=True)
@ -283,7 +292,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
else:
archive.extract_item(item, stdout=stdout, sparse=sparse)
except IOError as e:
self.print_error('%s: %s', remove_surrogates(orig_path), e)
self.print_warning('%s: %s', remove_surrogates(orig_path), e)
if not args.dry_run:
# need to set each directory's timestamps AFTER all files in it are
@ -293,7 +302,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
archive.extract_item(dirs.pop(-1))
for pattern in (patterns or []):
if isinstance(pattern, IncludePattern) and pattern.match_count == 0:
self.print_error("Warning: Include pattern '%s' never matched.", pattern)
self.print_warning("Include pattern '%s' never matched.", pattern)
return self.exit_code
def do_rename(self, args):
@ -321,21 +330,23 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
repository.commit()
cache.commit()
if args.stats:
stats.print_('Deleted data:', cache)
logger.info(stats.summary.format(label='Deleted data:', stats=stats))
logger.info(str(cache))
else:
if not args.cache_only:
print("You requested to completely DELETE the repository *including* all archives it contains:")
print("You requested to completely DELETE the repository *including* all archives it contains:", file=sys.stderr)
for archive_info in manifest.list_archive_infos(sort_by='ts'):
print(format_archive(archive_info))
print(format_archive(archive_info), file=sys.stderr)
if not os.environ.get('BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'):
print("""Type "YES" if you understand this and want to continue.\n""")
print("""Type "YES" if you understand this and want to continue.\n""", file=sys.stderr)
# XXX: prompt may end up on stdout, but we'll assume that input() does the right thing
if input('Do you want to continue? ') != 'YES':
self.exit_code = 1
self.exit_code = EXIT_ERROR
return self.exit_code
repository.destroy()
print("Repository deleted.")
logger.info("Repository deleted.")
cache.destroy()
print("Cache deleted.")
logger.info("Cache deleted.")
return self.exit_code
def do_mount(self, args):
@ -343,7 +354,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
try:
from .fuse import FuseOperations
except ImportError as e:
self.print_error('loading fuse support failed [ImportError: %s]' % str(e))
self.print_error('Loading fuse support failed [ImportError: %s]' % str(e))
return self.exit_code
if not os.path.isdir(args.mountpoint) or not os.access(args.mountpoint, os.R_OK | os.W_OK | os.X_OK):
@ -351,18 +362,21 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
return self.exit_code
repository = self.open_repository(args.src)
manifest, key = Manifest.load(repository)
if args.src.archive:
archive = Archive(repository, key, manifest, args.src.archive)
else:
archive = None
operations = FuseOperations(key, repository, manifest, archive)
self.print_verbose("Mounting filesystem")
try:
operations.mount(args.mountpoint, args.options, args.foreground)
except RuntimeError:
# Relevant error message already printed to stderr by fuse
self.exit_code = 1
manifest, key = Manifest.load(repository)
if args.src.archive:
archive = Archive(repository, key, manifest, args.src.archive)
else:
archive = None
operations = FuseOperations(key, repository, manifest, archive)
self.print_info("Mounting filesystem")
try:
operations.mount(args.mountpoint, args.options, args.foreground)
except RuntimeError:
# Relevant error message already printed to stderr by fuse
self.exit_code = EXIT_ERROR
finally:
repository.close()
return self.exit_code
def do_list(self, args):
@ -421,7 +435,9 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
print('Time: %s' % to_localtime(archive.ts).strftime('%c'))
print('Command line:', remove_surrogates(' '.join(archive.metadata[b'cmdline'])))
print('Number of files: %d' % stats.nfiles)
stats.print_('This archive:', cache)
print()
print(str(stats))
print(str(cache))
return self.exit_code
def do_prune(self, args):
@ -433,7 +449,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
if args.hourly + args.daily + args.weekly + args.monthly + args.yearly == 0 and args.within is None:
self.print_error('At least one of the "within", "keep-hourly", "keep-daily", "keep-weekly", '
'"keep-monthly" or "keep-yearly" settings must be specified')
return 1
return self.exit_code
if args.prefix:
archives = [archive for archive in archives if archive.name.startswith(args.prefix)]
keep = []
@ -454,19 +470,20 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
to_delete = [a for a in archives if a not in keep]
stats = Statistics()
for archive in keep:
self.print_verbose('Keeping archive: %s' % format_archive(archive))
self.print_info('Keeping archive: %s' % format_archive(archive))
for archive in to_delete:
if args.dry_run:
self.print_verbose('Would prune: %s' % format_archive(archive))
self.print_info('Would prune: %s' % format_archive(archive))
else:
self.print_verbose('Pruning archive: %s' % format_archive(archive))
self.print_info('Pruning archive: %s' % format_archive(archive))
Archive(repository, key, manifest, archive.name, cache).delete(stats)
if to_delete and not args.dry_run:
manifest.write()
repository.commit()
cache.commit()
if args.stats:
stats.print_('Deleted data:', cache)
logger.info(stats.summary.format(label='Deleted data:', stats=stats))
logger.info(str(cache))
return self.exit_code
def do_upgrade(self, args):
@ -482,7 +499,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
# XXX: should auto-detect if it is an attic repository here
repo = AtticRepositoryUpgrader(args.repository.path, create=False)
try:
repo.upgrade(args.dry_run)
repo.upgrade(args.dry_run, inplace=args.inplace)
except NotImplementedError as e:
print("warning: %s" % e)
return self.exit_code
@ -540,7 +557,9 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
('--daily', '--keep-daily', 'Warning: "--daily" has been deprecated. Use "--keep-daily" instead.'),
('--weekly', '--keep-weekly', 'Warning: "--weekly" has been deprecated. Use "--keep-weekly" instead.'),
('--monthly', '--keep-monthly', 'Warning: "--monthly" has been deprecated. Use "--keep-monthly" instead.'),
('--yearly', '--keep-yearly', 'Warning: "--yearly" has been deprecated. Use "--keep-yearly" instead.')
('--yearly', '--keep-yearly', 'Warning: "--yearly" has been deprecated. Use "--keep-yearly" instead.'),
('--do-not-cross-mountpoints', '--one-file-system',
'Warning: "--do-no-cross-mountpoints" has been deprecated. Use "--one-file-system" instead.'),
]
if args and args[0] == 'verify':
print('Warning: "borg verify" has been deprecated. Use "borg extract --dry-run" instead.')
@ -552,26 +571,9 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
print(warning)
return args
def run(self, args=None):
check_extension_modules()
keys_dir = get_keys_dir()
if not os.path.exists(keys_dir):
os.makedirs(keys_dir)
os.chmod(keys_dir, stat.S_IRWXU)
cache_dir = get_cache_dir()
if not os.path.exists(cache_dir):
os.makedirs(cache_dir)
os.chmod(cache_dir, stat.S_IRWXU)
with open(os.path.join(cache_dir, 'CACHEDIR.TAG'), 'w') as fd:
fd.write(textwrap.dedent("""
Signature: 8a477f597d28d172789f06886806bc55
# This file is a cache directory tag created by Borg.
# For information about cache directory tags, see:
# http://www.brynosaurus.com/cachedir/
""").lstrip())
common_parser = argparse.ArgumentParser(add_help=False)
common_parser.add_argument('-v', '--verbose', dest='verbose', action='store_true',
default=False,
def build_parser(self, args=None, prog=None):
common_parser = argparse.ArgumentParser(add_help=False, prog=prog)
common_parser.add_argument('-v', '--verbose', dest='verbose', action='store_true', default=False,
help='verbose output')
common_parser.add_argument('--no-files-cache', dest='cache_files', action='store_false',
help='do not load/update the file metadata cache used to detect unchanged files')
@ -580,11 +582,9 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
common_parser.add_argument('--remote-path', dest='remote_path', default=RemoteRepository.remote_path, metavar='PATH',
help='set remote path to executable (default: "%(default)s")')
# We can't use argparse for "serve" since we don't want it to show up in "Available commands"
if args:
args = self.preprocess_args(args)
parser = argparse.ArgumentParser(description='Borg %s - Deduplicated Backups' % __version__)
parser = argparse.ArgumentParser(prog=prog, description='Borg - Deduplicated Backups')
parser.add_argument('-V', '--version', action='version', version='%(prog)s ' + __version__,
help='show version number and exit')
subparsers = parser.add_subparsers(title='Available commands')
serve_epilog = textwrap.dedent("""
@ -699,9 +699,11 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
subparser.add_argument('-s', '--stats', dest='stats',
action='store_true', default=False,
help='print statistics for the created archive')
subparser.add_argument('-p', '--progress', dest='progress',
action='store_true', default=False,
help='print progress while creating the archive')
subparser.add_argument('-p', '--progress', dest='progress', const=not sys.stderr.isatty(),
action='store_const', default=sys.stdin.isatty(),
help="""toggle progress display while creating the archive, showing Original,
Compressed and Deduplicated sizes, followed by the Number of files seen
and the path being processed, default: %(default)s""")
subparser.add_argument('-e', '--exclude', dest='excludes',
type=ExcludePattern, action='append',
metavar="PATTERN", help='exclude paths matching PATTERN')
@ -714,9 +716,9 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
subparser.add_argument('-c', '--checkpoint-interval', dest='checkpoint_interval',
type=int, default=300, metavar='SECONDS',
help='write checkpoint every SECONDS seconds (Default: 300)')
subparser.add_argument('--do-not-cross-mountpoints', dest='dontcross',
subparser.add_argument('-x', '--one-file-system', dest='one_file_system',
action='store_true', default=False,
help='do not cross mount points')
help='stay in same file system, do not cross mount points')
subparser.add_argument('--numeric-owner', dest='numeric_owner',
action='store_true', default=False,
help='only store numeric user and group identifiers')
@ -925,7 +927,7 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
help='repository to prune')
upgrade_epilog = textwrap.dedent("""
upgrade an existing Borg repository in place. this currently
upgrade an existing Borg repository. this currently
only support converting an Attic repository, but may
eventually be extended to cover major Borg upgrades as well.
@ -940,13 +942,6 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
the first backup after the conversion takes longer than expected
due to the cache resync.
it is recommended you run this on a copy of the Attic
repository, in case something goes wrong, for example:
cp -a attic borg
borg upgrade -n borg
borg upgrade borg
upgrade should be able to resume if interrupted, although it
will still iterate over all segments. if you want to start
from scratch, use `borg delete` over the copied repository to
@ -954,11 +949,19 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
borg delete borg
the conversion can PERMANENTLY DAMAGE YOUR REPOSITORY! Attic
will also NOT BE ABLE TO READ THE BORG REPOSITORY ANYMORE, as
the magic strings will have changed.
unless ``--inplace`` is specified, the upgrade process first
creates a backup copy of the repository, in
REPOSITORY.upgrade-DATETIME, using hardlinks. this takes
longer than in place upgrades, but is much safer and gives
progress information (as opposed to ``cp -al``). once you are
satisfied with the conversion, you can safely destroy the
backup copy.
you have been warned.""")
WARNING: running the upgrade in place will make the current
copy unusable with older version, with no way of going back
to previous versions. this can PERMANENTLY DAMAGE YOUR
REPOSITORY! Attic CAN NOT READ BORG REPOSITORIES, as the
magic strings have changed. you have been warned.""")
subparser = subparsers.add_parser('upgrade', parents=[common_parser],
description=self.do_upgrade.__doc__,
epilog=upgrade_epilog,
@ -967,6 +970,10 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
subparser.add_argument('-n', '--dry-run', dest='dry_run',
default=False, action='store_true',
help='do not change repository')
subparser.add_argument('-i', '--inplace', dest='inplace',
default=False, action='store_true',
help="""rewrite repository in place, with no chance of going back to older
versions of the repository.""")
subparser.add_argument('repository', metavar='REPOSITORY', nargs='?', default='',
type=location_validator(archive=False),
help='path to the repository to be upgraded')
@ -980,9 +987,34 @@ Type "Yes I am sure" if you understand this and want to continue.\n""")
subparser.set_defaults(func=functools.partial(self.do_help, parser, subparsers.choices))
subparser.add_argument('topic', metavar='TOPIC', type=str, nargs='?',
help='additional help on TOPIC')
return parser
def run(self, args=None):
check_extension_modules()
keys_dir = get_keys_dir()
if not os.path.exists(keys_dir):
os.makedirs(keys_dir)
os.chmod(keys_dir, stat.S_IRWXU)
cache_dir = get_cache_dir()
if not os.path.exists(cache_dir):
os.makedirs(cache_dir)
os.chmod(cache_dir, stat.S_IRWXU)
with open(os.path.join(cache_dir, 'CACHEDIR.TAG'), 'w') as fd:
fd.write(textwrap.dedent("""
Signature: 8a477f597d28d172789f06886806bc55
# This file is a cache directory tag created by Borg.
# For information about cache directory tags, see:
# http://www.brynosaurus.com/cachedir/
""").lstrip())
# We can't use argparse for "serve" since we don't want it to show up in "Available commands"
if args:
args = self.preprocess_args(args)
parser = self.build_parser(args)
args = parser.parse_args(args or ['-h'])
self.verbose = args.verbose
setup_logging()
os.umask(args.umask)
RemoteRepository.remote_path = args.remote_path
RemoteRepository.umask = args.umask
@ -1001,7 +1033,7 @@ def sig_info_handler(signum, stack): # pragma: no cover
total = loc['st'].st_size
except Exception:
pos, total = 0, 0
print("{0} {1}/{2}".format(path, format_file_size(pos), format_file_size(total)))
logger.info("{0} {1}/{2}".format(path, format_file_size(pos), format_file_size(total)))
break
if func in ('extract_item', ): # extract op
path = loc['item'][b'path']
@ -1009,7 +1041,7 @@ def sig_info_handler(signum, stack): # pragma: no cover
pos = loc['fd'].tell()
except Exception:
pos = 0
print("{0} {1}/???".format(path, format_file_size(pos)))
logger.info("{0} {1}/???".format(path, format_file_size(pos)))
break
@ -1031,22 +1063,34 @@ def main(): # pragma: no cover
setup_signal_handlers()
archiver = Archiver()
try:
msg = None
exit_code = archiver.run(sys.argv[1:])
except Error as e:
archiver.print_error(e.get_message() + "\n%s" % traceback.format_exc())
msg = e.get_message() + "\n%s" % traceback.format_exc()
exit_code = e.exit_code
except RemoteRepository.RPCError as e:
archiver.print_error('Error: Remote Exception.\n%s' % str(e))
exit_code = 1
msg = 'Remote Exception.\n%s' % str(e)
exit_code = EXIT_ERROR
except Exception:
archiver.print_error('Error: Local Exception.\n%s' % traceback.format_exc())
exit_code = 1
msg = 'Local Exception.\n%s' % traceback.format_exc()
exit_code = EXIT_ERROR
except KeyboardInterrupt:
archiver.print_error('Error: Keyboard interrupt.\n%s' % traceback.format_exc())
exit_code = 1
if exit_code:
archiver.print_error('Exiting with failure status due to previous errors')
msg = 'Keyboard interrupt.\n%s' % traceback.format_exc()
exit_code = EXIT_ERROR
if msg:
logger.error(msg)
exit_msg = 'terminating with %s status, rc %d'
if exit_code == EXIT_SUCCESS:
logger.info(exit_msg % ('success', exit_code))
elif exit_code == EXIT_WARNING:
logger.warning(exit_msg % ('warning', exit_code))
elif exit_code == EXIT_ERROR:
logger.error(exit_msg % ('error', exit_code))
else:
# if you see 666 in output, it usually means exit_code was None
logger.error(exit_msg % ('abnormal', exit_code or 666))
sys.exit(exit_code)
if __name__ == '__main__':
main()

44
borg/cache.py
View file

@ -1,7 +1,7 @@
import configparser
from .remote import cache_if_remote
from collections import namedtuple
import errno
import msgpack
import os
import stat
import sys
@ -12,11 +12,16 @@ import tarfile
import tempfile
from .key import PlaintextKey
from .logger import create_logger
logger = create_logger()
from .helpers import Error, get_cache_dir, decode_dict, st_mtime_ns, unhexlify, int_to_bigint, \
bigint_to_int
bigint_to_int, format_file_size, have_cython
from .locking import UpgradableLock
from .hashindex import ChunkIndex
if have_cython():
import msgpack
class Cache:
"""Client Side cache
@ -47,6 +52,7 @@ class Cache:
self.manifest = manifest
self.path = path or os.path.join(get_cache_dir(), hexlify(repository.id).decode('ascii'))
self.do_files = do_files
logger.info('initializing cache')
# Warn user before sending data to a never seen before unencrypted repository
if not os.path.exists(self.path):
if warn_if_unencrypted and isinstance(key, PlaintextKey):
@ -68,16 +74,33 @@ class Cache:
# Make sure an encrypted repository has not been swapped for an unencrypted repository
if self.key_type is not None and self.key_type != str(key.TYPE):
raise self.EncryptionMethodMismatch()
logger.info('synchronizing cache')
self.sync()
self.commit()
def __del__(self):
self.close()
def __str__(self):
fmt = """\
All archives: {0.total_size:>20s} {0.total_csize:>20s} {0.unique_csize:>20s}
Unique chunks Total chunks
Chunk index: {0.total_unique_chunks:20d} {0.total_chunks:20d}"""
return fmt.format(self.format_tuple())
def format_tuple(self):
# XXX: this should really be moved down to `hashindex.pyx`
Summary = namedtuple('Summary', ['total_size', 'total_csize', 'unique_size', 'unique_csize', 'total_unique_chunks', 'total_chunks'])
stats = Summary(*self.chunks.summarize())._asdict()
for field in ['total_size', 'total_csize', 'unique_csize']:
stats[field] = format_file_size(stats[field])
return Summary(**stats)
def _confirm(self, message, env_var_override=None):
print(message, file=sys.stderr)
if env_var_override and os.environ.get(env_var_override):
print("Yes (From {})".format(env_var_override))
print("Yes (From {})".format(env_var_override), file=sys.stderr)
return True
if not sys.stdin.isatty():
return False
@ -146,6 +169,7 @@ class Cache:
def _read_files(self):
self.files = {}
self._newest_mtime = 0
logger.info('reading files cache')
with open(os.path.join(self.path, 'files'), 'rb') as fd:
u = msgpack.Unpacker(use_list=True)
while True:
@ -267,7 +291,7 @@ class Cache:
unpacker.feed(data)
for item in unpacker:
if not isinstance(item, dict):
print('Error: Did not get expected metadata dict - archive corrupted!')
logger.error('Error: Did not get expected metadata dict - archive corrupted!')
continue
if b'chunks' in item:
for chunk_id, size, csize in item[b'chunks']:
@ -289,10 +313,10 @@ class Cache:
return name
def create_master_idx(chunk_idx):
print('Synchronizing chunks cache...')
logger.info('Synchronizing chunks cache...')
cached_ids = cached_archives()
archive_ids = repo_archives()
print('Archives: %d, w/ cached Idx: %d, w/ outdated Idx: %d, w/o cached Idx: %d.' % (
logger.info('Archives: %d, w/ cached Idx: %d, w/ outdated Idx: %d, w/o cached Idx: %d.' % (
len(archive_ids), len(cached_ids),
len(cached_ids - archive_ids), len(archive_ids - cached_ids), ))
# deallocates old hashindex, creates empty hashindex:
@ -304,12 +328,12 @@ class Cache:
archive_name = lookup_name(archive_id)
if archive_id in cached_ids:
archive_chunk_idx_path = mkpath(archive_id)
print("Reading cached archive chunk index for %s ..." % archive_name)
logger.info("Reading cached archive chunk index for %s ..." % archive_name)
archive_chunk_idx = ChunkIndex.read(archive_chunk_idx_path)
else:
print('Fetching and building archive index for %s ...' % archive_name)
logger.info('Fetching and building archive index for %s ...' % archive_name)
archive_chunk_idx = fetch_and_build_idx(archive_id, repository, self.key)
print("Merging into master chunks index ...")
logger.info("Merging into master chunks index ...")
if chunk_idx is None:
# we just use the first archive's idx as starting point,
# to avoid growing the hash table from 0 size and also
@ -317,7 +341,7 @@ class Cache:
chunk_idx = archive_chunk_idx
else:
chunk_idx.merge(archive_chunk_idx)
print('Done.')
logger.info('Done.')
return chunk_idx
def legacy_cleanup():

19
borg/fuse.py
View file

@ -2,17 +2,19 @@ from collections import defaultdict
import errno
import io
import llfuse
import msgpack
import os
import stat
import tempfile
import time
from .archive import Archive
from .helpers import daemonize
from .helpers import daemonize, have_cython
from .remote import cache_if_remote
if have_cython():
import msgpack
# Does this version of llfuse support ns precision?
have_fuse_mtime_ns = hasattr(llfuse.EntryAttributes, 'st_mtime_ns')
have_fuse_xtime_ns = hasattr(llfuse.EntryAttributes, 'st_mtime_ns')
class ItemCache:
@ -153,14 +155,15 @@ class FuseOperations(llfuse.Operations):
entry.st_size = size
entry.st_blksize = 512
entry.st_blocks = dsize / 512
if have_fuse_mtime_ns:
entry.st_atime_ns = item[b'mtime']
# note: older archives only have mtime (not atime nor ctime)
if have_fuse_xtime_ns:
entry.st_atime_ns = item.get(b'atime') or item[b'mtime']
entry.st_mtime_ns = item[b'mtime']
entry.st_ctime_ns = item[b'mtime']
entry.st_ctime_ns = item.get(b'ctime') or item[b'mtime']
else:
entry.st_atime = item[b'mtime'] / 1e9
entry.st_atime = (item.get(b'atime') or item[b'mtime']) / 1e9
entry.st_mtime = item[b'mtime'] / 1e9
entry.st_ctime = item[b'mtime'] / 1e9
entry.st_ctime = (item.get(b'ctime') or item[b'mtime']) / 1e9
return entry
def listxattr(self, inode):

165
borg/helpers.py
View file

@ -1,5 +1,4 @@
from .support import argparse # see support/__init__.py docstring
# DEPRECATED - remove after requiring py 3.4
from .support import argparse # see support/__init__.py docstring, DEPRECATED - remove after requiring py 3.4
import binascii
from collections import namedtuple
@ -9,6 +8,12 @@ import os
import pwd
import queue
import re
try:
from shutil import get_terminal_size
except ImportError:
def get_terminal_size(fallback=(80, 24)):
TerminalSize = namedtuple('TerminalSize', ['columns', 'lines'])
return TerminalSize(int(os.environ.get('COLUMNS', fallback[0])), int(os.environ.get('LINES', fallback[1])))
import sys
import time
import unicodedata
@ -17,11 +22,34 @@ from datetime import datetime, timezone, timedelta
from fnmatch import translate
from operator import attrgetter
import msgpack
from . import hashindex
from . import chunker
from . import crypto
def have_cython():
"""allow for a way to disable Cython includes
this is used during usage docs build, in setup.py. It is to avoid
loading the Cython libraries which are built, but sometimes not in
the search path (namely, during Tox runs).
we simply check an environment variable (``BORG_CYTHON_DISABLE``)
which, when set (to anything) will disable includes of Cython
libraries in key places to enable usage docs to be built.
:returns: True if Cython is available, False otherwise.
"""
return not os.environ.get('BORG_CYTHON_DISABLE')
if have_cython():
from . import hashindex
from . import chunker
from . import crypto
import msgpack
# return codes returned by borg command
# when borg is killed by signal N, rc = 128 + N
EXIT_SUCCESS = 0 # everything done, no problems
EXIT_WARNING = 1 # reached normal end of operation, but there were issues
EXIT_ERROR = 2 # terminated abruptly, did not reach end of operation
QUEUE_DEBUG = False
@ -29,10 +57,17 @@ QUEUE_DEBUG = False
class Error(Exception):
"""Error base class"""
exit_code = 1
# if we raise such an Error and it is only catched by the uppermost
# exception handler (that exits short after with the given exit_code),
# it is always a (fatal and abrupt) EXIT_ERROR, never just a warning.
exit_code = EXIT_ERROR
def get_message(self):
return 'Error: ' + type(self).__doc__.format(*self.args)
return type(self).__doc__.format(*self.args)
class IntegrityError(Error):
"""Data integrity error"""
class ExtensionModuleError(Error):
@ -144,27 +179,41 @@ class Statistics:
if unique:
self.usize += csize
def print_(self, label, cache):
total_size, total_csize, unique_size, unique_csize, total_unique_chunks, total_chunks = cache.chunks.summarize()
print()
print(' Original size Compressed size Deduplicated size')
print('%-15s %20s %20s %20s' % (label, format_file_size(self.osize), format_file_size(self.csize), format_file_size(self.usize)))
print('All archives: %20s %20s %20s' % (format_file_size(total_size), format_file_size(total_csize), format_file_size(unique_csize)))
print()
print(' Unique chunks Total chunks')
print('Chunk index: %20d %20d' % (total_unique_chunks, total_chunks))
summary = """\
Original size Compressed size Deduplicated size
{label:15} {stats.osize_fmt:>20s} {stats.csize_fmt:>20s} {stats.usize_fmt:>20s}"""
def show_progress(self, item=None, final=False):
def __str__(self):
return self.summary.format(stats=self, label='This archive:')
def __repr__(self):
return "<{cls} object at {hash:#x} ({self.osize}, {self.csize}, {self.usize})>".format(cls=type(self).__name__, hash=id(self), self=self)
@property
def osize_fmt(self):
return format_file_size(self.osize)
@property
def usize_fmt(self):
return format_file_size(self.usize)
@property
def csize_fmt(self):
return format_file_size(self.csize)
def show_progress(self, item=None, final=False, stream=None):
columns, lines = get_terminal_size()
if not final:
msg = '{0.osize_fmt} O {0.csize_fmt} C {0.usize_fmt} D {0.nfiles} N '.format(self)
path = remove_surrogates(item[b'path']) if item else ''
if len(path) > 43:
path = '%s...%s' % (path[:20], path[-20:])
msg = '%9s O %9s C %9s D %-43s' % (
format_file_size(self.osize), format_file_size(self.csize), format_file_size(self.usize), path)
space = columns - len(msg)
if space < len('...') + len(path):
path = '%s...%s' % (path[:(space//2)-len('...')], path[-space//2:])
msg += "{0:<{space}}".format(path, space=space)
else:
msg = ' ' * 79
print(msg, end='\r')
sys.stdout.flush()
msg = ' ' * columns
print(msg, file=stream or sys.stderr, end="\r")
(stream or sys.stderr).flush()
def get_keys_dir():
@ -230,7 +279,7 @@ def exclude_path(path, patterns):
def normalized(func):
""" Decorator for the Pattern match methods, returning a wrapper that
normalizes OSX paths to match the normalized pattern on OSX, and
normalizes OSX paths to match the normalized pattern on OSX, and
returning the original method on other platforms"""
@wraps(func)
def normalize_wrapper(self, path):
@ -426,29 +475,35 @@ def format_file_mode(mod):
return '%s%s%s' % (x(mod // 64), x(mod // 8), x(mod))
def format_file_size(v):
def format_file_size(v, precision=2):
"""Format file size into a human friendly format
"""
if abs(v) > 10**12:
return '%.2f TB' % (v / 10**12)
elif abs(v) > 10**9:
return '%.2f GB' % (v / 10**9)
elif abs(v) > 10**6:
return '%.2f MB' % (v / 10**6)
elif abs(v) > 10**3:
return '%.2f kB' % (v / 10**3)
else:
return '%d B' % v
return sizeof_fmt_decimal(v, suffix='B', sep=' ', precision=precision)
def sizeof_fmt(num, suffix='B', units=None, power=None, sep='', precision=2):
for unit in units[:-1]:
if abs(round(num, precision)) < power:
if isinstance(num, int):
return "{}{}{}{}".format(num, sep, unit, suffix)
else:
return "{:3.{}f}{}{}{}".format(num, precision, sep, unit, suffix)
num /= float(power)
return "{:.{}f}{}{}{}".format(num, precision, sep, units[-1], suffix)
def sizeof_fmt_iec(num, suffix='B', sep='', precision=2):
return sizeof_fmt(num, suffix=suffix, sep=sep, precision=precision, units=['', 'Ki', 'Mi', 'Gi', 'Ti', 'Pi', 'Ei', 'Zi', 'Yi'], power=1024)
def sizeof_fmt_decimal(num, suffix='B', sep='', precision=2):
return sizeof_fmt(num, suffix=suffix, sep=sep, precision=precision, units=['', 'k', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y'], power=1000)
def format_archive(archive):
return '%-36s %s' % (archive.name, to_localtime(archive.ts).strftime('%c'))
class IntegrityError(Error):
"""Data integrity error"""
def memoize(function):
cache = {}
@ -498,14 +553,24 @@ def posix_acl_use_stored_uid_gid(acl):
"""Replace the user/group field with the stored uid/gid
"""
entries = []
for entry in acl.decode('ascii').split('\n'):
for entry in safe_decode(acl).split('\n'):
if entry:
fields = entry.split(':')
if len(fields) == 4:
entries.append(':'.join([fields[0], fields[3], fields[2]]))
else:
entries.append(entry)
return ('\n'.join(entries)).encode('ascii')
return safe_encode('\n'.join(entries))
def safe_decode(s, coding='utf-8', errors='surrogateescape'):
"""decode bytes to str, with round-tripping "invalid" bytes"""
return s.decode(coding, errors)
def safe_encode(s, coding='utf-8', errors='surrogateescape'):
"""encode str to bytes, with round-tripping "invalid" bytes"""
return s.encode(coding, errors)
class Location:
@ -685,7 +750,13 @@ class StableDict(dict):
if sys.version < '3.3':
# st_mtime_ns attribute only available in 3.3+
# st_xtime_ns attributes only available in 3.3+
def st_atime_ns(st):
return int(st.st_atime * 1e9)
def st_ctime_ns(st):
return int(st.st_ctime * 1e9)
def st_mtime_ns(st):
return int(st.st_mtime * 1e9)
@ -695,6 +766,12 @@ if sys.version < '3.3':
data = data.encode('ascii')
return binascii.unhexlify(data)
else:
def st_atime_ns(st):
return st.st_atime_ns
def st_ctime_ns(st):
return st.st_ctime_ns
def st_mtime_ns(st):
return st.st_mtime_ns

30
borg/key.py
View file

@ -2,14 +2,18 @@ from binascii import hexlify, a2b_base64, b2a_base64
import configparser
import getpass
import os
import msgpack
import textwrap
import hmac
from hashlib import sha256
from .crypto import pbkdf2_sha256, get_random_bytes, AES, bytes_to_long, long_to_bytes, bytes_to_int, num_aes_blocks
from .compress import Compressor, COMPR_BUFFER
from .helpers import IntegrityError, get_keys_dir, Error
from .helpers import IntegrityError, get_keys_dir, Error, have_cython
from .logger import create_logger
logger = create_logger()
if have_cython():
from .crypto import pbkdf2_sha256, get_random_bytes, AES, bytes_to_long, long_to_bytes, bytes_to_int, num_aes_blocks
from .compress import Compressor, COMPR_BUFFER
import msgpack
PREFIX = b'\0' * 8
@ -88,7 +92,7 @@ class PlaintextKey(KeyBase):
@classmethod
def create(cls, repository, args):
print('Encryption NOT enabled.\nUse the "--encryption=repokey|keyfile|passphrase" to enable encryption.')
logger.info('Encryption NOT enabled.\nUse the "--encryption=repokey|keyfile|passphrase" to enable encryption.')
return cls(repository)
@classmethod
@ -190,12 +194,12 @@ class Passphrase(str):
if allow_empty or passphrase:
passphrase2 = cls.getpass('Enter same passphrase again: ')
if passphrase == passphrase2:
print('Remember your passphrase. Your data will be inaccessible without it.')
logger.info('Remember your passphrase. Your data will be inaccessible without it.')
return passphrase
else:
print('Passphrases do not match')
print('Passphrases do not match', file=sys.stderr)
else:
print('Passphrase must not be blank')
print('Passphrase must not be blank', file=sys.stderr)
def __repr__(self):
return '<Passphrase "***hidden***">'
@ -215,8 +219,8 @@ class PassphraseKey(AESKeyBase):
@classmethod
def create(cls, repository, args):
key = cls(repository)
print('WARNING: "passphrase" mode is deprecated and will be removed in 1.0.')
print('If you want something similar (but with less issues), use "repokey" mode.')
logger.warning('WARNING: "passphrase" mode is deprecated and will be removed in 1.0.')
logger.warning('If you want something similar (but with less issues), use "repokey" mode.')
passphrase = Passphrase.new(allow_empty=False)
key.init(repository, passphrase)
return key
@ -324,7 +328,7 @@ class KeyfileKeyBase(AESKeyBase):
def change_passphrase(self):
passphrase = Passphrase.new(allow_empty=True)
self.save(self.target, passphrase)
print('Key updated')
logger.info('Key updated')
@classmethod
def create(cls, repository, args):
@ -335,8 +339,8 @@ class KeyfileKeyBase(AESKeyBase):
key.init_ciphers()
target = key.get_new_target(args)
key.save(target, passphrase)
print('Key in "%s" created.' % target)
print('Keep this key safe. Your data will be inaccessible without it.')
logger.info('Key in "%s" created.' % target)
logger.info('Keep this key safe. Your data will be inaccessible without it.')
return key
def save(self, target, passphrase):

13
borg/locking.py
View file

@ -2,7 +2,6 @@ import errno
import json
import os
import socket
import threading
import time
from borg.helpers import Error
@ -10,13 +9,17 @@ from borg.helpers import Error
ADD, REMOVE = 'add', 'remove'
SHARED, EXCLUSIVE = 'shared', 'exclusive'
# only determine the PID and hostname once.
# for FUSE mounts, we fork a child process that needs to release
# the lock made by the parent, so it needs to use the same PID for that.
_pid = os.getpid()
_hostname = socket.gethostname()
def get_id():
"""Get identification tuple for 'us'"""
hostname = socket.gethostname()
pid = os.getpid()
tid = threading.current_thread().ident & 0xffffffff
return hostname, pid, tid
thread_id = 0
return _hostname, _pid, thread_id
class TimeoutTimer:

85
borg/logger.py Normal file
View file

@ -0,0 +1,85 @@
"""logging facilities
The way to use this is as follows:
* each module declares its own logger, using:
from .logger import create_logger
logger = create_logger()
* then each module uses logger.info/warning/debug/etc according to the
level it believes is appropriate:
logger.debug('debugging info for developers or power users')
logger.info('normal, informational output')
logger.warning('warn about a non-fatal error or sth else')
logger.error('a fatal error')
... and so on. see the `logging documentation
<https://docs.python.org/3/howto/logging.html#when-to-use-logging>`_
for more information
* console interaction happens on stderr, that includes interactive
reporting functions like `help`, `info` and `list`
* ...except ``input()`` is special, because we can't control the
stream it is using, unfortunately. we assume that it won't clutter
stdout, because interaction would be broken then anyways
* what is output on INFO level is additionally controlled by commandline
flags
"""
import inspect
import logging
import sys
def setup_logging(stream=None):
"""setup logging module according to the arguments provided
this sets up a stream handler logger on stderr (by default, if no
stream is provided).
"""
logging.raiseExceptions = False
l = logging.getLogger('')
sh = logging.StreamHandler(stream)
# other formatters will probably want this, but let's remove
# clutter on stderr
# example:
# sh.setFormatter(logging.Formatter('%(name)s: %(message)s'))
l.addHandler(sh)
l.setLevel(logging.INFO)
return sh
def find_parent_module():
"""find the name of a the first module calling this module
if we cannot find it, we return the current module's name
(__name__) instead.
"""
try:
frame = inspect.currentframe().f_back
module = inspect.getmodule(frame)
while module is None or module.__name__ == __name__:
frame = frame.f_back
module = inspect.getmodule(frame)
return module.__name__
except AttributeError:
# somehow we failed to find our module
# return the logger module name by default
return __name__
def create_logger(name=None):
"""create a Logger object with the proper path, which is returned by
find_parent_module() by default, or is provided via the commandline
this is really a shortcut for:
logger = logging.getLogger(__name__)
we use it to avoid errors and provide a more standard API.
"""
return logging.getLogger(name or find_parent_module())

18
borg/platform_darwin.pyx
View file

@ -1,5 +1,5 @@
import os
from .helpers import user2uid, group2gid
from .helpers import user2uid, group2gid, safe_decode, safe_encode
API_VERSION = 2
@ -20,7 +20,7 @@ def _remove_numeric_id_if_possible(acl):
"""Replace the user/group field with the local uid/gid if possible
"""
entries = []
for entry in acl.decode('ascii').split('\n'):
for entry in safe_decode(acl).split('\n'):
if entry:
fields = entry.split(':')
if fields[0] == 'user':
@ -30,22 +30,22 @@ def _remove_numeric_id_if_possible(acl):
if group2gid(fields[2]) is not None:
fields[1] = fields[3] = ''
entries.append(':'.join(fields))
return ('\n'.join(entries)).encode('ascii')
return safe_encode('\n'.join(entries))
def _remove_non_numeric_identifier(acl):
"""Remove user and group names from the acl
"""
entries = []
for entry in acl.split(b'\n'):
for entry in safe_decode(acl).split('\n'):
if entry:
fields = entry.split(b':')
if fields[0] in (b'user', b'group'):
fields[2] = b''
entries.append(b':'.join(fields))
fields = entry.split(':')
if fields[0] in ('user', 'group'):
fields[2] = ''
entries.append(':'.join(fields))
else:
entries.append(entry)
return b'\n'.join(entries)
return safe_encode('\n'.join(entries))
def acl_get(path, item, st, numeric_owner=False):

6
borg/platform_freebsd.pyx
View file

@ -1,5 +1,5 @@
import os
from .helpers import posix_acl_use_stored_uid_gid
from .helpers import posix_acl_use_stored_uid_gid, safe_encode, safe_decode
API_VERSION = 2
@ -78,14 +78,14 @@ cdef _nfs4_use_stored_uid_gid(acl):
"""Replace the user/group field with the stored uid/gid
"""
entries = []
for entry in acl.decode('ascii').split('\n'):
for entry in safe_decode(acl).split('\n'):
if entry:
if entry.startswith('user:') or entry.startswith('group:'):
fields = entry.split(':')
entries.append(':'.join(fields[0], fields[5], *fields[2:-1]))
else:
entries.append(entry)
return ('\n'.join(entries)).encode('ascii')
return safe_encode('\n'.join(entries))
def acl_set(path, item, numeric_owner=False):

20
borg/platform_linux.pyx
View file

@ -1,7 +1,7 @@
import os
import re
from stat import S_ISLNK
from .helpers import posix_acl_use_stored_uid_gid, user2uid, group2gid
from .helpers import posix_acl_use_stored_uid_gid, user2uid, group2gid, safe_decode, safe_encode
API_VERSION = 2
@ -31,22 +31,22 @@ def acl_use_local_uid_gid(acl):
"""Replace the user/group field with the local uid/gid if possible
"""
entries = []
for entry in acl.decode('ascii').split('\n'):
for entry in safe_decode(acl).split('\n'):
if entry:
fields = entry.split(':')
if fields[0] == 'user' and fields[1]:
fields[1] = user2uid(fields[1], fields[3])
fields[1] = str(user2uid(fields[1], fields[3]))
elif fields[0] == 'group' and fields[1]:
fields[1] = group2gid(fields[1], fields[3])
entries.append(':'.join(entry.split(':')[:3]))
return ('\n'.join(entries)).encode('ascii')
fields[1] = str(group2gid(fields[1], fields[3]))
entries.append(':'.join(fields[:3]))
return safe_encode('\n'.join(entries))
cdef acl_append_numeric_ids(acl):
"""Extend the "POSIX 1003.1e draft standard 17" format with an additional uid/gid field
"""
entries = []
for entry in _comment_re.sub('', acl.decode('ascii')).split('\n'):
for entry in _comment_re.sub('', safe_decode(acl)).split('\n'):
if entry:
type, name, permission = entry.split(':')
if name and type == 'user':
@ -55,14 +55,14 @@ cdef acl_append_numeric_ids(acl):
entries.append(':'.join([type, name, permission, str(group2gid(name, name))]))
else:
entries.append(entry)
return ('\n'.join(entries)).encode('ascii')
return safe_encode('\n'.join(entries))
cdef acl_numeric_ids(acl):
"""Replace the "POSIX 1003.1e draft standard 17" user/group field with uid/gid
"""
entries = []
for entry in _comment_re.sub('', acl.decode('ascii')).split('\n'):
for entry in _comment_re.sub('', safe_decode(acl)).split('\n'):
if entry:
type, name, permission = entry.split(':')
if name and type == 'user':
@ -73,7 +73,7 @@ cdef acl_numeric_ids(acl):
entries.append(':'.join([type, gid, permission, gid]))
else:
entries.append(entry)
return ('\n'.join(entries)).encode('ascii')
return safe_encode('\n'.join(entries))
def acl_get(path, item, st, numeric_owner=False):

6
borg/remote.py
View file

@ -1,6 +1,5 @@
import errno
import fcntl
import msgpack
import os
import select
import shlex
@ -11,9 +10,12 @@ import traceback
from . import __version__
from .helpers import Error, IntegrityError
from .helpers import Error, IntegrityError, have_cython
from .repository import Repository
if have_cython():
import msgpack
BUFSIZE = 10 * 1024 * 1024

13
borg/repository.py
View file

@ -2,14 +2,18 @@ from configparser import RawConfigParser
from binascii import hexlify
from itertools import islice
import errno
import logging
logger = logging.getLogger(__name__)
import os
import shutil
import struct
import sys
from zlib import crc32
from .hashindex import NSIndex
from .helpers import Error, IntegrityError, read_msgpack, write_msgpack, unhexlify
from .helpers import Error, IntegrityError, read_msgpack, write_msgpack, unhexlify, have_cython
if have_cython():
from .hashindex import NSIndex
from .locking import UpgradableLock
from .lrucache import LRUCache
@ -278,7 +282,7 @@ class Repository:
def report_error(msg):
nonlocal error_found
error_found = True
print(msg, file=sys.stderr)
logger.error(msg)
assert not self._active_txn
try:
@ -546,11 +550,10 @@ class LoggedIO:
def recover_segment(self, segment, filename):
if segment in self.fds:
del self.fds[segment]
# FIXME: save a copy of the original file
with open(filename, 'rb') as fd:
data = memoryview(fd.read())
os.rename(filename, filename + '.beforerecover')
print('attempting to recover ' + filename, file=sys.stderr)
logger.info('attempting to recover ' + filename)
with open(filename, 'wb') as fd:
fd.write(MAGIC)
while len(data) >= self.header_fmt.size:

5
borg/testsuite/__init__.py
View file

@ -85,8 +85,9 @@ class BaseTestCase(unittest.TestCase):
if fuse and not have_fuse_mtime_ns:
d1.append(round(st_mtime_ns(s1), -4))
d2.append(round(st_mtime_ns(s2), -4))
d1.append(round(st_mtime_ns(s1), st_mtime_ns_round))
d2.append(round(st_mtime_ns(s2), st_mtime_ns_round))
else:
d1.append(round(st_mtime_ns(s1), st_mtime_ns_round))
d2.append(round(st_mtime_ns(s2), st_mtime_ns_round))
d1.append(get_all(path1, follow_symlinks=False))
d2.append(get_all(path2, follow_symlinks=False))
self.assert_equal(d1, d2)

184
borg/testsuite/archiver.py
View file

@ -1,5 +1,6 @@
from binascii import hexlify
from configparser import RawConfigParser
import errno
import os
from io import StringIO
import stat
@ -19,7 +20,7 @@ from ..archive import Archive, ChunkBuffer, CHUNK_MAX_EXP
from ..archiver import Archiver
from ..cache import Cache
from ..crypto import bytes_to_long, num_aes_blocks
from ..helpers import Manifest
from ..helpers import Manifest, EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR, st_atime_ns, st_mtime_ns
from ..remote import RemoteRepository, PathNotAllowed
from ..repository import Repository
from . import BaseTestCase
@ -70,14 +71,83 @@ class environment_variable:
else:
os.environ[k] = v
def exec_cmd(*args, archiver=None, fork=False, exe=None, **kw):
if fork:
try:
if exe is None:
borg = (sys.executable, '-m', 'borg.archiver')
elif isinstance(exe, str):
borg = (exe, )
elif not isinstance(exe, tuple):
raise ValueError('exe must be None, a tuple or a str')
output = subprocess.check_output(borg + args, stderr=subprocess.STDOUT)
ret = 0
except subprocess.CalledProcessError as e:
output = e.output
ret = e.returncode
return ret, os.fsdecode(output)
else:
stdin, stdout, stderr = sys.stdin, sys.stdout, sys.stderr
try:
sys.stdin = StringIO()
sys.stdout = sys.stderr = output = StringIO()
if archiver is None:
archiver = Archiver()
ret = archiver.run(list(args))
return ret, output.getvalue()
finally:
sys.stdin, sys.stdout, sys.stderr = stdin, stdout, stderr
# check if the binary "borg.exe" is available
try:
exec_cmd('help', exe='borg.exe', fork=True)
BORG_EXES = ['python', 'binary', ]
except (IOError, OSError) as err:
if err.errno != errno.ENOENT:
raise
BORG_EXES = ['python', ]
@pytest.fixture(params=BORG_EXES)
def cmd(request):
if request.param == 'python':
exe = None
elif request.param == 'binary':
exe = 'borg.exe'
else:
raise ValueError("param must be 'python' or 'binary'")
def exec_fn(*args, **kw):
return exec_cmd(*args, exe=exe, fork=True, **kw)
return exec_fn
def test_return_codes(cmd, tmpdir):
repo = tmpdir.mkdir('repo')
input = tmpdir.mkdir('input')
output = tmpdir.mkdir('output')
input.join('test_file').write('content')
rc, out = cmd('init', '%s' % str(repo))
assert rc == EXIT_SUCCESS
rc, out = cmd('create', '%s::archive' % repo, str(input))
assert rc == EXIT_SUCCESS
with changedir(str(output)):
rc, out = cmd('extract', '%s::archive' % repo)
assert rc == EXIT_SUCCESS
rc, out = cmd('extract', '%s::archive' % repo, 'does/not/match')
assert rc == EXIT_WARNING # pattern did not match
rc, out = cmd('create', '%s::archive' % repo, str(input))
assert rc == EXIT_ERROR # duplicate archive name
class ArchiverTestCaseBase(BaseTestCase):
EXE = None # python source based
FORK_DEFAULT = False
prefix = ''
def setUp(self):
os.environ['BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'] = '1'
self.archiver = Archiver()
self.archiver = not self.FORK_DEFAULT and Archiver() or None
self.tmpdir = tempfile.mkdtemp()
self.repository_path = os.path.join(self.tmpdir, 'repository')
self.repository_location = self.prefix + self.repository_path
@ -102,34 +172,15 @@ class ArchiverTestCaseBase(BaseTestCase):
shutil.rmtree(self.tmpdir)
def cmd(self, *args, **kw):
exit_code = kw.get('exit_code', 0)
fork = kw.get('fork', False)
if fork:
try:
output = subprocess.check_output((sys.executable, '-m', 'borg.archiver') + args)
ret = 0
except subprocess.CalledProcessError as e:
output = e.output
ret = e.returncode
output = os.fsdecode(output)
if ret != exit_code:
print(output)
self.assert_equal(exit_code, ret)
return output
args = list(args)
stdin, stdout, stderr = sys.stdin, sys.stdout, sys.stderr
try:
sys.stdin = StringIO()
output = StringIO()
sys.stdout = sys.stderr = output
ret = self.archiver.run(args)
sys.stdin, sys.stdout, sys.stderr = stdin, stdout, stderr
if ret != exit_code:
print(output.getvalue())
self.assert_equal(exit_code, ret)
return output.getvalue()
finally:
sys.stdin, sys.stdout, sys.stderr = stdin, stdout, stderr
exit_code = kw.pop('exit_code', 0)
fork = kw.pop('fork', None)
if fork is None:
fork = self.FORK_DEFAULT
ret, output = exec_cmd(*args, fork=fork, exe=self.EXE, archiver=self.archiver, **kw)
if ret != exit_code:
print(output)
self.assert_equal(ret, exit_code)
return output
def create_src_archive(self, name):
self.cmd('create', self.repository_location + '::' + name, src_dir)
@ -231,9 +282,37 @@ class ArchiverTestCase(ArchiverTestCaseBase):
shutil.rmtree(self.cache_path)
with environment_variable(BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK='1'):
info_output2 = self.cmd('info', self.repository_location + '::test')
# info_output2 starts with some "initializing cache" text but should
# end the same way as info_output
assert info_output2.endswith(info_output)
def filter(output):
# filter for interesting "info" output, ignore cache rebuilding related stuff
prefixes = ['Name:', 'Fingerprint:', 'Number of files:', 'This archive:',
'All archives:', 'Chunk index:', ]
result = []
for line in output.splitlines():
for prefix in prefixes:
if line.startswith(prefix):
result.append(line)
return '\n'.join(result)
# the interesting parts of info_output2 and info_output should be same
self.assert_equal(filter(info_output), filter(info_output2))
def test_atime(self):
have_root = self.create_test_files()
atime, mtime = 123456780, 234567890
os.utime('input/file1', (atime, mtime))
self.cmd('init', self.repository_location)
self.cmd('create', self.repository_location + '::test', 'input')
with changedir('output'):
self.cmd('extract', self.repository_location + '::test')
sti = os.stat('input/file1')
sto = os.stat('output/input/file1')
assert st_mtime_ns(sti) == st_mtime_ns(sto) == mtime * 1e9
if hasattr(os, 'O_NOATIME'):
assert st_atime_ns(sti) == st_atime_ns(sto) == atime * 1e9
else:
# it touched the input file's atime while backing it up
assert st_atime_ns(sto) == atime * 1e9
def _extract_repository_id(self, path):
return Repository(self.repository_path).id
@ -304,7 +383,10 @@ class ArchiverTestCase(ArchiverTestCaseBase):
self.cmd('init', '--encryption=none', self.repository_location)
self._set_repository_id(self.repository_path, repository_id)
self.assert_equal(repository_id, self._extract_repository_id(self.repository_path))
self.assert_raises(Cache.EncryptionMethodMismatch, lambda: self.cmd('create', self.repository_location + '::test.2', 'input'))
if self.FORK_DEFAULT:
self.cmd('create', self.repository_location + '::test.2', 'input', exit_code=1) # fails
else:
self.assert_raises(Cache.EncryptionMethodMismatch, lambda: self.cmd('create', self.repository_location + '::test.2', 'input'))
def test_repository_swap_detection2(self):
self.create_test_files()
@ -314,7 +396,10 @@ class ArchiverTestCase(ArchiverTestCaseBase):
self.cmd('create', self.repository_location + '_encrypted::test', 'input')
shutil.rmtree(self.repository_path + '_encrypted')
os.rename(self.repository_path + '_unencrypted', self.repository_path + '_encrypted')
self.assert_raises(Cache.RepositoryAccessAborted, lambda: self.cmd('create', self.repository_location + '_encrypted::test.2', 'input'))
if self.FORK_DEFAULT:
self.cmd('create', self.repository_location + '_encrypted::test.2', 'input', exit_code=1) # fails
else:
self.assert_raises(Cache.RepositoryAccessAborted, lambda: self.cmd('create', self.repository_location + '_encrypted::test.2', 'input'))
def test_strip_components(self):
self.cmd('init', self.repository_location)
@ -539,8 +624,12 @@ class ArchiverTestCase(ArchiverTestCaseBase):
self.assert_in('bar-2015-08-12-20:00', output)
def test_usage(self):
self.assert_raises(SystemExit, lambda: self.cmd())
self.assert_raises(SystemExit, lambda: self.cmd('-h'))
if self.FORK_DEFAULT:
self.cmd(exit_code=0)
self.cmd('-h', exit_code=0)
else:
self.assert_raises(SystemExit, lambda: self.cmd())
self.assert_raises(SystemExit, lambda: self.cmd('-h'))
def test_help(self):
assert 'Borg' in self.cmd('help')
@ -627,6 +716,12 @@ class ArchiverTestCase(ArchiverTestCaseBase):
self.verify_aes_counter_uniqueness('passphrase')
@unittest.skipUnless('binary' in BORG_EXES, 'no borg.exe available')
class ArchiverTestCaseBinary(ArchiverTestCase):
EXE = 'borg.exe'
FORK_DEFAULT = True
class ArchiverCheckTestCase(ArchiverTestCaseBase):
def setUp(self):
@ -716,3 +811,16 @@ if 0:
self.cmd('init', self.repository_location + '_2')
with patch.object(RemoteRepository, 'extra_test_args', ['--restrict-to-path', '/foo', '--restrict-to-path', path_prefix]):
self.cmd('init', self.repository_location + '_3')
# skip fuse tests here, they deadlock since this change in exec_cmd:
# -output = subprocess.check_output(borg + args, stderr=None)
# +output = subprocess.check_output(borg + args, stderr=subprocess.STDOUT)
# this was introduced because some tests expect stderr contents to show up
# in "output" also. Also, the non-forking exec_cmd catches both, too.
@unittest.skip('deadlock issues')
def test_fuse_mount_repository(self):
pass
@unittest.skip('deadlock issues')
def test_fuse_mount_archive(self):
pass

100
borg/testsuite/benchmark.py Normal file
View file

@ -0,0 +1,100 @@
"""
Do benchmarks using pytest-benchmark.
Usage:
py.test --benchmark-only
"""
import os
import pytest
from .archiver import changedir, cmd
@pytest.yield_fixture
def repo_url(request, tmpdir):
os.environ['BORG_PASSPHRASE'] = '123456'
os.environ['BORG_CHECK_I_KNOW_WHAT_I_AM_DOING'] = '1'
os.environ['BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK'] = '1'
os.environ['BORG_KEYS_DIR'] = str(tmpdir.join('keys'))
os.environ['BORG_CACHE_DIR'] = str(tmpdir.join('cache'))
yield str(tmpdir.join('repository'))
tmpdir.remove(rec=1)
@pytest.fixture(params=["none", "passphrase"])
def repo(request, cmd, repo_url):
cmd('init', '--encryption', request.param, repo_url)
return repo_url
@pytest.yield_fixture(scope='session', params=["zeros", "random"])
def testdata(request, tmpdir_factory):
count, size = 10, 1000*1000
p = tmpdir_factory.mktemp('data')
data_type = request.param
if data_type == 'zeros':
# do not use a binary zero (\0) to avoid sparse detection
data = lambda: b'0' * size
if data_type == 'random':
rnd = open('/dev/urandom', 'rb')
data = lambda: rnd.read(size)
for i in range(count):
with open(str(p.join(str(i))), "wb") as f:
f.write(data())
if data_type == 'random':
rnd.close()
yield str(p)
p.remove(rec=1)
@pytest.fixture(params=['none', 'lz4'])
def archive(request, cmd, repo, testdata):
archive_url = repo + '::test'
cmd('create', '--compression', request.param, archive_url, testdata)
return archive_url
def test_create_none(benchmark, cmd, repo, testdata):
result, out = benchmark.pedantic(cmd, ('create', '--compression', 'none', repo + '::test', testdata))
assert result == 0
def test_create_lz4(benchmark, cmd, repo, testdata):
result, out = benchmark.pedantic(cmd, ('create', '--compression', 'lz4', repo + '::test', testdata))
assert result == 0
def test_extract(benchmark, cmd, archive, tmpdir):
with changedir(str(tmpdir)):
result, out = benchmark.pedantic(cmd, ('extract', archive))
assert result == 0
def test_delete(benchmark, cmd, archive):
result, out = benchmark.pedantic(cmd, ('delete', archive))
assert result == 0
def test_list(benchmark, cmd, archive):
result, out = benchmark(cmd, 'list', archive)
assert result == 0
def test_info(benchmark, cmd, archive):
result, out = benchmark(cmd, 'info', archive)
assert result == 0
def test_check(benchmark, cmd, archive):
repo = archive.split('::')[0]
result, out = benchmark(cmd, 'check', repo)
assert result == 0
def test_help(benchmark, cmd):
result, out = benchmark(cmd, 'help')
assert result == 0

141
borg/testsuite/helpers.py
View file

@ -1,14 +1,15 @@
import hashlib
from time import mktime, strptime
from datetime import datetime, timezone, timedelta
from io import StringIO
import os
import pytest
import sys
import msgpack
from ..helpers import adjust_patterns, exclude_path, Location, format_timedelta, IncludePattern, ExcludePattern, make_path_safe, \
prune_within, prune_split, get_cache_dir, \
from ..helpers import adjust_patterns, exclude_path, Location, format_file_size, format_timedelta, IncludePattern, ExcludePattern, make_path_safe, \
prune_within, prune_split, get_cache_dir, Statistics, \
StableDict, int_to_bigint, bigint_to_int, parse_timestamp, CompressionSpec, ChunkerParams
from . import BaseTestCase
@ -29,44 +30,44 @@ class TestLocationWithoutEnv:
def test_ssh(self, monkeypatch):
monkeypatch.delenv('BORG_REPO', raising=False)
assert repr(Location('ssh://user@host:1234/some/path::archive')) == \
"Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive='archive')"
"Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive='archive')"
assert repr(Location('ssh://user@host:1234/some/path')) == \
"Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive=None)"
"Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive=None)"
def test_file(self, monkeypatch):
monkeypatch.delenv('BORG_REPO', raising=False)
assert repr(Location('file:///some/path::archive')) == \
"Location(proto='file', user=None, host=None, port=None, path='/some/path', archive='archive')"
"Location(proto='file', user=None, host=None, port=None, path='/some/path', archive='archive')"
assert repr(Location('file:///some/path')) == \
"Location(proto='file', user=None, host=None, port=None, path='/some/path', archive=None)"
"Location(proto='file', user=None, host=None, port=None, path='/some/path', archive=None)"
def test_scp(self, monkeypatch):
monkeypatch.delenv('BORG_REPO', raising=False)
assert repr(Location('user@host:/some/path::archive')) == \
"Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive='archive')"
"Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive='archive')"
assert repr(Location('user@host:/some/path')) == \
"Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)"
"Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)"
def test_folder(self, monkeypatch):
monkeypatch.delenv('BORG_REPO', raising=False)
assert repr(Location('path::archive')) == \
"Location(proto='file', user=None, host=None, port=None, path='path', archive='archive')"
"Location(proto='file', user=None, host=None, port=None, path='path', archive='archive')"
assert repr(Location('path')) == \
"Location(proto='file', user=None, host=None, port=None, path='path', archive=None)"
"Location(proto='file', user=None, host=None, port=None, path='path', archive=None)"
def test_abspath(self, monkeypatch):
monkeypatch.delenv('BORG_REPO', raising=False)
assert repr(Location('/some/absolute/path::archive')) == \
"Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive='archive')"
"Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive='archive')"
assert repr(Location('/some/absolute/path')) == \
"Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive=None)"
"Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive=None)"
def test_relpath(self, monkeypatch):
monkeypatch.delenv('BORG_REPO', raising=False)
assert repr(Location('some/relative/path::archive')) == \
"Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive='archive')"
"Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive='archive')"
assert repr(Location('some/relative/path')) == \
"Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive=None)"
"Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive=None)"
def test_underspecified(self, monkeypatch):
monkeypatch.delenv('BORG_REPO', raising=False)
@ -94,51 +95,51 @@ class TestLocationWithoutEnv:
'ssh://user@host:1234/some/path::archive']
for location in locations:
assert Location(location).canonical_path() == \
Location(Location(location).canonical_path()).canonical_path()
Location(Location(location).canonical_path()).canonical_path()
class TestLocationWithEnv:
def test_ssh(self, monkeypatch):
monkeypatch.setenv('BORG_REPO', 'ssh://user@host:1234/some/path')
assert repr(Location('::archive')) == \
"Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive='archive')"
"Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive='archive')"
assert repr(Location()) == \
"Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive=None)"
"Location(proto='ssh', user='user', host='host', port=1234, path='/some/path', archive=None)"
def test_file(self, monkeypatch):
monkeypatch.setenv('BORG_REPO', 'file:///some/path')
assert repr(Location('::archive')) == \
"Location(proto='file', user=None, host=None, port=None, path='/some/path', archive='archive')"
"Location(proto='file', user=None, host=None, port=None, path='/some/path', archive='archive')"
assert repr(Location()) == \
"Location(proto='file', user=None, host=None, port=None, path='/some/path', archive=None)"
"Location(proto='file', user=None, host=None, port=None, path='/some/path', archive=None)"
def test_scp(self, monkeypatch):
monkeypatch.setenv('BORG_REPO', 'user@host:/some/path')
assert repr(Location('::archive')) == \
"Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive='archive')"
"Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive='archive')"
assert repr(Location()) == \
"Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)"
"Location(proto='ssh', user='user', host='host', port=None, path='/some/path', archive=None)"
def test_folder(self, monkeypatch):
monkeypatch.setenv('BORG_REPO', 'path')
assert repr(Location('::archive')) == \
"Location(proto='file', user=None, host=None, port=None, path='path', archive='archive')"
"Location(proto='file', user=None, host=None, port=None, path='path', archive='archive')"
assert repr(Location()) == \
"Location(proto='file', user=None, host=None, port=None, path='path', archive=None)"
"Location(proto='file', user=None, host=None, port=None, path='path', archive=None)"
def test_abspath(self, monkeypatch):
monkeypatch.setenv('BORG_REPO', '/some/absolute/path')
assert repr(Location('::archive')) == \
"Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive='archive')"
"Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive='archive')"
assert repr(Location()) == \
"Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive=None)"
"Location(proto='file', user=None, host=None, port=None, path='/some/absolute/path', archive=None)"
def test_relpath(self, monkeypatch):
monkeypatch.setenv('BORG_REPO', 'some/relative/path')
assert repr(Location('::archive')) == \
"Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive='archive')"
"Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive='archive')"
assert repr(Location()) == \
"Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive=None)"
"Location(proto='file', user=None, host=None, port=None, path='some/relative/path', archive=None)"
def test_no_slashes(self, monkeypatch):
monkeypatch.setenv('BORG_REPO', '/some/absolute/path')
@ -211,7 +212,7 @@ class PatternNonAsciiTestCase(BaseTestCase):
assert i.match("ba\N{COMBINING ACUTE ACCENT}/foo")
assert not e.match("b\N{LATIN SMALL LETTER A WITH ACUTE}/foo")
assert e.match("ba\N{COMBINING ACUTE ACCENT}/foo")
def testInvalidUnicode(self):
pattern = str(b'ba\x80', 'latin1')
i = IncludePattern(pattern)
@ -234,7 +235,7 @@ class OSXPatternNormalizationTestCase(BaseTestCase):
assert i.match("ba\N{COMBINING ACUTE ACCENT}/foo")
assert e.match("b\N{LATIN SMALL LETTER A WITH ACUTE}/foo")
assert e.match("ba\N{COMBINING ACUTE ACCENT}/foo")
def testDecomposedUnicode(self):
pattern = 'ba\N{COMBINING ACUTE ACCENT}'
i = IncludePattern(pattern)
@ -244,7 +245,7 @@ class OSXPatternNormalizationTestCase(BaseTestCase):
assert i.match("ba\N{COMBINING ACUTE ACCENT}/foo")
assert e.match("b\N{LATIN SMALL LETTER A WITH ACUTE}/foo")
assert e.match("ba\N{COMBINING ACUTE ACCENT}/foo")
def testInvalidUnicode(self):
pattern = str(b'ba\x80', 'latin1')
i = IncludePattern(pattern)
@ -399,3 +400,83 @@ def test_get_cache_dir():
# reset old env
if old_env is not None:
os.environ['BORG_CACHE_DIR'] = old_env
@pytest.fixture()
def stats():
stats = Statistics()
stats.update(20, 10, unique=True)
return stats
def test_stats_basic(stats):
assert stats.osize == 20
assert stats.csize == stats.usize == 10
stats.update(20, 10, unique=False)
assert stats.osize == 40
assert stats.csize == 20
assert stats.usize == 10
def tests_stats_progress(stats, columns=80):
os.environ['COLUMNS'] = str(columns)
out = StringIO()
stats.show_progress(stream=out)
s = '20 B O 10 B C 10 B D 0 N '
buf = ' ' * (columns - len(s))
assert out.getvalue() == s + buf + "\r"
out = StringIO()
stats.update(10**3, 0, unique=False)
stats.show_progress(item={b'path': 'foo'}, final=False, stream=out)
s = '1.02 kB O 10 B C 10 B D 0 N foo'
buf = ' ' * (columns - len(s))
assert out.getvalue() == s + buf + "\r"
out = StringIO()
stats.show_progress(item={b'path': 'foo'*40}, final=False, stream=out)
s = '1.02 kB O 10 B C 10 B D 0 N foofoofoofoofoofoofoofo...oofoofoofoofoofoofoofoofoo'
buf = ' ' * (columns - len(s))
assert out.getvalue() == s + buf + "\r"
def test_stats_format(stats):
assert str(stats) == """\
Original size Compressed size Deduplicated size
This archive: 20 B 10 B 10 B"""
s = "{0.osize_fmt}".format(stats)
assert s == "20 B"
# kind of redundant, but id is variable so we can't match reliably
assert repr(stats) == '<Statistics object at {:#x} (20, 10, 10)>'.format(id(stats))
def test_file_size():
"""test the size formatting routines"""
si_size_map = {
0: '0 B', # no rounding necessary for those
1: '1 B',
142: '142 B',
999: '999 B',
1000: '1.00 kB', # rounding starts here
1001: '1.00 kB', # should be rounded away
1234: '1.23 kB', # should be rounded down
1235: '1.24 kB', # should be rounded up
1010: '1.01 kB', # rounded down as well
999990000: '999.99 MB', # rounded down
999990001: '999.99 MB', # rounded down
999995000: '1.00 GB', # rounded up to next unit
10**6: '1.00 MB', # and all the remaining units, megabytes
10**9: '1.00 GB', # gigabytes
10**12: '1.00 TB', # terabytes
10**15: '1.00 PB', # petabytes
10**18: '1.00 EB', # exabytes
10**21: '1.00 ZB', # zottabytes
10**24: '1.00 YB', # yottabytes
}
for size, fmt in si_size_map.items():
assert format_file_size(size) == fmt
def test_file_size_precision():
assert format_file_size(1234, precision=1) == '1.2 kB' # rounded down
assert format_file_size(1254, precision=1) == '1.3 kB' # rounded up
assert format_file_size(999990000, precision=1) == '1.0 GB' # and not 999.9 MB or 1000.0 MB

40
borg/testsuite/logger.py Normal file
View file

@ -0,0 +1,40 @@
import logging
from io import StringIO
from mock import Mock
import pytest
from ..logger import find_parent_module, create_logger, setup_logging
logger = create_logger()
@pytest.fixture()
def io_logger():
io = StringIO()
handler = setup_logging(io)
handler.setFormatter(logging.Formatter('%(name)s: %(message)s'))
logger.setLevel(logging.DEBUG)
return io
def test_setup_logging(io_logger):
logger.info('hello world')
assert io_logger.getvalue() == "borg.testsuite.logger: hello world\n"
def test_multiple_loggers(io_logger):
logger = logging.getLogger(__name__)
logger.info('hello world 1')
assert io_logger.getvalue() == "borg.testsuite.logger: hello world 1\n"
logger = logging.getLogger('borg.testsuite.logger')
logger.info('hello world 2')
assert io_logger.getvalue() == "borg.testsuite.logger: hello world 1\nborg.testsuite.logger: hello world 2\n"
io_logger.truncate(0)
io_logger.seek(0)
logger = logging.getLogger('borg.testsuite.logger')
logger.info('hello world 2')
assert io_logger.getvalue() == "borg.testsuite.logger: hello world 2\n"
def test_parent_module():
assert find_parent_module() == __name__

36
borg/testsuite/platform.py
View file

@ -72,6 +72,42 @@ class PlatformLinuxTestCase(BaseTestCase):
self.assert_equal(self.get_acl(self.tmpdir)[b'acl_access'], ACCESS_ACL)
self.assert_equal(self.get_acl(self.tmpdir)[b'acl_default'], DEFAULT_ACL)
def test_non_ascii_acl(self):
# Testing non-ascii ACL processing to see whether our code is robust.
# I have no idea whether non-ascii ACLs are allowed by the standard,
# but in practice they seem to be out there and must not make our code explode.
file = tempfile.NamedTemporaryFile()
self.assert_equal(self.get_acl(file.name), {})
nothing_special = 'user::rw-\ngroup::r--\nmask::rw-\nother::---\n'.encode('ascii')
# TODO: can this be tested without having an existing system user übel with uid 666 gid 666?
user_entry = 'user:übel:rw-:666'.encode('utf-8')
user_entry_numeric = 'user:666:rw-:666'.encode('ascii')
group_entry = 'group:übel:rw-:666'.encode('utf-8')
group_entry_numeric = 'group:666:rw-:666'.encode('ascii')
acl = b'\n'.join([nothing_special, user_entry, group_entry])
self.set_acl(file.name, access=acl, numeric_owner=False)
acl_access = self.get_acl(file.name, numeric_owner=False)[b'acl_access']
self.assert_in(user_entry, acl_access)
self.assert_in(group_entry, acl_access)
acl_access_numeric = self.get_acl(file.name, numeric_owner=True)[b'acl_access']
self.assert_in(user_entry_numeric, acl_access_numeric)
self.assert_in(group_entry_numeric, acl_access_numeric)
file2 = tempfile.NamedTemporaryFile()
self.set_acl(file2.name, access=acl, numeric_owner=True)
acl_access = self.get_acl(file2.name, numeric_owner=False)[b'acl_access']
self.assert_in(user_entry, acl_access)
self.assert_in(group_entry, acl_access)
acl_access_numeric = self.get_acl(file.name, numeric_owner=True)[b'acl_access']
self.assert_in(user_entry_numeric, acl_access_numeric)
self.assert_in(group_entry_numeric, acl_access_numeric)
def test_utils(self):
from ..platform_linux import acl_use_local_uid_gid
self.assert_equal(acl_use_local_uid_gid(b'user:nonexistent1234:rw-:1234'), b'user:1234:rw-')
self.assert_equal(acl_use_local_uid_gid(b'group:nonexistent1234:rw-:1234'), b'group:1234:rw-')
self.assert_equal(acl_use_local_uid_gid(b'user:root:rw-:0'), b'user:0:rw-')
self.assert_equal(acl_use_local_uid_gid(b'group:root:rw-:0'), b'group:0:rw-')
@unittest.skipUnless(sys.platform.startswith('darwin'), 'OS X only test')
@unittest.skipIf(fakeroot_detected(), 'not compatible with fakeroot')

69
borg/testsuite/upgrader.py
View file

@ -1,6 +1,4 @@
import os
import shutil
import tempfile
import pytest
@ -14,10 +12,8 @@ except ImportError:
from ..upgrader import AtticRepositoryUpgrader, AtticKeyfileKey
from ..helpers import get_keys_dir
from ..key import KeyfileKey
from ..repository import Repository, MAGIC
pytestmark = pytest.mark.skipif(attic is None,
reason='cannot find an attic install')
from ..remote import RemoteRepository
from ..repository import Repository
def repo_valid(path):
@ -64,7 +60,13 @@ def attic_repo(tmpdir):
return attic_repo
def test_convert_segments(tmpdir, attic_repo):
@pytest.fixture(params=[True, False])
def inplace(request):
return request.param
@pytest.mark.skipif(attic is None, reason='cannot find an attic install')
def test_convert_segments(tmpdir, attic_repo, inplace):
"""test segment conversion
this will load the given attic repository, list all the segments
@ -77,11 +79,10 @@ def test_convert_segments(tmpdir, attic_repo):
"""
# check should fail because of magic number
assert not repo_valid(tmpdir)
print("opening attic repository with borg and converting")
repo = AtticRepositoryUpgrader(str(tmpdir), create=False)
segments = [filename for i, filename in repo.io.segment_iterator()]
repo.close()
repo.convert_segments(segments, dryrun=False)
repo.convert_segments(segments, dryrun=False, inplace=inplace)
repo.convert_cache(dryrun=False)
assert repo_valid(tmpdir)
@ -124,6 +125,7 @@ def attic_key_file(attic_repo, tmpdir):
MockArgs(keys_dir))
@pytest.mark.skipif(attic is None, reason='cannot find an attic install')
def test_keys(tmpdir, attic_repo, attic_key_file):
"""test key conversion
@ -142,7 +144,8 @@ def test_keys(tmpdir, attic_repo, attic_key_file):
assert key_valid(attic_key_file.path)
def test_convert_all(tmpdir, attic_repo, attic_key_file):
@pytest.mark.skipif(attic is None, reason='cannot find an attic install')
def test_convert_all(tmpdir, attic_repo, attic_key_file, inplace):
"""test all conversion steps
this runs everything. mostly redundant test, since everything is
@ -156,8 +159,50 @@ def test_convert_all(tmpdir, attic_repo, attic_key_file):
"""
# check should fail because of magic number
assert not repo_valid(tmpdir)
print("opening attic repository with borg and converting")
def stat_segment(path):
return os.stat(os.path.join(path, 'data', '0', '0'))
def first_inode(path):
return stat_segment(path).st_ino
orig_inode = first_inode(attic_repo.path)
repo = AtticRepositoryUpgrader(str(tmpdir), create=False)
repo.upgrade(dryrun=False)
# replicate command dispatch, partly
os.umask(RemoteRepository.umask)
backup = repo.upgrade(dryrun=False, inplace=inplace)
if inplace:
assert backup is None
assert first_inode(repo.path) == orig_inode
else:
assert backup
assert first_inode(repo.path) != first_inode(backup)
# i have seen cases where the copied tree has world-readable
# permissions, which is wrong
assert stat_segment(backup).st_mode & 0o007 == 0
assert key_valid(attic_key_file.path)
assert repo_valid(tmpdir)
def test_hardlink(tmpdir, inplace):
"""test that we handle hard links properly
that is, if we are in "inplace" mode, hardlinks should *not*
change (ie. we write to the file directly, so we do not rewrite the
whole file, and we do not re-create the file).
if we are *not* in inplace mode, then the inode should change, as
we are supposed to leave the original inode alone."""
a = str(tmpdir.join('a'))
with open(a, 'wb') as tmp:
tmp.write(b'aXXX')
b = str(tmpdir.join('b'))
os.link(a, b)
AtticRepositoryUpgrader.header_replace(b, b'a', b'b', inplace=inplace)
if not inplace:
assert os.stat(a).st_ino != os.stat(b).st_ino
else:
assert os.stat(a).st_ino == os.stat(b).st_ino
with open(b, 'rb') as tmp:
assert tmp.read() == b'bXXX'

87
borg/upgrader.py
View file

@ -1,6 +1,10 @@
from binascii import hexlify
import datetime
import logging
logger = logging.getLogger(__name__)
import os
import shutil
import sys
import time
from .helpers import get_keys_dir, get_cache_dir
@ -12,7 +16,7 @@ ATTIC_MAGIC = b'ATTICSEG'
class AtticRepositoryUpgrader(Repository):
def upgrade(self, dryrun=True):
def upgrade(self, dryrun=True, inplace=False):
"""convert an attic repository to a borg repository
those are the files that need to be upgraded here, from most
@ -23,14 +27,20 @@ class AtticRepositoryUpgrader(Repository):
we nevertheless do the order in reverse, as we prefer to do
the fast stuff first, to improve interactivity.
"""
print("reading segments from attic repository using borg")
# we need to open it to load the configuration and other fields
backup = None
if not inplace:
backup = '{}.upgrade-{:%Y-%m-%d-%H:%M:%S}'.format(self.path, datetime.datetime.now())
logger.info('making a hardlink copy in %s', backup)
if not dryrun:
shutil.copytree(self.path, backup, copy_function=os.link)
logger.info("opening attic repository with borg and converting")
# we need to open the repo to load configuration, keyfiles and segments
self.open(self.path, exclusive=False)
segments = [filename for i, filename in self.io.segment_iterator()]
try:
keyfile = self.find_attic_keyfile()
except KeyfileNotFoundError:
print("no key file found for repository")
logger.warning("no key file found for repository")
else:
self.convert_keyfiles(keyfile, dryrun)
self.close()
@ -39,13 +49,14 @@ class AtticRepositoryUpgrader(Repository):
exclusive=True).acquire()
try:
self.convert_cache(dryrun)
self.convert_segments(segments, dryrun)
self.convert_segments(segments, dryrun=dryrun, inplace=inplace)
finally:
self.lock.release()
self.lock = None
return backup
@staticmethod
def convert_segments(segments, dryrun):
def convert_segments(segments, dryrun=True, inplace=False):
"""convert repository segments from attic to borg
replacement pattern is `s/ATTICSEG/BORG_SEG/` in files in
@ -53,26 +64,39 @@ class AtticRepositoryUpgrader(Repository):
luckily the magic string length didn't change so we can just
replace the 8 first bytes of all regular files in there."""
print("converting %d segments..." % len(segments))
logger.info("converting %d segments..." % len(segments))
i = 0
for filename in segments:
i += 1
print("\rconverting segment %d/%d in place, %.2f%% done (%s)"
% (i, len(segments), 100*float(i)/len(segments), filename), end='')
print("\rconverting segment %d/%d, %.2f%% done (%s)"
% (i, len(segments), 100*float(i)/len(segments), filename),
end='', file=sys.stderr)
if dryrun:
time.sleep(0.001)
else:
AtticRepositoryUpgrader.header_replace(filename, ATTIC_MAGIC, MAGIC)
print()
AtticRepositoryUpgrader.header_replace(filename, ATTIC_MAGIC, MAGIC, inplace=inplace)
print(file=sys.stderr)
@staticmethod
def header_replace(filename, old_magic, new_magic):
def header_replace(filename, old_magic, new_magic, inplace=True):
with open(filename, 'r+b') as segment:
segment.seek(0)
# only write if necessary
if segment.read(len(old_magic)) == old_magic:
segment.seek(0)
segment.write(new_magic)
if inplace:
segment.seek(0)
segment.write(new_magic)
else:
# rename the hardlink and rewrite the file. this works
# because the file is still open. so even though the file
# is renamed, we can still read it until it is closed.
os.rename(filename, filename + '.tmp')
with open(filename, 'wb') as new_segment:
new_segment.write(new_magic)
new_segment.write(segment.read())
# the little dance with the .tmp file is necessary
# because Windows won't allow overwriting an open file.
os.unlink(filename + '.tmp')
def find_attic_keyfile(self):
"""find the attic keyfiles
@ -107,12 +131,12 @@ class AtticRepositoryUpgrader(Repository):
key file because magic string length changed, but that's not a
problem because the keyfiles are small (compared to, say,
all the segments)."""
print("converting keyfile %s" % keyfile)
logger.info("converting keyfile %s" % keyfile)
with open(keyfile, 'r') as f:
data = f.read()
data = data.replace(AtticKeyfileKey.FILE_ID, KeyfileKey.FILE_ID, 1)
keyfile = os.path.join(get_keys_dir(), os.path.basename(keyfile))
print("writing borg keyfile to %s" % keyfile)
logger.info("writing borg keyfile to %s" % keyfile)
if not dryrun:
with open(keyfile, 'w') as f:
f.write(data)
@ -135,12 +159,14 @@ class AtticRepositoryUpgrader(Repository):
`Cache.open()`, edit in place and then `Cache.close()` to
make sure we have locking right
"""
caches = []
transaction_id = self.get_index_transaction_id()
if transaction_id is None:
print('no index file found for repository %s' % self.path)
logger.warning('no index file found for repository %s' % self.path)
else:
caches += [os.path.join(self.path, 'index.%d' % transaction_id).encode('utf-8')]
index = os.path.join(self.path, 'index.%d' % transaction_id).encode('utf-8')
logger.info("converting index index %s" % index)
if not dryrun:
AtticRepositoryUpgrader.header_replace(index, b'ATTICIDX', b'BORG_IDX')
# copy of attic's get_cache_dir()
attic_cache_dir = os.environ.get('ATTIC_CACHE_DIR',
@ -160,23 +186,23 @@ class AtticRepositoryUpgrader(Repository):
:params path: the basename of the cache file to copy
(example: "files" or "chunks") as a string
:returns: the borg file that was created or None if non
was created.
:returns: the borg file that was created or None if no
Attic cache file was found.
"""
attic_file = os.path.join(attic_cache_dir, path)
if os.path.exists(attic_file):
borg_file = os.path.join(borg_cache_dir, path)
if os.path.exists(borg_file):
print("borg cache file already exists in %s, skipping conversion of %s" % (borg_file, attic_file))
logger.warning("borg cache file already exists in %s, not copying from Attic", borg_file)
else:
print("copying attic cache file from %s to %s" % (attic_file, borg_file))
logger.info("copying attic cache file from %s to %s" % (attic_file, borg_file))
if not dryrun:
shutil.copyfile(attic_file, borg_file)
return borg_file
return borg_file
else:
print("no %s cache file found in %s" % (path, attic_file))
return None
logger.warning("no %s cache file found in %s" % (path, attic_file))
return None
# XXX: untested, because generating cache files is a PITA, see
# Archiver.do_create() for proof
@ -190,11 +216,10 @@ class AtticRepositoryUpgrader(Repository):
# we need to convert the headers of those files, copy first
for cache in ['chunks']:
copied = copy_cache_file(cache)
if copied:
print("converting cache %s" % cache)
if not dryrun:
AtticRepositoryUpgrader.header_replace(cache, b'ATTICIDX', b'BORG_IDX')
cache = copy_cache_file(cache)
logger.info("converting cache %s" % cache)
if not dryrun:
AtticRepositoryUpgrader.header_replace(cache, b'ATTICIDX', b'BORG_IDX')
class AtticKeyfileKey(KeyfileKey):

42
docs/Makefile
View file

@ -36,7 +36,7 @@ help:
clean:
-rm -rf $(BUILDDIR)/*
html: usage api.rst
html:
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
@ -128,43 +128,3 @@ doctest:
$(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest
@echo "Testing of doctests in the sources finished, look at the " \
"results in $(BUILDDIR)/doctest/output.txt."
gh-io: html
GH_IO_CLONE="`mktemp -d`" && \
git clone git@github.com:borgbackup/borgbackup.github.io.git $$GH_IO_CLONE && \
(cd $$GH_IO_CLONE && git rm -r *) && \
cp -r _build/html/* $$GH_IO_CLONE && \
(cd $$GH_IO_CLONE && git add -A && git commit -m 'Updated borgbackup.github.io' && git push) && \
rm -rf $$GH_IO_CLONE
inotify: html
while inotifywait -r . --exclude usage.rst --exclude '_build/*' ; do make html ; done
# generate list of targets
usage: $(shell borg help | grep -A1 "Available commands:" | tail -1 | sed 's/[{} ]//g;s/,\|^/.rst.inc usage\//g;s/^.rst.inc//;s/usage\/help//')
# generate help file based on usage
usage/%.rst.inc: ../borg/archiver.py
@echo generating usage for $*
@printf ".. _borg_$*:\n\n" > $@
@printf "borg $*\n" >> $@
@echo -n borg $* | tr 'a-z- ' '-' >> $@
@printf "\n::\n\n" >> $@
@borg help $* --usage-only | sed -e 's/^/ /' >> $@
@printf "\nDescription\n~~~~~~~~~~~\n" >> $@
@borg help $* --epilog-only >> $@
api.rst: Makefile
@echo "auto-generating API documentation"
@echo "Borg Backup API documentation" > $@
@echo "=============================" >> $@
@echo "" >> $@
@for mod in ../borg/*.pyx ../borg/*.py; do \
if echo "$$mod" | grep -q "/_"; then \
continue ; \
fi ; \
printf ".. automodule:: "; \
echo "$$mod" | sed "s!\.\./!!;s/\.pyx\?//;s!/!.!"; \
echo " :members:"; \
echo " :undoc-members:"; \
done >> $@

0
docs/_themes/local/static/favicon.ico → docs/_static/favicon.ico vendored
View file

Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 38 KiB

After

Width:  |  Height:  |  Size: 38 KiB

5
docs/_themes/local/sidebarlogo.html vendored
View file

@ -1,5 +0,0 @@
<div class="sidebarlogo">
<a href="{{ pathto('index') }}">
<div class="title">Borg</div>
</a>
</div>

20
docs/_themes/local/sidebarusefullinks.html vendored
View file

@ -1,20 +0,0 @@
<a href="https://github.com/borgbackup/borg"><img style="position: fixed; top: 0; right: 0; border: 0;"
src="https://s3.amazonaws.com/github/ribbons/forkme_right_gray_6d6d6d.png" alt="Fork me on GitHub"></a>
<h3>Useful Links</h3>
<ul>
<li><a href="https://borgbackup.github.io/borgbackup/">Main Web Site</a></li>
<li><a href="https://pypi.python.org/pypi/borgbackup">PyPI packages</a></li>
<li><a href="https://github.com/borgbackup/borg/issues/214">Binaries</a></li>
<li><a href="https://github.com/borgbackup/borg/blob/master/CHANGES.rst">Current ChangeLog</a></li>
<li><a href="https://github.com/borgbackup/borg">GitHub</a></li>
<li><a href="https://github.com/borgbackup/borg/issues">Issue Tracker</a></li>
<li><a href="https://www.bountysource.com/teams/borgbackup">Bounties &amp; Fundraisers</a></li>
<li><a href="http://librelist.com/browser/borgbackup/">Mailing List</a></li>
</ul>
<h3>Related Projects</h3>
<ul>
<li><a href="https://borgbackup.github.io/borgweb/">BorgWeb</a></li>
</ul>

173
docs/_themes/local/static/local.css_t vendored
View file

@ -1,173 +0,0 @@
@import url("basic.css");
@import url(//fonts.googleapis.com/css?family=Black+Ops+One);
body {
font-family: Arial, Helvetica, sans-serif;
background-color: black;
margin: 0;
padding: 0;
position: relative;
}
div.related {
display: none;
background-color: black;
padding: .4em;
width: 800px;
margin: 0 auto;
}
div.related a {
color: white;
text-decoration: none;
}
div.document {
width: 1030px;
margin: 0 auto;
}
div.documentwrapper {
float: right;
width: 760px;
padding: 0 20px 20px 20px;
color: #00cc00;
background-color: #000000;
margin-bottom: 2em;
}
div.sphinxsidebar {
margin-left: 0;
padding-right: 20px;
width: 230px;
background: #081008;
position: absolute;
top: 0;
min-height: 100%;
}
h1, h2, h3 {
font-weight: normal;
color: #33ff33;
}
h1 {
margin: .8em 0 .5em;
font-size: 200%;
}
h2 {
margin: 1.2em 0 .6em;
font-size: 140%;
}
h3 {
margin: 1.2em 0 .6em;
font-size: 110%;
}
ul {
padding-left: 1.2em;
margin-bottom: .3em;
}
ul ul {
font-size: 95%;
}
li {
margin: .1em 0;
}
a:link {
color: #dddd00;
text-decoration: none;
}
a:visited {
color: #990000;
text-decoration: none;
}
a:hover {
color: #dd0000;
border-bottom: 1px dotted #dd0000;
}
div.sphinxsidebar a:link, div.sphinxsidebar a:visited {
border-bottom: 1px dotted #555;
}
div.sphinxsidebar {
color: #00cc00;
background: 0000000;
}
div.sphinxsidebar input {
color: #00ff00;
background: 0000000;
border: 1px solid #444444;
}
pre {
padding: 10px 20px;
background: #101010;
color: #22cc22;
line-height: 1.5em;
border-bottom: 2px solid black;
}
pre a:link,
pre a:visited {
color: #00b0e4;
}
div.sidebarlogo .title {
font-family: 'Black Ops One', cursive;
font-size: 500%;
}
div.sidebarlogo a {
color: #00dd00;
}
div.sidebarlogo .subtitle {
font-style: italic;
color: #777;
}
tt span.pre {
font-size: 110%;
}
dt {
font-size: 95%;
}
div.admonition p.admonition-title + p {
display: inline;
}
div.admonition p {
margin-bottom: 5px;
}
p.admonition-title {
display: inline;
}
p.admonition-title:after {
content: ":";
}
div.note {
background-color: #002211;
border-bottom: 2px solid #22dd22;
}
div.seealso {
background-color: #0fe;
border: 1px solid #0f6;
border-radius: .4em;
box-shadow: 2px 2px #dd6;
}

6
docs/_themes/local/theme.conf vendored
View file

@ -1,6 +0,0 @@
[theme]
inherit = basic
stylesheet = local.css
pygments_style = tango
[options]

11
docs/authors.rst Normal file
View file

@ -0,0 +1,11 @@
.. include:: global.rst.inc
.. include:: ../AUTHORS
License
=======
.. _license:
.. include:: ../LICENSE
:literal:

578
docs/changes.rst
View file

@ -1,4 +1,576 @@
.. include:: global.rst.inc
.. _changelog:
Changelog
=========
.. include:: ../CHANGES.rst
Version 0.28.0
--------------
New features:
- refactor return codes (exit codes), fixes #61
- give a final status into the log output, including exit code, fixes #58
- borg create backups atime and ctime additionally to mtime, fixes #317
- extract: support atime additionally to mtime
- FUSE: support ctime and atime additionally to mtime
- support borg --version
Bug fixes:
- setup.py: fix bug related to BORG_LZ4_PREFIX processing
- borg mount: fix unlocking of repository at umount time, fixes #331
- fix reading files without touching their atime, #334
- non-ascii ACL fixes for Linux, FreeBSD and OS X, #277
- fix acl_use_local_uid_gid() and add a test for it, attic #359
- borg upgrade: do not upgrade repositories in place by default, #299
- fix cascading failure with the index conversion code, #269
- borg check: implement 'cmdline' archive metadata value decoding, #311
Other changes:
- improve file size displays
- convert to more flexible size formatters
- explicitely commit to the units standard, #289
- archiver: add E status (means that an error occured when processing this
(single) item
- do binary releases via "github releases", closes #214
- create: use -x and --one-file-system (was: --do-not-cross-mountpoints), #296
- a lot of changes related to using "logging" module and screen output, #233
- show progress display if on a tty, output more progress information, #303
- factor out status output so it is consistent, fix surrogates removal,
maybe fixes #309
- benchmarks: test create, extract, list, delete, info, check, help, fixes #146
- benchmarks: test with both the binary and the python code
- tests:
- travis: also run tests on Python 3.5
- travis: use tox -r so it rebuilds the tox environments
- test the generated pyinstaller-based binary by archiver unit tests, #215
- vagrant: tests: announce whether fakeroot is used or not
- vagrant: add vagrant user to fuse group for debianoid systems also
- vagrant: llfuse install on darwin needs pkgconfig installed
- archiver tests: test with both the binary and the python code, fixes #215
- docs:
- moved docs to borgbackup.readthedocs.org, #155
- a lot of fixes and improvements, use mobile-friendly RTD standard theme
- use zlib,6 compression in some examples, fixes #275
- add missing rename usage to docs, closes #279
- include the help offered by borg help <topic> in the usage docs, fixes #293
- include a list of major changes compared to attic into README, fixes #224
- add OS X install instructions, #197
- more details about the release process, #260
- fix linux glibc requirement (binaries built on debian7 now)
- build: move usage and API generation to setup.py
- update docs about return codes, #61
- remove api docs (too much breakage on rtd)
- borgbackup install + basics presentation (asciinema)
- describe the current style guide in documentation
Version 0.27.0
--------------
New features:
- "borg upgrade" command - attic -> borg one time converter / migration, #21
- temporary hack to avoid using lots of disk space for chunks.archive.d, #235:
To use it: rm -rf chunks.archive.d ; touch chunks.archive.d
- respect XDG_CACHE_HOME, attic #181
- add support for arbitrary SSH commands, attic #99
- borg delete --cache-only REPO (only delete cache, not REPO), attic #123
Bug fixes:
- use Debian 7 (wheezy) to build pyinstaller borgbackup binaries, fixes slow
down observed when running the Centos6-built binary on Ubuntu, #222
- do not crash on empty lock.roster, fixes #232
- fix multiple issues with the cache config version check, #234
- fix segment entry header size check, attic #352
plus other error handling improvements / code deduplication there.
- always give segment and offset in repo IntegrityErrors
Other changes:
- stop producing binary wheels, remove docs about it, #147
- docs:
- add warning about prune
- generate usage include files only as needed
- development docs: add Vagrant section
- update / improve / reformat FAQ
- hint to single-file pyinstaller binaries from README
Version 0.26.1
--------------
This is a minor update, just docs and new pyinstaller binaries.
- docs update about python and binary requirements
- better docs for --read-special, fix #220
- re-built the binaries, fix #218 and #213 (glibc version issue)
- update web site about single-file pyinstaller binaries
Note: if you did a python-based installation, there is no need to upgrade.
Version 0.26.0
--------------
New features:
- Faster cache sync (do all in one pass, remove tar/compression stuff), #163
- BORG_REPO env var to specify the default repo, #168
- read special files as if they were regular files, #79
- implement borg create --dry-run, attic issue #267
- Normalize paths before pattern matching on OS X, #143
- support OpenBSD and NetBSD (except xattrs/ACLs)
- support / run tests on Python 3.5
Bug fixes:
- borg mount repo: use absolute path, attic #200, attic #137
- chunker: use off_t to get 64bit on 32bit platform, #178
- initialize chunker fd to -1, so it's not equal to STDIN_FILENO (0)
- fix reaction to "no" answer at delete repo prompt, #182
- setup.py: detect lz4.h header file location
- to support python < 3.2.4, add less buggy argparse lib from 3.2.6 (#194)
- fix for obtaining ``char *`` from temporary Python value (old code causes
a compile error on Mint 17.2)
- llfuse 0.41 install troubles on some platforms, require < 0.41
(UnicodeDecodeError exception due to non-ascii llfuse setup.py)
- cython code: add some int types to get rid of unspecific python add /
subtract operations (avoid ``undefined symbol FPE_``... error on some platforms)
- fix verbose mode display of stdin backup
- extract: warn if a include pattern never matched, fixes #209,
implement counters for Include/ExcludePatterns
- archive names with slashes are invalid, attic issue #180
- chunker: add a check whether the POSIX_FADV_DONTNEED constant is defined -
fixes building on OpenBSD.
Other changes:
- detect inconsistency / corruption / hash collision, #170
- replace versioneer with setuptools_scm, #106
- docs:
- pkg-config is needed for llfuse installation
- be more clear about pruning, attic issue #132
- unit tests:
- xattr: ignore security.selinux attribute showing up
- ext3 seems to need a bit more space for a sparse file
- do not test lzma level 9 compression (avoid MemoryError)
- work around strange mtime granularity issue on netbsd, fixes #204
- ignore st_rdev if file is not a block/char device, fixes #203
- stay away from the setgid and sticky mode bits
- use Vagrant to do easy cross-platform testing (#196), currently:
- Debian 7 "wheezy" 32bit, Debian 8 "jessie" 64bit
- Ubuntu 12.04 32bit, Ubuntu 14.04 64bit
- Centos 7 64bit
- FreeBSD 10.2 64bit
- OpenBSD 5.7 64bit
- NetBSD 6.1.5 64bit
- Darwin (OS X Yosemite)
Version 0.25.0
--------------
Compatibility notes:
- lz4 compression library (liblz4) is a new requirement (#156)
- the new compression code is very compatible: as long as you stay with zlib
compression, older borg releases will still be able to read data from a
repo/archive made with the new code (note: this is not the case for the
default "none" compression, use "zlib,0" if you want a "no compression" mode
that can be read by older borg). Also the new code is able to read repos and
archives made with older borg versions (for all zlib levels 0..9).
Deprecations:
- --compression N (with N being a number, as in 0.24) is deprecated.
We keep the --compression 0..9 for now to not break scripts, but it is
deprecated and will be removed later, so better fix your scripts now:
--compression 0 (as in 0.24) is the same as --compression zlib,0 (now).
BUT: if you do not want compression, you rather want --compression none
(which is the default).
--compression 1 (in 0.24) is the same as --compression zlib,1 (now)
--compression 9 (in 0.24) is the same as --compression zlib,9 (now)
New features:
- create --compression none (default, means: do not compress, just pass through
data "as is". this is more efficient than zlib level 0 as used in borg 0.24)
- create --compression lz4 (super-fast, but not very high compression)
- create --compression zlib,N (slower, higher compression, default for N is 6)
- create --compression lzma,N (slowest, highest compression, default N is 6)
- honor the nodump flag (UF_NODUMP) and do not backup such items
- list --short just outputs a simple list of the files/directories in an archive
Bug fixes:
- fixed --chunker-params parameter order confusion / malfunction, fixes #154
- close fds of segments we delete (during compaction)
- close files which fell out the lrucache
- fadvise DONTNEED now is only called for the byte range actually read, not for
the whole file, fixes #158.
- fix issue with negative "all archives" size, fixes #165
- restore_xattrs: ignore if setxattr fails with EACCES, fixes #162
Other changes:
- remove fakeroot requirement for tests, tests run faster without fakeroot
(test setup does not fail any more without fakeroot, so you can run with or
without fakeroot), fixes #151 and #91.
- more tests for archiver
- recover_segment(): don't assume we have an fd for segment
- lrucache refactoring / cleanup, add dispose function, py.test tests
- generalize hashindex code for any key length (less hardcoding)
- lock roster: catch file not found in remove() method and ignore it
- travis CI: use requirements file
- improved docs:
- replace hack for llfuse with proper solution (install libfuse-dev)
- update docs about compression
- update development docs about fakeroot
- internals: add some words about lock files / locking system
- support: mention BountySource and for what it can be used
- theme: use a lighter green
- add pypi, wheel, dist package based install docs
- split install docs into system-specific preparations and generic instructions
Version 0.24.0
--------------
Incompatible changes (compared to 0.23):
- borg now always issues --umask NNN option when invoking another borg via ssh
on the repository server. By that, it's making sure it uses the same umask
for remote repos as for local ones. Because of this, you must upgrade both
server and client(s) to 0.24.
- the default umask is 077 now (if you do not specify via --umask) which might
be a different one as you used previously. The default umask avoids that
you accidentally give access permissions for group and/or others to files
created by borg (e.g. the repository).
Deprecations:
- "--encryption passphrase" mode is deprecated, see #85 and #97.
See the new "--encryption repokey" mode for a replacement.
New features:
- borg create --chunker-params ... to configure the chunker, fixes #16
(attic #302, attic #300, and somehow also #41).
This can be used to reduce memory usage caused by chunk management overhead,
so borg does not create a huge chunks index/repo index and eats all your RAM
if you back up lots of data in huge files (like VM disk images).
See docs/misc/create_chunker-params.txt for more information.
- borg info now reports chunk counts in the chunk index.
- borg create --compression 0..9 to select zlib compression level, fixes #66
(attic #295).
- borg init --encryption repokey (to store the encryption key into the repo),
fixes #85
- improve at-end error logging, always log exceptions and set exit_code=1
- LoggedIO: better error checks / exceptions / exception handling
- implement --remote-path to allow non-default-path borg locations, #125
- implement --umask M and use 077 as default umask for better security, #117
- borg check: give a named single archive to it, fixes #139
- cache sync: show progress indication
- cache sync: reimplement the chunk index merging in C
Bug fixes:
- fix segfault that happened for unreadable files (chunker: n needs to be a
signed size_t), #116
- fix the repair mode, #144
- repo delete: add destroy to allowed rpc methods, fixes issue #114
- more compatible repository locking code (based on mkdir), maybe fixes #92
(attic #317, attic #201).
- better Exception msg if no Borg is installed on the remote repo server, #56
- create a RepositoryCache implementation that can cope with >2GiB,
fixes attic #326.
- fix Traceback when running check --repair, attic #232
- clarify help text, fixes #73.
- add help string for --no-files-cache, fixes #140
Other changes:
- improved docs:
- added docs/misc directory for misc. writeups that won't be included
"as is" into the html docs.
- document environment variables and return codes (attic #324, attic #52)
- web site: add related projects, fix web site url, IRC #borgbackup
- Fedora/Fedora-based install instructions added to docs
- Cygwin-based install instructions added to docs
- updated AUTHORS
- add FAQ entries about redundancy / integrity
- clarify that borg extract uses the cwd as extraction target
- update internals doc about chunker params, memory usage and compression
- added docs about development
- add some words about resource usage in general
- document how to backup a raw disk
- add note about how to run borg from virtual env
- add solutions for (ll)fuse installation problems
- document what borg check does, fixes #138
- reorganize borgbackup.github.io sidebar, prev/next at top
- deduplicate and refactor the docs / README.rst
- use borg-tmp as prefix for temporary files / directories
- short prune options without "keep-" are deprecated, do not suggest them
- improved tox configuration
- remove usage of unittest.mock, always use mock from pypi
- use entrypoints instead of scripts, for better use of the wheel format and
modern installs
- add requirements.d/development.txt and modify tox.ini
- use travis-ci for testing based on Linux and (new) OS X
- use coverage.py, pytest-cov and codecov.io for test coverage support
I forgot to list some stuff already implemented in 0.23.0, here they are:
New features:
- efficient archive list from manifest, meaning a big speedup for slow
repo connections and "list <repo>", "delete <repo>", "prune" (attic #242,
attic #167)
- big speedup for chunks cache sync (esp. for slow repo connections), fixes #18
- hashindex: improve error messages
Other changes:
- explicitly specify binary mode to open binary files
- some easy micro optimizations
Version 0.23.0
--------------
Incompatible changes (compared to attic, fork related):
- changed sw name and cli command to "borg", updated docs
- package name (and name in urls) uses "borgbackup" to have less collisions
- changed repo / cache internal magic strings from ATTIC* to BORG*,
changed cache location to .cache/borg/ - this means that it currently won't
accept attic repos (see issue #21 about improving that)
Bug fixes:
- avoid defect python-msgpack releases, fixes attic #171, fixes attic #185
- fix traceback when trying to do unsupported passphrase change, fixes attic #189
- datetime does not like the year 10.000, fixes attic #139
- fix "info" all archives stats, fixes attic #183
- fix parsing with missing microseconds, fixes attic #282
- fix misleading hint the fuse ImportError handler gave, fixes attic #237
- check unpacked data from RPC for tuple type and correct length, fixes attic #127
- fix Repository._active_txn state when lock upgrade fails
- give specific path to xattr.is_enabled(), disable symlink setattr call that
always fails
- fix test setup for 32bit platforms, partial fix for attic #196
- upgraded versioneer, PEP440 compliance, fixes attic #257
New features:
- less memory usage: add global option --no-cache-files
- check --last N (only check the last N archives)
- check: sort archives in reverse time order
- rename repo::oldname newname (rename repository)
- create -v output more informative
- create --progress (backup progress indicator)
- create --timestamp (utc string or reference file/dir)
- create: if "-" is given as path, read binary from stdin
- extract: if --stdout is given, write all extracted binary data to stdout
- extract --sparse (simple sparse file support)
- extra debug information for 'fread failed'
- delete <repo> (deletes whole repo + local cache)
- FUSE: reflect deduplication in allocated blocks
- only allow whitelisted RPC calls in server mode
- normalize source/exclude paths before matching
- use posix_fadvise to not spoil the OS cache, fixes attic #252
- toplevel error handler: show tracebacks for better error analysis
- sigusr1 / sigint handler to print current file infos - attic PR #286
- RPCError: include the exception args we get from remote
Other changes:
- source: misc. cleanups, pep8, style
- docs and faq improvements, fixes, updates
- cleanup crypto.pyx, make it easier to adapt to other AES modes
- do os.fsync like recommended in the python docs
- source: Let chunker optionally work with os-level file descriptor.
- source: Linux: remove duplicate os.fsencode calls
- source: refactor _open_rb code a bit, so it is more consistent / regular
- source: refactor indicator (status) and item processing
- source: use py.test for better testing, flake8 for code style checks
- source: fix tox >=2.0 compatibility (test runner)
- pypi package: add python version classifiers, add FreeBSD to platforms
Attic Changelog
---------------
Here you can see the full list of changes between each Attic release until Borg
forked from Attic:
Version 0.17
~~~~~~~~~~~~
(bugfix release, released on X)
- Fix hashindex ARM memory alignment issue (#309)
- Improve hashindex error messages (#298)
Version 0.16
~~~~~~~~~~~~
(bugfix release, released on May 16, 2015)
- Fix typo preventing the security confirmation prompt from working (#303)
- Improve handling of systems with improperly configured file system encoding (#289)
- Fix "All archives" output for attic info. (#183)
- More user friendly error message when repository key file is not found (#236)
- Fix parsing of iso 8601 timestamps with zero microseconds (#282)
Version 0.15
~~~~~~~~~~~~
(bugfix release, released on Apr 15, 2015)
- xattr: Be less strict about unknown/unsupported platforms (#239)
- Reduce repository listing memory usage (#163).
- Fix BrokenPipeError for remote repositories (#233)
- Fix incorrect behavior with two character directory names (#265, #268)
- Require approval before accessing relocated/moved repository (#271)
- Require approval before accessing previously unknown unencrypted repositories (#271)
- Fix issue with hash index files larger than 2GB.
- Fix Python 3.2 compatibility issue with noatime open() (#164)
- Include missing pyx files in dist files (#168)
Version 0.14
~~~~~~~~~~~~
(feature release, released on Dec 17, 2014)
- Added support for stripping leading path segments (#95)
"attic extract --strip-segments X"
- Add workaround for old Linux systems without acl_extended_file_no_follow (#96)
- Add MacPorts' path to the default openssl search path (#101)
- HashIndex improvements, eliminates unnecessary IO on low memory systems.
- Fix "Number of files" output for attic info. (#124)
- limit create file permissions so files aren't read while restoring
- Fix issue with empty xattr values (#106)
Version 0.13
~~~~~~~~~~~~
(feature release, released on Jun 29, 2014)
- Fix sporadic "Resource temporarily unavailable" when using remote repositories
- Reduce file cache memory usage (#90)
- Faster AES encryption (utilizing AES-NI when available)
- Experimental Linux, OS X and FreeBSD ACL support (#66)
- Added support for backup and restore of BSDFlags (OSX, FreeBSD) (#56)
- Fix bug where xattrs on symlinks were not correctly restored
- Added cachedir support. CACHEDIR.TAG compatible cache directories
can now be excluded using ``--exclude-caches`` (#74)
- Fix crash on extreme mtime timestamps (year 2400+) (#81)
- Fix Python 3.2 specific lockf issue (EDEADLK)
Version 0.12
~~~~~~~~~~~~
(feature release, released on April 7, 2014)
- Python 3.4 support (#62)
- Various documentation improvements a new style
- ``attic mount`` now supports mounting an entire repository not only
individual archives (#59)
- Added option to restrict remote repository access to specific path(s):
``attic serve --restrict-to-path X`` (#51)
- Include "all archives" size information in "--stats" output. (#54)
- Added ``--stats`` option to ``attic delete`` and ``attic prune``
- Fixed bug where ``attic prune`` used UTC instead of the local time zone
when determining which archives to keep.
- Switch to SI units (Power of 1000 instead 1024) when printing file sizes
Version 0.11
~~~~~~~~~~~~
(feature release, released on March 7, 2014)
- New "check" command for repository consistency checking (#24)
- Documentation improvements
- Fix exception during "attic create" with repeated files (#39)
- New "--exclude-from" option for attic create/extract/verify.
- Improved archive metadata deduplication.
- "attic verify" has been deprecated. Use "attic extract --dry-run" instead.
- "attic prune --hourly|daily|..." has been deprecated.
Use "attic prune --keep-hourly|daily|..." instead.
- Ignore xattr errors during "extract" if not supported by the filesystem. (#46)
Version 0.10
~~~~~~~~~~~~
(bugfix release, released on Jan 30, 2014)
- Fix deadlock when extracting 0 sized files from remote repositories
- "--exclude" wildcard patterns are now properly applied to the full path
not just the file name part (#5).
- Make source code endianness agnostic (#1)
Version 0.9
~~~~~~~~~~~
(feature release, released on Jan 23, 2014)
- Remote repository speed and reliability improvements.
- Fix sorting of segment names to ignore NFS left over files. (#17)
- Fix incorrect display of time (#13)
- Improved error handling / reporting. (#12)
- Use fcntl() instead of flock() when locking repository/cache. (#15)
- Let ssh figure out port/user if not specified so we don't override .ssh/config (#9)
- Improved libcrypto path detection (#23).
Version 0.8.1
~~~~~~~~~~~~~
(bugfix release, released on Oct 4, 2013)
- Fix segmentation fault issue.
Version 0.8
~~~~~~~~~~~
(feature release, released on Oct 3, 2013)
- Fix xattr issue when backing up sshfs filesystems (#4)
- Fix issue with excessive index file size (#6)
- Support access of read only repositories.
- New syntax to enable repository encryption:
attic init --encryption="none|passphrase|keyfile".
- Detect and abort if repository is older than the cache.
Version 0.7
~~~~~~~~~~~
(feature release, released on Aug 5, 2013)
- Ported to FreeBSD
- Improved documentation
- Experimental: Archives mountable as fuse filesystems.
- The "user." prefix is no longer stripped from xattrs on Linux
Version 0.6.1
~~~~~~~~~~~~~
(bugfix release, released on July 19, 2013)
- Fixed an issue where mtime was not always correctly restored.
Version 0.6
~~~~~~~~~~~
First public release on July 9, 2013

14
docs/conf.py
View file

@ -19,6 +19,8 @@ sys.path.insert(0, os.path.abspath('..'))
from borg import __version__ as sw_version
on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
@ -92,7 +94,11 @@ pygments_style = 'sphinx'
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'local'
#html_theme = ''
if not on_rtd: # only import and set the theme if we're building docs locally
import sphinx_rtd_theme
html_theme = 'sphinx_rtd_theme'
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
@ -111,17 +117,17 @@ html_theme_path = ['_themes']
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
html_logo = '_static/favicon.ico'
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
html_favicon = 'favicon.ico'
html_favicon = '_static/favicon.ico'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = []
#html_static_path = ['_static']
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.

85
docs/development.rst
View file

@ -9,6 +9,15 @@ This chapter will get you started with |project_name|' development.
|project_name| is written in Python (with a little bit of Cython and C for
the performance critical parts).
Style guide
-----------
We generally follow `pep8
<https://www.python.org/dev/peps/pep-0008/>`_, with 120 columns
instead of 79. We do *not* use form-feed (``^L``) characters to
separate sections either. The `flake8
<https://flake8.readthedocs.org/>`_ commandline tool should be used to
check for style errors before sending pull requests.
Building a development environment
----------------------------------
@ -68,6 +77,9 @@ Now run::
Then point a web browser at docs/_build/html/index.html.
The website is updated automatically through Github web hooks on the
main repository.
Using Vagrant
-------------
@ -91,49 +103,54 @@ Usage::
vagrant scp OS:/vagrant/borg/borg/dist/borg .
Creating a new release
----------------------
Checklist::
- all issues for this milestone closed?
- any low hanging fruit left on the issue tracker?
- run tox on all supported platforms via vagrant, check for test fails.
- is Travis CI happy also?
- update CHANGES.rst (compare to git log). check version number of upcoming release.
- check MANIFEST.in and setup.py - are they complete?
- tag the release::
git tag -s -m "tagged release" 0.26.0
- cd docs ; make html # to update the usage include files
- update website with the html
- create a release on PyPi::
python setup.py register sdist upload --identity="Thomas Waldmann" --sign
- close release milestone.
- announce on::
- mailing list
- Twitter
- IRC channel (topic)
- create standalone binaries and link them from issue tracker: https://github.com/borgbackup/borg/issues/214
Creating standalone binaries
----------------------------
Make sure you have everything built and installed (including llfuse and fuse).
When using the Vagrant VMs, pyinstaller will already be installed.
With virtual env activated::
pip install pyinstaller>=3.0 # or git checkout master
pyinstaller -F -n borg-PLATFORM --hidden-import=logging.config borg/__main__.py
ls -l dist/*
for file in dist/borg-*; do gpg --armor --detach-sign $file; done
If you encounter issues, see also our `Vagrantfile` for details.
Note: Standalone binaries built with pyinstaller are supposed to work on same OS,
same architecture (x86 32bit, amd64 64bit) without external dependencies.
.. note:: Standalone binaries built with pyinstaller are supposed to
work on same OS, same architecture (x86 32bit, amd64 64bit)
without external dependencies.
Creating a new release
----------------------
Checklist:
- make sure all issues for this milestone are closed or moved to the
next milestone
- find and fix any low hanging fruit left on the issue tracker
- run tox on all supported platforms via vagrant, check for test failures
- check that Travis CI is also happy
- update ``CHANGES.rst``, based on ``git log $PREVIOUS_RELEASE..``
- check version number of upcoming release in ``CHANGES.rst``
- verify that ``MANIFEST.in`` and ``setup.py`` are complete
- tag the release::
git tag -s -m "tagged/signed release X.Y.Z" X.Y.Z
- build fresh docs and update the web site with them
- create a release on PyPi::
python setup.py register sdist upload --identity="Thomas Waldmann" --sign
- close release milestone on Github
- announce on:
- `mailing list <mailto:borgbackup@librelist.org>`_
- Twitter (follow @ThomasJWaldmann for these tweets)
- `IRC channel <irc://irc.freenode.net/borgbackup>`_ (change ``/topic``)
- create a Github release, include:
* standalone binaries (see above for how to create them)
* a link to ``CHANGES.rst``

203
docs/faq.rst
View file

@ -5,24 +5,30 @@ Frequently asked questions
==========================
Can I backup VM disk images?
Yes, the :ref:`deduplication <deduplication_def>` technique used by
|project_name| makes sure only the modified parts of the file are stored.
Also, we have optional simple sparse file support for extract.
----------------------------
Yes, the `deduplication`_ technique used by
|project_name| makes sure only the modified parts of the file are stored.
Also, we have optional simple sparse file support for extract.
Can I backup from multiple servers into a single repository?
Yes, but in order for the deduplication used by |project_name| to work, it
needs to keep a local cache containing checksums of all file
chunks already stored in the repository. This cache is stored in
``~/.cache/borg/``. If |project_name| detects that a repository has been
modified since the local cache was updated it will need to rebuild
the cache. This rebuild can be quite time consuming.
------------------------------------------------------------
So, yes it's possible. But it will be most efficient if a single
repository is only modified from one place. Also keep in mind that
|project_name| will keep an exclusive lock on the repository while creating
or deleting archives, which may make *simultaneous* backups fail.
Yes, but in order for the deduplication used by |project_name| to work, it
needs to keep a local cache containing checksums of all file
chunks already stored in the repository. This cache is stored in
``~/.cache/borg/``. If |project_name| detects that a repository has been
modified since the local cache was updated it will need to rebuild
the cache. This rebuild can be quite time consuming.
So, yes it's possible. But it will be most efficient if a single
repository is only modified from one place. Also keep in mind that
|project_name| will keep an exclusive lock on the repository while creating
or deleting archives, which may make *simultaneous* backups fail.
Which file types, attributes, etc. are preserved?
-------------------------------------------------
* Directories
* Regular files
* Hardlinks (considering all files in the same archive)
@ -40,6 +46,8 @@ Which file types, attributes, etc. are preserved?
* BSD flags on OS X and FreeBSD
Which file types, attributes, etc. are *not* preserved?
-------------------------------------------------------
* UNIX domain sockets (because it does not make sense - they are
meaningless without the running process that created them and the process
needs to recreate them in any case). So, don't panic if your backup
@ -50,91 +58,138 @@ Which file types, attributes, etc. are *not* preserved?
Archive extraction has optional support to extract all-zero chunks as
holes in a sparse file.
Why is my backup bigger than with attic? Why doesn't |project_name| do compression by default?
----------------------------------------------------------------------------------------------
Attic was rather unflexible when it comes to compression, it always
compressed using zlib level 6 (no way to switch compression off or
adjust the level or algorithm).
|project_name| offers a lot of different compression algorithms and
levels. Which of them is the best for you pretty much depends on your
use case, your data, your hardware - so you need to do an informed
decision about whether you want to use compression, which algorithm
and which level you want to use. This is why compression defaults to
none.
How can I specify the encryption passphrase programmatically?
The encryption passphrase can be specified programmatically using the
`BORG_PASSPHRASE` environment variable. This is convenient when setting up
automated encrypted backups. Another option is to use
key file based encryption with a blank passphrase. See
:ref:`encrypted_repos` for more details.
-------------------------------------------------------------
The encryption passphrase can be specified programmatically using the
`BORG_PASSPHRASE` environment variable. This is convenient when setting up
automated encrypted backups. Another option is to use
key file based encryption with a blank passphrase. See
:ref:`encrypted_repos` for more details.
.. _password_env:
.. note:: Be careful how you set the environment; using the ``env``
command, a ``system()`` call or using inline shell scripts
might expose the credentials in the process list directly
and they will be readable to all users on a system. Using
``export`` in a shell script file should be safe, however, as
the environment of a process is `accessible only to that
user
<http://security.stackexchange.com/questions/14000/environment-variable-accessibility-in-linux/14009#14009>`_.
When backing up to remote encrypted repos, is encryption done locally?
Yes, file and directory metadata and data is locally encrypted, before
leaving the local machine. We do not mean the transport layer encryption
by that, but the data/metadata itself. Transport layer encryption (e.g.
when ssh is used as a transport) applies additionally.
----------------------------------------------------------------------
Yes, file and directory metadata and data is locally encrypted, before
leaving the local machine. We do not mean the transport layer encryption
by that, but the data/metadata itself. Transport layer encryption (e.g.
when ssh is used as a transport) applies additionally.
When backing up to remote servers, do I have to trust the remote server?
Yes and No.
No, as far as data confidentiality is concerned - if you use encryption,
all your files/dirs data and metadata are stored in their encrypted form
into the repository.
Yes, as an attacker with access to the remote server could delete (or
otherwise make unavailable) all your backups.
------------------------------------------------------------------------
Yes and No.
No, as far as data confidentiality is concerned - if you use encryption,
all your files/dirs data and metadata are stored in their encrypted form
into the repository.
Yes, as an attacker with access to the remote server could delete (or
otherwise make unavailable) all your backups.
If a backup stops mid-way, does the already-backed-up data stay there?
Yes, |project_name| supports resuming backups.
During a backup a special checkpoint archive named ``<archive-name>.checkpoint``
is saved every checkpoint interval (the default value for this is 5
minutes) containing all the data backed-up until that point. This means
that at most <checkpoint interval> worth of data needs to be retransmitted
if a backup needs to be restarted.
Once your backup has finished successfully, you can delete all ``*.checkpoint``
archives.
----------------------------------------------------------------------
Yes, |project_name| supports resuming backups.
During a backup a special checkpoint archive named ``<archive-name>.checkpoint``
is saved every checkpoint interval (the default value for this is 5
minutes) containing all the data backed-up until that point. This means
that at most <checkpoint interval> worth of data needs to be retransmitted
if a backup needs to be restarted.
Once your backup has finished successfully, you can delete all ``*.checkpoint``
archives.
If it crashes with a UnicodeError, what can I do?
Check if your encoding is set correctly. For most POSIX-like systems, try::
-------------------------------------------------
export LANG=en_US.UTF-8 # or similar, important is correct charset
Check if your encoding is set correctly. For most POSIX-like systems, try::
export LANG=en_US.UTF-8 # or similar, important is correct charset
I can't extract non-ascii filenames by giving them on the commandline!?
This might be due to different ways to represent some characters in unicode
or due to other non-ascii encoding issues.
If you run into that, try this:
-----------------------------------------------------------------------
- avoid the non-ascii characters on the commandline by e.g. extracting
the parent directory (or even everything)
- mount the repo using FUSE and use some file manager
This might be due to different ways to represent some characters in unicode
or due to other non-ascii encoding issues.
If you run into that, try this:
- avoid the non-ascii characters on the commandline by e.g. extracting
the parent directory (or even everything)
- mount the repo using FUSE and use some file manager
Can |project_name| add redundancy to the backup data to deal with hardware malfunction?
No, it can't. While that at first sounds like a good idea to defend against
some defect HDD sectors or SSD flash blocks, dealing with this in a
reliable way needs a lot of low-level storage layout information and
control which we do not have (and also can't get, even if we wanted).
---------------------------------------------------------------------------------------
So, if you need that, consider RAID or a filesystem that offers redundant
storage or just make backups to different locations / different hardware.
No, it can't. While that at first sounds like a good idea to defend against
some defect HDD sectors or SSD flash blocks, dealing with this in a
reliable way needs a lot of low-level storage layout information and
control which we do not have (and also can't get, even if we wanted).
See also `ticket 225 <https://github.com/borgbackup/borg/issues/225>`_.
So, if you need that, consider RAID or a filesystem that offers redundant
storage or just make backups to different locations / different hardware.
See also `ticket 225 <https://github.com/borgbackup/borg/issues/225>`_.
Can |project_name| verify data integrity of a backup archive?
Yes, if you want to detect accidental data damage (like bit rot), use the
``check`` operation. It will notice corruption using CRCs and hashes.
If you want to be able to detect malicious tampering also, use a encrypted
repo. It will then be able to check using CRCs and HMACs.
-------------------------------------------------------------
Yes, if you want to detect accidental data damage (like bit rot), use the
``check`` operation. It will notice corruption using CRCs and hashes.
If you want to be able to detect malicious tampering also, use a encrypted
repo. It will then be able to check using CRCs and HMACs.
Why was Borg forked from Attic?
Borg was created in May 2015 in response to the difficulty of getting new
code or larger changes incorporated into Attic and establishing a bigger
developer community / more open development.
-------------------------------
More details can be found in `ticket 217
<https://github.com/jborg/attic/issues/217>`_ that led to the fork.
Borg was created in May 2015 in response to the difficulty of getting new
code or larger changes incorporated into Attic and establishing a bigger
developer community / more open development.
Borg intends to be:
More details can be found in `ticket 217
<https://github.com/jborg/attic/issues/217>`_ that led to the fork.
* simple:
Borg intends to be:
* as simple as possible, but no simpler
* do the right thing by default, but offer options
* open:
* simple:
* welcome feature requests
* accept pull requests of good quality and coding style
* give feedback on PRs that can't be accepted "as is"
* discuss openly, don't work in the dark
* changing:
* as simple as possible, but no simpler
* do the right thing by default, but offer options
* open:
* Borg is not compatible with Attic
* do not break compatibility accidentally, without a good reason
or without warning. allow compatibility breaking for other cases.
* if major version number changes, it may have incompatible changes
* welcome feature requests
* accept pull requests of good quality and coding style
* give feedback on PRs that can't be accepted "as is"
* discuss openly, don't work in the dark
* changing:
* Borg is not compatible with Attic
* do not break compatibility accidentally, without a good reason
or without warning. allow compatibility breaking for other cases.
* if major version number changes, it may have incompatible changes

4
docs/global.rst.inc
View file

@ -16,12 +16,12 @@
.. _libattr: http://savannah.nongnu.org/projects/attr/
.. _liblz4: https://github.com/Cyan4973/lz4
.. _OpenSSL: https://www.openssl.org/
.. _Python: http://www.python.org/
.. _`Python 3`: http://www.python.org/
.. _Buzhash: https://en.wikipedia.org/wiki/Buzhash
.. _msgpack: http://msgpack.org/
.. _`msgpack-python`: https://pypi.python.org/pypi/msgpack-python/
.. _llfuse: https://pypi.python.org/pypi/llfuse/
.. _homebrew: http://mxcl.github.io/homebrew/
.. _homebrew: http://brew.sh/
.. _userspace filesystems: https://en.wikipedia.org/wiki/Filesystem_in_Userspace
.. _librelist: http://librelist.com/
.. _Cython: http://cython.org/

5
docs/index.rst
View file

@ -4,10 +4,11 @@
Borg Documentation
==================
.. include:: ../README.rst
.. toctree::
:maxdepth: 2
intro
installation
quickstart
usage
@ -16,4 +17,4 @@ Borg Documentation
changes
internals
development
api
authors

241
docs/installation.rst
View file

@ -4,153 +4,126 @@
Installation
============
|project_name| pyinstaller binary installation requires:
There are different ways to install |project_name|:
* Linux: glibc >= 2.12 (ok for most supported Linux releases)
* MacOS X: 10.10 (unknown whether it works for older releases)
* FreeBSD: 10.2 (unknown whether it works for older releases)
|project_name| non-binary installation requires:
* Python_ >= 3.2.2
* OpenSSL_ >= 1.0.0
* libacl_ (that pulls in libattr_ also)
* liblz4_
* some python dependencies, see install_requires in setup.py
General notes
-------------
You need to do some platform specific preparation steps (to install libraries
and tools) followed by the generic installation of |project_name| itself:
Below, we describe different ways to install |project_name|.
- **dist package** - easy and fast, needs a distribution and platform specific
binary package (for your Linux/*BSD/OS X/... distribution).
- **pyinstaller binary** - easy and fast, we provide a ready-to-use binary file
that just works on the supported platforms
- **pypi** - installing a source package from pypi needs more installation steps
and will need a compiler, development headers, etc..
- **distribution package** - easy and fast if a package is available for your
Linux/BSD distribution.
- **PyInstaller binary** - easy and fast, we provide a ready-to-use binary file
that comes bundled with all dependencies.
- **pip** - installing a source package with pip needs more installation steps
and requires all dependencies with development headers and a compiler.
- **git** - for developers and power users who want to have the latest code or
use revision control (each release is tagged).
**Python 3**: Even though this is not the default Python version on many systems,
it is usually available as an optional install.
Virtualenv_ can be used to build and install |project_name| without affecting
the system Python or requiring root access.
Installation (Distribution Package)
-----------------------------------
Important:
If you install into a virtual environment, you need to **activate**
the virtual env first (``source borg-env/bin/activate``).
Alternatively, directly run ``borg-env/bin/borg`` (or symlink that into some
directory that is in your PATH so you can just run ``borg``).
Using a virtual environment is optional, but recommended except for the most
simple use cases.
Some Linux and BSD distributions might offer a ready-to-use ``borgbackup``
package which can be installed with the package manager. As |project_name| is
still a young project, such a package might be not available for your system
yet. Please ask package maintainers to build a package or, if you can package /
submit it yourself, please help us with that!
The llfuse_ python package is also required if you wish to mount an
archive as a FUSE filesystem. Only FUSE >= 2.8.0 can support llfuse.
* On **Arch Linux**, there is a package available in the AUR_.
You only need **Cython** to compile the .pyx files to the respective .c files
when using |project_name| code from git. For |project_name| releases, the .c
files will be bundled, so you won't need Cython to install a release.
If a package is available, it might be interesting to check its version
and compare that to our latest release and review the :doc:`changes`.
Platform notes
--------------
FreeBSD: You may need to get a recent enough OpenSSL version from FreeBSD ports.
Mac OS X: You may need to get a recent enough OpenSSL version from homebrew_.
Mac OS X: You need OS X FUSE >= 3.0.
.. _AUR: https://aur.archlinux.org/packages/borgbackup/
Installation (dist package)
---------------------------
Some Linux, BSD and OS X distributions might offer a ready-to-use
`borgbackup` package (which can be easily installed in the usual way).
As |project_name| is still relatively new, such a package might be not
available for your system yet. Please ask package maintainers to build a
package or, if you can package / submit it yourself, please help us with
that!
If a package is available, it might be interesting for you to check its version
and compare that to our latest release and review the change log (see links on
our web site).
Installation (pyinstaller binary)
Installation (PyInstaller Binary)
---------------------------------
For some platforms we offer a ready-to-use standalone borg binary.
It is supposed to work without requiring installation or preparations.
The |project_name| binary is available on the releases_ page for the following
platforms:
Check https://github.com/borgbackup/borg/issues/214 for available binaries.
* **Linux**: glibc >= 2.13 (ok for most supported Linux releases)
* **Mac OS X**: 10.10 (unknown whether it works for older releases)
* **FreeBSD**: 10.2 (unknown whether it works for older releases)
These binaries work without requiring specific installation steps. Just drop
them into a directory in your ``PATH`` and then you can run ``borg``. If a new
version is released, you will have to manually download it and replace the old
version.
Debian Jessie / Ubuntu 14.04 preparations (git/pypi)
----------------------------------------------------
.. _releases: https://github.com/borgbackup/borg/releases
.. parsed-literal::
Installing the Dependencies
---------------------------
# Python 3.x (>= 3.2) + Headers, Py Package Installer, VirtualEnv
apt-get install python3 python3-dev python3-pip python-virtualenv
To install |project_name| from a source package, you have to install the
following dependencies first:
# we need OpenSSL + Headers for Crypto
apt-get install libssl-dev openssl
* `Python 3`_ >= 3.2.2. Even though Python 3 is not the default Python version on
most systems, it is usually available as an optional install.
* OpenSSL_ >= 1.0.0
* libacl_ (that pulls in libattr_ also)
* liblz4_
* some Python dependencies, pip will automatically install them for you
* optionally, the llfuse_ Python package is required if you wish to mount an
archive as a FUSE filesystem. FUSE >= 2.8.0 is required for llfuse.
# ACL support Headers + Library
apt-get install libacl1-dev libacl1
In the following, the steps needed to install the dependencies are listed for a
selection of platforms. If your distribution is not covered by these
instructions, try to use your package manager to install the dependencies. On
FreeBSD, you may need to get a recent enough OpenSSL version from FreeBSD
ports.
# lz4 super fast compression support Headers + Library
apt-get install liblz4-dev liblz4-1
After you have installed the dependencies, you can proceed with steps outlined
under :ref:`pip-installation`.
# if you do not have gcc / make / etc. yet
apt-get install build-essential
Debian / Ubuntu
~~~~~~~~~~~~~~~
# optional: FUSE support - to mount backup archives
# in case you get complaints about permission denied on /etc/fuse.conf:
# on ubuntu this means your user is not in the "fuse" group. just add
# yourself there, log out and log in again.
apt-get install libfuse-dev fuse pkg-config
Install the dependencies with development headers::
# optional: for unit testing
apt-get install fakeroot
sudo apt-get install python3 python3-dev python3-pip python-virtualenv
sudo apt-get install libssl-dev openssl
sudo apt-get install libacl1-dev libacl1
sudo apt-get install liblz4-dev liblz4-1
sudo apt-get install build-essential
sudo apt-get install libfuse-dev fuse pkg-config # optional, for FUSE support
In case you get complaints about permission denied on ``/etc/fuse.conf``: on
Ubuntu this means your user is not in the ``fuse`` group. Add yourself to that
group, log out and log in again.
Korora / Fedora 21 preparations (git/pypi)
------------------------------------------
Fedora / Korora
~~~~~~~~~~~~~~~
.. parsed-literal::
Install the dependencies with development headers::
# Python 3.x (>= 3.2) + Headers, Py Package Installer, VirtualEnv
sudo dnf install python3 python3-devel python3-pip python3-virtualenv
# we need OpenSSL + Headers for Crypto
sudo dnf install openssl-devel openssl
# ACL support Headers + Library
sudo dnf install libacl-devel libacl
# lz4 super fast compression support Headers + Library
sudo dnf install lz4-devel
# optional: FUSE support - to mount backup archives
sudo dnf install fuse-devel fuse pkgconfig
# optional: for unit testing
sudo dnf install fakeroot
sudo dnf install fuse-devel fuse pkgconfig # optional, for FUSE support
Cygwin preparations (git/pypi)
------------------------------
Mac OS X
~~~~~~~~
Please note that running under cygwin is rather experimental, stuff has been
tested with CygWin (x86-64) v2.1.0.
Assuming you have installed homebrew_, the following steps will install all the
dependencies::
You'll need at least (use the cygwin installer to fetch/install these):
brew install python3 lz4 openssl
pip3 install virtualenv
::
For FUSE support to mount the backup archives, you need at least version 3.0 of
FUSE for OS X, which is available as a pre-release_.
.. _pre-release: https://github.com/osxfuse/osxfuse/releases
Cygwin
~~~~~~
.. note::
Running under Cygwin is experimental and has only been tested with Cygwin
(x86-64) v2.1.0.
Use the Cygwin installer to install the dependencies::
python3 python3-setuptools
python3-cython # not needed for releases
@ -159,50 +132,55 @@ You'll need at least (use the cygwin installer to fetch/install these):
liblz4_1 liblz4-devel # from cygwinports.org
git make openssh
You can then install ``pip`` and ``virtualenv``:
::
You can then install ``pip`` and ``virtualenv``::
easy_install-3.4 pip
pip install virtualenv
And now continue with the generic installation (see below).
In case that creation of the virtual env fails, try deleting this file:
::
In case the creation of the virtual environment fails, try deleting this file::
/usr/lib/python3.4/__pycache__/platform.cpython-34.pyc
Installation (pypi)
-------------------
.. _pip-installation:
This uses the latest (source package) release from PyPi.
Installation (pip)
------------------
.. parsed-literal::
Virtualenv_ can be used to build and install |project_name| without affecting
the system Python or requiring root access. Using a virtual environment is
optional, but recommended except for the most simple use cases.
.. note::
If you install into a virtual environment, you need to **activate** it
first (``source borg-env/bin/activate``), before running ``borg``.
Alternatively, symlink ``borg-env/bin/borg`` into some directory that is in
your ``PATH`` so you can just run ``borg``.
This will use ``pip`` to install the latest release from PyPi::
virtualenv --python=python3 borg-env
source borg-env/bin/activate # always before using!
source borg-env/bin/activate
# install borg + dependencies into virtualenv
# install Borg + Python dependencies into virtualenv
pip install 'llfuse<0.41' # optional, for FUSE support
# 0.41 and 0.41.1 have unicode issues at install time
pip install borgbackup
Note: we install into a virtual environment here, but this is not a requirement.
To upgrade |project_name| to a new version later, run the following after
activating your virtual environment::
pip install -U borgbackup
Installation (git)
------------------
This uses latest, unreleased development code from git.
While we try not to break master, there are no guarantees on anything.
While we try not to break master, there are no guarantees on anything. ::
.. parsed-literal::
# get |project_name| from github, install it
git clone |git_url|
# get borg from github
git clone https://github.com/borgbackup/borg.git
virtualenv --python=python3 borg-env
source borg-env/bin/activate # always before using!
@ -216,6 +194,7 @@ While we try not to break master, there are no guarantees on anything.
pip install -e . # in-place editable mode
# optional: run all the tests, on all supported Python versions
# requires fakeroot, available through your package manager
fakeroot -u tox
Note: as a developer or power user, you always want to use a virtual environment.
.. note:: As a developer or power user, you always want to use a virtual environment.

7
docs/intro.rst
View file

@ -1,7 +0,0 @@
.. include:: global.rst.inc
.. _foreword:
Introduction
============
.. include:: ../README.rst

5550
docs/misc/asciinema/install_and_basics.json Normal file
View file

File diff suppressed because it is too large Load diff

51
docs/misc/asciinema/install_and_basics.txt Normal file
View file

@ -0,0 +1,51 @@
# borgbackup - installation and basic usage
# I have already downloaded the binary release from github:
ls -l
# binary file + GPG signature
# verifying whether the binary is valid:
gpg --verify borg-linux64.asc borg-linux64
# install it as "borg":
cp borg-linux64 ~/bin/borg
# making it executable:
chmod +x ~/bin/borg
# yay, installation done! let's make backups!
# creating a repository:
borg init repo
# creating our first backup with stuff from "data" directory:
borg create --stats --progress --compression lz4 repo::backup1 data
# changing the data slightly:
echo "some more data" > data/one_file_more
# creating another backup:
borg create --stats --progress repo::backup2 data
# that was much faster! it recognized/deduplicated unchanged files.
# see the "Deduplicated size" column for "This archive"! :)
# extracting a backup archive:
mv data data.orig
borg extract repo::backup2
# checking if restored data differs from original data:
diff -r data.orig data
# no, it doesn't! :)
# listing the repo contents:
borg list repo
# listing the backup2 archive contents (shortened):
borg list repo::backup2 | tail
# easy, isn't it?
# if you like #borgbackup, spread the word!

12
docs/quickstart.rst
View file

@ -100,17 +100,17 @@ Backup compression
Default is no compression, but we support different methods with high speed
or high compression:
If you have a quick repo storage and you want a little compression:
If you have a quick repo storage and you want a little compression: ::
$ borg create --compression lz4 /mnt/backup::repo ~
If you have a medium fast repo storage and you want a bit more compression (N=0..9,
0 means no compression, 9 means high compression):
0 means no compression, 9 means high compression): ::
$ borg create --compression zlib,N /mnt/backup::repo ~
If you have a very slow repo storage and you want high compression (N=0..9, 0 means
low compression, 9 means high compression):
low compression, 9 means high compression): ::
$ borg create --compression lzma,N /mnt/backup::repo ~
@ -150,7 +150,11 @@ by providing the correct passphrase.
For automated backups the passphrase can be specified using the
`BORG_PASSPHRASE` environment variable.
**The repository data is totally inaccessible without the key:**
.. note:: Be careful about how you set that environment, see
:ref:`this note about password environments <password_env>`
for more information.
.. important:: The repository data is totally inaccessible without the key:**
Make a backup copy of the key file (``keyfile`` mode) or repo config
file (``repokey`` mode) and keep it at a safe place, so you still have
the key in case it gets corrupted or lost.

105
docs/usage.rst
View file

@ -22,19 +22,15 @@ Return codes
::
0 no error, normal termination
1 some error occurred (this can be a complete or a partial failure)
128+N killed by signal N (e.g. 137 == kill -9)
0 = success (logged as INFO)
1 = warning (operation reached its normal end, but there were warnings -
you should check the log, logged as WARNING)
2 = error (like a fatal error, a local or remote exception, the operation
did not reach its normal end, logged as ERROR)
128+N = killed by signal N (e.g. 137 == kill -9)
The return code is also logged at the indicated level as the last log entry.
Note: we are aware that more distinct return codes might be useful, but it is
not clear yet which return codes should be used for which precise conditions.
See issue #61 for a discussion about that. Depending on the outcome of the
discussion there, return codes may change in future (the only thing rather sure
is that 0 will always mean some sort of success and "not 0" will always mean
some sort of warning / error / failure - but the definition of success might
change).
Environment Variables
---------------------
@ -60,6 +56,12 @@ Some "yes" sayers (if set, they automatically confirm that you really want to do
For "Warning: The repository at location ... was previously located at ..."
BORG_CHECK_I_KNOW_WHAT_I_AM_DOING
For "Warning: 'check --repair' is an experimental feature that might result in data loss."
BORG_CYTHON_DISABLE
Disables the loading of Cython modules. This is currently
experimental and is used only to generate usage docs at build
time. It is unlikely to produce good results on a regular
run. The variable should be set to the name of the calling class, and
should be unique across all of borg. It is currently only used by ``build_usage``.
Directories:
BORG_KEYS_DIR
@ -128,6 +130,19 @@ Network:
In case you are interested in more details, please read the internals documentation.
Units
-----
To display quantities, |project_name| takes care of respecting the
usual conventions of scale. Disk sizes are displayed in `decimal
<https://en.wikipedia.org/wiki/Decimal>`_, using powers of ten (so
``kB`` means 1000 bytes). For memory usage, `binary prefixes
<https://en.wikipedia.org/wiki/Binary_prefix>`_ are used, and are
indicated using the `IEC binary prefixes
<https://en.wikipedia.org/wiki/IEC_80000-13#Prefixes_for_binary_multiples>`_,
using powers of two (so ``KiB`` means 1024 bytes).
.. include:: usage/init.rst.inc
Examples
@ -195,8 +210,9 @@ Examples
--exclude '*.pyc'
# Backup the root filesystem into an archive named "root-YYYY-MM-DD"
# use zlib compression (good, but slow) - default is no compression
NAME="root-`date +%Y-%m-%d`"
$ borg create /mnt/backup::$NAME / --do-not-cross-mountpoints
$ borg create -C zlib,6 /mnt/backup::$NAME / --do-not-cross-mountpoints
# Backup huge files with little chunk management overhead
$ borg create --chunker-params 19,23,21,4095 /mnt/backup::VMs /srv/VMs
@ -239,6 +255,8 @@ Note: currently, extract always writes into the current working directory ("."),
.. include:: usage/check.rst.inc
.. include:: usage/rename.rst.inc
.. include:: usage/delete.rst.inc
.. include:: usage/list.rst.inc
@ -309,7 +327,7 @@ Examples
Hostname: myhostname
Username: root
Time: Fri Aug 2 15:18:17 2013
Command line: /usr/bin/borg create --stats /mnt/backup::root-2013-08-02 / --do-not-cross-mountpoints
Command line: /usr/bin/borg create --stats -C zlib,6 /mnt/backup::root-2013-08-02 / --do-not-cross-mountpoints
Number of files: 147429
Original size: 5344169493 (4.98 GB)
Compressed size: 1748189642 (1.63 GB)
@ -362,65 +380,74 @@ Examples
command="borg serve --restrict-to-path /mnt/backup" ssh-rsa AAAAB3[...]
Miscellaneous Help
------------------
.. include:: usage/help.rst.inc
Additional Notes
================
----------------
Here are misc. notes about topics that are maybe not covered in enough detail in the usage section.
--read-special
--------------
~~~~~~~~~~~~~~
The option --read-special is not intended for normal, filesystem-level (full or
The option ``--read-special`` is not intended for normal, filesystem-level (full or
partly-recursive) backups. You only give this option if you want to do something
rather ... special - and if you have hand-picked some files that you want to treat
rather ... special -- and if you have hand-picked some files that you want to treat
that way.
`borg create --read-special` will open all files without doing any special treatment
according to the file type (the only exception here are directories: they will be
recursed into). Just imagine what happens if you do `cat filename` - the content
you will see there is what borg will backup for that filename.
``borg create --read-special`` will open all files without doing any special
treatment according to the file type (the only exception here are directories:
they will be recursed into). Just imagine what happens if you do ``cat
filename`` --- the content you will see there is what borg will backup for that
filename.
So, for example, symlinks will be followed, block device content will be read,
named pipes / UNIX domain sockets will be read.
You need to be careful with what you give as filename when using --read-special,
e.g. if you give /dev/zero, your backup will never terminate.
You need to be careful with what you give as filename when using ``--read-special``,
e.g. if you give ``/dev/zero``, your backup will never terminate.
The given files' metadata is saved as it would be saved without --read-special
(e.g. its name, its size [might be 0], its mode, etc.) - but additionally, also
the content read from it will be saved for it.
The given files' metadata is saved as it would be saved without
``--read-special`` (e.g. its name, its size [might be 0], its mode, etc.) - but
additionally, also the content read from it will be saved for it.
Restoring such files' content is currently only supported one at a time via --stdout
option (and you have to redirect stdout to where ever it shall go, maybe directly
into an existing device file of your choice or indirectly via dd).
Restoring such files' content is currently only supported one at a time via
``--stdout`` option (and you have to redirect stdout to where ever it shall go,
maybe directly into an existing device file of your choice or indirectly via
``dd``).
Example
~~~~~~~
Imagine you have made some snapshots of logical volumes (LVs) you want to backup.
Note: For some scenarios, this is a good method to get "crash-like" consistency
(I call it crash-like because it is the same as you would get if you just hit the
reset button or your machine would abrubtly and completely crash).
This is better than no consistency at all and a good method for some use cases,
but likely not good enough if you have databases running.
.. note::
For some scenarios, this is a good method to get "crash-like" consistency
(I call it crash-like because it is the same as you would get if you just
hit the reset button or your machine would abrubtly and completely crash).
This is better than no consistency at all and a good method for some use
cases, but likely not good enough if you have databases running.
Then you create a backup archive of all these snapshots. The backup process will
see a "frozen" state of the logical volumes, while the processes working in the
original volumes continue changing the data stored there.
You also add the output of `lvdisplay` to your backup, so you can see the LV sizes
in case you ever need to recreate and restore them.
You also add the output of ``lvdisplay`` to your backup, so you can see the LV
sizes in case you ever need to recreate and restore them.
After the backup has completed, you remove the snapshots again.
After the backup has completed, you remove the snapshots again. ::
::
$ # create snapshots here
$ lvdisplay > lvdisplay.txt
$ borg create --read-special /mnt/backup::repo lvdisplay.txt /dev/vg0/*-snapshot
$ # remove snapshots here
Now, let's see how to restore some LVs from such a backup.
Now, let's see how to restore some LVs from such a backup. ::
$ borg extract /mnt/backup::repo lvdisplay.txt
$ # create empty LVs with correct sizes here (look into lvdisplay.txt).

1
requirements.d/development.txt
View file

@ -2,4 +2,5 @@ tox
mock
pytest
pytest-cov<2.0.0
pytest-benchmark==3.0.0b1
Cython

6
setup.cfg
View file

@ -2,7 +2,5 @@
python_files = testsuite/*.py
[flake8]
ignore = E226,F403
max-line-length = 250
exclude = docs/conf.py,borg/_version.py,build,dist,.git,.idea,.cache
max-complexity = 100
max-line-length = 120
exclude = build,dist,.git,.idea,.cache,.tox

156
setup.py
View file

@ -1,8 +1,15 @@
# -*- encoding: utf-8 *-*
import os
import re
import sys
from glob import glob
from distutils.command.build import build
from distutils.core import Command
from distutils.errors import DistutilsOptionError
from distutils import log
from setuptools.command.build_py import build_py
min_python = (3, 2)
my_python = sys.version_info
@ -10,6 +17,9 @@ if my_python < min_python:
print("Borg requires Python %d.%d or later" % min_python)
sys.exit(1)
# Are we building on ReadTheDocs?
on_rtd = os.environ.get('READTHEDOCS')
# msgpack pure python data corruption was fixed in 0.4.6.
# Also, we might use some rather recent API features.
install_requires=['msgpack-python>=0.4.6', ]
@ -62,7 +72,7 @@ except ImportError:
platform_freebsd_source = platform_freebsd_source.replace('.pyx', '.c')
platform_darwin_source = platform_darwin_source.replace('.pyx', '.c')
from distutils.command.build_ext import build_ext
if not all(os.path.exists(path) for path in [
if not on_rtd and not all(os.path.exists(path) for path in [
compress_source, crypto_source, chunker_source, hashindex_source,
platform_linux_source, platform_freebsd_source]):
raise ImportError('The GIT version of Borg needs Cython. Install Cython or use a released version.')
@ -101,31 +111,149 @@ library_dirs.append(os.path.join(ssl_prefix, 'lib'))
possible_lz4_prefixes = ['/usr', '/usr/local', '/usr/local/opt/lz4', '/usr/local/lz4', '/usr/local/borg', '/opt/local']
if os.environ.get('BORG_LZ4_PREFIX'):
possible_openssl_prefixes.insert(0, os.environ.get('BORG_LZ4_PREFIX'))
possible_lz4_prefixes.insert(0, os.environ.get('BORG_LZ4_PREFIX'))
lz4_prefix = detect_lz4(possible_lz4_prefixes)
if not lz4_prefix:
if lz4_prefix:
include_dirs.append(os.path.join(lz4_prefix, 'include'))
library_dirs.append(os.path.join(lz4_prefix, 'lib'))
elif not on_rtd:
raise Exception('Unable to find LZ4 headers. (Looked here: {})'.format(', '.join(possible_lz4_prefixes)))
include_dirs.append(os.path.join(lz4_prefix, 'include'))
library_dirs.append(os.path.join(lz4_prefix, 'lib'))
with open('README.rst', 'r') as fd:
long_description = fd.read()
cmdclass = {'build_ext': build_ext, 'sdist': Sdist}
class build_usage(Command):
description = "generate usage for each command"
ext_modules = [
user_options = [
('output=', 'O', 'output directory'),
]
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
print('generating usage docs')
# allows us to build docs without the C modules fully loaded during help generation
if 'BORG_CYTHON_DISABLE' not in os.environ:
os.environ['BORG_CYTHON_DISABLE'] = self.__class__.__name__
from borg.archiver import Archiver
parser = Archiver().build_parser(prog='borg')
choices = {}
for action in parser._actions:
if action.choices is not None:
choices.update(action.choices)
print('found commands: %s' % list(choices.keys()))
if not os.path.exists('docs/usage'):
os.mkdir('docs/usage')
for command, parser in choices.items():
print('generating help for %s' % command)
with open('docs/usage/%s.rst.inc' % command, 'w') as doc:
if command == 'help':
for topic in Archiver.helptext:
params = {"topic": topic,
"underline": '~' * len('borg help ' + topic)}
doc.write(".. _borg_{topic}:\n\n".format(**params))
doc.write("borg help {topic}\n{underline}\n::\n\n".format(**params))
doc.write(Archiver.helptext[topic])
else:
params = {"command": command,
"underline": '-' * len('borg ' + command)}
doc.write(".. _borg_{command}:\n\n".format(**params))
doc.write("borg {command}\n{underline}\n::\n\n".format(**params))
epilog = parser.epilog
parser.epilog = None
doc.write(re.sub("^", " ", parser.format_help(), flags=re.M))
doc.write("\nDescription\n~~~~~~~~~~~\n")
doc.write(epilog)
# return to regular Cython configuration, if we changed it
if os.environ.get('BORG_CYTHON_DISABLE') == self.__class__.__name__:
del os.environ['BORG_CYTHON_DISABLE']
class build_api(Command):
description = "generate a basic api.rst file based on the modules available"
user_options = [
('output=', 'O', 'output directory'),
]
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
print("auto-generating API documentation")
with open("docs/api.rst", "w") as doc:
doc.write("""
API Documentation
=================
""")
for mod in glob('borg/*.py') + glob('borg/*.pyx'):
print("examining module %s" % mod)
mod = mod.replace('.pyx', '').replace('.py', '').replace('/', '.')
if "._" not in mod:
doc.write("""
.. automodule:: %s
:members:
:undoc-members:
""" % mod)
# (function, predicate), see http://docs.python.org/2/distutils/apiref.html#distutils.cmd.Command.sub_commands
# seems like this doesn't work on RTD, see below for build_py hack.
build.sub_commands.append(('build_api', None))
build.sub_commands.append(('build_usage', None))
class build_py_custom(build_py):
"""override build_py to also build our stuff
it is unclear why this is necessary, but in some environments
(Readthedocs.org, specifically), the above
``build.sub_commands.append()`` doesn't seem to have an effect:
our custom build commands seem to be ignored when running
``setup.py install``.
This class overrides the ``build_py`` target by forcing it to run
our custom steps as well.
See also the `bug report on RTD
<https://github.com/rtfd/readthedocs.org/issues/1740>`_.
"""
def run(self):
super().run()
self.announce('calling custom build steps', level=log.INFO)
self.run_command('build_ext')
self.run_command('build_api')
self.run_command('build_usage')
cmdclass = {
'build_ext': build_ext,
'build_api': build_api,
'build_usage': build_usage,
'build_py': build_py_custom,
'sdist': Sdist
}
ext_modules = []
if not on_rtd:
ext_modules += [
Extension('borg.compress', [compress_source], libraries=['lz4'], include_dirs=include_dirs, library_dirs=library_dirs),
Extension('borg.crypto', [crypto_source], libraries=['crypto'], include_dirs=include_dirs, library_dirs=library_dirs),
Extension('borg.chunker', [chunker_source]),
Extension('borg.hashindex', [hashindex_source])
]
if sys.platform.startswith('linux'):
ext_modules.append(Extension('borg.platform_linux', [platform_linux_source], libraries=['acl']))
elif sys.platform.startswith('freebsd'):
ext_modules.append(Extension('borg.platform_freebsd', [platform_freebsd_source]))
elif sys.platform == 'darwin':
ext_modules.append(Extension('borg.platform_darwin', [platform_darwin_source]))
if sys.platform.startswith('linux'):
ext_modules.append(Extension('borg.platform_linux', [platform_linux_source], libraries=['acl']))
elif sys.platform.startswith('freebsd'):
ext_modules.append(Extension('borg.platform_freebsd', [platform_freebsd_source]))
elif sys.platform == 'darwin':
ext_modules.append(Extension('borg.platform_darwin', [platform_darwin_source]))
setup(
name='borgbackup',
@ -134,7 +262,7 @@ setup(
},
author='The Borg Collective (see AUTHORS file)',
author_email='borgbackup@librelist.com',
url='https://borgbackup.github.io/',
url='https://borgbackup.readthedocs.org/',
description='Deduplicated, encrypted, authenticated and compressed backups',
long_description=long_description,
license='BSD',

2
tox.ini
View file

@ -11,6 +11,6 @@ changedir = {toxworkdir}
deps =
-rrequirements.d/development.txt
attic
commands = py.test --cov=borg --pyargs {posargs:borg.testsuite}
commands = py.test --cov=borg --benchmark-skip --pyargs {posargs:borg.testsuite}
# fakeroot -u needs some env vars:
passenv = *