python setup.py build_usage

This commit is contained in:
Thomas Waldmann 2022-04-14 21:09:23 +02:00
parent 801751e44e
commit d7fe23c515
10 changed files with 186 additions and 133 deletions

View file

@ -138,15 +138,18 @@ Second, the consistency and correctness of the archive metadata is verified:
The ``--max-duration`` option can be used to split a long-running repository check
into multiple partial checks. After the given number of seconds the check is
interrupted. The next partial check will continue where the previous one stopped,
until the complete repository has been checked. Example: Assuming a full check took 7
until the complete repository has been checked. Example: Assuming a complete check took 7
hours, then running a daily check with --max-duration=3600 (1 hour) resulted in one
full check per week.
completed check per week.
Attention: Partial checks can only do way less checking than a full check (only the
CRC32 checks on segment file entries are done), and cannot be combined with the
``--repair`` option. Partial checks may therefore be useful only with very large
repositories where a full check took too long. Doing a full repository check aborts a
partial check; the next partial check will restart from the beginning.
Attention: A partial --repository-only check can only do way less checking than a full
--repository-only check: only the non-cryptographic checksum checks on segment file
entries are done, while a full --repository-only check would also do a repo index check.
A partial check cannot be combined with the ``--repair`` option. Partial checks
may therefore be useful only with very large repositories where a full check would take
too long.
Doing a full repository check aborts a partial check; the next partial check will restart
from the beginning.
The ``--verify-data`` option will perform a full integrity verification (as opposed to
checking the CRC32 of the segment) of data, which means reading the data from the

View file

@ -71,6 +71,7 @@ When using ``--verbose``, borg will output an estimate of the freed space.
After upgrading borg (server) to 1.2+, you can use ``borg compact --cleanup-commits``
to clean up the numerous 17byte commit-only segments that borg 1.1 did not clean up
due to a bug. It is enough to do that once per repository.
due to a bug. It is enough to do that once per repository. After cleaning up the
commits, borg will also do a normal compaction.
See :ref:`separate_compaction` in Additional Notes for more details.

View file

@ -37,7 +37,7 @@ borg create
+-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--stdin-user USER`` | set user USER in archive for stdin data (default: 'root') |
+-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--stdin-group GROUP`` | set group GROUP in archive for stdin data (default: 'wheel') |
| | ``--stdin-group GROUP`` | set group GROUP in archive for stdin data (default: 'root') |
+-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--stdin-mode M`` | set mode to M in archive for stdin data (default: 0660) |
+-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
@ -139,7 +139,7 @@ borg create
--no-cache-sync experimental: do not synchronize the cache. Implies not using the files cache.
--stdin-name NAME use NAME in archive for stdin data (default: 'stdin')
--stdin-user USER set user USER in archive for stdin data (default: 'root')
--stdin-group GROUP set group GROUP in archive for stdin data (default: 'wheel')
--stdin-group GROUP set group GROUP in archive for stdin data (default: 'root')
--stdin-mode M set mode to M in archive for stdin data (default: 0660)
--content-from-command interpret PATH as command and store its stdout. See also section Reading from stdin below.
--paths-from-stdin read DELIM-separated list of paths to backup from stdin. Will not recurse into directories.

View file

@ -27,6 +27,8 @@ borg export-tar
+-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+
| | ``--list`` | output verbose list of items (files, dirs, ...) |
+-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+
| | ``--tar-format FMT`` | select tar format: BORG, PAX or GNU |
+-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+
| .. class:: borg-common-opt-ref |
| |
| :ref:`common_options` |
@ -63,8 +65,9 @@ borg export-tar
optional arguments
--tar-filter filter program to pipe data through
--list output verbose list of items (files, dirs, ...)
--tar-filter filter program to pipe data through
--list output verbose list of items (files, dirs, ...)
--tar-format FMT select tar format: BORG, PAX or GNU
:ref:`common_options`
@ -99,12 +102,18 @@ Alternatively, a ``--tar-filter`` program may be explicitly specified. It should
read the uncompressed tar stream from stdin and write a compressed/filtered
tar stream to stdout.
The generated tarball uses the GNU tar format.
Depending on the ``-tar-format`` option, these formats are created:
export-tar is a lossy conversion:
BSD flags, ACLs, extended attributes (xattrs), atime and ctime are not exported.
Timestamp resolution is limited to whole seconds, not the nanosecond resolution
otherwise supported by Borg.
+--------------+---------------------------+----------------------------+
| --tar-format | Specification | Metadata |
+--------------+---------------------------+----------------------------+
| BORG | BORG specific, like PAX | all as supported by borg |
+--------------+---------------------------+----------------------------+
| PAX | POSIX.1-2001 (pax) format | GNU + atime/ctime/mtime ns |
+--------------+---------------------------+----------------------------+
| GNU | GNU tar format | mtime s, no atime/ctime, |
| | | no ACLs/xattrs/bsdflags |
+--------------+---------------------------+----------------------------+
A ``--sparse`` option (as found in borg extract) is not supported.

View file

@ -249,11 +249,11 @@ and ``--remote-path`` values support these placeholders:
{now}
The current local date and time, by default in ISO-8601 format.
You can also supply your own `format string <https://docs.python.org/3.8/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {now:%Y-%m-%d_%H:%M:%S}
You can also supply your own `format string <https://docs.python.org/3.9/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {now:%Y-%m-%d_%H:%M:%S}
{utcnow}
The current UTC date and time, by default in ISO-8601 format.
You can also supply your own `format string <https://docs.python.org/3.8/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S}
You can also supply your own `format string <https://docs.python.org/3.9/library/datetime.html#strftime-and-strptime-behavior>`_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S}
{user}
The user name (or UID, if no name is available) of the user running borg.

View file

@ -106,12 +106,15 @@ stdout.
Most documentation of borg create applies. Note that this command does not
support excluding files.
import-tar is a lossy conversion:
BSD flags, ACLs, extended attributes (xattrs), atime and ctime are not exported.
Timestamp resolution is limited to whole seconds, not the nanosecond resolution
otherwise supported by Borg.
A ``--sparse`` option (as found in borg create) is not supported.
import-tar reads POSIX.1-1988 (ustar), POSIX.1-2001 (pax), GNU tar, UNIX V7 tar
and SunOS tar with extended attributes.
About tar formats and metadata conservation or loss, please see ``borg export-tar``.
import-tar reads these tar formats:
- BORG: borg specific (PAX-based)
- PAX: POSIX.1-2001
- GNU: GNU tar
- POSIX.1-1988 (ustar)
- UNIX V7 tar
- SunOS tar with extended attributes

View file

@ -27,6 +27,8 @@ borg init
+-------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--make-parent-dirs`` | create the parent directories of the repository directory, if they are missing. |
+-------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| | ``--key-algorithm`` | the algorithm we use to derive a key encryption key from your passphrase. Default: argon2 |
+-------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| .. class:: borg-common-opt-ref |
| |
| :ref:`common_options` |
@ -51,6 +53,7 @@ borg init
--append-only create an append-only mode repository. Note that this only affects the low level structure of the repository, and running `delete` or `prune` will still be allowed. See :ref:`append_only_mode` in Additional Notes for more details.
--storage-quota QUOTA Set storage quota of the new repository (e.g. 5G, 1.5T). Default: no quota.
--make-parent-dirs create the parent directories of the repository directory, if they are missing.
--key-algorithm the algorithm we use to derive a key encryption key from your passphrase. Default: argon2
:ref:`common_options`
@ -65,26 +68,24 @@ directory containing the deduplicated data from zero or more archives.
Encryption mode TLDR
++++++++++++++++++++
The encryption mode can only be configured when creating a new repository -
you can neither configure it on a per-archive basis nor change the
encryption mode of an existing repository.
The encryption mode can only be configured when creating a new repository - you can
neither configure it on a per-archive basis nor change the mode of an existing repository.
This example will likely NOT give optimum performance on your machine (performance
tips will come below):
Use ``repokey``::
::
borg init --encryption repokey /path/to/repo
Or ``repokey-blake2`` depending on which is faster on your client machines (see below)::
borg init --encryption repokey-blake2 /path/to/repo
Borg will:
1. Ask you to come up with a passphrase.
2. Create a borg key (which contains 3 random secrets. See :ref:`key_files`).
3. Encrypt the key with your passphrase.
4. Store the encrypted borg key inside the repository directory (in the repo config).
2. Create a borg key (which contains some random secrets. See :ref:`key_files`).
3. Derive a "key encryption key" from your passphrase
4. Encrypt and sign the key with the key encryption key
5. Store the encrypted borg key inside the repository directory (in the repo config).
This is why it is essential to use a secure passphrase.
5. Encrypt and sign your backups to prevent anyone from reading or forging them unless they
6. Encrypt and sign your backups to prevent anyone from reading or forging them unless they
have the key and know the passphrase. Make sure to keep a backup of
your key **outside** the repository - do not lock yourself out by
"leaving your keys inside your car" (see :ref:`borg_key_export`).
@ -92,7 +93,7 @@ Borg will:
never sees your passphrase, your unencrypted key or your unencrypted files.
Chunking and id generation are also based on your key to improve
your privacy.
6. Use the key when extracting files to decrypt them and to verify that the contents of
7. Use the key when extracting files to decrypt them and to verify that the contents of
the backups have not been accidentally or maliciously altered.
Picking a passphrase
@ -116,76 +117,69 @@ a different keyboard layout.
You can change your passphrase for existing repos at any time, it won't affect
the encryption/decryption key or other secrets.
More encryption modes
+++++++++++++++++++++
Choosing an encryption mode
+++++++++++++++++++++++++++
Only use ``--encryption none`` if you are OK with anyone who has access to
your repository being able to read your backups and tamper with their
contents without you noticing.
Depending on your hardware, hashing and crypto performance may vary widely.
The easiest way to find out about what's fastest is to run ``borg benchmark cpu``.
If you want "passphrase and having-the-key" security, use ``--encryption keyfile``.
The key will be stored in your home directory (in ``~/.config/borg/keys``).
`repokey` modes: if you want ease-of-use and "passphrase" security is good enough -
the key will be stored in the repository (in ``repo_dir/config``).
If you do **not** want to encrypt the contents of your backups, but still
want to detect malicious tampering use ``--encryption authenticated``.
`keyfile` modes: if you rather want "passphrase and having-the-key" security -
the key will be stored in your home directory (in ``~/.config/borg/keys``).
If ``BLAKE2b`` is faster than ``SHA-256`` on your hardware, use ``--encryption authenticated-blake2``,
``--encryption repokey-blake2`` or ``--encryption keyfile-blake2``. Note: for remote backups
the hashing is done on your local machine.
The following table is roughly sorted in order of preference, the better ones are
in the upper part of the table, in the lower part is the old and/or unsafe(r) stuff:
.. nanorst: inline-fill
+----------+---------------+------------------------+--------------------------+
| Hash/MAC | Not encrypted | Not encrypted, | Encrypted (AEAD w/ AES) |
| | no auth | but authenticated | and authenticated |
+----------+---------------+------------------------+--------------------------+
| SHA-256 | none | `authenticated` | repokey |
| | | | keyfile |
+----------+---------------+------------------------+--------------------------+
| BLAKE2b | n/a | `authenticated-blake2` | `repokey-blake2` |
| | | | `keyfile-blake2` |
+----------+---------------+------------------------+--------------------------+
+---------------------------------+---------------+---------------+------------------+-------+
|**mode (* = keyfile or repokey)**|**ID-Hash** |**Encryption** |**Authentication**|**V>=**|
+---------------------------------+---------------+---------------+------------------+-------+
| ``*-blake2-chacha20-poly1305`` | BLAKE2b | CHACHA20 | POLY1305 | 1.3 |
+---------------------------------+---------------+---------------+------------------+-------+
| ``*-chacha20-poly1305`` | HMAC-SHA-256 | CHACHA20 | POLY1305 | 1.3 |
+---------------------------------+---------------+---------------+------------------+-------+
| ``*-blake2-aes-ocb`` | BLAKE2b | AES256-OCB | AES256-OCB | 1.3 |
+---------------------------------+---------------+---------------+------------------+-------+
| ``*-aes-ocb`` | HMAC-SHA-256 | AES256-OCB | AES256-OCB | 1.3 |
+---------------------------------+---------------+---------------+------------------+-------+
| ``*-blake2`` | BLAKE2b | AES256-CTR | BLAKE2b | 1.1 |
+---------------------------------+---------------+---------------+------------------+-------+
| ``*`` | HMAC-SHA-256 | AES256-CTR | HMAC-SHA256 | any |
+---------------------------------+---------------+---------------+------------------+-------+
| authenticated-blake2 | BLAKE2b | none | BLAKE2b | 1.1 |
+---------------------------------+---------------+---------------+------------------+-------+
| authenticated | HMAC-SHA-256 | none | HMAC-SHA256 | 1.1 |
+---------------------------------+---------------+---------------+------------------+-------+
| none | SHA-256 | none | none | any |
+---------------------------------+---------------+---------------+------------------+-------+
.. nanorst: inline-replace
Modes `marked like this` in the above table are new in Borg 1.1 and are not
backwards-compatible with Borg 1.0.x.
`none` mode uses no encryption and no authentication. You're advised to NOT use this mode
as it would expose you to all sorts of issues (DoS, confidentiality, tampering, ...) in
case of malicious activity in the repository.
On modern Intel/AMD CPUs (except very cheap ones), AES is usually
hardware-accelerated.
BLAKE2b is faster than SHA256 on Intel/AMD 64-bit CPUs
(except AMD Ryzen and future CPUs with SHA extensions),
which makes `authenticated-blake2` faster than `none` and `authenticated`.
If you do **not** want to encrypt the contents of your backups, but still want to detect
malicious tampering use an `authenticated` mode. It's like `repokey` minus encryption.
On modern ARM CPUs, NEON provides hardware acceleration for SHA256 making it faster
than BLAKE2b-256 there. NEON accelerates AES as well.
Key derivation functions
++++++++++++++++++++++++
Hardware acceleration is always used automatically when available.
- ``--key-algorithm argon2`` is the default and is recommended.
The key encryption key is derived from your passphrase via argon2-id.
Argon2 is considered more modern and secure than pbkdf2.
`repokey` and `keyfile` use AES-CTR-256 for encryption and HMAC-SHA256 for
authentication in an encrypt-then-MAC (EtM) construction. The chunk ID hash
is HMAC-SHA256 as well (with a separate key).
These modes are compatible with Borg 1.0.x.
- You can use ``--key-algorithm pbkdf2`` if you want to access your repo via old versions of borg.
`repokey-blake2` and `keyfile-blake2` are also authenticated encryption modes,
but use BLAKE2b-256 instead of HMAC-SHA256 for authentication. The chunk ID
hash is a keyed BLAKE2b-256 hash.
These modes are new and *not* compatible with Borg 1.0.x.
Our implementation of argon2-based key algorithm follows the cryptographic best practices:
`authenticated` mode uses no encryption, but authenticates repository contents
through the same HMAC-SHA256 hash as the `repokey` and `keyfile` modes (it uses it
as the chunk ID hash). The key is stored like `repokey`.
This mode is new and *not* compatible with Borg 1.0.x.
- It derives two separate keys from your passphrase: one to encrypt your key and another one
to sign it. ``--key-algorithm pbkdf2`` uses the same key for both.
`authenticated-blake2` is like `authenticated`, but uses the keyed BLAKE2b-256 hash
from the other blake2 modes.
This mode is new and *not* compatible with Borg 1.0.x.
- It uses encrypt-then-mac instead of encrypt-and-mac used by ``--key-algorithm pbkdf2``
`none` mode uses no encryption and no authentication. It uses SHA256 as chunk
ID hash. This mode is not recommended, you should rather consider using an authenticated
or authenticated/encrypted mode. This mode has possible denial-of-service issues
when running ``borg create`` on contents controlled by an attacker.
Use it only for new repositories where no encryption is wanted **and** when compatibility
with 1.0.x is important. If compatibility with 1.0.x is not important, use
`authenticated-blake2` or `authenticated` instead.
This mode is compatible with Borg 1.0.x.
Neither is inherently linked to the key derivation function, but since we were going
to break backwards compatibility anyway we took the opportunity to fix all 3 issues at once.

View file

@ -0,0 +1,76 @@
.. IMPORTANT: this file is auto-generated from borg's built-in help, do not edit!
.. _borg_key_change-algorithm:
borg key change-algorithm
-------------------------
.. code-block:: none
borg [common options] key change-algorithm [options] [REPOSITORY] ALGORITHM
.. only:: html
.. class:: borg-options-table
+-------------------------------------------------------+----------------+----------------------+
| **positional arguments** |
+-------------------------------------------------------+----------------+----------------------+
| | ``REPOSITORY`` | |
+-------------------------------------------------------+----------------+----------------------+
| | ``ALGORITHM`` | select key algorithm |
+-------------------------------------------------------+----------------+----------------------+
| .. class:: borg-common-opt-ref |
| |
| :ref:`common_options` |
+-------------------------------------------------------+----------------+----------------------+
.. raw:: html
<script type='text/javascript'>
$(document).ready(function () {
$('.borg-options-table colgroup').remove();
})
</script>
.. only:: latex
REPOSITORY
ALGORITHM
select key algorithm
:ref:`common_options`
|
Description
~~~~~~~~~~~
Change the algorithm we use to encrypt and authenticate the borg key.
Important: In a `repokey` mode (e.g. repokey-blake2) all users share the same key.
In this mode upgrading to `argon2` will make it impossible to access the repo for users who use an old version of borg.
We recommend upgrading to the latest stable version.
Important: In a `keyfile` mode (e.g. keyfile-blake2) each user has their own key (in ``~/.config/borg/keys``).
In this mode this command will only change the key used by the current user.
If you want to upgrade to `argon2` to strengthen security, you will have to upgrade each user's key individually.
Your repository is encrypted and authenticated with a key that is randomly generated by ``borg init``.
The key is encrypted and authenticated with your passphrase.
We currently support two choices:
1. argon2 - recommended. This algorithm is used by default when initialising a new repository.
The key encryption key is derived from your passphrase via argon2-id.
Argon2 is considered more modern and secure than pbkdf2.
2. pbkdf2 - the legacy algorithm. Use this if you want to access your repo via old versions of borg.
The key encryption key is derived from your passphrase via PBKDF2-HMAC-SHA256.
Examples::
# Upgrade an existing key to argon2
borg key change-algorithm /path/to/repo argon2
# Downgrade to pbkdf2 - use this if upgrading borg is not an option
borg key change-algorithm /path/to/repo pbkdf2

View file

@ -113,7 +113,7 @@ The FORMAT specifier syntax
+++++++++++++++++++++++++++
The ``--format`` option uses python's `format string syntax
<https://docs.python.org/3.8/library/string.html#formatstrings>`_.
<https://docs.python.org/3.9/library/string.html#formatstrings>`_.
Examples:
::

View file

@ -99,47 +99,14 @@ See
https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability
for details.
Attic and Borg 0.xx to Borg 1.x
+++++++++++++++++++++++++++++++
Borg 0.xx to Borg 1.x
+++++++++++++++++++++
This currently supports converting an Attic repository to Borg and also
helps with converting Borg 0.xx to 1.0.
This currently supports converting Borg 0.xx to 1.0.
Currently, only LOCAL repositories can be upgraded (issue #465).
Please note that ``borg create`` (since 1.0.0) uses bigger chunks by
default than old borg or attic did, so the new chunks won't deduplicate
default than old borg did, so the new chunks won't deduplicate
with the old chunks in the upgraded repository.
See ``--chunker-params`` option of ``borg create`` and ``borg recreate``.
``borg upgrade`` will change the magic strings in the repository's
segments to match the new Borg magic strings. The keyfiles found in
$ATTIC_KEYS_DIR or ~/.attic/keys/ will also be converted and
copied to $BORG_KEYS_DIR or ~/.config/borg/keys.
The cache files are converted, from $ATTIC_CACHE_DIR or
~/.cache/attic to $BORG_CACHE_DIR or ~/.cache/borg, but the
cache layout between Borg and Attic changed, so it is possible
the first backup after the conversion takes longer than expected
due to the cache resync.
Upgrade should be able to resume if interrupted, although it
will still iterate over all segments. If you want to start
from scratch, use `borg delete` over the copied repository to
make sure the cache files are also removed::
borg delete borg
Unless ``--inplace`` is specified, the upgrade process first creates a backup
copy of the repository, in REPOSITORY.before-upgrade-DATETIME, using hardlinks.
This requires that the repository and its parent directory reside on same
filesystem so the hardlink copy can work.
This takes longer than in place upgrades, but is much safer and gives
progress information (as opposed to ``cp -al``). Once you are satisfied
with the conversion, you can safely destroy the backup copy.
WARNING: Running the upgrade in place will make the current
copy unusable with older version, with no way of going back
to previous versions. This can PERMANENTLY DAMAGE YOUR
REPOSITORY! Attic CAN NOT READ BORG REPOSITORIES, as the
magic strings have changed. You have been warned.
See ``--chunker-params`` option of ``borg create`` and ``borg recreate``.