diff --git a/docs/usage/check.rst.inc b/docs/usage/check.rst.inc index e4e530a2e..0f4ff7afb 100644 --- a/docs/usage/check.rst.inc +++ b/docs/usage/check.rst.inc @@ -138,15 +138,18 @@ Second, the consistency and correctness of the archive metadata is verified: The ``--max-duration`` option can be used to split a long-running repository check into multiple partial checks. After the given number of seconds the check is interrupted. The next partial check will continue where the previous one stopped, -until the complete repository has been checked. Example: Assuming a full check took 7 +until the complete repository has been checked. Example: Assuming a complete check took 7 hours, then running a daily check with --max-duration=3600 (1 hour) resulted in one -full check per week. +completed check per week. -Attention: Partial checks can only do way less checking than a full check (only the -CRC32 checks on segment file entries are done), and cannot be combined with the -``--repair`` option. Partial checks may therefore be useful only with very large -repositories where a full check took too long. Doing a full repository check aborts a -partial check; the next partial check will restart from the beginning. +Attention: A partial --repository-only check can only do way less checking than a full +--repository-only check: only the non-cryptographic checksum checks on segment file +entries are done, while a full --repository-only check would also do a repo index check. +A partial check cannot be combined with the ``--repair`` option. Partial checks +may therefore be useful only with very large repositories where a full check would take +too long. +Doing a full repository check aborts a partial check; the next partial check will restart +from the beginning. The ``--verify-data`` option will perform a full integrity verification (as opposed to checking the CRC32 of the segment) of data, which means reading the data from the diff --git a/docs/usage/compact.rst.inc b/docs/usage/compact.rst.inc index 3d67fe5d3..8187bbf7f 100644 --- a/docs/usage/compact.rst.inc +++ b/docs/usage/compact.rst.inc @@ -71,6 +71,7 @@ When using ``--verbose``, borg will output an estimate of the freed space. After upgrading borg (server) to 1.2+, you can use ``borg compact --cleanup-commits`` to clean up the numerous 17byte commit-only segments that borg 1.1 did not clean up -due to a bug. It is enough to do that once per repository. +due to a bug. It is enough to do that once per repository. After cleaning up the +commits, borg will also do a normal compaction. See :ref:`separate_compaction` in Additional Notes for more details. \ No newline at end of file diff --git a/docs/usage/create.rst.inc b/docs/usage/create.rst.inc index cb377b2ac..b61fb9794 100644 --- a/docs/usage/create.rst.inc +++ b/docs/usage/create.rst.inc @@ -37,7 +37,7 @@ borg create +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--stdin-user USER`` | set user USER in archive for stdin data (default: 'root') | +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--stdin-group GROUP`` | set group GROUP in archive for stdin data (default: 'wheel') | + | | ``--stdin-group GROUP`` | set group GROUP in archive for stdin data (default: 'root') | +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--stdin-mode M`` | set mode to M in archive for stdin data (default: 0660) | +-------------------------------------------------------+---------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -139,7 +139,7 @@ borg create --no-cache-sync experimental: do not synchronize the cache. Implies not using the files cache. --stdin-name NAME use NAME in archive for stdin data (default: 'stdin') --stdin-user USER set user USER in archive for stdin data (default: 'root') - --stdin-group GROUP set group GROUP in archive for stdin data (default: 'wheel') + --stdin-group GROUP set group GROUP in archive for stdin data (default: 'root') --stdin-mode M set mode to M in archive for stdin data (default: 0660) --content-from-command interpret PATH as command and store its stdout. See also section Reading from stdin below. --paths-from-stdin read DELIM-separated list of paths to backup from stdin. Will not recurse into directories. diff --git a/docs/usage/export-tar.rst.inc b/docs/usage/export-tar.rst.inc index c02a04ee6..ea7a20b8c 100644 --- a/docs/usage/export-tar.rst.inc +++ b/docs/usage/export-tar.rst.inc @@ -27,6 +27,8 @@ borg export-tar +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ | | ``--list`` | output verbose list of items (files, dirs, ...) | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ + | | ``--tar-format FMT`` | select tar format: BORG, PAX or GNU | + +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ | .. class:: borg-common-opt-ref | | | | :ref:`common_options` | @@ -63,8 +65,9 @@ borg export-tar optional arguments - --tar-filter filter program to pipe data through - --list output verbose list of items (files, dirs, ...) + --tar-filter filter program to pipe data through + --list output verbose list of items (files, dirs, ...) + --tar-format FMT select tar format: BORG, PAX or GNU :ref:`common_options` @@ -99,12 +102,18 @@ Alternatively, a ``--tar-filter`` program may be explicitly specified. It should read the uncompressed tar stream from stdin and write a compressed/filtered tar stream to stdout. -The generated tarball uses the GNU tar format. +Depending on the ``-tar-format`` option, these formats are created: -export-tar is a lossy conversion: -BSD flags, ACLs, extended attributes (xattrs), atime and ctime are not exported. -Timestamp resolution is limited to whole seconds, not the nanosecond resolution -otherwise supported by Borg. ++--------------+---------------------------+----------------------------+ +| --tar-format | Specification | Metadata | ++--------------+---------------------------+----------------------------+ +| BORG | BORG specific, like PAX | all as supported by borg | ++--------------+---------------------------+----------------------------+ +| PAX | POSIX.1-2001 (pax) format | GNU + atime/ctime/mtime ns | ++--------------+---------------------------+----------------------------+ +| GNU | GNU tar format | mtime s, no atime/ctime, | +| | | no ACLs/xattrs/bsdflags | ++--------------+---------------------------+----------------------------+ A ``--sparse`` option (as found in borg extract) is not supported. diff --git a/docs/usage/help.rst.inc b/docs/usage/help.rst.inc index 39dcca80f..7f9b4e9e0 100644 --- a/docs/usage/help.rst.inc +++ b/docs/usage/help.rst.inc @@ -249,11 +249,11 @@ and ``--remote-path`` values support these placeholders: {now} The current local date and time, by default in ISO-8601 format. - You can also supply your own `format string `_, e.g. {now:%Y-%m-%d_%H:%M:%S} + You can also supply your own `format string `_, e.g. {now:%Y-%m-%d_%H:%M:%S} {utcnow} The current UTC date and time, by default in ISO-8601 format. - You can also supply your own `format string `_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S} + You can also supply your own `format string `_, e.g. {utcnow:%Y-%m-%d_%H:%M:%S} {user} The user name (or UID, if no name is available) of the user running borg. diff --git a/docs/usage/import-tar.rst.inc b/docs/usage/import-tar.rst.inc index b0b5822a7..c76bbfe20 100644 --- a/docs/usage/import-tar.rst.inc +++ b/docs/usage/import-tar.rst.inc @@ -106,12 +106,15 @@ stdout. Most documentation of borg create applies. Note that this command does not support excluding files. -import-tar is a lossy conversion: -BSD flags, ACLs, extended attributes (xattrs), atime and ctime are not exported. -Timestamp resolution is limited to whole seconds, not the nanosecond resolution -otherwise supported by Borg. - A ``--sparse`` option (as found in borg create) is not supported. -import-tar reads POSIX.1-1988 (ustar), POSIX.1-2001 (pax), GNU tar, UNIX V7 tar -and SunOS tar with extended attributes. \ No newline at end of file +About tar formats and metadata conservation or loss, please see ``borg export-tar``. + +import-tar reads these tar formats: + +- BORG: borg specific (PAX-based) +- PAX: POSIX.1-2001 +- GNU: GNU tar +- POSIX.1-1988 (ustar) +- UNIX V7 tar +- SunOS tar with extended attributes diff --git a/docs/usage/init.rst.inc b/docs/usage/init.rst.inc index d3d3f301d..5bf1cb647 100644 --- a/docs/usage/init.rst.inc +++ b/docs/usage/init.rst.inc @@ -27,6 +27,8 @@ borg init +-------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--make-parent-dirs`` | create the parent directories of the repository directory, if they are missing. | +-------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--key-algorithm`` | the algorithm we use to derive a key encryption key from your passphrase. Default: argon2 | + +-------------------------------------------------------+------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | .. class:: borg-common-opt-ref | | | | :ref:`common_options` | @@ -51,6 +53,7 @@ borg init --append-only create an append-only mode repository. Note that this only affects the low level structure of the repository, and running `delete` or `prune` will still be allowed. See :ref:`append_only_mode` in Additional Notes for more details. --storage-quota QUOTA Set storage quota of the new repository (e.g. 5G, 1.5T). Default: no quota. --make-parent-dirs create the parent directories of the repository directory, if they are missing. + --key-algorithm the algorithm we use to derive a key encryption key from your passphrase. Default: argon2 :ref:`common_options` @@ -65,26 +68,24 @@ directory containing the deduplicated data from zero or more archives. Encryption mode TLDR ++++++++++++++++++++ -The encryption mode can only be configured when creating a new repository - -you can neither configure it on a per-archive basis nor change the -encryption mode of an existing repository. +The encryption mode can only be configured when creating a new repository - you can +neither configure it on a per-archive basis nor change the mode of an existing repository. +This example will likely NOT give optimum performance on your machine (performance +tips will come below): -Use ``repokey``:: +:: borg init --encryption repokey /path/to/repo -Or ``repokey-blake2`` depending on which is faster on your client machines (see below):: - - borg init --encryption repokey-blake2 /path/to/repo - Borg will: 1. Ask you to come up with a passphrase. -2. Create a borg key (which contains 3 random secrets. See :ref:`key_files`). -3. Encrypt the key with your passphrase. -4. Store the encrypted borg key inside the repository directory (in the repo config). +2. Create a borg key (which contains some random secrets. See :ref:`key_files`). +3. Derive a "key encryption key" from your passphrase +4. Encrypt and sign the key with the key encryption key +5. Store the encrypted borg key inside the repository directory (in the repo config). This is why it is essential to use a secure passphrase. -5. Encrypt and sign your backups to prevent anyone from reading or forging them unless they +6. Encrypt and sign your backups to prevent anyone from reading or forging them unless they have the key and know the passphrase. Make sure to keep a backup of your key **outside** the repository - do not lock yourself out by "leaving your keys inside your car" (see :ref:`borg_key_export`). @@ -92,7 +93,7 @@ Borg will: never sees your passphrase, your unencrypted key or your unencrypted files. Chunking and id generation are also based on your key to improve your privacy. -6. Use the key when extracting files to decrypt them and to verify that the contents of +7. Use the key when extracting files to decrypt them and to verify that the contents of the backups have not been accidentally or maliciously altered. Picking a passphrase @@ -116,76 +117,69 @@ a different keyboard layout. You can change your passphrase for existing repos at any time, it won't affect the encryption/decryption key or other secrets. -More encryption modes -+++++++++++++++++++++ +Choosing an encryption mode ++++++++++++++++++++++++++++ -Only use ``--encryption none`` if you are OK with anyone who has access to -your repository being able to read your backups and tamper with their -contents without you noticing. +Depending on your hardware, hashing and crypto performance may vary widely. +The easiest way to find out about what's fastest is to run ``borg benchmark cpu``. -If you want "passphrase and having-the-key" security, use ``--encryption keyfile``. -The key will be stored in your home directory (in ``~/.config/borg/keys``). +`repokey` modes: if you want ease-of-use and "passphrase" security is good enough - +the key will be stored in the repository (in ``repo_dir/config``). -If you do **not** want to encrypt the contents of your backups, but still -want to detect malicious tampering use ``--encryption authenticated``. +`keyfile` modes: if you rather want "passphrase and having-the-key" security - +the key will be stored in your home directory (in ``~/.config/borg/keys``). -If ``BLAKE2b`` is faster than ``SHA-256`` on your hardware, use ``--encryption authenticated-blake2``, -``--encryption repokey-blake2`` or ``--encryption keyfile-blake2``. Note: for remote backups -the hashing is done on your local machine. +The following table is roughly sorted in order of preference, the better ones are +in the upper part of the table, in the lower part is the old and/or unsafe(r) stuff: .. nanorst: inline-fill -+----------+---------------+------------------------+--------------------------+ -| Hash/MAC | Not encrypted | Not encrypted, | Encrypted (AEAD w/ AES) | -| | no auth | but authenticated | and authenticated | -+----------+---------------+------------------------+--------------------------+ -| SHA-256 | none | `authenticated` | repokey | -| | | | keyfile | -+----------+---------------+------------------------+--------------------------+ -| BLAKE2b | n/a | `authenticated-blake2` | `repokey-blake2` | -| | | | `keyfile-blake2` | -+----------+---------------+------------------------+--------------------------+ ++---------------------------------+---------------+---------------+------------------+-------+ +|**mode (* = keyfile or repokey)**|**ID-Hash** |**Encryption** |**Authentication**|**V>=**| ++---------------------------------+---------------+---------------+------------------+-------+ +| ``*-blake2-chacha20-poly1305`` | BLAKE2b | CHACHA20 | POLY1305 | 1.3 | ++---------------------------------+---------------+---------------+------------------+-------+ +| ``*-chacha20-poly1305`` | HMAC-SHA-256 | CHACHA20 | POLY1305 | 1.3 | ++---------------------------------+---------------+---------------+------------------+-------+ +| ``*-blake2-aes-ocb`` | BLAKE2b | AES256-OCB | AES256-OCB | 1.3 | ++---------------------------------+---------------+---------------+------------------+-------+ +| ``*-aes-ocb`` | HMAC-SHA-256 | AES256-OCB | AES256-OCB | 1.3 | ++---------------------------------+---------------+---------------+------------------+-------+ +| ``*-blake2`` | BLAKE2b | AES256-CTR | BLAKE2b | 1.1 | ++---------------------------------+---------------+---------------+------------------+-------+ +| ``*`` | HMAC-SHA-256 | AES256-CTR | HMAC-SHA256 | any | ++---------------------------------+---------------+---------------+------------------+-------+ +| authenticated-blake2 | BLAKE2b | none | BLAKE2b | 1.1 | ++---------------------------------+---------------+---------------+------------------+-------+ +| authenticated | HMAC-SHA-256 | none | HMAC-SHA256 | 1.1 | ++---------------------------------+---------------+---------------+------------------+-------+ +| none | SHA-256 | none | none | any | ++---------------------------------+---------------+---------------+------------------+-------+ .. nanorst: inline-replace -Modes `marked like this` in the above table are new in Borg 1.1 and are not -backwards-compatible with Borg 1.0.x. +`none` mode uses no encryption and no authentication. You're advised to NOT use this mode +as it would expose you to all sorts of issues (DoS, confidentiality, tampering, ...) in +case of malicious activity in the repository. -On modern Intel/AMD CPUs (except very cheap ones), AES is usually -hardware-accelerated. -BLAKE2b is faster than SHA256 on Intel/AMD 64-bit CPUs -(except AMD Ryzen and future CPUs with SHA extensions), -which makes `authenticated-blake2` faster than `none` and `authenticated`. +If you do **not** want to encrypt the contents of your backups, but still want to detect +malicious tampering use an `authenticated` mode. It's like `repokey` minus encryption. -On modern ARM CPUs, NEON provides hardware acceleration for SHA256 making it faster -than BLAKE2b-256 there. NEON accelerates AES as well. +Key derivation functions +++++++++++++++++++++++++ -Hardware acceleration is always used automatically when available. +- ``--key-algorithm argon2`` is the default and is recommended. + The key encryption key is derived from your passphrase via argon2-id. + Argon2 is considered more modern and secure than pbkdf2. -`repokey` and `keyfile` use AES-CTR-256 for encryption and HMAC-SHA256 for -authentication in an encrypt-then-MAC (EtM) construction. The chunk ID hash -is HMAC-SHA256 as well (with a separate key). -These modes are compatible with Borg 1.0.x. +- You can use ``--key-algorithm pbkdf2`` if you want to access your repo via old versions of borg. -`repokey-blake2` and `keyfile-blake2` are also authenticated encryption modes, -but use BLAKE2b-256 instead of HMAC-SHA256 for authentication. The chunk ID -hash is a keyed BLAKE2b-256 hash. -These modes are new and *not* compatible with Borg 1.0.x. +Our implementation of argon2-based key algorithm follows the cryptographic best practices: -`authenticated` mode uses no encryption, but authenticates repository contents -through the same HMAC-SHA256 hash as the `repokey` and `keyfile` modes (it uses it -as the chunk ID hash). The key is stored like `repokey`. -This mode is new and *not* compatible with Borg 1.0.x. +- It derives two separate keys from your passphrase: one to encrypt your key and another one + to sign it. ``--key-algorithm pbkdf2`` uses the same key for both. -`authenticated-blake2` is like `authenticated`, but uses the keyed BLAKE2b-256 hash -from the other blake2 modes. -This mode is new and *not* compatible with Borg 1.0.x. +- It uses encrypt-then-mac instead of encrypt-and-mac used by ``--key-algorithm pbkdf2`` -`none` mode uses no encryption and no authentication. It uses SHA256 as chunk -ID hash. This mode is not recommended, you should rather consider using an authenticated -or authenticated/encrypted mode. This mode has possible denial-of-service issues -when running ``borg create`` on contents controlled by an attacker. -Use it only for new repositories where no encryption is wanted **and** when compatibility -with 1.0.x is important. If compatibility with 1.0.x is not important, use -`authenticated-blake2` or `authenticated` instead. -This mode is compatible with Borg 1.0.x. \ No newline at end of file +Neither is inherently linked to the key derivation function, but since we were going +to break backwards compatibility anyway we took the opportunity to fix all 3 issues at once. \ No newline at end of file diff --git a/docs/usage/key_change-algorithm.rst.inc b/docs/usage/key_change-algorithm.rst.inc new file mode 100644 index 000000000..27b2e4e05 --- /dev/null +++ b/docs/usage/key_change-algorithm.rst.inc @@ -0,0 +1,76 @@ +.. IMPORTANT: this file is auto-generated from borg's built-in help, do not edit! + +.. _borg_key_change-algorithm: + +borg key change-algorithm +------------------------- +.. code-block:: none + + borg [common options] key change-algorithm [options] [REPOSITORY] ALGORITHM + +.. only:: html + + .. class:: borg-options-table + + +-------------------------------------------------------+----------------+----------------------+ + | **positional arguments** | + +-------------------------------------------------------+----------------+----------------------+ + | | ``REPOSITORY`` | | + +-------------------------------------------------------+----------------+----------------------+ + | | ``ALGORITHM`` | select key algorithm | + +-------------------------------------------------------+----------------+----------------------+ + | .. class:: borg-common-opt-ref | + | | + | :ref:`common_options` | + +-------------------------------------------------------+----------------+----------------------+ + + .. raw:: html + + + +.. only:: latex + + REPOSITORY + + ALGORITHM + select key algorithm + + + :ref:`common_options` + | + +Description +~~~~~~~~~~~ + +Change the algorithm we use to encrypt and authenticate the borg key. + +Important: In a `repokey` mode (e.g. repokey-blake2) all users share the same key. +In this mode upgrading to `argon2` will make it impossible to access the repo for users who use an old version of borg. +We recommend upgrading to the latest stable version. + +Important: In a `keyfile` mode (e.g. keyfile-blake2) each user has their own key (in ``~/.config/borg/keys``). +In this mode this command will only change the key used by the current user. +If you want to upgrade to `argon2` to strengthen security, you will have to upgrade each user's key individually. + +Your repository is encrypted and authenticated with a key that is randomly generated by ``borg init``. +The key is encrypted and authenticated with your passphrase. + +We currently support two choices: + +1. argon2 - recommended. This algorithm is used by default when initialising a new repository. + The key encryption key is derived from your passphrase via argon2-id. + Argon2 is considered more modern and secure than pbkdf2. +2. pbkdf2 - the legacy algorithm. Use this if you want to access your repo via old versions of borg. + The key encryption key is derived from your passphrase via PBKDF2-HMAC-SHA256. + +Examples:: + + # Upgrade an existing key to argon2 + borg key change-algorithm /path/to/repo argon2 + # Downgrade to pbkdf2 - use this if upgrading borg is not an option + borg key change-algorithm /path/to/repo pbkdf2 + diff --git a/docs/usage/list.rst.inc b/docs/usage/list.rst.inc index 2c1bdc14e..7b42cf812 100644 --- a/docs/usage/list.rst.inc +++ b/docs/usage/list.rst.inc @@ -113,7 +113,7 @@ The FORMAT specifier syntax +++++++++++++++++++++++++++ The ``--format`` option uses python's `format string syntax -`_. +`_. Examples: :: diff --git a/docs/usage/upgrade.rst.inc b/docs/usage/upgrade.rst.inc index 9f94d2cf8..c07b73093 100644 --- a/docs/usage/upgrade.rst.inc +++ b/docs/usage/upgrade.rst.inc @@ -99,47 +99,14 @@ See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details. -Attic and Borg 0.xx to Borg 1.x -+++++++++++++++++++++++++++++++ +Borg 0.xx to Borg 1.x ++++++++++++++++++++++ -This currently supports converting an Attic repository to Borg and also -helps with converting Borg 0.xx to 1.0. +This currently supports converting Borg 0.xx to 1.0. Currently, only LOCAL repositories can be upgraded (issue #465). Please note that ``borg create`` (since 1.0.0) uses bigger chunks by -default than old borg or attic did, so the new chunks won't deduplicate +default than old borg did, so the new chunks won't deduplicate with the old chunks in the upgraded repository. -See ``--chunker-params`` option of ``borg create`` and ``borg recreate``. - -``borg upgrade`` will change the magic strings in the repository's -segments to match the new Borg magic strings. The keyfiles found in -$ATTIC_KEYS_DIR or ~/.attic/keys/ will also be converted and -copied to $BORG_KEYS_DIR or ~/.config/borg/keys. - -The cache files are converted, from $ATTIC_CACHE_DIR or -~/.cache/attic to $BORG_CACHE_DIR or ~/.cache/borg, but the -cache layout between Borg and Attic changed, so it is possible -the first backup after the conversion takes longer than expected -due to the cache resync. - -Upgrade should be able to resume if interrupted, although it -will still iterate over all segments. If you want to start -from scratch, use `borg delete` over the copied repository to -make sure the cache files are also removed:: - - borg delete borg - -Unless ``--inplace`` is specified, the upgrade process first creates a backup -copy of the repository, in REPOSITORY.before-upgrade-DATETIME, using hardlinks. -This requires that the repository and its parent directory reside on same -filesystem so the hardlink copy can work. -This takes longer than in place upgrades, but is much safer and gives -progress information (as opposed to ``cp -al``). Once you are satisfied -with the conversion, you can safely destroy the backup copy. - -WARNING: Running the upgrade in place will make the current -copy unusable with older version, with no way of going back -to previous versions. This can PERMANENTLY DAMAGE YOUR -REPOSITORY! Attic CAN NOT READ BORG REPOSITORIES, as the -magic strings have changed. You have been warned. \ No newline at end of file +See ``--chunker-params`` option of ``borg create`` and ``borg recreate``. \ No newline at end of file