From 70602bda521ad904e3d3044e5805488f8de02797 Mon Sep 17 00:00:00 2001 From: Thomas Waldmann Date: Sun, 11 Jun 2023 23:09:20 +0200 Subject: [PATCH] build_usage --- docs/usage/common-options.rst.inc | 1 + docs/usage/create.rst.inc | 30 +++---- docs/usage/delete.rst.inc | 3 - docs/usage/diff.rst.inc | 140 ++++++++++++++++++++---------- docs/usage/export-tar.rst.inc | 4 +- docs/usage/extract.rst.inc | 7 +- docs/usage/help.rst.inc | 91 ++++++++++++------- docs/usage/import-tar.rst.inc | 7 ++ docs/usage/list.rst.inc | 45 +++++----- docs/usage/mount.rst.inc | 16 ++-- docs/usage/prune.rst.inc | 18 +++- docs/usage/recreate.rst.inc | 4 +- docs/usage/rlist.rst.inc | 12 ++- docs/usage/serve.rst.inc | 12 ++- docs/usage/transfer.rst.inc | 86 +++++++++++------- 15 files changed, 308 insertions(+), 168 deletions(-) diff --git a/docs/usage/common-options.rst.inc b/docs/usage/common-options.rst.inc index 77f4728f8..793aedd84 100644 --- a/docs/usage/common-options.rst.inc +++ b/docs/usage/common-options.rst.inc @@ -18,4 +18,5 @@ --upload-buffer UPLOAD_BUFFER set network upload buffer size in MiB. (default: 0=no buffer) --debug-profile FILE Write execution profile in Borg format into FILE. For local use a Python-compatible file can be generated by suffixing FILE with ".pyprof". --rsh RSH Use this command to connect to the 'borg serve' process (default: 'ssh') +--socket PATH Use UNIX DOMAIN (IPC) socket at PATH for client/server communication with socket: protocol. -r REPO, --repo REPO repository to use diff --git a/docs/usage/create.rst.inc b/docs/usage/create.rst.inc index 0e60cc7e0..92f8259ae 100644 --- a/docs/usage/create.rst.inc +++ b/docs/usage/create.rst.inc @@ -43,17 +43,17 @@ borg create +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--content-from-command`` | interpret PATH as command and store its stdout. See also section Reading from stdin below. | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--paths-from-stdin`` | read DELIM-separated list of paths to back up from stdin. Will not recurse into directories. | + | | ``--paths-from-stdin`` | read DELIM-separated list of paths to back up from stdin. All control is external: it will back up all files given - no more, no less. | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--paths-from-command`` | interpret PATH as command and treat its output as ``--paths-from-stdin`` | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``--paths-delimiter DELIM`` | set path delimiter for ``--paths-from-stdin`` and ``--paths-from-command`` (default: \n) | + | | ``--paths-delimiter DELIM`` | set path delimiter for ``--paths-from-stdin`` and ``--paths-from-command`` (default: ``\n``) | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | .. class:: borg-common-opt-ref | | | | :ref:`common_options` | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | **Exclusion options** | + | **Include/Exclude options** | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -73,7 +73,7 @@ borg create +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | **Filesystem options** | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | | ``-x``, ``--one-file-system`` | stay in the same file system and do not store mount points of other file systems. This might behave different from your expectations, see the docs. | + | | ``-x``, ``--one-file-system`` | stay in the same file system and do not store mount points of other file systems - this might behave different from your expectations, see the description below. | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--numeric-ids`` | only store numeric user and group identifiers | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -138,15 +138,15 @@ borg create --stdin-group GROUP set group GROUP in archive for stdin data (default: do not store group/gid) --stdin-mode M set mode to M in archive for stdin data (default: 0660) --content-from-command interpret PATH as command and store its stdout. See also section Reading from stdin below. - --paths-from-stdin read DELIM-separated list of paths to back up from stdin. Will not recurse into directories. + --paths-from-stdin read DELIM-separated list of paths to back up from stdin. All control is external: it will back up all files given - no more, no less. --paths-from-command interpret PATH as command and treat its output as ``--paths-from-stdin`` - --paths-delimiter DELIM set path delimiter for ``--paths-from-stdin`` and ``--paths-from-command`` (default: \n) + --paths-delimiter DELIM set path delimiter for ``--paths-from-stdin`` and ``--paths-from-command`` (default: ``\n``) :ref:`common_options` | - Exclusion options + Include/Exclude options -e PATTERN, --exclude PATTERN exclude paths matching PATTERN --exclude-from EXCLUDEFILE read exclude patterns from EXCLUDEFILE, one per line --pattern PATTERN include/exclude paths matching PATTERN @@ -158,7 +158,7 @@ borg create Filesystem options - -x, --one-file-system stay in the same file system and do not store mount points of other file systems. This might behave different from your expectations, see the docs. + -x, --one-file-system stay in the same file system and do not store mount points of other file systems - this might behave different from your expectations, see the description below. --numeric-ids only store numeric user and group identifiers --atime do store atime into archive --noctime do not store ctime into archive @@ -276,13 +276,13 @@ through using the ``--keep-exclude-tags`` option. The ``-x`` or ``--one-file-system`` option excludes directories, that are mountpoints (and everything in them). It detects mountpoints by comparing the device number from the output of ``stat()`` of the directory and its parent directory. Specifically, it excludes directories for which ``stat()`` reports a device number different -from the device number of their parent. Be aware that in Linux (and possibly elsewhere) there are directories -with device number different from their parent, which the kernel does not consider a mountpoint and also the -other way around. Examples are bind mounts (possibly same device number, but always a mountpoint) and ALL -subvolumes of a btrfs (different device number from parent but not necessarily a mountpoint). Therefore when -using ``--one-file-system``, one should make doubly sure that the backup works as intended especially when using -btrfs. This is even more important, if the btrfs layout was created by someone else, e.g. a distribution -installer. +from the device number of their parent. +In general: be aware that there are directories with device number different from their parent, which the kernel +does not consider a mountpoint and also the other way around. +Linux examples for this are bind mounts (possibly same device number, but always a mountpoint) and ALL +subvolumes of a btrfs (different device number from parent but not necessarily a mountpoint). +macOS examples are the apfs mounts of a typical macOS installation. +Therefore, when using ``--one-file-system``, you should double-check that the backup works as intended. .. _list_item_flags: diff --git a/docs/usage/delete.rst.inc b/docs/usage/delete.rst.inc index 3eed03e5d..ab7bc7c85 100644 --- a/docs/usage/delete.rst.inc +++ b/docs/usage/delete.rst.inc @@ -23,8 +23,6 @@ borg delete +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ | | ``-s``, ``--stats`` | print statistics for the deleted archive | +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``--cache-only`` | delete only the local cache for the given repository | - +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ | | ``--force`` | force deletion of corrupted archives, use ``--force --force`` in case ``--force`` does not work. | +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ | | ``-c SECONDS``, ``--checkpoint-interval SECONDS`` | write checkpoint every SECONDS seconds (Default: 1800) | @@ -69,7 +67,6 @@ borg delete --list output verbose list of archives --consider-checkpoints consider checkpoint archives for deletion (default: not considered). -s, --stats print statistics for the deleted archive - --cache-only delete only the local cache for the given repository --force force deletion of corrupted archives, use ``--force --force`` in case ``--force`` does not work. -c SECONDS, --checkpoint-interval SECONDS write checkpoint every SECONDS seconds (Default: 1800) diff --git a/docs/usage/diff.rst.inc b/docs/usage/diff.rst.inc index b3bd6797c..b7f166b96 100644 --- a/docs/usage/diff.rst.inc +++ b/docs/usage/diff.rst.inc @@ -12,39 +12,43 @@ borg diff .. class:: borg-options-table - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | **positional arguments** | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``ARCHIVE1`` | ARCHIVE1 name | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``ARCHIVE2`` | ARCHIVE2 name | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``PATH`` | paths of items inside the archives to compare; patterns are supported | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | **options** | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``--numeric-ids`` | only consider numeric user and group identifiers | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``--same-chunker-params`` | Override check of chunker parameters. | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``--sort`` | Sort the output lines by file path. | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``--json-lines`` | Format output as JSON Lines. | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | .. class:: borg-common-opt-ref | - | | - | :ref:`common_options` | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | **Exclusion options** | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``--exclude-from EXCLUDEFILE`` | read exclude patterns from EXCLUDEFILE, one per line | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``--pattern PATTERN`` | include/exclude paths matching PATTERN | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ - | | ``--patterns-from PATTERNFILE`` | read include/exclude patterns from PATTERNFILE, one per line | - +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------+ + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | **positional arguments** | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``ARCHIVE1`` | ARCHIVE1 name | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``ARCHIVE2`` | ARCHIVE2 name | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``PATH`` | paths of items inside the archives to compare; patterns are supported | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | **options** | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``--numeric-ids`` | only consider numeric user and group identifiers | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``--same-chunker-params`` | Override check of chunker parameters. | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``--sort`` | Sort the output lines by file path. | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``--format FORMAT`` | specify format for differences between archives (default: "{change} {path}{NL}") | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``--json-lines`` | Format output as JSON Lines. | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``--content-only`` | Only compare differences in content (exclude metadata differences) | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | .. class:: borg-common-opt-ref | + | | + | :ref:`common_options` | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | **Include/Exclude options** | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``--exclude-from EXCLUDEFILE`` | read exclude patterns from EXCLUDEFILE, one per line | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``--pattern PATTERN`` | include/exclude paths matching PATTERN | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ + | | ``--patterns-from PATTERNFILE`` | read include/exclude patterns from PATTERNFILE, one per line | + +-------------------------------------------------------+---------------------------------------+----------------------------------------------------------------------------------+ .. raw:: html @@ -68,13 +72,15 @@ borg diff --numeric-ids only consider numeric user and group identifiers --same-chunker-params Override check of chunker parameters. --sort Sort the output lines by file path. + --format FORMAT specify format for differences between archives (default: "{change} {path}{NL}") --json-lines Format output as JSON Lines. + --content-only Only compare differences in content (exclude metadata differences) :ref:`common_options` | - Exclusion options + Include/Exclude options -e PATTERN, --exclude PATTERN exclude paths matching PATTERN --exclude-from EXCLUDEFILE read exclude patterns from EXCLUDEFILE, one per line --pattern PATTERN include/exclude paths matching PATTERN @@ -84,19 +90,61 @@ borg diff Description ~~~~~~~~~~~ -This command finds differences (file contents, user/group/mode) between archives. +This command finds differences (file contents, metadata) between ARCHIVE1 and ARCHIVE2. -A repository location and an archive name must be specified for REPO::ARCHIVE1. -ARCHIVE2 is just another archive name in same repository (no repository location -allowed). +For more help on include/exclude patterns, see the :ref:`borg_patterns` command output. -For archives created with Borg 1.1 or newer diff automatically detects whether -the archives are created with the same chunker params. If so, only chunk IDs -are compared, which is very fast. +.. man NOTES -For archives prior to Borg 1.1 chunk contents are compared by default. -If you did not create the archives with different chunker params, -pass ``--same-chunker-params``. -Note that the chunker params changed from Borg 0.xx to 1.0. +The FORMAT specifier syntax ++++++++++++++++++++++++++++ -For more help on include/exclude patterns, see the :ref:`borg_patterns` command output. \ No newline at end of file +The ``--format`` option uses python's `format string syntax +`_. + +Examples: +:: + + $ borg diff --format '{content:30} {path}{NL}' ArchiveFoo ArchiveBar + modified: +4.1 kB -1.0 kB file-diff + ... + + # {VAR:NUMBER} - pad to NUMBER columns right-aligned. + $ borg diff --format '{content:>30} {path}{NL}' ArchiveFoo ArchiveBar + modified: +4.1 kB -1.0 kB file-diff + ... + +The following keys are always available: + +- NEWLINE: OS dependent line separator +- NL: alias of NEWLINE +- NUL: NUL character for creating print0 / xargs -0 like output +- SPACE: space character +- TAB: tab character +- CR: carriage return character +- LF: line feed character + + +Keys available only when showing differences between archives: + +- path: archived file path +- change: all available changes + +- content: file content change +- mode: file mode change +- type: file type change +- owner: file owner (user/group) change +- group: file group change +- user: file user change + +- link: file link change +- directory: file directory change +- blkdev: file block device change +- chrdev: file character device change +- fifo: file fifo change + +- mtime: file modification time change +- ctime: file change time change +- isomtime: file modification time change (ISO 8601) +- isoctime: file creation time change (ISO 8601) diff --git a/docs/usage/export-tar.rst.inc b/docs/usage/export-tar.rst.inc index 1da3bc419..848d9d9c0 100644 --- a/docs/usage/export-tar.rst.inc +++ b/docs/usage/export-tar.rst.inc @@ -33,7 +33,7 @@ borg export-tar | | | :ref:`common_options` | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ - | **Exclusion options** | + | **Include/Exclude options** | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ @@ -73,7 +73,7 @@ borg export-tar :ref:`common_options` | - Exclusion options + Include/Exclude options -e PATTERN, --exclude PATTERN exclude paths matching PATTERN --exclude-from EXCLUDEFILE read exclude patterns from EXCLUDEFILE, one per line --pattern PATTERN include/exclude paths matching PATTERN diff --git a/docs/usage/extract.rst.inc b/docs/usage/extract.rst.inc index bb0eb0352..b99dd03dd 100644 --- a/docs/usage/extract.rst.inc +++ b/docs/usage/extract.rst.inc @@ -37,11 +37,13 @@ borg extract +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ | | ``--sparse`` | create holes in output sparse file from all-zero chunks | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ + | | ``--continue`` | continue a previously interrupted extraction of same archive | + +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ | .. class:: borg-common-opt-ref | | | | :ref:`common_options` | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ - | **Exclusion options** | + | **Include/Exclude options** | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN | +-------------------------------------------------------+---------------------------------------+-----------------------------------------------------------------------------------------------------------+ @@ -79,12 +81,13 @@ borg extract --noxattrs do not extract/set xattrs --stdout write all extracted data to stdout --sparse create holes in output sparse file from all-zero chunks + --continue continue a previously interrupted extraction of same archive :ref:`common_options` | - Exclusion options + Include/Exclude options -e PATTERN, --exclude PATTERN exclude paths matching PATTERN --exclude-from EXCLUDEFILE read exclude patterns from EXCLUDEFILE, one per line --pattern PATTERN include/exclude paths matching PATTERN diff --git a/docs/usage/help.rst.inc b/docs/usage/help.rst.inc index ef4df4760..1fee15d4e 100644 --- a/docs/usage/help.rst.inc +++ b/docs/usage/help.rst.inc @@ -24,15 +24,13 @@ confuse the root with the ``PATH`` argument of e.g. `borg extract`. Starting with Borg 1.2, paths that are matched against patterns always appear relative. If you give ``/absolute/`` as root, the paths going -into the matcher will look relative like ``absolute/.../file.ext``. -If you give ``../some/path`` as root, the paths will look like -``some/path/.../file.ext``. +into the matcher will start with ``absolute/``. +If you give ``../../relative`` as root, the paths will be normalized +as ``relative/``. -File patterns support five different styles. If followed by a colon ':', -the first two characters of a pattern are used as a style selector. -Explicit style selection is necessary if a non-default style is desired -or when the desired pattern starts with two alphanumeric characters -followed by a colon (i.e. ``aa:something/*``). +Borg supports different pattern styles. To define a non-default +style for a specific pattern, prefix it with two characters followed +by a colon ':' (i.e. ``fm:path/*``, ``sh:path/**``). `Fnmatch `_, selector ``fm:`` This is the default style for ``--exclude`` and ``--exclude-from``. @@ -55,7 +53,8 @@ Shell-style patterns, selector ``sh:`` Like fnmatch patterns these are similar to shell patterns. The difference is that the pattern may include ``**/`` for matching zero or more directory levels, ``*`` for matching zero or more arbitrary characters with the - exception of any path separator. A leading path separator is always removed. + exception of any path separator, ``{}`` containing comma-separated + alternative patterns. A leading path separator is always removed. `Regular expressions `_, selector ``re:`` Unlike shell patterns, regular expressions are not required to match the full @@ -175,6 +174,25 @@ before an exclude pattern, the file is backed up. Note that a no-recurse exclude stops examination of subdirectories so that potential includes will not match - use normal excludes for such use cases. +Example:: + + # Define the recursion root + R / + # Exclude all iso files in any directory + - **/*.iso + # Explicitly include all inside etc and root + + etc/** + + root/** + # Exclude a specific directory under each user's home directories + - home/*/.cache + # Explicitly include everything in /home + + home/** + # Explicitly exclude some directories without recursing into them + ! re:^(dev|proc|run|sys|tmp) + # Exclude all other files and directories + # that are not specifically included earlier. + - ** + **Tip: You can easily test your patterns with --dry-run and --list**:: $ borg create --dry-run --list --patterns-from patterns.txt archive @@ -393,29 +411,44 @@ auto,C[,L] obfuscate,SPEC,C[,L] Use compressed-size obfuscation to make fingerprinting attacks based on - the observable stored chunk size more difficult. - Note: - - you must combine this with encryption or it won't make any sense. - - your repo size will be bigger, of course. + the observable stored chunk size more difficult. Note: - The SPEC value will determine how the size obfuscation will work: + - You must combine this with encryption, or it won't make any sense. + - Your repo size will be bigger, of course. + - A chunk is limited by the constant ``MAX_DATA_SIZE`` (cur. ~20MiB). + + The SPEC value determines how the size obfuscation works: + + *Relative random reciprocal size variation* (multiplicative) - Relative random reciprocal size variation: Size will increase by a factor, relative to the compressed data size. - Smaller factors are often used, larger factors rarely. - 1: factor 0.01 .. 100.0 - 2: factor 0.1 .. 1000.0 - 3: factor 1.0 .. 10000.0 - 4: factor 10.0 .. 100000.0 - 5: factor 100.0 .. 1000000.0 - 6: factor 1000.0 .. 10000000.0 + Smaller factors are used often, larger factors rarely. - Add a randomly sized padding up to the given size: - 110: 1kiB - ... - 120: 1MiB - ... - 123: 8MiB (max.) + Available factors:: + + 1: 0.01 .. 100 + 2: 0.1 .. 1,000 + 3: 1 .. 10,000 + 4: 10 .. 100,000 + 5: 100 .. 1,000,000 + 6: 1,000 .. 10,000,000 + + Example probabilities for SPEC ``1``:: + + 90 % 0.01 .. 0.1 + 9 % 0.1 .. 1 + 0.9 % 1 .. 10 + 0.09% 10 .. 100 + + *Randomly sized padding up to the given size* (additive) + + :: + + 110: 1kiB (2 ^ (SPEC - 100)) + ... + 120: 1MiB + ... + 123: 8MiB (max.) Examples:: @@ -426,7 +459,7 @@ Examples:: borg create --compression zlib,1 REPO::ARCHIVE data borg create --compression auto,lzma,6 REPO::ARCHIVE data borg create --compression auto,lzma ... - borg create --compression obfuscate,3,none ... + borg create --compression obfuscate,110,none ... borg create --compression obfuscate,3,auto,zstd,10 ... borg create --compression obfuscate,2,zstd,6 ... diff --git a/docs/usage/import-tar.rst.inc b/docs/usage/import-tar.rst.inc index 0cb7d631b..24ce2fb0c 100644 --- a/docs/usage/import-tar.rst.inc +++ b/docs/usage/import-tar.rst.inc @@ -31,6 +31,8 @@ borg import-tar +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``--json`` | output stats as JSON (implies --stats) | +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--ignore-zeros`` | ignore zero-filled blocks in the input tarball | + +-------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | .. class:: borg-common-opt-ref | | | | :ref:`common_options` | @@ -72,6 +74,7 @@ borg import-tar --list output verbose list of items (files, dirs, ...) --filter STATUSCHARS only display items with the given status characters --json output stats as JSON (implies --stats) + --ignore-zeros ignore zero-filled blocks in the input tarball :ref:`common_options` @@ -121,3 +124,7 @@ import-tar reads these tar formats: - POSIX.1-1988 (ustar) - UNIX V7 tar - SunOS tar with extended attributes + +To import multiple tarballs into a single archive, they can be simply +concatenated (e.g. using "cat") into a single file, and imported with an +``--ignore-zeros`` option to skip through the stop markers between them. \ No newline at end of file diff --git a/docs/usage/list.rst.inc b/docs/usage/list.rst.inc index 756c81b64..ab50a2e06 100644 --- a/docs/usage/list.rst.inc +++ b/docs/usage/list.rst.inc @@ -31,7 +31,7 @@ borg list | | | :ref:`common_options` | +-------------------------------------------------------+---------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | **Exclusion options** | + | **Include/Exclude options** | +-------------------------------------------------------+---------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN | +-------------------------------------------------------+---------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -67,7 +67,7 @@ borg list :ref:`common_options` | - Exclusion options + Include/Exclude options -e PATTERN, --exclude PATTERN exclude paths matching PATTERN --exclude-from EXCLUDEFILE read exclude patterns from EXCLUDEFILE, one per line --pattern PATTERN include/exclude paths matching PATTERN @@ -107,35 +107,36 @@ The following keys are always available: - NEWLINE: OS dependent line separator - NL: alias of NEWLINE - NUL: NUL character for creating print0 / xargs -0 like output -- SPACE -- TAB -- CR -- LF +- SPACE: space character +- TAB: tab character +- CR: carriage return character +- LF: line feed character + Keys available only when listing files in an archive: -- type -- mode -- uid -- gid -- user -- group +- type: file type (file, dir, symlink, ...) +- mode: file mode (as in stat) +- uid: user id of file owner +- gid: group id of file owner +- user: user name of file owner +- group: group name of file owner - path: file path - target: link target for symlinks - hlid: hard link identity (same if hardlinking same fs object) -- flags +- flags: file flags -- size +- size: file size - dsize: deduplicated size - num_chunks: number of chunks in this file - unique_chunks: number of unique chunks in this file -- mtime -- ctime -- atime -- isomtime -- isoctime -- isoatime +- mtime: file modification time +- ctime: file change time +- atime: file access time +- isomtime: file modification time (ISO 8601 format) +- isoctime: file change time (ISO 8601 format) +- isoatime: file access time (ISO 8601 format) - blake2b - blake2s @@ -151,8 +152,8 @@ Keys available only when listing files in an archive: - sha512 - xxh64: XXH64 checksum of this file (note: this is NOT a cryptographic hash!) -- archiveid -- archivename +- archiveid: internal ID of the archive +- archivename: name of the archive - extra: prepends {target} with " -> " for soft links and " link to " for hard links - health: either "healthy" (file ok) or "broken" (if file has all-zero replacement chunks) diff --git a/docs/usage/mount.rst.inc b/docs/usage/mount.rst.inc index 9a321d43b..90498607c 100644 --- a/docs/usage/mount.rst.inc +++ b/docs/usage/mount.rst.inc @@ -51,7 +51,7 @@ borg mount +-----------------------------------------------------------------------------+----------------------------------------------+-----------------------------------------------------------------------------------------------------------+ | | ``--newer TIMESPAN`` | consider archives newer than (now - TIMESPAN), e.g. 7d or 12m. | +-----------------------------------------------------------------------------+----------------------------------------------+-----------------------------------------------------------------------------------------------------------+ - | **Exclusion options** | + | **Include/Exclude options** | +-----------------------------------------------------------------------------+----------------------------------------------+-----------------------------------------------------------------------------------------------------------+ | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN | +-----------------------------------------------------------------------------+----------------------------------------------+-----------------------------------------------------------------------------------------------------------+ @@ -101,7 +101,7 @@ borg mount --newer TIMESPAN consider archives newer than (now - TIMESPAN), e.g. 7d or 12m. - Exclusion options + Include/Exclude options -e PATTERN, --exclude PATTERN exclude paths matching PATTERN --exclude-from EXCLUDEFILE read exclude patterns from EXCLUDEFILE, one per line --pattern PATTERN include/exclude paths matching PATTERN @@ -148,14 +148,14 @@ manually. Unlike the ``uid`` and ``gid`` mount options which affect all files, Additional mount options supported by borg: -- versions: when used with a repository mount, this gives a merged, versioned +- ``versions``: when used with a repository mount, this gives a merged, versioned view of the files in the archives. EXPERIMENTAL, layout may change in future. -- allow_damaged_files: by default damaged files (where missing chunks were - replaced with runs of zeros by borg check ``--repair``) are not readable and +- ``allow_damaged_files``: by default damaged files (where missing chunks were + replaced with runs of zeros by ``borg check --repair``) are not readable and return EIO (I/O error). Set this option to read such files. -- ignore_permissions: for security reasons the "default_permissions" mount - option is internally enforced by borg. "ignore_permissions" can be given to - not enforce "default_permissions". +- ``ignore_permissions``: for security reasons the ``default_permissions`` mount + option is internally enforced by borg. ``ignore_permissions`` can be given to + not enforce ``default_permissions``. The BORG_MOUNT_DATA_CACHE_ENTRIES environment variable is meant for advanced users to tweak the performance. It sets the number of cached data chunks; additional diff --git a/docs/usage/prune.rst.inc b/docs/usage/prune.rst.inc index 01a462ace..c95788269 100644 --- a/docs/usage/prune.rst.inc +++ b/docs/usage/prune.rst.inc @@ -23,6 +23,14 @@ borg prune +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ | | ``--list`` | output verbose list of archives it keeps/prunes | +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ + | | ``--short`` | use a less wide archive part format | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ + | | ``--list-pruned`` | output verbose list of archives it prunes | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ + | | ``--list-kept`` | output verbose list of archives it keeps | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ + | | ``--format FORMAT`` | specify format for the archive part (default: "{archive:<36} {time} [{id}]") | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ | | ``--keep-within INTERVAL`` | keep all archives within this time interval | +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------+ | | ``--keep-last``, ``--keep-secondly`` | number of secondly archives to keep | @@ -75,6 +83,10 @@ borg prune --force force pruning of corrupted archives, use ``--force --force`` in case ``--force`` does not work. -s, --stats print statistics for the deleted archive --list output verbose list of archives it keeps/prunes + --short use a less wide archive part format + --list-pruned output verbose list of archives it prunes + --list-kept output verbose list of archives it keeps + --format FORMAT specify format for the archive part (default: "{archive:<36} {time} [{id}]") --keep-within INTERVAL keep all archives within this time interval --keep-last, --keep-secondly number of secondly archives to keep --keep-minutely number of minutely archives to keep @@ -153,4 +165,8 @@ backup archive in the same second). When using ``--stats``, you will get some statistics about how much data was deleted - the "Deleted data" deduplicated size there is most interesting as that is how much your repository will shrink. -Please note that the "All archives" stats refer to the state after pruning. \ No newline at end of file +Please note that the "All archives" stats refer to the state after pruning. + +You can influence how the ``--list`` output is formatted by using the ``--short`` +option (less wide output) or by giving a custom format using ``--format`` (see +the ``borg rlist`` description for more details about the format string). \ No newline at end of file diff --git a/docs/usage/recreate.rst.inc b/docs/usage/recreate.rst.inc index 1953c4d2a..ca28e1c18 100644 --- a/docs/usage/recreate.rst.inc +++ b/docs/usage/recreate.rst.inc @@ -31,7 +31,7 @@ borg recreate | | | :ref:`common_options` | +-----------------------------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ - | **Exclusion options** | + | **Include/Exclude options** | +-----------------------------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | | ``-e PATTERN``, ``--exclude PATTERN`` | exclude paths matching PATTERN | +-----------------------------------------------------------------------------+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ @@ -106,7 +106,7 @@ borg recreate :ref:`common_options` | - Exclusion options + Include/Exclude options -e PATTERN, --exclude PATTERN exclude paths matching PATTERN --exclude-from EXCLUDEFILE read exclude patterns from EXCLUDEFILE, one per line --pattern PATTERN include/exclude paths matching PATTERN diff --git a/docs/usage/rlist.rst.inc b/docs/usage/rlist.rst.inc index 72fa78184..f4fc67ab5 100644 --- a/docs/usage/rlist.rst.inc +++ b/docs/usage/rlist.rst.inc @@ -112,10 +112,11 @@ The following keys are always available: - NEWLINE: OS dependent line separator - NL: alias of NEWLINE - NUL: NUL character for creating print0 / xargs -0 like output -- SPACE -- TAB -- CR -- LF +- SPACE: space character +- TAB: tab character +- CR: carriage return character +- LF: line feed character + Keys available only when listing archives in a repository: @@ -131,3 +132,6 @@ Keys available only when listing archives in a repository: - hostname: hostname of host on which this archive was created - username: username of user who created this archive + +- size: size of this archive (data plus metadata, not considering compression and deduplication) +- nfiles: count of files in this archive diff --git a/docs/usage/serve.rst.inc b/docs/usage/serve.rst.inc index c4a709bfc..87fd13f05 100644 --- a/docs/usage/serve.rst.inc +++ b/docs/usage/serve.rst.inc @@ -53,4 +53,14 @@ borg serve Description ~~~~~~~~~~~ -This command starts a repository server process. This command is usually not used manually. \ No newline at end of file +This command starts a repository server process. + +borg serve can currently support: + +- Getting automatically started via ssh when the borg client uses a ssh://... + remote repository. In this mode, `borg serve` will live until that ssh connection + gets terminated. + +- Getting started by some other means (not by the borg client) as a long-running socket + server to be used for borg clients using a socket://... repository (see the `--socket` + option if you do not want to use the default path for the socket and pid file). \ No newline at end of file diff --git a/docs/usage/transfer.rst.inc b/docs/usage/transfer.rst.inc index d9df5e548..fabf9ef14 100644 --- a/docs/usage/transfer.rst.inc +++ b/docs/usage/transfer.rst.inc @@ -12,37 +12,41 @@ borg transfer .. class:: borg-options-table - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | **options** | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``-n``, ``--dry-run`` | do not change repository, just check | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``--other-repo SRC_REPOSITORY`` | transfer archives from the other repository | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``--upgrader UPGRADER`` | use the upgrader to convert transferred data (default: no conversion) | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | .. class:: borg-common-opt-ref | - | | - | :ref:`common_options` | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | **Archive filters** — Archive filters can be applied to repository targets. | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``-a PATTERN``, ``--match-archives PATTERN`` | only consider archive names matching the pattern. see "borg help match-archives". | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``--sort-by KEYS`` | Comma-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``--first N`` | consider first N archives after other filters were applied | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``--last N`` | consider last N archives after other filters were applied | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``--oldest TIMESPAN`` | consider archives between the oldest archive's timestamp and (oldest + TIMESPAN), e.g. 7d or 12m. | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``--newest TIMESPAN`` | consider archives between the newest archive's timestamp and (newest - TIMESPAN), e.g. 7d or 12m. | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``--older TIMESPAN`` | consider archives older than (now - TIMESPAN), e.g. 7d oder 12m. | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ - | | ``--newer TIMESPAN`` | consider archives newer than (now - TIMESPAN), e.g. 7d or 12m. | - +-----------------------------------------------------------------------------+----------------------------------------------+---------------------------------------------------------------------------------------------------+ + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | **options** | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``-n``, ``--dry-run`` | do not change repository, just check | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--other-repo SRC_REPOSITORY`` | transfer archives from the other repository | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--upgrader UPGRADER`` | use the upgrader to convert transferred data (default: no conversion) | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``-C COMPRESSION``, ``--compression COMPRESSION`` | select compression algorithm, see the output of the "borg help compression" command for details. | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--recompress MODE`` | recompress data chunks according to `MODE` and ``--compression``. Possible modes are `always`: recompress unconditionally; and `never`: do not recompress (faster: re-uses compressed data chunks w/o change).If no MODE is given, `always` will be used. Not passing --recompress is equivalent to "--recompress never". | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | .. class:: borg-common-opt-ref | + | | + | :ref:`common_options` | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | **Archive filters** — Archive filters can be applied to repository targets. | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``-a PATTERN``, ``--match-archives PATTERN`` | only consider archive names matching the pattern. see "borg help match-archives". | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--sort-by KEYS`` | Comma-separated list of sorting keys; valid keys are: timestamp, name, id; default is: timestamp | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--first N`` | consider first N archives after other filters were applied | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--last N`` | consider last N archives after other filters were applied | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--oldest TIMESPAN`` | consider archives between the oldest archive's timestamp and (oldest + TIMESPAN), e.g. 7d or 12m. | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--newest TIMESPAN`` | consider archives between the newest archive's timestamp and (newest - TIMESPAN), e.g. 7d or 12m. | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--older TIMESPAN`` | consider archives older than (now - TIMESPAN), e.g. 7d oder 12m. | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ + | | ``--newer TIMESPAN`` | consider archives newer than (now - TIMESPAN), e.g. 7d or 12m. | + +-----------------------------------------------------------------------------+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ .. raw:: html @@ -60,6 +64,8 @@ borg transfer -n, --dry-run do not change repository, just check --other-repo SRC_REPOSITORY transfer archives from the other repository --upgrader UPGRADER use the upgrader to convert transferred data (default: no conversion) + -C COMPRESSION, --compression COMPRESSION select compression algorithm, see the output of the "borg help compression" command for details. + --recompress MODE recompress data chunks according to `MODE` and ``--compression``. Possible modes are `always`: recompress unconditionally; and `never`: do not recompress (faster: re-uses compressed data chunks w/o change).If no MODE is given, `always` will be used. Not passing --recompress is equivalent to "--recompress never". :ref:`common_options` @@ -81,6 +87,14 @@ Description This command transfers archives from one repository to another repository. Optionally, it can also upgrade the transferred data. +Optionally, it can also recompress the transferred data. + +It is easiest (and fastest) to give ``--compression=COMPRESSION --recompress=never`` using +the same COMPRESSION mode as in the SRC_REPO - borg will use that COMPRESSION for metadata (in +any case) and keep data compressed "as is" (saves time as no data compression is needed). + +If you want to globally change compression while transferring archives to the DST_REPO, +give ``--compress=WANTED_COMPRESSION --recompress=always``. Suggested use for general purpose archive transfer (not repo upgrades):: @@ -96,11 +110,17 @@ Suggested use for general purpose archive transfer (not repo upgrades):: The default is to transfer all archives, including checkpoint archives. You could use the misc. archive filter options to limit which archives it will -transfer, e.g. using the -a option. This is recommended for big +transfer, e.g. using the ``-a`` option. This is recommended for big repositories with multiple data sets to keep the runtime per invocation lower. For repository upgrades (e.g. from a borg 1.2 repo to a related borg 2.0 repo), usage is quite similar to the above:: - borg --repo=DST_REPO transfer --other-repo=SRC_REPO --upgrader=From12To20 + # fast: compress metadata with zstd,3, but keep data chunks compressed as they are: + borg --repo=DST_REPO transfer --other-repo=SRC_REPO --upgrader=From12To20 \ + --compress=zstd,3 --recompress=never + + # compress metadata and recompress data with zstd,3 + borg --repo=DST_REPO transfer --other-repo=SRC_REPO --upgrader=From12To20 \ + --compress=zstd,3 --recompress=always