While writing my own out-of-band decoder, I had a hard time figuring out
how to unpack the manifest. From the description, I was only able to
read that the manifest is msgpack'd, but I had not been able to figure
out that it's also going through the same encryption+compression logic
as all other things do.
This should make it a little clearer and provide the necessary
information to understand how the compression works.
in the finished == true message, these are missing:
- message
- current / total
- info
This is to be somewhat consistent with #6683 by only providing a
minimal set of values for the finished case.
The finished messages is primarily intended for cleanup purposes,
e.g. clearing the progress display.
Backport from master: Make clear that absolute paths always go into the matcher as if they are relative (without leading slash). Adapt all examples accordingly.
The previous sample for creating a ~/.borg-passphrase file creates it first and then chmod's it to 400 permissions. That's probably fine in practice, but means there's a tiny window where the passphrase file is sitting with default permissions (likely world readable, depending on the system umask).
It seems safer to first change the umask to remove all group & world bits (0077) _before_ creating the file. To be polite and avoid messing with the user's previous umask, we do this in a subshell. (Note that umask 0077 leads to a mode of 600 rather than the previous 400, because removing the owner write bit doesn't seem to buy much since the owner can just chmod the file anyway.)
docs: clarify on-disk order and size of log entry fields
The order of the fields of a log entry on disk is CRC32 first, the docs had the size first.
I tried to make this list similar to the HashIndex struct description.
https://github.com/borgbackup/borg/pull/6188#discussion_r794752672
> Well, guess one could also use max(list of trusted nonce values).
>
> The real issue is if you have lost all or some of the trusted
> (client side) nonce values and you also have reason to not trust the
> server side nonce, because someone might attack you on the server.
docs: permissions note rewritten to make it less confusing
Original wording was confusing "Avoid to create a mixup of users and permissions in your repository (or cache)." is not clear, what should be avoided?
Also implement some feedback of @jdchristensen.
Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
It was unclear that the user _only_ needs to have borg installed on a remote system to use client/server mode. Hopefully this change makes it apparent that the user doesn't start anything on the remote system themselves.
the movement of the start of the hashing window stops at (file_size - window_size), thus THAT would be the factor in that formula, not just file_size.
for medium and big files, window_size is much smaller than file_size, so guess we can just say "approximately" for the general case.
While doing some doc updates I needed a way to test them - to build
the documentation and inspect the output. I ran into an issue:
running python setup.py build_man was throwing exceptions:
1. The import-tar parser had a None description causing:
File "/home/user/borg/setup_docs.py", line 451, in write_heading
write(char * len(header))
TypeError: object of type 'NoneType' has no len()
2. There was no docs/usage/import-tar.rst causing an exception too
add remote upload buffer (--remote-buffer)
- added new option --remote-buffer
- allow to_send to grow to selected size
- don't grow if wait is specified
- fill pipe on any command (including 'async_response')
- add new option to docs
- create EfficientBytesQueue to prevent recreation of buffer each time we send something
- add tests for EfficientBytesQueue
a file map can be:
- created internally inside chunkify by calling sparsemap, which uses
SEEK_DATA / SEEK_HOLE to determine data and hole ranges inside a
seekable sparse file.
Usage: borg create --sparse --chunker-params=fixed,BLOCKSIZE ...
BLOCKSIZE is the chunker blocksize here, not the filesystem blocksize!
- made by some other means and given to the chunkify function.
this is not used yet, but in future this could be used to only read
the changed parts and seek over the (known) unchanged parts of a file.
sparsemap: the generate range sizes are multiples of the fs block size.
the tests assume 4kiB fs block size.
Using "package" ansible module make the ansible playbook able to run
against most unix OSes. Pacman module only works with Arch and derivatives.
Also : changing state from "latest" to "present". Ansible should not be
a way to keep your system up-to-date : it's a configuration management
system and, as such, should not change anything if neither the playbook
nor the machine changed its state (idempotency).
Reference : https://github.com/ansible/ansible-lint/blob/master/lib/ansiblelint/rules/PackageIsNotLatestRule.py#L24
We know use only "target chunk size" when speaking of the chunk size
that is expected to happen most of the time. This removes statistical
and mathematical inacurracies that could be troublesome for mathematical
people.
Fixes#5336
rephrase some warnings, fixes#5164
borg check --repair and borg recreate are now present in the code since rather long, so they are not experimental any more.
borg recreate might be used wrongly (e.g. accidentally excluding everything / not matching anything when recreating an archive). added some warning words in the docs, but it will not ask for confirmation any more.
borg check: there might be kinds of corruption borg check --repair can not fix and it might make things even worse while trying to fix. so this will still ask for confirmation, just with different wording.
locking: fix ExclusiveLock race condition bug, fixes#4923
- ExclusiveLock is now based on os.rename instead of os.mkdir.
- catch FileNotFoundError observed under race condition in ExclusiveLock.release()
and .kill_stale_lock()
- added TestExclusiveLock.test_race_condition() which reveals issue #4923
- updated docs
- locking: use "raise LockTimeout from None" for prettier traceback
Co-authored-by: Thomas Portmann <thomas@portmann.org>
Co-authored-by: Thomas Waldmann <tw@waldmann-edv.de>
The umask value is NOT transmitted from client to server any more,
so the borg client can not influence the borg server umask any more.
If one wants to have a specific umask on the server side, one needs to
use a ssh forced command in .ssh/authorized_keys file.
OTOH, as the default value is 077 (in general, for client as well as for
the server) and the server does not take the value from the client any more,
there usually should be no need to give it on the server side, IF you are
happy with the default value.
Running 'borg key import' on a keyfile repository with the BORG_KEY_FILE
environment variable set works correctly if the BORG_KEY_FILE file
already exists. However, the command crashes if the BORG_KEY_FILE file
does not exist:
$ BORG_KEY_FILE=newborgkey borg key import /home/strager/borg-backups/straglum borgkey
Local Exception
Traceback (most recent call last):
[snip]
File "[snip]/borg/crypto/key.py", line 713, in sanity_check
with open(filename, 'rb') as fd:
FileNotFoundError: [Errno 2] No such file or directory: '[snip]/newborgkey'
Platform: Linux straglum 5.0.0-25-generic #26~18.04.1-Ubuntu SMP Thu Aug 1 13:51:02 UTC 2019 x86_64
Linux: debian buster/sid
Borg: 1.1.11 Python: CPython 3.7.7 msgpack: 0.5.6
PID: 15306 CWD: /home/strager/Projects/borg
sys.argv: ['[snip]/borg', 'key', 'import', '/home/strager/borg-backups/straglum', 'borgkey']
SSH_ORIGINAL_COMMAND: None
Make 'borg key import' not require the BORG_KEY_FILE file to already
exist.
This commit does not change the behavior of 'borg key import' without
BORG_KEY_FILE. This commit also does not change the behavior of 'borg
key import' on a repokey repository.
