mirror of
https://github.com/borgbackup/borg.git
synced 2026-05-28 04:03:21 -04:00
enkore
GitHub
commit
82455a0863
15 changed files with 779 additions and 148 deletions
248
docs/changes.rst
248
docs/changes.rst
|
|
@ -1,7 +1,62 @@
|
|||
Important notes
|
||||
===============
|
||||
|
||||
This section is used for infos about e.g. security and corruption issues.
|
||||
This section is used for infos about security and corruption issues.
|
||||
|
||||
.. _tam_vuln:
|
||||
|
||||
Pre-1.0.9 manifest spoofing vulnerability
|
||||
-----------------------------------------
|
||||
|
||||
A flaw in the cryptographic authentication scheme in Borg allowed an attacker
|
||||
to spoof the manifest. The attack requires an attacker to be able to
|
||||
|
||||
1. insert files (with no additional headers) into backups
|
||||
2. gain write access to the repository
|
||||
|
||||
This vulnerability does not disclose plaintext to the attacker, nor does it
|
||||
affect the authenticity of existing archives.
|
||||
|
||||
The vulnerability allows an attacker to create a spoofed manifest (the list of archives).
|
||||
Creating plausible fake archives may be feasible for small archives, but is unlikely
|
||||
for large archives.
|
||||
|
||||
The fix adds a separate authentication tag to the manifest. For compatibility
|
||||
with prior versions this authentication tag is *not* required by default
|
||||
for existing repositories. Repositories created with 1.0.9 and later require it.
|
||||
|
||||
Steps you should take:
|
||||
|
||||
1. Upgrade all clients to 1.0.9 or later.
|
||||
2. Run ``borg upgrade --tam <repository>`` *on every client* for *each* repository.
|
||||
3. This will list all archives, including archive IDs, for easy comparison with your logs.
|
||||
4. Done.
|
||||
|
||||
Prior versions can access and modify repositories with this measure enabled, however,
|
||||
to 1.0.9 or later their modifications are indiscernible from an attack and will
|
||||
raise an error until the below procedure is followed. We are aware that this can
|
||||
be be annoying in some circumstances, but don't see a way to fix the vulnerability
|
||||
otherwise.
|
||||
|
||||
In case a version prior to 1.0.9 is used to modify a repository where above procedure
|
||||
was completed, and now you get an error message from other clients:
|
||||
|
||||
1. ``borg upgrade --tam --force <repository>`` once with *any* client suffices.
|
||||
|
||||
This attack is mitigated by:
|
||||
|
||||
- Noting/logging ``borg list``, ``borg info``, or ``borg create --stats``, which
|
||||
contain the archive IDs.
|
||||
|
||||
We are not aware of others having discovered, disclosed or exploited this vulnerability.
|
||||
|
||||
Vulnerability time line:
|
||||
|
||||
* 2016-11-14: Vulnerability and fix discovered during review of cryptography by Marian Beermann (@enkore)
|
||||
* 2016-11-20: First patch
|
||||
* 2016-12-18: Released fixed versions: 1.0.9, 1.1.0b3
|
||||
|
||||
.. _attic013_check_corruption:
|
||||
|
||||
Pre-1.0.9 potential data loss
|
||||
-----------------------------
|
||||
|
|
@ -71,98 +126,6 @@ The best check that everything is ok is to run a dry-run extraction::
|
|||
Changelog
|
||||
=========
|
||||
|
||||
Version 1.0.9 (not released yet)
|
||||
--------------------------------
|
||||
|
||||
Bug fixes:
|
||||
|
||||
- borg check:
|
||||
|
||||
- rebuild manifest if it's corrupted
|
||||
- skip corrupted chunks during manifest rebuild
|
||||
- fix TypeError in integrity error handler, #1903, #1894
|
||||
- fix location parser for archives with @ char (regression introduced in 1.0.8), #1930
|
||||
- fix wrong duration/timestamps if system clock jumped during a create
|
||||
- fix progress display not updating if system clock jumps backwards
|
||||
- fix checkpoint interval being incorrect if system clock jumps
|
||||
|
||||
Other changes:
|
||||
|
||||
- docs:
|
||||
|
||||
- add python3-devel as a dependency for cygwin-based installation
|
||||
- clarify extract is relative to current directory
|
||||
- FAQ: fix link to changelog
|
||||
- markup fixes
|
||||
- tests:
|
||||
|
||||
- test_get_(cache|keys)_dir: clean env state, #1897
|
||||
- get back pytest's pretty assertion failures, #1938
|
||||
- setup.py build_usage:
|
||||
|
||||
- fixed build_usage not processing all commands
|
||||
- fixed build_usage not generating includes for debug commands
|
||||
|
||||
|
||||
Version 1.0.9rc1 (2016-11-27)
|
||||
-----------------------------
|
||||
|
||||
Bug fixes:
|
||||
|
||||
- files cache: fix determination of newest mtime in backup set (which is
|
||||
used in cache cleanup and led to wrong "A" [added] status for unchanged
|
||||
files in next backup), #1860.
|
||||
|
||||
- borg check:
|
||||
|
||||
- fix incorrectly reporting attic 0.13 and earlier archives as corrupt
|
||||
- handle repo w/o objects gracefully and also bail out early if repo is
|
||||
*completely* empty, #1815.
|
||||
- fix tox/pybuild in 1.0-maint
|
||||
- at xattr module import time, loggers are not initialized yet
|
||||
|
||||
New features:
|
||||
|
||||
- borg umount <mountpoint>
|
||||
exposed already existing umount code via the CLI api, so users can use it,
|
||||
which is more consistent than using borg to mount and fusermount -u (or
|
||||
umount) to un-mount, #1855.
|
||||
- implement borg create --noatime --noctime, fixes #1853
|
||||
|
||||
Other changes:
|
||||
|
||||
- docs:
|
||||
|
||||
- display README correctly on PyPI
|
||||
- improve cache / index docs, esp. files cache docs, fixes #1825
|
||||
- different pattern matching for --exclude, #1779
|
||||
- datetime formatting examples for {now} placeholder, #1822
|
||||
- clarify passphrase mode attic repo upgrade, #1854
|
||||
- clarify --umask usage, #1859
|
||||
- clarify how to choose PR target branch
|
||||
- clarify prune behavior for different archive contents, #1824
|
||||
- fix PDF issues, add logo, fix authors, headings, TOC
|
||||
- move security verification to support section
|
||||
- fix links in standalone README (:ref: tags)
|
||||
- add link to security contact in README
|
||||
- add FAQ about security
|
||||
- move fork differences to FAQ
|
||||
- add more details about resource usage
|
||||
- tests: skip remote tests on cygwin, #1268
|
||||
- travis:
|
||||
|
||||
- allow OS X failures until the brew cask osxfuse issue is fixed
|
||||
- caskroom osxfuse-beta gone, it's osxfuse now (3.5.3)
|
||||
- vagrant:
|
||||
|
||||
- upgrade OSXfuse / FUSE for macOS to 3.5.3
|
||||
- remove llfuse from tox.ini at a central place
|
||||
- do not try to install llfuse on centos6
|
||||
- fix fuse test for darwin, #1546
|
||||
- add windows virtual machine with cygwin
|
||||
- Vagrantfile cleanup / code deduplication
|
||||
|
||||
|
||||
Version 1.1.0b3 (not released yet)
|
||||
----------------------------------
|
||||
|
||||
|
|
@ -235,6 +198,105 @@ Other changes:
|
|||
- point XDG_*_HOME to temp dirs for tests, #1714
|
||||
- remove all BORG_* env vars from the outer environment
|
||||
|
||||
Version 1.0.9 (2016-12-20)
|
||||
--------------------------
|
||||
|
||||
Security fixes:
|
||||
|
||||
- A flaw in the cryptographic authentication scheme in Borg allowed an attacker
|
||||
to spoof the manifest. See :ref:`tam_vuln` above for the steps you should
|
||||
take.
|
||||
- borg check: When rebuilding the manifest (which should only be needed very rarely)
|
||||
duplicate archive names would be handled on a "first come first serve" basis, allowing
|
||||
an attacker to apparently replace archives.
|
||||
|
||||
Bug fixes:
|
||||
|
||||
- borg check:
|
||||
|
||||
- rebuild manifest if it's corrupted
|
||||
- skip corrupted chunks during manifest rebuild
|
||||
- fix TypeError in integrity error handler, #1903, #1894
|
||||
- fix location parser for archives with @ char (regression introduced in 1.0.8), #1930
|
||||
- fix wrong duration/timestamps if system clock jumped during a create
|
||||
- fix progress display not updating if system clock jumps backwards
|
||||
- fix checkpoint interval being incorrect if system clock jumps
|
||||
|
||||
Other changes:
|
||||
|
||||
- docs:
|
||||
|
||||
- add python3-devel as a dependency for cygwin-based installation
|
||||
- clarify extract is relative to current directory
|
||||
- FAQ: fix link to changelog
|
||||
- markup fixes
|
||||
- tests:
|
||||
|
||||
- test_get\_(cache|keys)_dir: clean env state, #1897
|
||||
- get back pytest's pretty assertion failures, #1938
|
||||
- setup.py build_usage:
|
||||
|
||||
- fixed build_usage not processing all commands
|
||||
- fixed build_usage not generating includes for debug commands
|
||||
|
||||
|
||||
Version 1.0.9rc1 (2016-11-27)
|
||||
-----------------------------
|
||||
|
||||
Bug fixes:
|
||||
|
||||
- files cache: fix determination of newest mtime in backup set (which is
|
||||
used in cache cleanup and led to wrong "A" [added] status for unchanged
|
||||
files in next backup), #1860.
|
||||
|
||||
- borg check:
|
||||
|
||||
- fix incorrectly reporting attic 0.13 and earlier archives as corrupt
|
||||
- handle repo w/o objects gracefully and also bail out early if repo is
|
||||
*completely* empty, #1815.
|
||||
- fix tox/pybuild in 1.0-maint
|
||||
- at xattr module import time, loggers are not initialized yet
|
||||
|
||||
New features:
|
||||
|
||||
- borg umount <mountpoint>
|
||||
exposed already existing umount code via the CLI api, so users can use it,
|
||||
which is more consistent than using borg to mount and fusermount -u (or
|
||||
umount) to un-mount, #1855.
|
||||
- implement borg create --noatime --noctime, fixes #1853
|
||||
|
||||
Other changes:
|
||||
|
||||
- docs:
|
||||
|
||||
- display README correctly on PyPI
|
||||
- improve cache / index docs, esp. files cache docs, fixes #1825
|
||||
- different pattern matching for --exclude, #1779
|
||||
- datetime formatting examples for {now} placeholder, #1822
|
||||
- clarify passphrase mode attic repo upgrade, #1854
|
||||
- clarify --umask usage, #1859
|
||||
- clarify how to choose PR target branch
|
||||
- clarify prune behavior for different archive contents, #1824
|
||||
- fix PDF issues, add logo, fix authors, headings, TOC
|
||||
- move security verification to support section
|
||||
- fix links in standalone README (:ref: tags)
|
||||
- add link to security contact in README
|
||||
- add FAQ about security
|
||||
- move fork differences to FAQ
|
||||
- add more details about resource usage
|
||||
- tests: skip remote tests on cygwin, #1268
|
||||
- travis:
|
||||
|
||||
- allow OS X failures until the brew cask osxfuse issue is fixed
|
||||
- caskroom osxfuse-beta gone, it's osxfuse now (3.5.3)
|
||||
- vagrant:
|
||||
|
||||
- upgrade OSXfuse / FUSE for macOS to 3.5.3
|
||||
- remove llfuse from tox.ini at a central place
|
||||
- do not try to install llfuse on centos6
|
||||
- fix fuse test for darwin, #1546
|
||||
- add windows virtual machine with cygwin
|
||||
- Vagrantfile cleanup / code deduplication
|
||||
|
||||
Version 1.1.0b2 (2016-10-01)
|
||||
----------------------------
|
||||
|
|
|
|||
|
|
@ -144,7 +144,7 @@ certain number of old archives:
|
|||
# archives of THIS machine. The '{hostname}-' prefix is very important to
|
||||
# limit prune's operation to this machine's archives and not apply to
|
||||
# other machine's archives also.
|
||||
borg prune -v --prefix '{hostname}-' \
|
||||
borg prune --list $REPOSITORY --prefix '{hostname}-' \
|
||||
--keep-daily=7 --keep-weekly=4 --keep-monthly=6
|
||||
|
||||
Pitfalls with shell variables and environment variables
|
||||
|
|
|
|||
|
|
@ -20,6 +20,12 @@ optional arguments
|
|||
``-i``, ``--inplace``
|
||||
| rewrite repository in place, with no chance of going back to older
|
||||
| versions of the repository.
|
||||
``--force``
|
||||
| Force upgrade
|
||||
``--tam``
|
||||
| Enable manifest authentication (in key and cache) (Borg 1.0.9 and later)
|
||||
``--disable-tam``
|
||||
| Disable manifest authentication (in key and cache)
|
||||
|
||||
`Common options`_
|
||||
|
|
||||
|
|
@ -28,6 +34,32 @@ Description
|
|||
~~~~~~~~~~~
|
||||
|
||||
Upgrade an existing Borg repository.
|
||||
|
||||
Borg 1.x.y upgrades
|
||||
-------------------
|
||||
|
||||
Use ``borg upgrade --tam REPO`` to require manifest authentication
|
||||
introduced with Borg 1.0.9 to address security issues. This means
|
||||
that modifying the repository after doing this with a version prior
|
||||
to 1.0.9 will raise a validation error, so only perform this upgrade
|
||||
after updating all clients using the repository to 1.0.9 or newer.
|
||||
|
||||
This upgrade should be done on each client for safety reasons.
|
||||
|
||||
If a repository is accidentally modified with a pre-1.0.9 client after
|
||||
this upgrade, use ``borg upgrade --tam --force REPO`` to remedy it.
|
||||
|
||||
If you routinely do this you might not want to enable this upgrade
|
||||
(which will leave you exposed to the security issue). You can
|
||||
reverse the upgrade by issuing ``borg upgrade --disable-tam REPO``.
|
||||
|
||||
See
|
||||
https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability
|
||||
for details.
|
||||
|
||||
Attic and Borg 0.xx to Borg 1.x
|
||||
-------------------------------
|
||||
|
||||
This currently supports converting an Attic repository to Borg and also
|
||||
helps with converting Borg 0.xx to 1.0.
|
||||
|
||||
|
|
|
|||
2
setup.py
2
setup.py
|
|
@ -109,7 +109,7 @@ except ImportError:
|
|||
platform_darwin_source = platform_darwin_source.replace('.pyx', '.c')
|
||||
from distutils.command.build_ext import build_ext
|
||||
if not on_rtd and not all(os.path.exists(path) for path in [
|
||||
compress_source, crypto_source, chunker_source, hashindex_source,
|
||||
compress_source, crypto_source, chunker_source, hashindex_source, item_source,
|
||||
platform_posix_source, platform_linux_source, platform_freebsd_source, platform_darwin_source]):
|
||||
raise ImportError('The GIT version of Borg needs Cython. Install Cython or use a released version.')
|
||||
|
||||
|
|
|
|||
|
|
@ -307,7 +307,7 @@ class Archive:
|
|||
|
||||
def _load_meta(self, id):
|
||||
_, data = self.key.decrypt(id, self.repository.get(id))
|
||||
metadata = ArchiveItem(internal_dict=msgpack.unpackb(data))
|
||||
metadata = ArchiveItem(internal_dict=msgpack.unpackb(data, unicode_errors='surrogateescape'))
|
||||
if metadata.version != 1:
|
||||
raise Exception('Unknown archive metadata version')
|
||||
return metadata
|
||||
|
|
@ -409,7 +409,7 @@ Number of files: {0.stats.nfiles}'''.format(
|
|||
}
|
||||
metadata.update(additional_metadata or {})
|
||||
metadata = ArchiveItem(metadata)
|
||||
data = msgpack.packb(metadata.as_dict(), unicode_errors='surrogateescape')
|
||||
data = self.key.pack_and_authenticate_metadata(metadata.as_dict(), context=b'archive')
|
||||
self.id = self.key.id_hash(data)
|
||||
self.cache.add_chunk(self.id, Chunk(data), self.stats)
|
||||
self.manifest.archives[name] = (self.id, metadata.time)
|
||||
|
|
@ -1197,8 +1197,18 @@ class ArchiveChecker:
|
|||
continue
|
||||
if valid_archive(archive):
|
||||
archive = ArchiveItem(internal_dict=archive)
|
||||
logger.info('Found archive %s', archive.name)
|
||||
manifest.archives[archive.name] = (chunk_id, archive.time)
|
||||
name = archive.name
|
||||
logger.info('Found archive %s', name)
|
||||
if name in manifest.archives:
|
||||
i = 1
|
||||
while True:
|
||||
new_name = '%s.%d' % (name, i)
|
||||
if new_name not in manifest.archives:
|
||||
break
|
||||
i += 1
|
||||
logger.warning('Duplicate archive name %s, storing as %s', name, new_name)
|
||||
name = new_name
|
||||
manifest.archives[name] = (chunk_id, archive.time)
|
||||
logger.info('Manifest rebuild complete.')
|
||||
return manifest
|
||||
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ from .helpers import signal_handler, raising_signal_handler, SigHup, SigTerm
|
|||
from .helpers import ErrorIgnoringTextIOWrapper
|
||||
from .helpers import ProgressIndicatorPercent
|
||||
from .item import Item
|
||||
from .key import key_creator, RepoKey, PassphraseKey
|
||||
from .key import key_creator, tam_required_file, tam_required, RepoKey, PassphraseKey
|
||||
from .keymanager import KeyManager
|
||||
from .platform import get_flags, umount
|
||||
from .remote import RepositoryServer, RemoteRepository, cache_if_remote
|
||||
|
|
@ -61,10 +61,12 @@ def argument(args, str_or_bool):
|
|||
"""If bool is passed, return it. If str is passed, retrieve named attribute from args."""
|
||||
if isinstance(str_or_bool, str):
|
||||
return getattr(args, str_or_bool)
|
||||
if isinstance(str_or_bool, (list, tuple)):
|
||||
return any(getattr(args, item) for item in str_or_bool)
|
||||
return str_or_bool
|
||||
|
||||
|
||||
def with_repository(fake=False, create=False, lock=True, exclusive=False, manifest=True, cache=False):
|
||||
def with_repository(fake=False, invert_fake=False, create=False, lock=True, exclusive=False, manifest=True, cache=False):
|
||||
"""
|
||||
Method decorator for subcommand-handling methods: do_XYZ(self, args, repository, …)
|
||||
|
||||
|
|
@ -81,7 +83,7 @@ def with_repository(fake=False, create=False, lock=True, exclusive=False, manife
|
|||
def wrapper(self, args, **kwargs):
|
||||
location = args.location # note: 'location' must be always present in args
|
||||
append_only = getattr(args, 'append_only', False)
|
||||
if argument(args, fake):
|
||||
if argument(args, fake) ^ invert_fake:
|
||||
return method(self, args, repository=None, **kwargs)
|
||||
elif location.proto == 'ssh':
|
||||
repository = RemoteRepository(location, create=create, exclusive=argument(args, exclusive),
|
||||
|
|
@ -182,7 +184,8 @@ class Archiver:
|
|||
@with_repository(create=True, exclusive=True, manifest=False)
|
||||
def do_init(self, args, repository):
|
||||
"""Initialize an empty repository"""
|
||||
logger.info('Initializing repository at "%s"' % args.location.canonical_path())
|
||||
path = args.location.canonical_path()
|
||||
logger.info('Initializing repository at "%s"' % path)
|
||||
try:
|
||||
key = key_creator(repository, args)
|
||||
except (EOFError, KeyboardInterrupt):
|
||||
|
|
@ -194,6 +197,19 @@ class Archiver:
|
|||
repository.commit()
|
||||
with Cache(repository, key, manifest, warn_if_unencrypted=False):
|
||||
pass
|
||||
if key.tam_required:
|
||||
tam_file = tam_required_file(repository)
|
||||
open(tam_file, 'w').close()
|
||||
logger.warning(
|
||||
'\n'
|
||||
'By default repositories initialized with this version will produce security\n'
|
||||
'errors if written to with an older version (up to and including Borg 1.0.8).\n'
|
||||
'\n'
|
||||
'If you want to use these older versions, you can disable the check by runnning:\n'
|
||||
'borg upgrade --disable-tam \'%s\'\n'
|
||||
'\n'
|
||||
'See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability '
|
||||
'for details about the security implications.', path)
|
||||
return self.exit_code
|
||||
|
||||
@with_repository(exclusive=True, manifest=False)
|
||||
|
|
@ -224,6 +240,7 @@ class Archiver:
|
|||
def do_change_passphrase(self, args, repository, manifest, key):
|
||||
"""Change repository key file passphrase"""
|
||||
key.change_passphrase()
|
||||
logger.info('Key updated')
|
||||
return EXIT_SUCCESS
|
||||
|
||||
@with_repository(lock=False, exclusive=False, manifest=False, cache=False)
|
||||
|
|
@ -272,6 +289,7 @@ class Archiver:
|
|||
key_new.id_key = key_old.id_key
|
||||
key_new.chunk_seed = key_old.chunk_seed
|
||||
key_new.change_passphrase() # option to change key protection passphrase, save
|
||||
logger.info('Key updated')
|
||||
return EXIT_SUCCESS
|
||||
|
||||
@with_repository(fake='dry_run', exclusive=True)
|
||||
|
|
@ -1046,21 +1064,57 @@ class Archiver:
|
|||
DASHES, logger=logging.getLogger('borg.output.stats'))
|
||||
return self.exit_code
|
||||
|
||||
def do_upgrade(self, args):
|
||||
@with_repository(fake=('tam', 'disable_tam'), invert_fake=True, manifest=False, exclusive=True)
|
||||
def do_upgrade(self, args, repository, manifest=None, key=None):
|
||||
"""upgrade a repository from a previous version"""
|
||||
# mainly for upgrades from Attic repositories,
|
||||
# but also supports borg 0.xx -> 1.0 upgrade.
|
||||
if args.tam:
|
||||
manifest, key = Manifest.load(repository, force_tam_not_required=args.force)
|
||||
|
||||
repo = AtticRepositoryUpgrader(args.location.path, create=False)
|
||||
try:
|
||||
repo.upgrade(args.dry_run, inplace=args.inplace, progress=args.progress)
|
||||
except NotImplementedError as e:
|
||||
print("warning: %s" % e)
|
||||
repo = BorgRepositoryUpgrader(args.location.path, create=False)
|
||||
try:
|
||||
repo.upgrade(args.dry_run, inplace=args.inplace, progress=args.progress)
|
||||
except NotImplementedError as e:
|
||||
print("warning: %s" % e)
|
||||
if not manifest.tam_verified or not manifest.config.get(b'tam_required', False):
|
||||
# The standard archive listing doesn't include the archive ID like in borg 1.1.x
|
||||
print('Manifest contents:')
|
||||
for archive_info in manifest.archives.list(sort_by=['ts']):
|
||||
print(format_archive(archive_info), '[%s]' % bin_to_hex(archive_info.id))
|
||||
manifest.config[b'tam_required'] = True
|
||||
manifest.write()
|
||||
repository.commit()
|
||||
if not key.tam_required:
|
||||
key.tam_required = True
|
||||
key.change_passphrase(key._passphrase)
|
||||
print('Key updated')
|
||||
if hasattr(key, 'find_key'):
|
||||
print('Key location:', key.find_key())
|
||||
if not tam_required(repository):
|
||||
tam_file = tam_required_file(repository)
|
||||
open(tam_file, 'w').close()
|
||||
print('Updated security database')
|
||||
elif args.disable_tam:
|
||||
manifest, key = Manifest.load(repository, force_tam_not_required=True)
|
||||
if tam_required(repository):
|
||||
os.unlink(tam_required_file(repository))
|
||||
if key.tam_required:
|
||||
key.tam_required = False
|
||||
key.change_passphrase(key._passphrase)
|
||||
print('Key updated')
|
||||
if hasattr(key, 'find_key'):
|
||||
print('Key location:', key.find_key())
|
||||
manifest.config[b'tam_required'] = False
|
||||
manifest.write()
|
||||
repository.commit()
|
||||
else:
|
||||
# mainly for upgrades from Attic repositories,
|
||||
# but also supports borg 0.xx -> 1.0 upgrade.
|
||||
|
||||
repo = AtticRepositoryUpgrader(args.location.path, create=False)
|
||||
try:
|
||||
repo.upgrade(args.dry_run, inplace=args.inplace, progress=args.progress)
|
||||
except NotImplementedError as e:
|
||||
print("warning: %s" % e)
|
||||
repo = BorgRepositoryUpgrader(args.location.path, create=False)
|
||||
try:
|
||||
repo.upgrade(args.dry_run, inplace=args.inplace, progress=args.progress)
|
||||
except NotImplementedError as e:
|
||||
print("warning: %s" % e)
|
||||
return self.exit_code
|
||||
|
||||
@with_repository(cache=True, exclusive=True)
|
||||
|
|
@ -1735,7 +1789,7 @@ class Archiver:
|
|||
help='manage repository key')
|
||||
|
||||
key_parsers = subparser.add_subparsers(title='required arguments', metavar='<command>')
|
||||
subparser.set_defaults(func=functools.partial(self.do_subcommand_help, subparser))
|
||||
subparser.set_defaults(fallback_func=functools.partial(self.do_subcommand_help, subparser))
|
||||
|
||||
key_export_epilog = textwrap.dedent("""
|
||||
If repository encryption is used, the repository is inaccessible
|
||||
|
|
@ -2303,6 +2357,32 @@ class Archiver:
|
|||
|
||||
upgrade_epilog = textwrap.dedent("""
|
||||
Upgrade an existing Borg repository.
|
||||
|
||||
Borg 1.x.y upgrades
|
||||
-------------------
|
||||
|
||||
Use ``borg upgrade --tam REPO`` to require manifest authentication
|
||||
introduced with Borg 1.0.9 to address security issues. This means
|
||||
that modifying the repository after doing this with a version prior
|
||||
to 1.0.9 will raise a validation error, so only perform this upgrade
|
||||
after updating all clients using the repository to 1.0.9 or newer.
|
||||
|
||||
This upgrade should be done on each client for safety reasons.
|
||||
|
||||
If a repository is accidentally modified with a pre-1.0.9 client after
|
||||
this upgrade, use ``borg upgrade --tam --force REPO`` to remedy it.
|
||||
|
||||
If you routinely do this you might not want to enable this upgrade
|
||||
(which will leave you exposed to the security issue). You can
|
||||
reverse the upgrade by issuing ``borg upgrade --disable-tam REPO``.
|
||||
|
||||
See
|
||||
https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability
|
||||
for details.
|
||||
|
||||
Attic and Borg 0.xx to Borg 1.x
|
||||
-------------------------------
|
||||
|
||||
This currently supports converting an Attic repository to Borg and also
|
||||
helps with converting Borg 0.xx to 1.0.
|
||||
|
||||
|
|
@ -2355,6 +2435,12 @@ class Archiver:
|
|||
default=False, action='store_true',
|
||||
help="""rewrite repository in place, with no chance of going back to older
|
||||
versions of the repository.""")
|
||||
subparser.add_argument('--force', dest='force', action='store_true',
|
||||
help="""Force upgrade""")
|
||||
subparser.add_argument('--tam', dest='tam', action='store_true',
|
||||
help="""Enable manifest authentication (in key and cache) (Borg 1.0.9 and later)""")
|
||||
subparser.add_argument('--disable-tam', dest='disable_tam', action='store_true',
|
||||
help="""Disable manifest authentication (in key and cache)""")
|
||||
subparser.add_argument('location', metavar='REPOSITORY', nargs='?', default='',
|
||||
type=location_validator(archive=False),
|
||||
help='path to the repository to be upgraded')
|
||||
|
|
@ -2525,7 +2611,7 @@ class Archiver:
|
|||
help='debugging command (not intended for normal use)')
|
||||
|
||||
debug_parsers = subparser.add_subparsers(title='required arguments', metavar='<command>')
|
||||
subparser.set_defaults(func=functools.partial(self.do_subcommand_help, subparser))
|
||||
subparser.set_defaults(fallback_func=functools.partial(self.do_subcommand_help, subparser))
|
||||
|
||||
debug_info_epilog = textwrap.dedent("""
|
||||
This command displays some system information that might be useful for bug
|
||||
|
|
@ -2698,7 +2784,9 @@ class Archiver:
|
|||
def run(self, args):
|
||||
os.umask(args.umask) # early, before opening files
|
||||
self.lock_wait = args.lock_wait
|
||||
setup_logging(level=args.log_level, is_serve=args.func == self.do_serve) # do not use loggers before this!
|
||||
# This works around http://bugs.python.org/issue9351
|
||||
func = getattr(args, 'func', None) or getattr(args, 'fallback_func')
|
||||
setup_logging(level=args.log_level, is_serve=func == self.do_serve) # do not use loggers before this!
|
||||
self._setup_implied_logging(vars(args))
|
||||
self._setup_topic_debugging(args)
|
||||
if args.show_version:
|
||||
|
|
@ -2706,7 +2794,7 @@ class Archiver:
|
|||
self.prerun_checks(logger)
|
||||
if is_slow_msgpack():
|
||||
logger.warning("Using a pure-python msgpack! This will result in lower performance.")
|
||||
return args.func(args)
|
||||
return func(args)
|
||||
|
||||
|
||||
def sig_info_handler(sig_no, stack): # pragma: no cover
|
||||
|
|
|
|||
|
|
@ -1,9 +1,13 @@
|
|||
"""A thin OpenSSL wrapper"""
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
from math import ceil
|
||||
|
||||
from libc.stdlib cimport malloc, free
|
||||
from cpython.buffer cimport PyBUF_SIMPLE, PyObject_GetBuffer, PyBuffer_Release
|
||||
|
||||
API_VERSION = 3
|
||||
API_VERSION = 4
|
||||
|
||||
|
||||
cdef extern from "blake2-libselect.h":
|
||||
|
|
@ -247,3 +251,30 @@ def blake2b_256(key, data):
|
|||
raise Exception('blake2b_final() failed')
|
||||
|
||||
return md
|
||||
|
||||
|
||||
def hkdf_hmac_sha512(ikm, salt, info, output_length):
|
||||
"""
|
||||
Compute HKDF-HMAC-SHA512 with input key material *ikm*, *salt* and *info* to produce *output_length* bytes.
|
||||
|
||||
This is the "HMAC-based Extract-and-Expand Key Derivation Function (HKDF)" (RFC 5869)
|
||||
instantiated with HMAC-SHA512.
|
||||
|
||||
*output_length* must not be greater than 64 * 255 bytes.
|
||||
"""
|
||||
digest_length = 64
|
||||
assert output_length <= (255 * digest_length), 'output_length must be <= 255 * 64 bytes'
|
||||
# Step 1. HKDF-Extract (ikm, salt) -> prk
|
||||
if salt is None:
|
||||
salt = bytes(64)
|
||||
prk = hmac.HMAC(salt, ikm, hashlib.sha512).digest()
|
||||
|
||||
# Step 2. HKDF-Expand (prk, info, output_length) -> output key
|
||||
n = ceil(output_length / digest_length)
|
||||
t_n = b''
|
||||
output = b''
|
||||
for i in range(n):
|
||||
msg = t_n + info + (i + 1).to_bytes(1, 'little')
|
||||
t_n = hmac.HMAC(prk, msg, hashlib.sha512).digest()
|
||||
output += t_n
|
||||
return output[:output_length]
|
||||
|
|
|
|||
|
|
@ -93,7 +93,7 @@ def check_extension_modules():
|
|||
raise ExtensionModuleError
|
||||
if compress.API_VERSION != 2:
|
||||
raise ExtensionModuleError
|
||||
if crypto.API_VERSION != 3:
|
||||
if crypto.API_VERSION != 4:
|
||||
raise ExtensionModuleError
|
||||
if platform.API_VERSION != platform.OS_API_VERSION != 5:
|
||||
raise ExtensionModuleError
|
||||
|
|
@ -192,15 +192,16 @@ class Manifest:
|
|||
self.key = key
|
||||
self.repository = repository
|
||||
self.item_keys = frozenset(item_keys) if item_keys is not None else ITEM_KEYS
|
||||
self.tam_verified = False
|
||||
|
||||
@property
|
||||
def id_str(self):
|
||||
return bin_to_hex(self.id)
|
||||
|
||||
@classmethod
|
||||
def load(cls, repository, key=None):
|
||||
def load(cls, repository, key=None, force_tam_not_required=False):
|
||||
from .item import ManifestItem
|
||||
from .key import key_factory
|
||||
from .key import key_factory, tam_required_file, tam_required
|
||||
from .repository import Repository
|
||||
try:
|
||||
cdata = repository.get(cls.MANIFEST_ID)
|
||||
|
|
@ -209,9 +210,10 @@ class Manifest:
|
|||
if not key:
|
||||
key = key_factory(repository, cdata)
|
||||
manifest = cls(key, repository)
|
||||
_, data = key.decrypt(None, cdata)
|
||||
data = key.decrypt(None, cdata).data
|
||||
manifest_dict, manifest.tam_verified = key.unpack_and_verify_manifest(data, force_tam_not_required=force_tam_not_required)
|
||||
m = ManifestItem(internal_dict=manifest_dict)
|
||||
manifest.id = key.id_hash(data)
|
||||
m = ManifestItem(internal_dict=msgpack.unpackb(data))
|
||||
if m.get('version') != 1:
|
||||
raise ValueError('Invalid manifest version')
|
||||
manifest.archives.set_raw_dict(m.archives)
|
||||
|
|
@ -219,21 +221,35 @@ class Manifest:
|
|||
manifest.config = m.config
|
||||
# valid item keys are whatever is known in the repo or every key we know
|
||||
manifest.item_keys = ITEM_KEYS | frozenset(key.decode() for key in m.get('item_keys', []))
|
||||
|
||||
if manifest.tam_verified:
|
||||
manifest_required = manifest.config.get(b'tam_required', False)
|
||||
security_required = tam_required(repository)
|
||||
if manifest_required and not security_required:
|
||||
logger.debug('Manifest is TAM verified and says TAM is required, updating security database...')
|
||||
file = tam_required_file(repository)
|
||||
open(file, 'w').close()
|
||||
if not manifest_required and security_required:
|
||||
logger.debug('Manifest is TAM verified and says TAM is *not* required, updating security database...')
|
||||
os.unlink(tam_required_file(repository))
|
||||
return manifest, key
|
||||
|
||||
def write(self):
|
||||
from .item import ManifestItem
|
||||
if self.key.tam_required:
|
||||
self.config[b'tam_required'] = True
|
||||
self.timestamp = datetime.utcnow().isoformat()
|
||||
manifest = ManifestItem(
|
||||
version=1,
|
||||
archives=self.archives.get_raw_dict(),
|
||||
archives=StableDict(self.archives.get_raw_dict()),
|
||||
timestamp=self.timestamp,
|
||||
config=self.config,
|
||||
item_keys=tuple(self.item_keys),
|
||||
config=StableDict(self.config),
|
||||
item_keys=tuple(sorted(self.item_keys)),
|
||||
)
|
||||
data = msgpack.packb(manifest.as_dict())
|
||||
self.tam_verified = True
|
||||
data = self.key.pack_and_authenticate_metadata(manifest.as_dict())
|
||||
self.id = self.key.id_hash(data)
|
||||
self.repository.put(self.MANIFEST_ID, self.key.encrypt(Chunk(data)))
|
||||
self.repository.put(self.MANIFEST_ID, self.key.encrypt(Chunk(data, compression={'name': 'none'})))
|
||||
|
||||
|
||||
def prune_within(archives, within):
|
||||
|
|
@ -292,7 +308,6 @@ def get_keys_dir():
|
|||
|
||||
def get_security_dir(repository_id=None):
|
||||
"""Determine where to store local security information."""
|
||||
|
||||
xdg_config = os.environ.get('XDG_CONFIG_HOME', os.path.join(get_home_dir(), '.config'))
|
||||
security_dir = os.environ.get('BORG_SECURITY_DIR', os.path.join(xdg_config, 'borg', 'security'))
|
||||
if repository_id:
|
||||
|
|
|
|||
|
|
@ -213,7 +213,7 @@ class Key(PropDict):
|
|||
If a Key shall be serialized, give as_dict() method output to msgpack packer.
|
||||
"""
|
||||
|
||||
VALID_KEYS = {'version', 'repository_id', 'enc_key', 'enc_hmac_key', 'id_key', 'chunk_seed'} # str-typed keys
|
||||
VALID_KEYS = {'version', 'repository_id', 'enc_key', 'enc_hmac_key', 'id_key', 'chunk_seed', 'tam_required'} # str-typed keys
|
||||
|
||||
__slots__ = ("_dict", ) # avoid setting attributes not supported by properties
|
||||
|
||||
|
|
@ -223,6 +223,7 @@ class Key(PropDict):
|
|||
enc_hmac_key = PropDict._make_property('enc_hmac_key', bytes)
|
||||
id_key = PropDict._make_property('id_key', bytes)
|
||||
chunk_seed = PropDict._make_property('chunk_seed', int)
|
||||
tam_required = PropDict._make_property('tam_required', bool)
|
||||
|
||||
|
||||
class ArchiveItem(PropDict):
|
||||
|
|
|
|||
131
src/borg/key.py
131
src/borg/key.py
|
|
@ -4,8 +4,8 @@ import os
|
|||
import sys
|
||||
import textwrap
|
||||
from binascii import a2b_base64, b2a_base64, hexlify, unhexlify
|
||||
from hashlib import sha256, pbkdf2_hmac
|
||||
from hmac import compare_digest
|
||||
from hashlib import sha256, sha512, pbkdf2_hmac
|
||||
from hmac import HMAC, compare_digest
|
||||
|
||||
import msgpack
|
||||
|
||||
|
|
@ -14,11 +14,11 @@ logger = create_logger()
|
|||
|
||||
from .constants import * # NOQA
|
||||
from .compress import Compressor, get_compressor
|
||||
from .crypto import AES, bytes_to_long, bytes_to_int, num_aes_blocks, hmac_sha256, blake2b_256
|
||||
from .helpers import Chunk
|
||||
from .crypto import AES, bytes_to_long, bytes_to_int, num_aes_blocks, hmac_sha256, blake2b_256, hkdf_hmac_sha512
|
||||
from .helpers import Chunk, StableDict
|
||||
from .helpers import Error, IntegrityError
|
||||
from .helpers import yes
|
||||
from .helpers import get_keys_dir
|
||||
from .helpers import get_keys_dir, get_security_dir
|
||||
from .helpers import bin_to_hex
|
||||
from .helpers import CompressionDecider2, CompressionSpec
|
||||
from .item import Key, EncryptedKey
|
||||
|
|
@ -41,6 +41,10 @@ class UnsupportedPayloadError(Error):
|
|||
"""Unsupported payload type {}. A newer version is required to access this repository."""
|
||||
|
||||
|
||||
class UnsupportedManifestError(Error):
|
||||
"""Unsupported manifest envelope. A newer version is required to access this repository."""
|
||||
|
||||
|
||||
class KeyfileNotFoundError(Error):
|
||||
"""No key file for repository {} found in {}."""
|
||||
|
||||
|
|
@ -57,6 +61,32 @@ class RepoKeyNotFoundError(Error):
|
|||
"""No key entry found in the config of repository {}."""
|
||||
|
||||
|
||||
class TAMRequiredError(IntegrityError):
|
||||
__doc__ = textwrap.dedent("""
|
||||
Manifest is unauthenticated, but it is required for this repository.
|
||||
|
||||
This either means that you are under attack, or that you modified this repository
|
||||
with a Borg version older than 1.0.9 after TAM authentication was enabled.
|
||||
|
||||
In the latter case, use "borg upgrade --tam --force '{}'" to re-authenticate the manifest.
|
||||
""").strip()
|
||||
traceback = False
|
||||
|
||||
|
||||
class TAMInvalid(IntegrityError):
|
||||
__doc__ = IntegrityError.__doc__
|
||||
traceback = False
|
||||
|
||||
def __init__(self):
|
||||
# Error message becomes: "Data integrity error: Manifest authentication did not verify"
|
||||
super().__init__('Manifest authentication did not verify')
|
||||
|
||||
|
||||
class TAMUnsupportedSuiteError(IntegrityError):
|
||||
"""Could not verify manifest: Unsupported suite {!r}; a newer version is needed."""
|
||||
traceback = False
|
||||
|
||||
|
||||
def key_creator(repository, args):
|
||||
if args.encryption == 'keyfile':
|
||||
return KeyfileKey.create(repository, args)
|
||||
|
|
@ -94,6 +124,16 @@ def key_factory(repository, manifest_data):
|
|||
raise UnsupportedPayloadError(key_type)
|
||||
|
||||
|
||||
def tam_required_file(repository):
|
||||
security_dir = get_security_dir(bin_to_hex(repository.id))
|
||||
return os.path.join(security_dir, 'tam_required')
|
||||
|
||||
|
||||
def tam_required(repository):
|
||||
file = tam_required_file(repository)
|
||||
return os.path.isfile(file)
|
||||
|
||||
|
||||
class KeyBase:
|
||||
TYPE = None # override in subclasses
|
||||
|
||||
|
|
@ -103,6 +143,7 @@ class KeyBase:
|
|||
self.target = None # key location file path / repo obj
|
||||
self.compression_decider2 = CompressionDecider2(CompressionSpec('none'))
|
||||
self.compressor = Compressor('none') # for decompression
|
||||
self.tam_required = True
|
||||
|
||||
def id_hash(self, data):
|
||||
"""Return HMAC hash using the "id" HMAC key
|
||||
|
|
@ -127,6 +168,68 @@ class KeyBase:
|
|||
if not compare_digest(id_computed, id):
|
||||
raise IntegrityError('Chunk %s: id verification failed' % bin_to_hex(id))
|
||||
|
||||
def _tam_key(self, salt, context):
|
||||
return hkdf_hmac_sha512(
|
||||
ikm=self.id_key + self.enc_key + self.enc_hmac_key,
|
||||
salt=salt,
|
||||
info=b'borg-metadata-authentication-' + context,
|
||||
output_length=64
|
||||
)
|
||||
|
||||
def pack_and_authenticate_metadata(self, metadata_dict, context=b'manifest'):
|
||||
metadata_dict = StableDict(metadata_dict)
|
||||
tam = metadata_dict['tam'] = StableDict({
|
||||
'type': 'HKDF_HMAC_SHA512',
|
||||
'hmac': bytes(64),
|
||||
'salt': os.urandom(64),
|
||||
})
|
||||
packed = msgpack.packb(metadata_dict, unicode_errors='surrogateescape')
|
||||
tam_key = self._tam_key(tam['salt'], context)
|
||||
tam['hmac'] = HMAC(tam_key, packed, sha512).digest()
|
||||
return msgpack.packb(metadata_dict, unicode_errors='surrogateescape')
|
||||
|
||||
def unpack_and_verify_manifest(self, data, force_tam_not_required=False):
|
||||
"""Unpack msgpacked *data* and return (object, did_verify)."""
|
||||
if data.startswith(b'\xc1' * 4):
|
||||
# This is a manifest from the future, we can't read it.
|
||||
raise UnsupportedManifestError()
|
||||
tam_required = self.tam_required
|
||||
if force_tam_not_required and tam_required:
|
||||
logger.warning('Manifest authentication DISABLED.')
|
||||
tam_required = False
|
||||
data = bytearray(data)
|
||||
# Since we don't trust these bytes we use the slower Python unpacker,
|
||||
# which is assumed to have a lower probability of security issues.
|
||||
unpacked = msgpack.fallback.unpackb(data, object_hook=StableDict, unicode_errors='surrogateescape')
|
||||
if b'tam' not in unpacked:
|
||||
if tam_required:
|
||||
raise TAMRequiredError(self.repository._location.canonical_path())
|
||||
else:
|
||||
logger.debug('TAM not found and not required')
|
||||
return unpacked, False
|
||||
tam = unpacked.pop(b'tam', None)
|
||||
if not isinstance(tam, dict):
|
||||
raise TAMInvalid()
|
||||
tam_type = tam.get(b'type', b'<none>').decode('ascii', 'replace')
|
||||
if tam_type != 'HKDF_HMAC_SHA512':
|
||||
if tam_required:
|
||||
raise TAMUnsupportedSuiteError(repr(tam_type))
|
||||
else:
|
||||
logger.debug('Ignoring TAM made with unsupported suite, since TAM is not required: %r', tam_type)
|
||||
return unpacked, False
|
||||
tam_hmac = tam.get(b'hmac')
|
||||
tam_salt = tam.get(b'salt')
|
||||
if not isinstance(tam_salt, bytes) or not isinstance(tam_hmac, bytes):
|
||||
raise TAMInvalid()
|
||||
offset = data.index(tam_hmac)
|
||||
data[offset:offset + 64] = bytes(64)
|
||||
tam_key = self._tam_key(tam_salt, context=b'manifest')
|
||||
calculated_hmac = HMAC(tam_key, data, sha512).digest()
|
||||
if not compare_digest(calculated_hmac, tam_hmac):
|
||||
raise TAMInvalid()
|
||||
logger.debug('TAM-verified manifest')
|
||||
return unpacked, True
|
||||
|
||||
|
||||
class PlaintextKey(KeyBase):
|
||||
TYPE = 0x02
|
||||
|
|
@ -134,6 +237,10 @@ class PlaintextKey(KeyBase):
|
|||
|
||||
chunk_seed = 0
|
||||
|
||||
def __init__(self, repository):
|
||||
super().__init__(repository)
|
||||
self.tam_required = False
|
||||
|
||||
@classmethod
|
||||
def create(cls, repository, args):
|
||||
logger.info('Encryption NOT enabled.\nUse the "--encryption=repokey|keyfile" to enable encryption.')
|
||||
|
|
@ -161,6 +268,9 @@ class PlaintextKey(KeyBase):
|
|||
self.assert_id(id, data)
|
||||
return Chunk(data)
|
||||
|
||||
def _tam_key(self, salt, context):
|
||||
return salt + context
|
||||
|
||||
|
||||
def random_blake2b_256_key():
|
||||
# This might look a bit curious, but is the same construction used in the keyed mode of BLAKE2b.
|
||||
|
|
@ -373,6 +483,7 @@ class PassphraseKey(ID_HMAC_SHA_256, AESKeyBase):
|
|||
key.decrypt(None, manifest_data)
|
||||
num_blocks = num_aes_blocks(len(manifest_data) - 41)
|
||||
key.init_ciphers(key.extract_nonce(manifest_data) + num_blocks)
|
||||
key._passphrase = passphrase
|
||||
return key
|
||||
except IntegrityError:
|
||||
passphrase = Passphrase.getpass(prompt)
|
||||
|
|
@ -388,6 +499,7 @@ class PassphraseKey(ID_HMAC_SHA_256, AESKeyBase):
|
|||
def init(self, repository, passphrase):
|
||||
self.init_from_random_data(passphrase.kdf(repository.id, self.iterations, 100))
|
||||
self.init_ciphers()
|
||||
self.tam_required = False
|
||||
|
||||
|
||||
class KeyfileKeyBase(AESKeyBase):
|
||||
|
|
@ -411,6 +523,7 @@ class KeyfileKeyBase(AESKeyBase):
|
|||
raise PassphraseWrong
|
||||
num_blocks = num_aes_blocks(len(manifest_data) - 41)
|
||||
key.init_ciphers(key.extract_nonce(manifest_data) + num_blocks)
|
||||
key._passphrase = passphrase
|
||||
return key
|
||||
|
||||
def find_key(self):
|
||||
|
|
@ -432,6 +545,7 @@ class KeyfileKeyBase(AESKeyBase):
|
|||
self.enc_hmac_key = key.enc_hmac_key
|
||||
self.id_key = key.id_key
|
||||
self.chunk_seed = key.chunk_seed
|
||||
self.tam_required = key.get('tam_required', tam_required(self.repository))
|
||||
return True
|
||||
return False
|
||||
|
||||
|
|
@ -469,15 +583,16 @@ class KeyfileKeyBase(AESKeyBase):
|
|||
enc_hmac_key=self.enc_hmac_key,
|
||||
id_key=self.id_key,
|
||||
chunk_seed=self.chunk_seed,
|
||||
tam_required=self.tam_required,
|
||||
)
|
||||
data = self.encrypt_key_file(msgpack.packb(key.as_dict()), passphrase)
|
||||
key_data = '\n'.join(textwrap.wrap(b2a_base64(data).decode('ascii')))
|
||||
return key_data
|
||||
|
||||
def change_passphrase(self):
|
||||
passphrase = Passphrase.new(allow_empty=True)
|
||||
def change_passphrase(self, passphrase=None):
|
||||
if passphrase is None:
|
||||
passphrase = Passphrase.new(allow_empty=True)
|
||||
self.save(self.target, passphrase)
|
||||
logger.info('Key updated')
|
||||
|
||||
@classmethod
|
||||
def create(cls, repository, args):
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ SELFTEST_CASES = [
|
|||
ChunkerTestCase,
|
||||
]
|
||||
|
||||
SELFTEST_COUNT = 30
|
||||
SELFTEST_COUNT = 35
|
||||
|
||||
|
||||
class SelfTestResult(TestResult):
|
||||
|
|
|
|||
|
|
@ -3,6 +3,8 @@ from configparser import ConfigParser
|
|||
import errno
|
||||
import os
|
||||
import inspect
|
||||
from datetime import datetime
|
||||
from datetime import timedelta
|
||||
from io import StringIO
|
||||
import logging
|
||||
import random
|
||||
|
|
@ -17,6 +19,7 @@ import unittest
|
|||
from unittest.mock import patch
|
||||
from hashlib import sha256
|
||||
|
||||
import msgpack
|
||||
import pytest
|
||||
try:
|
||||
import llfuse
|
||||
|
|
@ -34,7 +37,7 @@ from ..helpers import Chunk, Manifest
|
|||
from ..helpers import EXIT_SUCCESS, EXIT_WARNING, EXIT_ERROR
|
||||
from ..helpers import bin_to_hex
|
||||
from ..item import Item
|
||||
from ..key import KeyfileKeyBase, RepoKey, KeyfileKey, Passphrase
|
||||
from ..key import KeyfileKeyBase, RepoKey, KeyfileKey, Passphrase, TAMRequiredError
|
||||
from ..keymanager import RepoIdMismatch, NotABorgKeyFile
|
||||
from ..remote import RemoteRepository, PathNotAllowed
|
||||
from ..repository import Repository
|
||||
|
|
@ -42,6 +45,7 @@ from . import has_lchflags, has_llfuse
|
|||
from . import BaseTestCase, changedir, environment_variable, no_selinux
|
||||
from . import are_symlinks_supported, are_hardlinks_supported, are_fifos_supported, is_utime_fully_supported
|
||||
from .platform import fakeroot_detected
|
||||
from . import key
|
||||
|
||||
|
||||
src_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
|
||||
|
|
@ -1645,8 +1649,8 @@ class ArchiverTestCase(ArchiverTestCaseBase):
|
|||
|
||||
def verify_uniqueness():
|
||||
with Repository(self.repository_path) as repository:
|
||||
for key, _ in repository.open_index(repository.get_transaction_id()).iteritems():
|
||||
data = repository.get(key)
|
||||
for id, _ in repository.open_index(repository.get_transaction_id()).iteritems():
|
||||
data = repository.get(id)
|
||||
hash = sha256(data).digest()
|
||||
if hash not in seen:
|
||||
seen.add(hash)
|
||||
|
|
@ -1947,7 +1951,7 @@ class ArchiverTestCase(ArchiverTestCaseBase):
|
|||
repo_key = RepoKey(repository)
|
||||
repo_key.load(None, Passphrase.env_passphrase())
|
||||
|
||||
backup_key = KeyfileKey(None)
|
||||
backup_key = KeyfileKey(key.TestKey.MockRepository())
|
||||
backup_key.load(export_file, Passphrase.env_passphrase())
|
||||
|
||||
assert repo_key.enc_key == backup_key.enc_key
|
||||
|
|
@ -2176,6 +2180,33 @@ class ArchiverCheckTestCase(ArchiverTestCaseBase):
|
|||
self.assert_in('archive2', output)
|
||||
self.cmd('check', self.repository_location, exit_code=0)
|
||||
|
||||
def test_manifest_rebuild_duplicate_archive(self):
|
||||
archive, repository = self.open_archive('archive1')
|
||||
key = archive.key
|
||||
with repository:
|
||||
manifest = repository.get(Manifest.MANIFEST_ID)
|
||||
corrupted_manifest = manifest + b'corrupted!'
|
||||
repository.put(Manifest.MANIFEST_ID, corrupted_manifest)
|
||||
|
||||
archive = msgpack.packb({
|
||||
'cmdline': [],
|
||||
'items': [],
|
||||
'hostname': 'foo',
|
||||
'username': 'bar',
|
||||
'name': 'archive1',
|
||||
'time': '2016-12-15T18:49:51.849711',
|
||||
'version': 1,
|
||||
})
|
||||
archive_id = key.id_hash(archive)
|
||||
repository.put(archive_id, key.encrypt(Chunk(archive)))
|
||||
repository.commit()
|
||||
self.cmd('check', self.repository_location, exit_code=1)
|
||||
self.cmd('check', '--repair', self.repository_location, exit_code=0)
|
||||
output = self.cmd('list', self.repository_location)
|
||||
self.assert_in('archive1', output)
|
||||
self.assert_in('archive1.1', output)
|
||||
self.assert_in('archive2', output)
|
||||
|
||||
def test_extra_chunks(self):
|
||||
self.cmd('check', self.repository_location, exit_code=0)
|
||||
with Repository(self.repository_location, exclusive=True) as repository:
|
||||
|
|
@ -2251,6 +2282,82 @@ class ArchiverCheckTestCase(ArchiverTestCaseBase):
|
|||
self.cmd('list', self.repository_location + '::0.13', exit_code=0)
|
||||
|
||||
|
||||
class ManifestAuthenticationTest(ArchiverTestCaseBase):
|
||||
def spoof_manifest(self, repository):
|
||||
with repository:
|
||||
_, key = Manifest.load(repository)
|
||||
repository.put(Manifest.MANIFEST_ID, key.encrypt(Chunk(msgpack.packb({
|
||||
'version': 1,
|
||||
'archives': {},
|
||||
'config': {},
|
||||
'timestamp': (datetime.utcnow() + timedelta(days=1)).isoformat(),
|
||||
}))))
|
||||
repository.commit()
|
||||
|
||||
def test_fresh_init_tam_required(self):
|
||||
self.cmd('init', self.repository_location)
|
||||
repository = Repository(self.repository_path, exclusive=True)
|
||||
with repository:
|
||||
manifest, key = Manifest.load(repository)
|
||||
repository.put(Manifest.MANIFEST_ID, key.encrypt(Chunk(msgpack.packb({
|
||||
'version': 1,
|
||||
'archives': {},
|
||||
'timestamp': (datetime.utcnow() + timedelta(days=1)).isoformat(),
|
||||
}))))
|
||||
repository.commit()
|
||||
|
||||
with pytest.raises(TAMRequiredError):
|
||||
self.cmd('list', self.repository_location)
|
||||
|
||||
def test_not_required(self):
|
||||
self.cmd('init', self.repository_location)
|
||||
self.create_src_archive('archive1234')
|
||||
repository = Repository(self.repository_path, exclusive=True)
|
||||
with repository:
|
||||
shutil.rmtree(get_security_dir(bin_to_hex(repository.id)))
|
||||
_, key = Manifest.load(repository)
|
||||
key.tam_required = False
|
||||
key.change_passphrase(key._passphrase)
|
||||
|
||||
manifest = msgpack.unpackb(key.decrypt(None, repository.get(Manifest.MANIFEST_ID)).data)
|
||||
del manifest[b'tam']
|
||||
repository.put(Manifest.MANIFEST_ID, key.encrypt(Chunk(msgpack.packb(manifest))))
|
||||
repository.commit()
|
||||
output = self.cmd('list', '--debug', self.repository_location)
|
||||
assert 'archive1234' in output
|
||||
assert 'TAM not found and not required' in output
|
||||
# Run upgrade
|
||||
self.cmd('upgrade', '--tam', self.repository_location)
|
||||
# Manifest must be authenticated now
|
||||
output = self.cmd('list', '--debug', self.repository_location)
|
||||
assert 'archive1234' in output
|
||||
assert 'TAM-verified manifest' in output
|
||||
# Try to spoof / modify pre-1.0.9
|
||||
self.spoof_manifest(repository)
|
||||
# Fails
|
||||
with pytest.raises(TAMRequiredError):
|
||||
self.cmd('list', self.repository_location)
|
||||
# Force upgrade
|
||||
self.cmd('upgrade', '--tam', '--force', self.repository_location)
|
||||
self.cmd('list', self.repository_location)
|
||||
|
||||
def test_disable(self):
|
||||
self.cmd('init', self.repository_location)
|
||||
self.create_src_archive('archive1234')
|
||||
self.cmd('upgrade', '--disable-tam', self.repository_location)
|
||||
repository = Repository(self.repository_path, exclusive=True)
|
||||
self.spoof_manifest(repository)
|
||||
assert not self.cmd('list', self.repository_location)
|
||||
|
||||
def test_disable2(self):
|
||||
self.cmd('init', self.repository_location)
|
||||
self.create_src_archive('archive1234')
|
||||
repository = Repository(self.repository_path, exclusive=True)
|
||||
self.spoof_manifest(repository)
|
||||
self.cmd('upgrade', '--disable-tam', self.repository_location)
|
||||
assert not self.cmd('list', self.repository_location)
|
||||
|
||||
|
||||
@pytest.mark.skipif(sys.platform == 'cygwin', reason='remote is broken on cygwin and hangs')
|
||||
class RemoteArchiverTestCase(ArchiverTestCase):
|
||||
prefix = '__testsuite__:'
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@ from binascii import hexlify, unhexlify
|
|||
|
||||
from ..crypto import AES, bytes_to_long, bytes_to_int, long_to_bytes, hmac_sha256, blake2b_256
|
||||
from ..crypto import increment_iv, bytes16_to_int, int_to_bytes16
|
||||
|
||||
from ..crypto import hkdf_hmac_sha512
|
||||
from . import BaseTestCase
|
||||
|
||||
# Note: these tests are part of the self test, do not use or import py.test functionality here.
|
||||
|
|
@ -96,3 +96,55 @@ class CryptoTestCase(BaseTestCase):
|
|||
key = unhexlify('e944973af2256d4d670c12dd75304c319f58f4e40df6fb18ef996cb47e063676')
|
||||
data = memoryview(b'1234567890' * 100)
|
||||
assert blake2b_256(key, data) == unhexlify('97ede832378531dd0f4c668685d166e797da27b47d8cd441e885b60abd5e0cb2')
|
||||
|
||||
# These test vectors come from https://www.kullo.net/blog/hkdf-sha-512-test-vectors/
|
||||
# who claims to have verified these against independent Python and C++ implementations.
|
||||
|
||||
def test_hkdf_hmac_sha512(self):
|
||||
ikm = b'\x0b' * 22
|
||||
salt = bytes.fromhex('000102030405060708090a0b0c')
|
||||
info = bytes.fromhex('f0f1f2f3f4f5f6f7f8f9')
|
||||
l = 42
|
||||
|
||||
okm = hkdf_hmac_sha512(ikm, salt, info, l)
|
||||
assert okm == bytes.fromhex('832390086cda71fb47625bb5ceb168e4c8e26a1a16ed34d9fc7fe92c1481579338da362cb8d9f925d7cb')
|
||||
|
||||
def test_hkdf_hmac_sha512_2(self):
|
||||
ikm = bytes.fromhex('000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f2021222324252627'
|
||||
'28292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f')
|
||||
salt = bytes.fromhex('606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868'
|
||||
'788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeaf')
|
||||
info = bytes.fromhex('b0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7'
|
||||
'd8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff')
|
||||
l = 82
|
||||
|
||||
okm = hkdf_hmac_sha512(ikm, salt, info, l)
|
||||
assert okm == bytes.fromhex('ce6c97192805b346e6161e821ed165673b84f400a2b514b2fe23d84cd189ddf1b695b48cbd1c838844'
|
||||
'1137b3ce28f16aa64ba33ba466b24df6cfcb021ecff235f6a2056ce3af1de44d572097a8505d9e7a93')
|
||||
|
||||
def test_hkdf_hmac_sha512_3(self):
|
||||
ikm = bytes.fromhex('0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b')
|
||||
salt = None
|
||||
info = b''
|
||||
l = 42
|
||||
|
||||
okm = hkdf_hmac_sha512(ikm, salt, info, l)
|
||||
assert okm == bytes.fromhex('f5fa02b18298a72a8c23898a8703472c6eb179dc204c03425c970e3b164bf90fff22d04836d0e2343bac')
|
||||
|
||||
def test_hkdf_hmac_sha512_4(self):
|
||||
ikm = bytes.fromhex('0b0b0b0b0b0b0b0b0b0b0b')
|
||||
salt = bytes.fromhex('000102030405060708090a0b0c')
|
||||
info = bytes.fromhex('f0f1f2f3f4f5f6f7f8f9')
|
||||
l = 42
|
||||
|
||||
okm = hkdf_hmac_sha512(ikm, salt, info, l)
|
||||
assert okm == bytes.fromhex('7413e8997e020610fbf6823f2ce14bff01875db1ca55f68cfcf3954dc8aff53559bd5e3028b080f7c068')
|
||||
|
||||
def test_hkdf_hmac_sha512_5(self):
|
||||
ikm = bytes.fromhex('0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c0c')
|
||||
salt = None
|
||||
info = b''
|
||||
l = 42
|
||||
|
||||
okm = hkdf_hmac_sha512(ikm, salt, info, l)
|
||||
assert okm == bytes.fromhex('1407d46013d98bc6decefcfee55f0f90b0c7f63d68eb1a80eaf07e953cfc0a3a5240a155d6e4daa965bb')
|
||||
|
|
|
|||
|
|
@ -659,6 +659,7 @@ def test_get_keys_dir(monkeypatch):
|
|||
|
||||
def test_get_security_dir(monkeypatch):
|
||||
"""test that get_security_dir respects environment"""
|
||||
monkeypatch.delenv('BORG_SECURITY_DIR', raising=False)
|
||||
monkeypatch.delenv('XDG_CONFIG_HOME', raising=False)
|
||||
assert get_security_dir() == os.path.join(os.path.expanduser('~'), '.config', 'borg', 'security')
|
||||
assert get_security_dir(repository_id='1234') == os.path.join(os.path.expanduser('~'), '.config', 'borg', 'security', '1234')
|
||||
|
|
|
|||
|
|
@ -5,14 +5,16 @@ import os.path
|
|||
from binascii import hexlify, unhexlify
|
||||
|
||||
import pytest
|
||||
import msgpack
|
||||
|
||||
from ..crypto import bytes_to_long, num_aes_blocks
|
||||
from ..helpers import Location
|
||||
from ..helpers import Chunk
|
||||
from ..helpers import Chunk, StableDict
|
||||
from ..helpers import IntegrityError
|
||||
from ..helpers import get_security_dir
|
||||
from ..key import PlaintextKey, PassphraseKey, KeyfileKey, RepoKey, Blake2KeyfileKey, Blake2RepoKey, AuthenticatedKey
|
||||
from ..key import Passphrase, PasswordRetriesExceeded, bin_to_hex
|
||||
from ..key import TAMRequiredError, TAMInvalid, TAMUnsupportedSuiteError, UnsupportedManifestError
|
||||
|
||||
|
||||
class TestKey:
|
||||
|
|
@ -74,6 +76,9 @@ class TestKey:
|
|||
class _Location:
|
||||
orig = '/some/place'
|
||||
|
||||
def canonical_path(self):
|
||||
return self.orig
|
||||
|
||||
_location = _Location()
|
||||
id = bytes(32)
|
||||
id_str = bin_to_hex(id)
|
||||
|
|
@ -277,3 +282,115 @@ class TestPassphrase:
|
|||
|
||||
def test_passphrase_repr(self):
|
||||
assert "secret" not in repr(Passphrase("secret"))
|
||||
|
||||
|
||||
class TestTAM:
|
||||
@pytest.fixture
|
||||
def key(self, monkeypatch):
|
||||
monkeypatch.setenv('BORG_PASSPHRASE', 'test')
|
||||
return KeyfileKey.create(TestKey.MockRepository(), TestKey.MockArgs())
|
||||
|
||||
def test_unpack_future(self, key):
|
||||
blob = b'\xc1\xc1\xc1\xc1foobar'
|
||||
with pytest.raises(UnsupportedManifestError):
|
||||
key.unpack_and_verify_manifest(blob)
|
||||
|
||||
blob = b'\xc1\xc1\xc1'
|
||||
with pytest.raises(msgpack.UnpackException):
|
||||
key.unpack_and_verify_manifest(blob)
|
||||
|
||||
def test_missing_when_required(self, key):
|
||||
blob = msgpack.packb({})
|
||||
with pytest.raises(TAMRequiredError):
|
||||
key.unpack_and_verify_manifest(blob)
|
||||
|
||||
def test_missing(self, key):
|
||||
blob = msgpack.packb({})
|
||||
key.tam_required = False
|
||||
unpacked, verified = key.unpack_and_verify_manifest(blob)
|
||||
assert unpacked == {}
|
||||
assert not verified
|
||||
|
||||
def test_unknown_type_when_required(self, key):
|
||||
blob = msgpack.packb({
|
||||
'tam': {
|
||||
'type': 'HMAC_VOLLBIT',
|
||||
},
|
||||
})
|
||||
with pytest.raises(TAMUnsupportedSuiteError):
|
||||
key.unpack_and_verify_manifest(blob)
|
||||
|
||||
def test_unknown_type(self, key):
|
||||
blob = msgpack.packb({
|
||||
'tam': {
|
||||
'type': 'HMAC_VOLLBIT',
|
||||
},
|
||||
})
|
||||
key.tam_required = False
|
||||
unpacked, verified = key.unpack_and_verify_manifest(blob)
|
||||
assert unpacked == {}
|
||||
assert not verified
|
||||
|
||||
@pytest.mark.parametrize('tam, exc', (
|
||||
({}, TAMUnsupportedSuiteError),
|
||||
({'type': b'\xff'}, TAMUnsupportedSuiteError),
|
||||
(None, TAMInvalid),
|
||||
(1234, TAMInvalid),
|
||||
))
|
||||
def test_invalid(self, key, tam, exc):
|
||||
blob = msgpack.packb({
|
||||
'tam': tam,
|
||||
})
|
||||
with pytest.raises(exc):
|
||||
key.unpack_and_verify_manifest(blob)
|
||||
|
||||
@pytest.mark.parametrize('hmac, salt', (
|
||||
({}, bytes(64)),
|
||||
(bytes(64), {}),
|
||||
(None, bytes(64)),
|
||||
(bytes(64), None),
|
||||
))
|
||||
def test_wrong_types(self, key, hmac, salt):
|
||||
data = {
|
||||
'tam': {
|
||||
'type': 'HKDF_HMAC_SHA512',
|
||||
'hmac': hmac,
|
||||
'salt': salt
|
||||
},
|
||||
}
|
||||
tam = data['tam']
|
||||
if hmac is None:
|
||||
del tam['hmac']
|
||||
if salt is None:
|
||||
del tam['salt']
|
||||
blob = msgpack.packb(data)
|
||||
with pytest.raises(TAMInvalid):
|
||||
key.unpack_and_verify_manifest(blob)
|
||||
|
||||
def test_round_trip(self, key):
|
||||
data = {'foo': 'bar'}
|
||||
blob = key.pack_and_authenticate_metadata(data)
|
||||
assert blob.startswith(b'\x82')
|
||||
|
||||
unpacked = msgpack.unpackb(blob)
|
||||
assert unpacked[b'tam'][b'type'] == b'HKDF_HMAC_SHA512'
|
||||
|
||||
unpacked, verified = key.unpack_and_verify_manifest(blob)
|
||||
assert verified
|
||||
assert unpacked[b'foo'] == b'bar'
|
||||
assert b'tam' not in unpacked
|
||||
|
||||
@pytest.mark.parametrize('which', (b'hmac', b'salt'))
|
||||
def test_tampered(self, key, which):
|
||||
data = {'foo': 'bar'}
|
||||
blob = key.pack_and_authenticate_metadata(data)
|
||||
assert blob.startswith(b'\x82')
|
||||
|
||||
unpacked = msgpack.unpackb(blob, object_hook=StableDict)
|
||||
assert len(unpacked[b'tam'][which]) == 64
|
||||
unpacked[b'tam'][which] = unpacked[b'tam'][which][0:32] + bytes(32)
|
||||
assert len(unpacked[b'tam'][which]) == 64
|
||||
blob = msgpack.packb(unpacked)
|
||||
|
||||
with pytest.raises(TAMInvalid):
|
||||
key.unpack_and_verify_manifest(blob)
|
||||
|
|
|
|||
Loading…
Reference in a new issue