Winfried Plappert
bcd4168428
Enhancement: calls to SnapshotFilter.FindLatest() can be simplified ( #5688 )
Create and publish a Docker image / build-and-push-image (push) Waiting to run
Create and publish a Docker image / provenance (push) Blocked by required conditions
test / Linux Go 1.24.x (push) Waiting to run
test / Linux (race) Go 1.25.x (push) Waiting to run
test / Windows Go 1.25.x (push) Waiting to run
test / macOS Go 1.25.x (push) Waiting to run
test / Linux Go 1.25.x (push) Waiting to run
test / Cross Compile for subset 0/3 (push) Waiting to run
test / Cross Compile for subset 1/3 (push) Waiting to run
test / Cross Compile for subset 2/3 (push) Waiting to run
test / lint (push) Waiting to run
test / Analyze results (push) Blocked by required conditions
test / docker (push) Waiting to run
2026-01-31 23:04:01 +01:00
Michael Eischer
24d56fe2a6
diff: switch to efficient DualTreeIterator
...
The previous implementation stored the whole tree in a map and used it
for checking overlap between trees. This is now replaced with the
DualTreeIterator, which iterates over two trees in parallel and returns
the merge stream in order. In case of overlap between both trees, it
returns both nodes at the same time. Otherwise, only a single node is
returned.
2026-01-31 20:03:38 +01:00
Michael Eischer
350f29d921
data: replace Tree with TreeNodeIterator
...
The TreeNodeIterator decodes nodes while iterating over a tree blob.
This should reduce peak memory usage as now only the serialized tree
blob and a single node have to alive at the same time. Using the
iterator has implications for the error handling however. Now it is
necessary that all loops that iterate through a tree check for errors
before using the node returned by the iterator.
The other change is that it is no longer possible to iterate over a tree
multiple times. Instead it must be loaded a second time. This only
affects the tree rewriting code.
2026-01-31 20:03:38 +01:00
Michael Eischer
1e183509d4
data: rework StreamTrees to use synchronous callbacks
...
The tree.Nodes will be replaced by an iterator to loads and serializes
tree node ondemand. Thus, the processing moves from StreamTrees into the
callback. Schedule them onto the workers used by StreamTrees for proper
load distribution.
2026-01-31 20:03:38 +01:00
Michael Eischer
278e457e1f
data: use data.TreeWriter to serialize&write data.Tree
...
Always serialize trees via TreeJSONBuilder. Add a wrapper called
TreeWriter which combines serialization and saving the tree blob in the
repository. In the future, TreeJSONBuilder will have to upload tree
chunks while the tree is still serialized. This will a wrapper like
TreeWriter, so add it right now already.
The archiver.treeSaver still directly uses the TreeJSONBuilder as it
requires special handling.
2026-01-31 19:18:36 +01:00
Winfried Plappert
86ccc6d445
Bugfix: restic check: add missing finalizeSnapshotFilter() ( #5644 )
...
Create and publish a Docker image / build-and-push-image (push) Has been cancelled
test / Linux Go 1.24.x (push) Has been cancelled
test / Linux (race) Go 1.25.x (push) Has been cancelled
test / Windows Go 1.25.x (push) Has been cancelled
test / macOS Go 1.25.x (push) Has been cancelled
test / Linux Go 1.25.x (push) Has been cancelled
test / Cross Compile for subset 0/3 (push) Has been cancelled
test / Cross Compile for subset 1/3 (push) Has been cancelled
test / Cross Compile for subset 2/3 (push) Has been cancelled
test / lint (push) Has been cancelled
test / docker (push) Has been cancelled
Create and publish a Docker image / provenance (push) Has been cancelled
test / Analyze results (push) Has been cancelled
add missing finalizeSnapshotFilter() to cmd.RunE()
---------
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2026-01-26 21:08:15 +00:00
Michael Eischer
a8be8e36fa
Merge pull request #5621 from MichaelEischer/copy-stream-snapshots
...
copy: iterate through snapshots
2025-12-03 21:21:05 +01:00
Michael Eischer
1e6ed458ff
remove old // +build comments
2025-11-30 11:53:23 +01:00
Winfried Plappert
ce57961f14
restic check with snapshot filters ( #5469 )
...
---------
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2025-11-28 19:12:38 +00:00
Michael Eischer
e1bc2fb71a
copy: iterate through snapshots
2025-11-26 22:48:54 +01:00
Michael Eischer
046b0e711d
repository: add SaveBlobAsync method
2025-11-26 21:18:21 +01:00
Michael Eischer
134893bd35
copy: use AssociatedBlobSet to keep track of processed trees
2025-11-26 21:00:18 +01:00
Michael Eischer
46ebee948f
stats: use AssociatedBlobSet
2025-11-26 20:59:39 +01:00
Michael Eischer
d91fe1d7e1
diff: use AssociatedBlobSet
2025-11-26 20:59:39 +01:00
Michael Eischer
ff099a216a
copy: use AssociatedBlobSet
2025-11-26 20:59:38 +01:00
Michael Eischer
7e80536a9b
Merge pull request #5472 from wplapper/cmd_copy_stream
...
restic copy --stream: run one large copy operation crossing snapshot boundaries - issue #5453
2025-11-26 20:57:46 +01:00
Michael Eischer
f9e5660e75
output which source and target snapshot belong together
2025-11-23 22:01:53 +01:00
Michael Eischer
e79b01d82f
more aggressive batching
2025-11-23 21:46:03 +01:00
Michael Eischer
857b42fca4
merge into existing copy test
2025-11-23 19:08:49 +01:00
Michael Eischer
39db78446f
Simplify test
2025-11-23 19:05:55 +01:00
Michael Eischer
cf409b7c66
automatically batch snapshots in copy
2025-11-23 17:40:37 +01:00
Michael Eischer
f95dc73d38
deduplicate blob enqueuing
2025-11-23 17:13:10 +01:00
Michael Eischer
63bc1405ea
unify snapshot copy codepaths
2025-11-23 17:12:54 +01:00
Michael Eischer
05364500b6
use correct context
2025-11-23 16:25:09 +01:00
Michael Eischer
e775192fe7
don't sort snapshots, drop duplicate code and cleanup copyTreeBatched function signature
2025-11-23 16:20:40 +01:00
Michael Eischer
4395a77154
copy: remove bugous seenBlobs set
2025-11-23 16:06:45 +01:00
Michael Eischer
81d8bc4ade
repository: replace CopyBlobs with Repack implementation
2025-11-23 16:06:29 +01:00
Michael Eischer
6174c91042
Merge pull request #5588 from seqizz/g_timezoneshow
...
Create and publish a Docker image / build-and-push-image (push) Has been cancelled
test / Linux Go 1.23.x (push) Has been cancelled
test / Linux Go 1.24.x (push) Has been cancelled
test / Linux (race) Go 1.25.x (push) Has been cancelled
test / Windows Go 1.25.x (push) Has been cancelled
test / macOS Go 1.25.x (push) Has been cancelled
test / Linux Go 1.25.x (push) Has been cancelled
test / Cross Compile for subset 0/3 (push) Has been cancelled
test / Cross Compile for subset 1/3 (push) Has been cancelled
test / Cross Compile for subset 2/3 (push) Has been cancelled
test / lint (push) Has been cancelled
test / docker (push) Has been cancelled
Create and publish a Docker image / provenance (push) Has been cancelled
test / Analyze results (push) Has been cancelled
snapshots: Show timezone in non-compact output
2025-11-19 22:06:37 +01:00
Winfried Plappert
b87f7586e4
restic copy --batch: a fresh start from commit 382616747
...
Instead of rebasing my code, I decided to start fresh, since WithBlobUploader()
has been introduced.
changelog/unreleased/issue-5453:
doc/045_working_with_repos.rst:
the usual
cmd/restic/cmd_copy.go:
gather all snaps to be collected - collectAllSnapshots()
run overall copy step - func copyTreeBatched()
helper copySaveSnapshot() to save the corresponding snapshot
internal/repository/repack.go:
introduce wrapper CopyBlobs(), which passes parameter `uploader restic.BlobSaver` from
WithBlobUploader() via copyTreeBatched() to repack().
internal/backend/local/local_windows.go:
I did not touch it, but gofmt did: whitespace
2025-11-19 07:09:24 +00:00
Gürkan
dc4e9b31f6
snapshots: Show timezone in non-compact output
2025-11-18 13:32:44 +01:00
Michael Eischer
8767549367
Merge pull request #5601 from MichaelEischer/snapshots-fix-groupby-with-latest
...
Create and publish a Docker image / build-and-push-image (push) Has been cancelled
test / Linux Go 1.23.x (push) Has been cancelled
test / Linux Go 1.24.x (push) Has been cancelled
test / Linux (race) Go 1.25.x (push) Has been cancelled
test / Windows Go 1.25.x (push) Has been cancelled
test / macOS Go 1.25.x (push) Has been cancelled
test / Linux Go 1.25.x (push) Has been cancelled
test / Cross Compile for subset 0/3 (push) Has been cancelled
test / Cross Compile for subset 1/3 (push) Has been cancelled
test / Cross Compile for subset 2/3 (push) Has been cancelled
test / lint (push) Has been cancelled
test / docker (push) Has been cancelled
Create and publish a Docker image / provenance (push) Has been cancelled
test / Analyze results (push) Has been cancelled
snapshots: correctly handle --latest in combination with --group-by
2025-11-17 22:50:50 +01:00
Michael Eischer
5afe61585b
snapshots: correctly handle --latest in combination with --group-by
2025-11-17 22:26:57 +01:00
Michael Eischer
0ff3e20c4b
prune: return proper error if blob cannot be found
2025-11-16 17:04:03 +01:00
Michael Eischer
3b854d9c04
Merge pull request #5449 from provokateurin/restore-ownership-by-name
...
Create and publish a Docker image / build-and-push-image (push) Waiting to run
Create and publish a Docker image / provenance (push) Blocked by required conditions
test / Linux Go 1.23.x (push) Waiting to run
test / Linux Go 1.24.x (push) Waiting to run
test / Linux (race) Go 1.25.x (push) Waiting to run
test / Windows Go 1.25.x (push) Waiting to run
test / macOS Go 1.25.x (push) Waiting to run
test / Linux Go 1.25.x (push) Waiting to run
test / Cross Compile for subset 0/3 (push) Waiting to run
test / Cross Compile for subset 1/3 (push) Waiting to run
test / Cross Compile for subset 2/3 (push) Waiting to run
test / lint (push) Waiting to run
test / Analyze results (push) Blocked by required conditions
test / docker (push) Waiting to run
feat(internal/fs/node): Restore ownership by name
2025-11-16 16:50:36 +01:00
provokateurin
8fae46011a
feat(internal/fs/node): Restore ownership by name
2025-11-16 16:40:58 +01:00
Winfried Plappert
25611f4628
restic copy - add statistics counters
...
cmd/restic/cmd_copy.go:
add function copyStats() and call it before the actual copying starts.
changelog/unreleased/pull-5319:
rephrased wording of the statistics counters.
2025-11-16 13:47:10 +01:00
Winfried Plappert
90ac3efa88
restic copy - add additional status counters
...
'copyTree()' now counts and sizes the blobs in 'copyBlobs' and prints them out
via 'Verbosef()'.
2025-11-16 13:46:27 +01:00
Michael Eischer
14f3bc8232
Merge pull request #5560 from MichaelEischer/index-iterators
...
index: port to modern Go iterators
2025-11-16 13:24:48 +01:00
Michael Eischer
38c543457e
index: convert to implement modern go iterators
2025-11-16 12:56:37 +01:00
Michael Eischer
a0925fa922
repository: set progress bar maximum in Repack
2025-11-16 12:51:46 +01:00
Michael Eischer
b2afccbd96
repository: remove unused obsoletePacks return values from Repack
2025-11-16 12:51:46 +01:00
Michael Eischer
0624b656b8
Merge pull request #5558 from MichaelEischer/simplify-blob-upload
...
repository: enforce correct usage of SaveBlob
2025-11-16 12:51:01 +01:00
Michael Eischer
157f174dd9
Merge pull request #5370 from hashier/feat/exclude-macOS-cloud-files
...
feat(backup): add possibility to exclude macOS cloud-only files
2025-11-16 11:57:37 +01:00
Michael Eischer
c6e33c3954
repository: enforce that SaveBlob is called within WithBlobUploader
...
This is achieved by removing SaveBlob from the public API and only
returning it via a uploader object that is passed in by
WithBlobUploader.
2025-10-12 18:26:26 +02:00
Michael Eischer
aa7bd241d9
init: move more logic into global package
2025-10-12 18:08:26 +02:00
Michael Eischer
a816b827cf
extract GlobalOptions into internal/global package
...
Rough steps:
```
mv cmd/restic/global* cmd/restic/secondary_repo* internal/global/
sed -i "s/package main/package global/" internal/global/*.go
Rename "GlobalOptions" to "Options" in internal/global/
Replace everywhere " GlobalOptions" -> " global.Options"
Replace everywhere "\*GlobalOptions" -> " *global.Options"
Make SecondaryRepoOptions public
Make create public
Make version public
```
2025-10-12 17:56:28 +02:00
Michael Eischer
2c677d8db4
global: make private fields public
2025-10-12 17:56:28 +02:00
Michael Eischer
394c8de502
add package to create a prepopulated backend registry
2025-10-12 17:56:28 +02:00
Michael Eischer
a632f490fa
Merge pull request #5550 from MichaelEischer/refactor-check-data-selection
...
check: refactor pack selection for read data
2025-10-12 17:51:00 +02:00
Michael Eischer
ac4642b479
repository: replace StartPackUploader+Flush with WithBlobUploader
...
The new method combines both step into a single wrapper function. Thus
it ensures that both are always called in pairs. As an additional
benefit this slightly reduces the boilerplate to upload blobs.
2025-10-08 22:49:45 +02:00
