Michael Eischer
666a0b0bdb
repository: streamPack: replace streaming with chunked download
...
Due to the interface of streamPack, we cannot guarantee that operations
progress fast enough that the underlying connections remains open. This
introduces partial failures which massively complicate the error
handling.
Switch to a simpler approach that retrieves the pack in chunks of 32MB.
If a blob is larger than this limit, then it is downloaded separately.
To avoid multiple copies in memory, an auxiliary interface
`discardReader` is introduced that allows directly accessing the
downloaded byte slices, while still supporting the streaming used by the
`check` command.
2024-04-22 21:21:23 +02:00
Michael Eischer
621012dac0
repository: Add blob loading fallback to LoadBlobsFromPack
...
Try to retrieve individual blobs via LoadBlob if streaming did not work.
2024-04-21 21:35:55 +02:00
Michael Eischer
5e98f1e2eb
repository: fix test setup race conditions
2024-03-28 23:17:02 +01:00
Michael Eischer
dc441c57a7
repository: unify repository initialization in tests
...
Tests should use a helper from internal/repository/testing.go to
construct a Repository object.
2024-03-28 23:17:02 +01:00
Michael Eischer
3ba1fa3cee
repository: remove a few global variables
2024-03-28 23:17:02 +01:00
Michael Eischer
044e8bf821
repository: parallelize lock tests
2024-03-28 23:17:02 +01:00
Michael Eischer
e8df50fa3c
repository: remove global list of locks
2024-03-28 22:46:33 +01:00
Michael Eischer
cbb5f89252
lock: move code to repository package
2024-03-28 22:46:33 +01:00
Michael Eischer
4c3218ef9f
repository: include packID in StreamPack for decrypt/decompress errors
2024-02-17 19:38:01 +01:00
Michael Eischer
18b0bbbf42
repository: use fmt.Errorf in StreamPacks
2024-02-17 19:37:32 +01:00
Alexander Neumann
c0514dd8ba
Fix linter errors (except for tests)
2024-02-10 22:58:10 +01:00
Michael Eischer
5957417b1f
Apply changelog entry / documentation improvements from review
2024-02-04 18:55:41 +01:00
Michael Eischer
86b38a0b17
rename --no-verify-pack to --no-extra-verify
2024-02-04 17:01:05 +01:00
Michael Eischer
c97a271e89
repository: ask users to report corrupted data while saving blobs
2024-02-04 15:31:42 +01:00
Michael Eischer
193140525c
repository: test verification of blobs/unpacked data
2024-02-04 15:31:42 +01:00
Michael Eischer
2dbb18128c
repository: Allow skipping verification for tests
...
Some tests have to explicitly create pack files with blobs that don't
match their ID. For those blobs the builtin verification of the
repository must be disabled.
2024-02-03 18:22:47 +01:00
Michael Eischer
30a84e9003
backup: verify unpacked files before upload
2024-02-03 18:22:47 +01:00
Michael Eischer
c01a0c6da7
backup: verify blobs before upload
...
This only covers the blobs themselves, the pack header is not verified
so far. Unpacked files are also not covered by the integrity check.
2024-02-03 18:22:47 +01:00
Michael Eischer
16e3f79e8b
repository: make repo.Options configurable for test repos
2024-02-03 18:22:47 +01:00
Michael Eischer
bb92b487f7
repository: fix repack test
2024-02-03 18:22:47 +01:00
Michael Eischer
bfb56b78e1
replace some usages of restic.Repository with more specific interface
...
This should eventually make it easier to test the code.
2024-01-27 13:02:02 +01:00
Michael Eischer
3424088274
Merge pull request #4644 from MichaelEischer/refactor-repair-packs
...
Refactor and test `repair packs`
2024-01-27 13:00:51 +01:00
Michael Eischer
f0e1ad2285
fix linter warning
2024-01-27 12:51:45 +01:00
Michael Eischer
fd579421dd
repository: deduplicate test
2024-01-27 12:51:45 +01:00
Michael Eischer
42c9318b9c
repair pack: add tests
2024-01-27 12:51:45 +01:00
Michael Eischer
764b0bacd6
repair pack: add support for truncated files
2024-01-27 12:51:45 +01:00
Michael Eischer
7c351bc53c
repair pack: reenable auto index updates
...
The method is not available on the restic.Repository interface that is
used for testing. Drop the call as a small amount of additional index
writes is not a problem.
2024-01-27 12:51:45 +01:00
Michael Eischer
feeab84204
repair pack: extract the repair logic into the repository package
...
Currently, the cmd/restic package contains a significant amount of code
that modifies repository internals. This code should in the mid-term
move into the repository package.
2024-01-27 12:51:45 +01:00
Michael Eischer
cb50832d50
index: let MasterIndex.Save also delete obsolete indexes
2024-01-27 12:51:08 +01:00
Michael Eischer
c13bf0b607
repository: Introduce RemoveKey function
...
This replaces directly removing keys via the backend.
2024-01-27 12:42:58 +01:00
Michael Eischer
2c310a526e
repository: Replace StreamPack function with LoadBlobsFromPack method
...
LoadBlobsFromPack is now part of the repository struct. This ensures
that users of that method don't have to deal will internals of the
repository implementation.
The filerestorer tests now also contain far fewer pack file
implementation details.
2024-01-19 21:40:43 +01:00
Michael Eischer
6b7b5c89e9
repository: prepare StreamPack refactor
2024-01-19 21:40:43 +01:00
Michael Eischer
fb422497af
repository: split StreamPack implementation
...
Move the actual decoding of the pack data into a separate iterator.
2024-01-19 21:39:55 +01:00
Michael Eischer
77b1c52673
repository: test that StreamPack only delivers blobs once
2024-01-07 10:54:53 +01:00
Michael Eischer
fe5c337ca2
repository: StreamPack delivers blobs at most once
...
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
2024-01-07 10:54:49 +01:00
Andrea Gelmini
241916d55b
Fix typos
2023-12-06 13:11:55 +01:00
Michael Eischer
45962c2847
Merge pull request #4499 from MichaelEischer/modular-backend-code
...
Split backend code from restic package
2023-10-27 20:19:20 +02:00
Leo R. Lundgren
aafb806a8c
doc: Correct two typos
2023-10-27 18:56:32 +02:00
Michael Eischer
c7b770eb1f
convert MemorizeList to be repository based
...
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
2023-10-25 23:01:35 +02:00
Michael Eischer
1b8a67fe76
move Backend interface to backend package
2023-10-25 23:00:18 +02:00
Michael Eischer
b6d79bdf6f
restic: decouple restic.Handle
2023-10-25 22:54:07 +02:00
Michael Eischer
cb9cbe55d9
repository: store oversized blobs in separate pack files
...
Store oversized blobs in separate pack files as the blobs is large
enough to warrant its own pack file. This simplifies the garbage
collection of such blobs and keeps the cache smaller, as oversize (tree)
blobs only have to be downloaded if they are actually used.
2023-10-17 22:52:16 +02:00
Michael Eischer
3fd0ad7448
repository: list index files only once
2023-10-01 19:53:26 +02:00
arjunajesh
ed65a7dbca
implement progress bar for index loading
2023-10-01 19:52:59 +02:00
Michael Eischer
191c47d30e
Merge pull request #4353 from MichaelEischer/tune-gc
...
Tune Go garbage collector
2023-06-16 23:24:39 +02:00
Michael Eischer
eef0ee7a85
repository: trigger GC after loading the index
...
Loading the index requires some scratch space, thus make sure that this
memory does not factor into the targeted gc memory usage limit.
2023-06-02 21:56:14 +02:00
Michael Eischer
ffca602315
repository: Fix panic in benchmarkLoadIndex
2023-05-28 23:55:47 +02:00
Michael Eischer
d1a5ec7839
Rename unused testing parameter to _
...
The parameter is an additional marker that the test helper must only be
used for tests.
2023-05-18 21:17:53 +02:00
Michael Eischer
1514593f22
Remove unused context or testing parameters
2023-05-18 21:17:53 +02:00
Michael Eischer
e01baeabba
Use either test or rtest to refer to internal test helpers
...
A single test file should not use both names.
2023-05-18 21:15:45 +02:00
