Michael Eischer
d2c26e33f3
repository: remove further usages of repo.Backend()
2024-05-18 21:42:51 +02:00
Michael Eischer
8a425c2f0a
remove usages of repo.Backend() from tests
2024-05-18 21:42:51 +02:00
Michael Eischer
aa4647f773
repository: unexport PackBlobIterator
2024-05-18 21:42:51 +02:00
Michael Eischer
94e863885c
check: move verification of individual pack file to repository
2024-05-18 21:42:50 +02:00
Michael Eischer
67e2ba0d40
repository: Lock requires *repository.Repository
...
This allows the Lock function to access the backend, even once the
Backend method is removed from the interface.
2024-05-18 21:38:31 +02:00
Michael Eischer
d8b184b3d3
repository: convert test helper to return *repository.Repository
2024-05-18 21:38:31 +02:00
Michael Eischer
a1ca5e15c4
migrations: add temporary hack for s3_layout
...
The migration will be removed after the next restic release anyways.
Thus, there's no need for a clean implementation.
2024-05-18 21:38:31 +02:00
Michael Eischer
34d90aecf9
migrations: move logic of upgrade_repo_v2 to repository package
...
The migration modifies repository internals and thus should live within
the repository package.
2024-05-18 21:38:31 +02:00
Michael Eischer
ab9077bc13
replace usages of backend.Remove() with repository.RemoveUnpacked()
...
RemoveUnpacked will eventually block removal of all filetypes other than
snapshots. However, getting there requires a major refactor to provide
some components with privileged access.
2024-05-18 21:38:31 +02:00
Michael Eischer
8f8d872a68
fix compatibility with go 1.19
2024-05-18 21:28:54 +02:00
Michael Eischer
987c3b250c
repository: test retries of ListPack
2024-05-18 21:28:54 +02:00
Michael Eischer
bf16096771
repository: test LoadBlob retries
2024-05-18 21:28:54 +02:00
Michael Eischer
4f45668b7c
repository: rework and extend LoadRaw tests
2024-05-18 21:28:54 +02:00
Michael Eischer
385cee09dc
repository: fix caching of tree packs in LoadBlobsFromPack
2024-05-18 21:28:54 +02:00
Michael Eischer
433a6aad29
repository: remove redundant blob loading fallback from RepairPacks
...
LoadBlobsFromPack already implements the same fallback behavior.
2024-05-18 21:28:54 +02:00
Michael Eischer
7017adb7e9
repository: retry failed ListPack once
2024-05-18 21:28:54 +02:00
Michael Eischer
e33ce7f408
repository: retry failed LoadBlob once
2024-05-18 21:28:54 +02:00
Michael Eischer
2ace242f36
repository: make reloading broken files explicit
2024-05-18 21:28:54 +02:00
Michael Eischer
503c8140b1
repository: unify blob decoding code
2024-05-18 21:26:00 +02:00
Michael Eischer
6563f1d2ca
repository: remove redundant debug log
2024-05-18 21:26:00 +02:00
Michael Eischer
021fb49559
repository: Implement repository.LoadUnpacked using LoadRaw
...
Both functions were using a similar implementation.
2024-05-18 21:26:00 +02:00
Michael Eischer
1d6d3656b0
repository: move backend.LoadAll to repository.LoadRaw
...
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.
The removed cache backend test will be added again later on.
2024-05-18 21:26:00 +02:00
Michael Eischer
6328b7e1f5
replace "too small" with "too short" in error messages
2024-05-18 19:59:26 +02:00
Michael Eischer
ffe5439149
Merge pull request #4605 from MichaelEischer/better-restorer-error-handling
...
Rework repository.StreamPacks & better restorer error handling
2024-05-01 16:37:41 +02:00
Michael Eischer
940a3159b5
let index.Each() and pack.Size() return error on canceled context
...
This forces a caller to actually check that the function did complete.
2024-04-22 22:39:32 +02:00
Michael Eischer
31624aeffd
Improve command shutdown on context cancellation
2024-04-22 22:31:38 +02:00
Michael Eischer
b15d867414
Merge pull request #4763 from MichaelEischer/refactor-prune
...
Refactor repair index / prune into the repository package
2024-04-22 22:24:53 +02:00
Michael Eischer
20d8eed400
repository: streamPack: separate requests for gap larger than 1MB
...
With most cloud providers, traffic is much more expensive than API
calls. Thus slightly bias streamPack towards a bit more API calls in
exchange for slightly less traffic.
2024-04-22 21:21:23 +02:00
Michael Eischer
cf700d8794
repository: streamPack: reuse zstd decoder
2024-04-22 21:21:23 +02:00
Michael Eischer
666a0b0bdb
repository: streamPack: replace streaming with chunked download
...
Due to the interface of streamPack, we cannot guarantee that operations
progress fast enough that the underlying connections remains open. This
introduces partial failures which massively complicate the error
handling.
Switch to a simpler approach that retrieves the pack in chunks of 32MB.
If a blob is larger than this limit, then it is downloaded separately.
To avoid multiple copies in memory, an auxiliary interface
`discardReader` is introduced that allows directly accessing the
downloaded byte slices, while still supporting the streaming used by the
`check` command.
2024-04-22 21:21:23 +02:00
Michael Eischer
621012dac0
repository: Add blob loading fallback to LoadBlobsFromPack
...
Try to retrieve individual blobs via LoadBlob if streaming did not work.
2024-04-21 21:35:55 +02:00
Michael Eischer
10355c3fb6
repository: Better error message if blob is larger than 4GB
2024-04-19 22:00:35 +02:00
Michael Eischer
09587e6c08
repository: duplicate a few blobs in prune tests
2024-04-14 13:57:19 +02:00
Michael Eischer
defd7ae729
prune/repair index: reset in-memory index after command
...
The current in-memory index becomes stale after prune or repair index
have run. Thus, just drop the in-memory index altogether once these
commands have finished.
2024-04-14 13:46:24 +02:00
Michael Eischer
038586dc9d
repository: add minimal test for prune
2024-04-14 13:45:17 +02:00
Michael Eischer
d8622c86eb
prune: clean up internal interface
2024-04-14 13:45:15 +02:00
Michael Eischer
8d507c1372
repository: add basic test for RepairIndex
2024-04-14 13:45:15 +02:00
Michael Eischer
310db03c0e
repair index: improve log output if index cannot be deleted
...
The operation will always fail with an error if an index cannot be
deleted. Thus, this change is purely cosmetic.
2024-04-14 13:45:13 +02:00
Michael Eischer
7d1b9cde34
repository: use normal Init method in tests
2024-04-14 13:45:11 +02:00
Michael Eischer
b25fc2c89d
repository: remove redundant flushes from tests
2024-04-14 13:45:10 +02:00
Michael Eischer
c65459cd8a
repository: speed up tests
2024-04-14 13:45:10 +02:00
Michael Eischer
4c9a10ca37
repair packs: deduplicate index rebuild
2024-04-14 13:45:02 +02:00
Michael Eischer
85e4021619
prune: move additional option checks to repository
2024-04-14 13:44:58 +02:00
Michael Eischer
fc3b548625
prune: move logic into repository package
2024-04-10 21:30:52 +02:00
Michael Eischer
866ddf5698
repair index: refactor code into repository package
2024-04-10 21:30:52 +02:00
Michael Eischer
5e98f1e2eb
repository: fix test setup race conditions
2024-03-28 23:17:02 +01:00
Michael Eischer
dc441c57a7
repository: unify repository initialization in tests
...
Tests should use a helper from internal/repository/testing.go to
construct a Repository object.
2024-03-28 23:17:02 +01:00
Michael Eischer
3ba1fa3cee
repository: remove a few global variables
2024-03-28 23:17:02 +01:00
Michael Eischer
044e8bf821
repository: parallelize lock tests
2024-03-28 23:17:02 +01:00
Michael Eischer
e8df50fa3c
repository: remove global list of locks
2024-03-28 22:46:33 +01:00
