Michael Eischer
57d69aa640
index: cleanup SaveIndex method
2024-05-24 21:33:17 +02:00
Michael Eischer
864995271e
repository: unwrap BlobHandle parameters of LookupBlob
...
The method now uses the same parameters as LookupBlobSize.
2024-05-24 21:33:17 +02:00
Michael Eischer
4df887406f
repository: inline MasterIndex interface into Repository interface
2024-05-24 21:33:17 +02:00
Michael Eischer
8a425c2f0a
remove usages of repo.Backend() from tests
2024-05-18 21:42:51 +02:00
Michael Eischer
d8b184b3d3
repository: convert test helper to return *repository.Repository
2024-05-18 21:38:31 +02:00
Michael Eischer
8f8d872a68
fix compatibility with go 1.19
2024-05-18 21:28:54 +02:00
Michael Eischer
987c3b250c
repository: test retries of ListPack
2024-05-18 21:28:54 +02:00
Michael Eischer
bf16096771
repository: test LoadBlob retries
2024-05-18 21:28:54 +02:00
Michael Eischer
2ace242f36
repository: make reloading broken files explicit
2024-05-18 21:28:54 +02:00
Michael Eischer
1d6d3656b0
repository: move backend.LoadAll to repository.LoadRaw
...
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.
The removed cache backend test will be added again later on.
2024-05-18 21:26:00 +02:00
Michael Eischer
940a3159b5
let index.Each() and pack.Size() return error on canceled context
...
This forces a caller to actually check that the function did complete.
2024-04-22 22:39:32 +02:00
Michael Eischer
c65459cd8a
repository: speed up tests
2024-04-14 13:45:10 +02:00
Michael Eischer
dc441c57a7
repository: unify repository initialization in tests
...
Tests should use a helper from internal/repository/testing.go to
construct a Repository object.
2024-03-28 23:17:02 +01:00
Michael Eischer
bfb56b78e1
replace some usages of restic.Repository with more specific interface
...
This should eventually make it easier to test the code.
2024-01-27 13:02:02 +01:00
Michael Eischer
fd579421dd
repository: deduplicate test
2024-01-27 12:51:45 +01:00
Michael Eischer
6b7b5c89e9
repository: prepare StreamPack refactor
2024-01-19 21:40:43 +01:00
Michael Eischer
77b1c52673
repository: test that StreamPack only delivers blobs once
2024-01-07 10:54:53 +01:00
Andrea Gelmini
241916d55b
Fix typos
2023-12-06 13:11:55 +01:00
Michael Eischer
1b8a67fe76
move Backend interface to backend package
2023-10-25 23:00:18 +02:00
arjunajesh
ed65a7dbca
implement progress bar for index loading
2023-10-01 19:52:59 +02:00
Michael Eischer
ffca602315
repository: Fix panic in benchmarkLoadIndex
2023-05-28 23:55:47 +02:00
Michael Eischer
1514593f22
Remove unused context or testing parameters
2023-05-18 21:17:53 +02:00
Michael Eischer
e01baeabba
Use either test or rtest to refer to internal test helpers
...
A single test file should not use both names.
2023-05-18 21:15:45 +02:00
greatroar
d129baba7a
repository: Reuse buffers in Repository.LoadUnpacked
...
This method had a buffer argument, but that was nil at all call sites.
That's removed, and instead LoadUnpacked now reuses whatever it
allocates inside its retry loop.
2023-01-30 22:01:01 +01:00
greatroar
c0b5ec55ab
repository: Remove empty cleanup functions in tests
...
TestRepository and its variants always returned no-op cleanup functions.
If they ever do need to do cleanup, using testing.T.Cleanup is easier
than passing these functions around.
2022-12-11 11:06:25 +01:00
Alexander Neumann
8dd95b710e
Merge pull request #3992 from MichaelEischer/err-on-invalid-compression
...
Return error if RESTIC_COMPRESSION env variable is invalid
2022-11-04 19:41:34 +01:00
greatroar
137f0bc944
repository: Fix benchmarkSaveAndEncrypt
2022-10-29 23:09:17 +02:00
Michael Eischer
01f0db4e56
return error if RESTIC_COMPRESSION env variable is invalid
2022-10-29 22:03:39 +02:00
Michael Eischer
2e3f1c08c5
repository: split index into a separate package
2022-10-08 21:15:34 +02:00
Michael Eischer
5c6b6edefe
retry index, lock and snapshot loading on hash mismatch
2022-09-25 11:35:35 +02:00
Michael Eischer
1ebd57247a
repository: optimize MasterIndex.Each
...
Sending data through a channel at very high frequency is extremely
inefficient. Thus use simple callbacks instead of channels.
> name old time/op new time/op delta
> MasterIndexEach-16 6.68s ±24% 0.96s ± 2% -85.64% (p=0.008 n=5+5)
2022-09-24 12:21:59 +02:00
Michael Eischer
7266f07c87
repository: StreamPack in parts if there are too large gaps
...
For large pack sizes we might be only interested in the first and last
blob of a pack file. Thus stream a pack file in multiple parts if the
gaps between requested blobs grow too large.
2022-08-05 23:48:36 +02:00
MichaelEischer
443cc49afd
Merge pull request #3830 from MichaelEischer/cleanup-repo
...
Extract Load/SaveTree/JSONUnpacked from repository
2022-07-23 10:46:13 +02:00
Michael Eischer
8c11fc3ec9
crypto: move crypto buffer helpers
2022-07-17 13:42:23 +02:00
Michael Eischer
89d3ce852b
repository: extract Load/StoreJSONUnpacked
...
A Load/Store method for each data type is much clearer. As a result the
repository no longer needs a method to load / store json.
2022-07-17 13:22:00 +02:00
Michael Eischer
fbcbd5318c
repository: extract LoadTree/SaveTree
...
The repository has no real idea what a Tree is. So these methods never
belonged there.
2022-07-17 13:11:28 +02:00
Michael Eischer
120ccc8754
repository: Rework blob saving to use an async pack uploader
...
Previously, SaveAndEncrypt would assemble blobs into packs and either
return immediately if the pack is not yet full or upload the pack file
otherwise. The upload will block the current goroutine until it
finishes.
Now, the upload is done using separate goroutines. This requires changes
to the error handling. As uploads are no longer tied to a SaveAndEncrypt
call, failed uploads are signaled using an errgroup.
To count the uploaded amount of data, the pack header overhead is no
longer returned by `packer.Finalize` but rather by
`packer.HeaderOverhead`. This helper method is necessary to continue
returning the pack header overhead directly to the responsible call to
`repository.SaveBlob`. Without the method this would not be possible,
as packs are finalized asynchronously.
2022-07-02 22:42:34 +02:00
Alexander Neumann
99634c0936
Return real size from SaveBlob
2022-07-02 18:55:12 +02:00
Michael Eischer
e0a7852b8b
repository: remove unused (Master)Index.Count
2022-07-02 18:39:58 +02:00
Michael Eischer
fe5a8e137a
repository: remove unused index.Store
2022-07-02 18:39:12 +02:00
Michael Eischer
628ae799ca
repository: make flushPacks private
2022-07-02 18:39:12 +02:00
Michael Eischer
a77d5c4d11
repository: index saving belongs into the MasterIndex
2022-07-02 18:38:56 +02:00
Michael Eischer
4b01b06f2f
repository: Test compressed blobs in StreamPack
2022-04-30 11:34:10 +02:00
Michael Eischer
9ffb8920f1
repository: run blackbox tests using old and new repo version
2022-04-30 11:34:10 +02:00
Michael Eischer
7132df529e
repository: Increase index size for repo version 2
...
A compressed index is only about one third the size of an uncompressed
one. Thus increase the number of entries in an index to avoid cluttering
the repository with small indexes.
2022-04-30 11:34:10 +02:00
Michael Eischer
6877e7edbb
repository: Rename LoadAndDecrypt to LoadUnpacked
...
The method is the complement for SaveUnpacked and not for
SaveAndEncrypt. The latter assembles blobs into pack files.
2022-03-28 22:09:49 +02:00
Alexander Neumann
e682f7c0d6
Add tests for StreamPack
2022-03-21 21:15:03 +01:00
Michael Eischer
becebf5d88
repository: remove unused DownloadAndHash
2022-02-12 20:18:25 +01:00
Michael Eischer
a12c5f1d37
repository: move otherwise unused LoadIndex to tests
2020-12-22 22:36:18 +01:00
Alexander Neumann
7facc8ccc1
Merge pull request #2505 from aawsome/fix-repo-configfile
...
Fix repo configfile
2020-12-07 07:52:37 +01:00
