octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-12-29 13:22:43 +00:00
Code Issues Releases Wiki Activity
7,871 Commits 7 Branches 33 Tags 66 MiB
7ed560a201
Commit Graph

53 Commits

Author SHA1 Message Date
Michael Eischer
940a3159b5 let index.Each() and pack.Size() return error on canceled context 
This forces a caller to actually check that the function did complete.
 2024-04-22 22:39:32 +02:00
Michael Eischer
31624aeffd Improve command shutdown on context cancellation 2024-04-22 22:31:38 +02:00
Michael Eischer
10355c3fb6 repository: Better error message if blob is larger than 4GB 2024-04-19 22:00:35 +02:00
Michael Eischer
dc441c57a7 repository: unify repository initialization in tests 
Tests should use a helper from internal/repository/testing.go to
construct a Repository object.
 2024-03-28 23:17:02 +01:00
Michael Eischer
f8852f0eb6 repair index: fix deletion of legacy indexes 2024-03-09 18:21:22 +01:00
Michael Eischer
1a8bf358f1 index: deprecate legacy index format 2024-03-09 18:21:14 +01:00
Alexander Neumann
c0514dd8ba Fix linter errors (except for tests) 2024-02-10 22:58:10 +01:00
Michael Eischer
bfb56b78e1 replace some usages of restic.Repository with more specific interface 
This should eventually make it easier to test the code.
 2024-01-27 13:02:02 +01:00
Michael Eischer
cb50832d50 index: let MasterIndex.Save also delete obsolete indexes 2024-01-27 12:51:08 +01:00
Andrea Gelmini
241916d55b
Fix typos 2023-12-06 13:11:55 +01:00
Michael Eischer
c7b770eb1f convert MemorizeList to be repository based 
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
 2023-10-25 23:01:35 +02:00
Michael Eischer
1b8a67fe76 move Backend interface to backend package 2023-10-25 23:00:18 +02:00
Michael Eischer
91aef00df3 check: add index loading progress bar 2023-10-01 19:55:29 +02:00
Michael Eischer
3fd0ad7448 repository: list index files only once 2023-10-01 19:53:26 +02:00
arjunajesh
ed65a7dbca implement progress bar for index loading 2023-10-01 19:52:59 +02:00
Michael Eischer
2bec99dc6f master_index: fix inconsistent length blob length in test 
Two blobs with the same hash must always have the same content length.
 2023-08-19 20:04:25 +02:00
Michael Eischer
090f9d6237 restic: Cleanup and simplify TestCreateSnapshot 2023-07-22 19:55:57 +02:00
Michael Eischer
b2ed42cec4 index: add basic hat test 2023-06-16 23:12:30 +02:00
Michael Eischer
55c21846b1 Revert "index: remove redundant storage of indexmap size" 
This reverts commit f1c388c623.

For an uninitialized indexmap the returned size was `-1` which is
unexpected and could cause problems.
 2023-06-08 18:08:46 +02:00
Michael Eischer
ac1dfc99bb index: fix blocklist size 2023-06-02 19:39:12 +02:00
Michael Eischer
9a7056a479 index: implement indexmap.grow() without random access 2023-05-30 20:13:33 +02:00
Michael Eischer
fc05e35a08 index: let indexmap.Each iterate in allocation order 
Iterating through the indexmap according to the bucket order has the
problem that all indexEntries are accessed in random order which is
rather cache inefficient.

As we already keep a list of all allocated blocks, just iterate through
it. This allows iterating through a batch of indexEntries without random
memory accesses. In addition, the packID will likely remain similar
across multiple blobs as all blobs of a pack file are added as a single
batch.
 2023-05-30 20:12:36 +02:00
Michael Eischer
f1c388c623 index: remove redundant storage of indexmap size 2023-05-30 20:11:53 +02:00
Michael Eischer
12141afbad index: Allow inlining of HAT 2023-05-30 20:11:14 +02:00
Michael Eischer
fed33295c3 index: store indexEntries in hashed array tree 
This data structure reduces the wasted memory to O(sqrt(n)). The
top-layer of the hashed array tree (HAT) also has a size of O(sqrt(n)),
which makes it cache efficient. The top-layer should be small enough to
easily fit into the CPU cache and thus only adds little overhead
compared to directly accessing an index entry via a pointer.
 2023-05-29 00:24:15 +02:00
Michael Eischer
b217f38ee7 index: Remove pointers from within indexentrys 
The indexEntry objects are now allocated in a separate array. References
to an indexEntry are now stored as array indices. This has the benefit
of allowing the garbage collector to ignore the indexEntry objects as
these do not contain pointers and are part of a single large allocation.
 2023-05-29 00:24:15 +02:00
Michael Eischer
0c1240360d index: add garbage collection benchmark 
Allocates an index and repeatedly triggers the GC.
 2023-05-29 00:23:04 +02:00
Michael Eischer
0058745881 test: use parameter instead of hardcoded constant 2023-05-18 21:17:53 +02:00
greatroar
d129baba7a repository: Reuse buffers in Repository.LoadUnpacked 
This method had a buffer argument, but that was nil at all call sites.
That's removed, and instead LoadUnpacked now reuses whatever it
allocates inside its retry loop.
 2023-01-30 22:01:01 +01:00
greatroar
99755c634b index: Optimize generatePackList 
name                 old time/op    new time/op    delta
EncodeIndex/100-8      1.56ms ± 2%    1.48ms ± 3%   -5.37%  (p=0.000 n=10+10)
EncodeIndex/1000-8     14.5ms ± 2%    13.1ms ± 2%   -9.49%  (p=0.000 n=9+10)
EncodeIndex/10000-8     120ms ± 2%     116ms ± 2%   -3.58%  (p=0.000 n=10+10)

name                 old alloc/op   new alloc/op   delta
EncodeIndex/100-8       306kB ± 1%     275kB ± 1%  -10.28%  (p=0.000 n=10+10)
EncodeIndex/1000-8     3.69MB ±11%    2.88MB ± 5%  -22.07%  (p=0.000 n=10+9)
EncodeIndex/10000-8    35.9MB ±11%    31.9MB ±10%  -11.13%  (p=0.005 n=10+10)

name                 old allocs/op  new allocs/op  delta
EncodeIndex/100-8       3.39k ± 0%     2.39k ± 0%  -29.61%  (p=0.000 n=10+10)
EncodeIndex/1000-8      32.6k ± 0%     22.9k ± 0%  -29.63%  (p=0.000 n=10+9)
EncodeIndex/10000-8      326k ± 0%      229k ± 0%  -29.71%  (p=0.000 n=10+10)

The bulk of the allocation rate improvement comes from just removing the
debug.Log calls: every one of those copied a restic.ID to the heap.
 2023-01-14 20:41:07 +01:00
greatroar
c0b5ec55ab repository: Remove empty cleanup functions in tests 
TestRepository and its variants always returned no-op cleanup functions.
If they ever do need to do cleanup, using testing.T.Cleanup is easier
than passing these functions around.
 2022-12-11 11:06:25 +01:00
greatroar
0e8893dae9 index: Compact data structure for Index.EachByPack 2022-10-29 23:09:17 +02:00
Michael Eischer
ae45f3b04f restic: Unify code to load Index/Lock/Snapshot 2022-10-21 21:25:11 +02:00
Michael Eischer
2e3f1c08c5 repository: split index into a separate package 2022-10-08 21:15:34 +02:00
Alexander Weiss
9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
greatroar
21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter 
This fixes two race conditions while cleaning up the code.
 2020-11-09 12:12:35 +01:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" (#2885) 
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
 2020-08-16 11:16:38 +02:00
greatroar
9f7cd69f13 Move Index.FindBlob to tests 2020-04-29 10:57:01 +02:00
Alexander Neumann
0b600d6cef index.New: Test various error conditions 2018-10-28 14:02:31 +01:00
Alexander Neumann
f7c0893f76 index: Add tests for error conditions 2018-10-28 13:16:36 +01:00
Alexander Neumann
c3de301fc8 index: Use interfaces instead of *repository.Repository 2018-10-28 12:58:34 +01:00
Alexander Neumann
944b446ac0 index: Cancel context for tests 2018-10-28 11:17:04 +01:00
Alexander Neumann
b096fc7abf index: Correctly process errors listing all files 
This also removes the now unused `list` and `worker` packages.
 2018-10-28 11:16:29 +01:00
Alexander Neumann
ff3de66ddf Merge pull request #1582 from restic/optimize-debug-log 
Optimize debug logs
 2018-01-26 21:57:18 +01:00
Alexander Neumann
f63c7048c7 index: Limit index files to 3000 pack files 2018-01-26 21:07:16 +01:00
Alexander Neumann
ca3c566a99 index: Check structure after saving a new index 2018-01-26 21:07:16 +01:00
Alexander Neumann
663c57ab4d debug: Remove manual Str() call Log() 2018-01-25 20:49:41 +01:00
Igor Fedorenko
0084e42cb6 Optimize Repository.ListPack() 
Use pack file size returned by Backend.List() to avoid extra per-pack
Backend.Stat() requests

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-23 22:39:51 -05:00
Alexander Neumann
b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00