octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-30 00:33:57 +00:00
Code Issues Releases Wiki Activity
7,243 Commits 7 Branches 33 Tags 66 MiB
b6593ad7df
Commit Graph

39 Commits

Author SHA1 Message Date
arjunajesh
ed65a7dbca implement progress bar for index loading 2023-10-01 19:52:59 +02:00
Michael Eischer
2bec99dc6f master_index: fix inconsistent length blob length in test 
Two blobs with the same hash must always have the same content length.
 2023-08-19 20:04:25 +02:00
Michael Eischer
090f9d6237 restic: Cleanup and simplify TestCreateSnapshot 2023-07-22 19:55:57 +02:00
Michael Eischer
b2ed42cec4 index: add basic hat test 2023-06-16 23:12:30 +02:00
Michael Eischer
55c21846b1 Revert "index: remove redundant storage of indexmap size" 
This reverts commit f1c388c623.

For an uninitialized indexmap the returned size was `-1` which is
unexpected and could cause problems.
 2023-06-08 18:08:46 +02:00
Michael Eischer
ac1dfc99bb index: fix blocklist size 2023-06-02 19:39:12 +02:00
Michael Eischer
9a7056a479 index: implement indexmap.grow() without random access 2023-05-30 20:13:33 +02:00
Michael Eischer
fc05e35a08 index: let indexmap.Each iterate in allocation order 
Iterating through the indexmap according to the bucket order has the
problem that all indexEntries are accessed in random order which is
rather cache inefficient.

As we already keep a list of all allocated blocks, just iterate through
it. This allows iterating through a batch of indexEntries without random
memory accesses. In addition, the packID will likely remain similar
across multiple blobs as all blobs of a pack file are added as a single
batch.
 2023-05-30 20:12:36 +02:00
Michael Eischer
f1c388c623 index: remove redundant storage of indexmap size 2023-05-30 20:11:53 +02:00
Michael Eischer
12141afbad index: Allow inlining of HAT 2023-05-30 20:11:14 +02:00
Michael Eischer
fed33295c3 index: store indexEntries in hashed array tree 
This data structure reduces the wasted memory to O(sqrt(n)). The
top-layer of the hashed array tree (HAT) also has a size of O(sqrt(n)),
which makes it cache efficient. The top-layer should be small enough to
easily fit into the CPU cache and thus only adds little overhead
compared to directly accessing an index entry via a pointer.
 2023-05-29 00:24:15 +02:00
Michael Eischer
b217f38ee7 index: Remove pointers from within indexentrys 
The indexEntry objects are now allocated in a separate array. References
to an indexEntry are now stored as array indices. This has the benefit
of allowing the garbage collector to ignore the indexEntry objects as
these do not contain pointers and are part of a single large allocation.
 2023-05-29 00:24:15 +02:00
Michael Eischer
0c1240360d index: add garbage collection benchmark 
Allocates an index and repeatedly triggers the GC.
 2023-05-29 00:23:04 +02:00
Michael Eischer
0058745881 test: use parameter instead of hardcoded constant 2023-05-18 21:17:53 +02:00
greatroar
d129baba7a repository: Reuse buffers in Repository.LoadUnpacked 
This method had a buffer argument, but that was nil at all call sites.
That's removed, and instead LoadUnpacked now reuses whatever it
allocates inside its retry loop.
 2023-01-30 22:01:01 +01:00
greatroar
99755c634b index: Optimize generatePackList 
name                 old time/op    new time/op    delta
EncodeIndex/100-8      1.56ms ± 2%    1.48ms ± 3%   -5.37%  (p=0.000 n=10+10)
EncodeIndex/1000-8     14.5ms ± 2%    13.1ms ± 2%   -9.49%  (p=0.000 n=9+10)
EncodeIndex/10000-8     120ms ± 2%     116ms ± 2%   -3.58%  (p=0.000 n=10+10)

name                 old alloc/op   new alloc/op   delta
EncodeIndex/100-8       306kB ± 1%     275kB ± 1%  -10.28%  (p=0.000 n=10+10)
EncodeIndex/1000-8     3.69MB ±11%    2.88MB ± 5%  -22.07%  (p=0.000 n=10+9)
EncodeIndex/10000-8    35.9MB ±11%    31.9MB ±10%  -11.13%  (p=0.005 n=10+10)

name                 old allocs/op  new allocs/op  delta
EncodeIndex/100-8       3.39k ± 0%     2.39k ± 0%  -29.61%  (p=0.000 n=10+10)
EncodeIndex/1000-8      32.6k ± 0%     22.9k ± 0%  -29.63%  (p=0.000 n=10+9)
EncodeIndex/10000-8      326k ± 0%      229k ± 0%  -29.71%  (p=0.000 n=10+10)

The bulk of the allocation rate improvement comes from just removing the
debug.Log calls: every one of those copied a restic.ID to the heap.
 2023-01-14 20:41:07 +01:00
greatroar
c0b5ec55ab repository: Remove empty cleanup functions in tests 
TestRepository and its variants always returned no-op cleanup functions.
If they ever do need to do cleanup, using testing.T.Cleanup is easier
than passing these functions around.
 2022-12-11 11:06:25 +01:00
greatroar
0e8893dae9 index: Compact data structure for Index.EachByPack 2022-10-29 23:09:17 +02:00
Michael Eischer
ae45f3b04f restic: Unify code to load Index/Lock/Snapshot 2022-10-21 21:25:11 +02:00
Michael Eischer
2e3f1c08c5 repository: split index into a separate package 2022-10-08 21:15:34 +02:00
Alexander Weiss
9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
greatroar
21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter 
This fixes two race conditions while cleaning up the code.
 2020-11-09 12:12:35 +01:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" (#2885) 
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
 2020-08-16 11:16:38 +02:00
greatroar
9f7cd69f13 Move Index.FindBlob to tests 2020-04-29 10:57:01 +02:00
Alexander Neumann
0b600d6cef index.New: Test various error conditions 2018-10-28 14:02:31 +01:00
Alexander Neumann
f7c0893f76 index: Add tests for error conditions 2018-10-28 13:16:36 +01:00
Alexander Neumann
c3de301fc8 index: Use interfaces instead of *repository.Repository 2018-10-28 12:58:34 +01:00
Alexander Neumann
944b446ac0 index: Cancel context for tests 2018-10-28 11:17:04 +01:00
Alexander Neumann
b096fc7abf index: Correctly process errors listing all files 
This also removes the now unused `list` and `worker` packages.
 2018-10-28 11:16:29 +01:00
Alexander Neumann
ff3de66ddf Merge pull request #1582 from restic/optimize-debug-log 
Optimize debug logs
 2018-01-26 21:57:18 +01:00
Alexander Neumann
f63c7048c7 index: Limit index files to 3000 pack files 2018-01-26 21:07:16 +01:00
Alexander Neumann
ca3c566a99 index: Check structure after saving a new index 2018-01-26 21:07:16 +01:00
Alexander Neumann
663c57ab4d debug: Remove manual Str() call Log() 2018-01-25 20:49:41 +01:00
Igor Fedorenko
0084e42cb6 Optimize Repository.ListPack() 
Use pack file size returned by Backend.List() to avoid extra per-pack
Backend.Stat() requests

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-23 22:39:51 -05:00
Alexander Neumann
b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00
Alexander Neumann
23c903074c Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
Alexander Neumann
6caeff2408 Run goimports 2017-07-23 14:21:03 +02:00
Alexander Neumann
83d1a46526 Moves files 2017-07-23 14:19:13 +02:00