octoleo
/
restic
mirror of https://github.com/octoleo/restic.git
2
2
Fork 0
Code Issues Releases Wiki Activity
7,630 Commits 7 Branches 33 Tags 64 MiB
Commit Graph

46 Commits

Author SHA1 Message Date
Michael Eischer bfb56b78e1 replace some usages of restic.Repository with more specific interface 
This should eventually make it easier to test the code.
 2024-01-27 13:02:02 +01:00
Michael Eischer cb50832d50 index: let MasterIndex.Save also delete obsolete indexes 2024-01-27 12:51:08 +01:00
Andrea Gelmini 241916d55b
Fix typos 2023-12-06 13:11:55 +01:00
Michael Eischer c7b770eb1f convert MemorizeList to be repository based 
Ideally, code that uses a repository shouldn't directly interact with
the underlying backend. Thus, move MemorizeList one layer up.
 2023-10-25 23:01:35 +02:00
Michael Eischer 1b8a67fe76 move Backend interface to backend package 2023-10-25 23:00:18 +02:00
Michael Eischer 91aef00df3 check: add index loading progress bar 2023-10-01 19:55:29 +02:00
Michael Eischer 3fd0ad7448 repository: list index files only once 2023-10-01 19:53:26 +02:00
arjunajesh ed65a7dbca implement progress bar for index loading 2023-10-01 19:52:59 +02:00
Michael Eischer 2bec99dc6f master_index: fix inconsistent length blob length in test 
Two blobs with the same hash must always have the same content length.
 2023-08-19 20:04:25 +02:00
Michael Eischer 090f9d6237 restic: Cleanup and simplify TestCreateSnapshot 2023-07-22 19:55:57 +02:00
Michael Eischer b2ed42cec4 index: add basic hat test 2023-06-16 23:12:30 +02:00
Michael Eischer 55c21846b1 Revert "index: remove redundant storage of indexmap size" 
This reverts commit f1c388c623.

For an uninitialized indexmap the returned size was `-1` which is
unexpected and could cause problems.
 2023-06-08 18:08:46 +02:00
Michael Eischer ac1dfc99bb index: fix blocklist size 2023-06-02 19:39:12 +02:00
Michael Eischer 9a7056a479 index: implement indexmap.grow() without random access 2023-05-30 20:13:33 +02:00
Michael Eischer fc05e35a08 index: let indexmap.Each iterate in allocation order 
Iterating through the indexmap according to the bucket order has the
problem that all indexEntries are accessed in random order which is
rather cache inefficient.

As we already keep a list of all allocated blocks, just iterate through
it. This allows iterating through a batch of indexEntries without random
memory accesses. In addition, the packID will likely remain similar
across multiple blobs as all blobs of a pack file are added as a single
batch.
 2023-05-30 20:12:36 +02:00
Michael Eischer f1c388c623 index: remove redundant storage of indexmap size 2023-05-30 20:11:53 +02:00
Michael Eischer 12141afbad index: Allow inlining of HAT 2023-05-30 20:11:14 +02:00
Michael Eischer fed33295c3 index: store indexEntries in hashed array tree 
This data structure reduces the wasted memory to O(sqrt(n)). The
top-layer of the hashed array tree (HAT) also has a size of O(sqrt(n)),
which makes it cache efficient. The top-layer should be small enough to
easily fit into the CPU cache and thus only adds little overhead
compared to directly accessing an index entry via a pointer.
 2023-05-29 00:24:15 +02:00
Michael Eischer b217f38ee7 index: Remove pointers from within indexentrys 
The indexEntry objects are now allocated in a separate array. References
to an indexEntry are now stored as array indices. This has the benefit
of allowing the garbage collector to ignore the indexEntry objects as
these do not contain pointers and are part of a single large allocation.
 2023-05-29 00:24:15 +02:00
Michael Eischer 0c1240360d index: add garbage collection benchmark 
Allocates an index and repeatedly triggers the GC.
 2023-05-29 00:23:04 +02:00
Michael Eischer 0058745881 test: use parameter instead of hardcoded constant 2023-05-18 21:17:53 +02:00
greatroar d129baba7a repository: Reuse buffers in Repository.LoadUnpacked 
This method had a buffer argument, but that was nil at all call sites.
That's removed, and instead LoadUnpacked now reuses whatever it
allocates inside its retry loop.
 2023-01-30 22:01:01 +01:00
greatroar 99755c634b index: Optimize generatePackList 
name                 old time/op    new time/op    delta
EncodeIndex/100-8      1.56ms ± 2%    1.48ms ± 3%   -5.37%  (p=0.000 n=10+10)
EncodeIndex/1000-8     14.5ms ± 2%    13.1ms ± 2%   -9.49%  (p=0.000 n=9+10)
EncodeIndex/10000-8     120ms ± 2%     116ms ± 2%   -3.58%  (p=0.000 n=10+10)

name                 old alloc/op   new alloc/op   delta
EncodeIndex/100-8       306kB ± 1%     275kB ± 1%  -10.28%  (p=0.000 n=10+10)
EncodeIndex/1000-8     3.69MB ±11%    2.88MB ± 5%  -22.07%  (p=0.000 n=10+9)
EncodeIndex/10000-8    35.9MB ±11%    31.9MB ±10%  -11.13%  (p=0.005 n=10+10)

name                 old allocs/op  new allocs/op  delta
EncodeIndex/100-8       3.39k ± 0%     2.39k ± 0%  -29.61%  (p=0.000 n=10+10)
EncodeIndex/1000-8      32.6k ± 0%     22.9k ± 0%  -29.63%  (p=0.000 n=10+9)
EncodeIndex/10000-8      326k ± 0%      229k ± 0%  -29.71%  (p=0.000 n=10+10)

The bulk of the allocation rate improvement comes from just removing the
debug.Log calls: every one of those copied a restic.ID to the heap.
 2023-01-14 20:41:07 +01:00
greatroar c0b5ec55ab repository: Remove empty cleanup functions in tests 
TestRepository and its variants always returned no-op cleanup functions.
If they ever do need to do cleanup, using testing.T.Cleanup is easier
than passing these functions around.
 2022-12-11 11:06:25 +01:00
greatroar 0e8893dae9 index: Compact data structure for Index.EachByPack 2022-10-29 23:09:17 +02:00
Michael Eischer ae45f3b04f restic: Unify code to load Index/Lock/Snapshot 2022-10-21 21:25:11 +02:00
Michael Eischer 2e3f1c08c5 repository: split index into a separate package 2022-10-08 21:15:34 +02:00
Alexander Weiss 9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
greatroar 21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar ddca699cd2 Replace restic.Progress with new progress.Counter 
This fixes two race conditions while cleaning up the code.
 2020-11-09 12:12:35 +01:00
aawsome 0fed6a8dfc
Use "pack file" instead of "data file" (#2885) 
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
 2020-08-16 11:16:38 +02:00
greatroar 9f7cd69f13 Move Index.FindBlob to tests 2020-04-29 10:57:01 +02:00
Alexander Neumann 0b600d6cef index.New: Test various error conditions 2018-10-28 14:02:31 +01:00
Alexander Neumann f7c0893f76 index: Add tests for error conditions 2018-10-28 13:16:36 +01:00
Alexander Neumann c3de301fc8 index: Use interfaces instead of *repository.Repository 2018-10-28 12:58:34 +01:00
Alexander Neumann 944b446ac0 index: Cancel context for tests 2018-10-28 11:17:04 +01:00
Alexander Neumann b096fc7abf index: Correctly process errors listing all files 
This also removes the now unused `list` and `worker` packages.
 2018-10-28 11:16:29 +01:00
Alexander Neumann ff3de66ddf Merge pull request #1582 from restic/optimize-debug-log 
Optimize debug logs
 2018-01-26 21:57:18 +01:00
Alexander Neumann f63c7048c7 index: Limit index files to 3000 pack files 2018-01-26 21:07:16 +01:00
Alexander Neumann ca3c566a99 index: Check structure after saving a new index 2018-01-26 21:07:16 +01:00
Alexander Neumann 663c57ab4d debug: Remove manual Str() call Log() 2018-01-25 20:49:41 +01:00
Igor Fedorenko 0084e42cb6 Optimize Repository.ListPack() 
Use pack file size returned by Backend.List() to avoid extra per-pack
Backend.Stat() requests

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-01-23 22:39:51 -05:00
Alexander Neumann b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00
Alexander Neumann 23c903074c Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
Alexander Neumann 6caeff2408 Run goimports 2017-07-23 14:21:03 +02:00
Alexander Neumann 83d1a46526 Moves files 2017-07-23 14:19:13 +02:00