Michael Eischer
fc05e35a08
index: let indexmap.Each iterate in allocation order
...
Iterating through the indexmap according to the bucket order has the
problem that all indexEntries are accessed in random order which is
rather cache inefficient.
As we already keep a list of all allocated blocks, just iterate through
it. This allows iterating through a batch of indexEntries without random
memory accesses. In addition, the packID will likely remain similar
across multiple blobs as all blobs of a pack file are added as a single
batch.
2023-05-30 20:12:36 +02:00
Michael Eischer
f1c388c623
index: remove redundant storage of indexmap size
2023-05-30 20:11:53 +02:00
Michael Eischer
12141afbad
index: Allow inlining of HAT
2023-05-30 20:11:14 +02:00
Michael Eischer
fed33295c3
index: store indexEntries in hashed array tree
...
This data structure reduces the wasted memory to O(sqrt(n)). The
top-layer of the hashed array tree (HAT) also has a size of O(sqrt(n)),
which makes it cache efficient. The top-layer should be small enough to
easily fit into the CPU cache and thus only adds little overhead
compared to directly accessing an index entry via a pointer.
2023-05-29 00:24:15 +02:00
Michael Eischer
b217f38ee7
index: Remove pointers from within indexentrys
...
The indexEntry objects are now allocated in a separate array. References
to an indexEntry are now stored as array indices. This has the benefit
of allowing the garbage collector to ignore the indexEntry objects as
these do not contain pointers and are part of a single large allocation.
2023-05-29 00:24:15 +02:00
Michael Eischer
0c1240360d
index: add garbage collection benchmark
...
Allocates an index and repeatedly triggers the GC.
2023-05-29 00:23:04 +02:00
Michael Eischer
0058745881
test: use parameter instead of hardcoded constant
2023-05-18 21:17:53 +02:00
greatroar
d129baba7a
repository: Reuse buffers in Repository.LoadUnpacked
...
This method had a buffer argument, but that was nil at all call sites.
That's removed, and instead LoadUnpacked now reuses whatever it
allocates inside its retry loop.
2023-01-30 22:01:01 +01:00
greatroar
99755c634b
index: Optimize generatePackList
...
name old time/op new time/op delta
EncodeIndex/100-8 1.56ms ± 2% 1.48ms ± 3% -5.37% (p=0.000 n=10+10)
EncodeIndex/1000-8 14.5ms ± 2% 13.1ms ± 2% -9.49% (p=0.000 n=9+10)
EncodeIndex/10000-8 120ms ± 2% 116ms ± 2% -3.58% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
EncodeIndex/100-8 306kB ± 1% 275kB ± 1% -10.28% (p=0.000 n=10+10)
EncodeIndex/1000-8 3.69MB ±11% 2.88MB ± 5% -22.07% (p=0.000 n=10+9)
EncodeIndex/10000-8 35.9MB ±11% 31.9MB ±10% -11.13% (p=0.005 n=10+10)
name old allocs/op new allocs/op delta
EncodeIndex/100-8 3.39k ± 0% 2.39k ± 0% -29.61% (p=0.000 n=10+10)
EncodeIndex/1000-8 32.6k ± 0% 22.9k ± 0% -29.63% (p=0.000 n=10+9)
EncodeIndex/10000-8 326k ± 0% 229k ± 0% -29.71% (p=0.000 n=10+10)
The bulk of the allocation rate improvement comes from just removing the
debug.Log calls: every one of those copied a restic.ID to the heap.
2023-01-14 20:41:07 +01:00
greatroar
c0b5ec55ab
repository: Remove empty cleanup functions in tests
...
TestRepository and its variants always returned no-op cleanup functions.
If they ever do need to do cleanup, using testing.T.Cleanup is easier
than passing these functions around.
2022-12-11 11:06:25 +01:00
greatroar
0e8893dae9
index: Compact data structure for Index.EachByPack
2022-10-29 23:09:17 +02:00
Michael Eischer
ae45f3b04f
restic: Unify code to load Index/Lock/Snapshot
2022-10-21 21:25:11 +02:00
Michael Eischer
2e3f1c08c5
repository: split index into a separate package
2022-10-08 21:15:34 +02:00
Alexander Weiss
9607cad267
Remove internal/index
2020-11-15 07:05:09 +01:00
greatroar
21b787a4d1
Stop Counters where they're constructed and started
2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2
Replace restic.Progress with new progress.Counter
...
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" ( #2885 )
...
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
greatroar
9f7cd69f13
Move Index.FindBlob to tests
2020-04-29 10:57:01 +02:00
Alexander Neumann
0b600d6cef
index.New: Test various error conditions
2018-10-28 14:02:31 +01:00
Alexander Neumann
f7c0893f76
index: Add tests for error conditions
2018-10-28 13:16:36 +01:00
Alexander Neumann
c3de301fc8
index: Use interfaces instead of *repository.Repository
2018-10-28 12:58:34 +01:00
Alexander Neumann
944b446ac0
index: Cancel context for tests
2018-10-28 11:17:04 +01:00
Alexander Neumann
b096fc7abf
index: Correctly process errors listing all files
...
This also removes the now unused `list` and `worker` packages.
2018-10-28 11:16:29 +01:00
Alexander Neumann
ff3de66ddf
Merge pull request #1582 from restic/optimize-debug-log
...
Optimize debug logs
2018-01-26 21:57:18 +01:00
Alexander Neumann
f63c7048c7
index: Limit index files to 3000 pack files
2018-01-26 21:07:16 +01:00
Alexander Neumann
ca3c566a99
index: Check structure after saving a new index
2018-01-26 21:07:16 +01:00
Alexander Neumann
663c57ab4d
debug: Remove manual Str() call Log()
2018-01-25 20:49:41 +01:00
Igor Fedorenko
0084e42cb6
Optimize Repository.ListPack()
...
Use pack file size returned by Backend.List() to avoid extra per-pack
Backend.Stat() requests
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-23 22:39:51 -05:00
Alexander Neumann
b0c6e53241
Fix calls to repo/backend.List() everywhere
2018-01-21 21:15:09 +01:00
Alexander Neumann
23c903074c
Move restic package to internal/restic
2017-07-24 17:43:32 +02:00
Alexander Neumann
6caeff2408
Run goimports
2017-07-23 14:21:03 +02:00
Alexander Neumann
83d1a46526
Moves files
2017-07-23 14:19:13 +02:00
