Commit Graph

330 Commits

Author SHA1 Message Date
Matthew Dawson 539599d1f1 repository/index: Optimize index.Has()
When backing up several million files (>14M tested here) with few changes,
a large amount of time is spent failing to find an id in an index and creating
an error to signify this.  Since this is checked using the Has method,
which doesn't use this error, this time creating the error is wasted.

Instead, directly check if the given id and type are present in the index.
This also avoids reporting all the packs containing this blob, further
reducing cpu usage.
2018-01-08 21:46:17 +01:00
Alexander Neumann d77a326bb0 Add benchmark for Index.Has() 2018-01-08 21:10:30 +01:00
Alexander Neumann 7d8765a937 backend: Only return top-level files for most dirs
Fixes #1478
2017-12-14 19:14:16 +01:00
George Armhold d886cb5c27 replace ad-hoc context.TODO() with gopts.ctx, so that cancellation
can properly trickle down from cmd_*.

gh-1434
2017-12-03 07:22:14 -05:00
Alexander Neumann 6d46824fb0 Pass in a nil buffer to Open() 2017-11-01 10:30:40 +01:00
Alexander Neumann 931e6ed2ac Use Seal/Open everywhere 2017-11-01 10:30:40 +01:00
Alexander Neumann 178e946fc7 Rename KDFParams -> Params 2017-10-28 10:28:29 +02:00
Alexander Neumann e56370eb5b Remove Deleter interface 2017-10-14 16:04:29 +02:00
Felix Lee cc5ada63a4 fixes #1251, race when writing indexes 2017-10-07 05:11:42 -07:00
Alexander Neumann d886bc6c48 Ignore invalid index files, print warning 2017-10-05 20:39:53 +02:00
Alexander Neumann cdef55bb88 Merge pull request #1312 from hgfischer/issue-1119
nit: please remove dot imports #1119
2017-10-04 21:01:40 +02:00
Alexander Neumann f26492fc2d prune: Warn about wrong plaintext blob ID 2017-10-02 16:27:08 +02:00
Herbert 3473c3f7b6 Remove all dot-imports 2017-10-02 15:06:39 +02:00
Alexander Neumann db5ec5d876 repo: Automatically cache tree-only pack files 2017-09-24 23:11:23 +02:00
Alexander Neumann 1eaad6cebb index: Add TreePacks() 2017-09-24 21:54:53 +02:00
Alexander Neumann 3890a947ca Clear data files in cache 2017-09-24 21:54:53 +02:00
Alexander Neumann e299272378 repo: Try cached pack files first 2017-09-24 21:54:53 +02:00
Alexander Neumann 70248bd05a repo: Clear indexes 2017-09-24 21:54:53 +02:00
Alexander Neumann 7a5fde8f5a repository: Save pack files for trees in cache 2017-09-24 21:54:53 +02:00
Alexander Neumann 9be24a1c9f Add cache
This commits adds rudimentary support for a cache directory, enabled by
default. The cache directory is created if it does not exist. The cache
is used if there's anything in it, newly created snapshot and index
files are written to the cache automatically.
2017-09-24 21:54:53 +02:00
Alexander Neumann 3541d06d07 repo: Split packers for tree and data
The code now bundles tree blobs and data blobs into different pack
files, so we'll end up with pack files that either only contain data or
trees. This is in preparation to adding a cache (#1040), because
tree-only pack files can easily be cached later on.
2017-09-22 15:36:47 +02:00
Alexander Neumann db0e3cd772 repo: Remove packer limits
This commit simplifies finding a packer: The first open packer is taken,
and the upper limit for the pack file is removed.
2017-09-22 15:36:47 +02:00
Alexander Neumann d610c60991 repo: Remove unused sync.Pool 2017-09-22 12:37:10 +02:00
Alexander Neumann e66adc42da Always use long name for keys
Otherwise the code panics if a file with a short name is tried.
2017-09-10 15:35:10 +02:00
Alexander Neumann e846e14965 Ignore files with invalid name in the repo 2017-09-10 11:00:07 +02:00
Alexander Neumann 36e70228f2 Handle invalid key file 2017-09-10 10:55:01 +02:00
Alexander Neumann 1e48141648 Fix panic when file name is too short
Closes #1204
2017-09-09 10:50:32 +02:00
Alexander Neumann 23c903074c Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
Alexander Neumann 6caeff2408 Run goimports 2017-07-23 14:21:03 +02:00
Alexander Neumann 83d1a46526 Moves files 2017-07-23 14:19:13 +02:00