octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-12-23 03:18:55 +00:00
Code Issues Releases Wiki Activity
7,019 Commits 7 Branches 33 Tags 66 MiB
191c47d30e
Commit Graph

7019 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Michael Eischer
191c47d30e
Merge pull request #4353 from MichaelEischer/tune-gc 
Tune Go garbage collector
 2023-06-16 23:24:39 +02:00
Michael Eischer
dd1ef13c1c
Merge pull request #4352 from MichaelEischer/pointerless-index 
index: optimize go GC performance
 2023-06-16 23:23:14 +02:00
Michael Eischer
b2ed42cec4 index: add basic hat test 2023-06-16 23:12:30 +02:00
Michael Eischer
237f32c651
Merge pull request #3261 from DRON-666/prealloc-packs 
Prevent local backend file fragmentation by file preallocation.
 2023-06-09 11:50:46 +02:00
DRON-666
74e4656850 Update changelog 2023-06-09 11:49:06 +03:00
DRON-666
c37d587f81 Use PreallocateFile in local backend 2023-06-09 11:49:06 +03:00
DRON-666
ffc6b3d887 Move PreallocateFile to fs package 2023-06-09 11:49:06 +03:00
Michael Eischer
88c63a029c
Merge pull request #3563 from matta/improve-eta 
Improve the ETA displayed during backup
 2023-06-08 20:49:19 +02:00
Michael Eischer
0b908bb1fb Address review comments 2023-06-08 20:24:21 +02:00
Matt Armstrong
0372c7ef04 Improve the ETA displayed during backup 
The ETA restic displays was based on a rate computed across the entire
backup operation. Often restic can progress at uneven rates. In the worst
case, restic progresses over most of the backup at a very high rate and
then finds new data to back up. The displayed ETA is then unrealistic and
never adapts.

Restic now estimates the transfer rate based on a sliding window, with the
goal of adapting to observed changes in rate. To avoid wild changes in the
estimate, several heuristics are used to keep the sliding window wide
enough to be relatively stable.
 2023-06-08 20:05:56 +02:00
Michael Eischer
55c21846b1 Revert "index: remove redundant storage of indexmap size" 
This reverts commit f1c388c623.

For an uninitialized indexmap the returned size was `-1` which is
unexpected and could cause problems.
 2023-06-08 18:08:46 +02:00
Michael Eischer
0f80b6a137 add changelog for gc tuning 2023-06-08 18:02:46 +02:00
Michael Eischer
e14ccb1142
Merge pull request #4298 from restic/backend-parseconfig-cleanup 
Unified and slightly type-safer backend config parsing
 2023-06-08 12:02:27 +02:00
Michael Eischer
609367195a backend: Fix test compilation with Go 1.18 and 1.19 2023-06-07 22:31:15 +02:00
Michael Eischer
18eb1d3ab0 backend: test cleanup 2023-06-07 22:31:15 +02:00
Michael Eischer
32a6b66267 backend: add standardized Config.ApplyEnvironment 
This removes the backend specific special cases while parsing the
configuration in `global.go`.
 2023-06-07 22:31:15 +02:00
Michael Eischer
f903db492c backend: let ParseConfig return a Config pointer 
In order to change the backend initialization in `global.go` to be able
to generically call cfg.ApplyEnvironment() for supported backends, the
`interface{}` returned by `ParseConfig` must contain a pointer to the
configuration.

An alternative would be to use reflection to convert the type from
`interface{}(Config)` to `interface{}(*Config)` (from value to pointer
type). However, this would just complicate the type mess further.
 2023-06-07 22:31:15 +02:00
Michael Eischer
25a0be7f26 backend: fix linter warnings 2023-06-07 22:31:15 +02:00
Michael Eischer
a27b7f1370 backend: use generic instead of any type for test suite 2023-06-07 22:31:14 +02:00
Michael Eischer
fa361dbfbd backend: use generic implementation for ParseConfig tests 2023-06-07 22:30:35 +02:00
Michael Eischer
5260d38980 backend: let ParseConfig return concrete type 2023-06-07 22:30:35 +02:00
Michael Eischer
2f7b4ceae1 backend: Move environment based configuration into backend 2023-06-07 22:30:35 +02:00
Michael Eischer
49a6a4f5bf
Merge pull request #4356 from MichaelEischer/fix-unlock-hint 
lock: fix missing hint to unlock command if repository is locked
 2023-06-06 21:03:06 +02:00
Michael Eischer
237d00000e lock: fix missing hint to unlock command if repository is locked 2023-06-02 23:16:49 +02:00
dependabot[bot]
0c727f6ad1
Merge pull request #4355 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.56 2023-06-02 20:51:26 +00:00
Michael Eischer
4e7d3efad4
Merge pull request #4345 from DanielGibson/patch-1 
restic restore docs: Mention that restic restore is fastest
 2023-06-02 22:44:38 +02:00
Michael Eischer
17446da5fd
Merge pull request #4343 from greatroar/cache 
cache: Restructure New to remove redundant operations
 2023-06-02 22:41:17 +02:00
dependabot[bot]
a3cee840d2
build(deps): bump github.com/minio/minio-go/v7 from 7.0.52 to 7.0.56 
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.52 to 7.0.56.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.52...v7.0.56)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-02 20:35:42 +00:00
dependabot[bot]
c76f3a1e27
Merge pull request #4348 from restic/dependabot/go_modules/github.com/spf13/cobra-1.7.0 2023-06-02 20:34:38 +00:00
dependabot[bot]
1e9714088d
Merge pull request #4347 from restic/dependabot/go_modules/golang.org/x/sync-0.2.0 2023-06-02 20:32:27 +00:00
Michael Eischer
58e3f5955c
Merge pull request #4350 from restic/dependabot/go_modules/github.com/Azure/azure-sdk-for-go/sdk/azcore-1.6.0 
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.4.0 to 1.6.0
 2023-06-02 22:26:55 +02:00
dependabot[bot]
dd8d8b1ae0
build(deps): bump github.com/spf13/cobra from 1.6.1 to 1.7.0 
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.6.1 to 1.7.0.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.6.1...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-02 20:22:39 +00:00
dependabot[bot]
e1ac0f0e0c
build(deps): bump golang.org/x/sync from 0.1.0 to 0.2.0 
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.1.0 to 0.2.0.
- [Commits](https://github.com/golang/sync/compare/v0.1.0...v0.2.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-02 20:22:36 +00:00
Michael Eischer
a88d90b8e5
Merge pull request #4346 from restic/dependabot/go_modules/golang.org/x/oauth2-0.8.0 
build(deps): bump golang.org/x/oauth2 from 0.7.0 to 0.8.0
 2023-06-02 22:21:22 +02:00
Michael Eischer
2fcb3947df prune: trigger GC after prune planning 2023-06-02 21:57:40 +02:00
Michael Eischer
eef0ee7a85 repository: trigger GC after loading the index 
Loading the index requires some scratch space, thus make sure that this
memory does not factor into the targeted gc memory usage limit.
 2023-06-02 21:56:14 +02:00
Michael Eischer
f1b73c9301 Reduce GOGC to 50 
The index used by restic consumes a major part of the total memory
usage. This leads to an unnecessarily large amount of memory that
contains ephemeral objects that are only used for a short time.
 2023-06-02 21:51:50 +02:00
Michael Eischer
ac1dfc99bb index: fix blocklist size 2023-06-02 19:39:12 +02:00
dependabot[bot]
379282299a
build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/azcore 
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.4.0 to 1.6.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.4.0...sdk/azcore/v1.6.0)

---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-01 01:59:16 +00:00
dependabot[bot]
0c796dbd9b
build(deps): bump golang.org/x/oauth2 from 0.7.0 to 0.8.0 
Bumps [golang.org/x/oauth2](https://github.com/golang/oauth2) from 0.7.0 to 0.8.0.
- [Commits](https://github.com/golang/oauth2/compare/v0.7.0...v0.8.0)

---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-06-01 01:58:29 +00:00
Daniel Gibson
f9dded83b3 doc/050_restore.rst: Mention that restic restore is fastest 
Especially with lots of files (like when backing up git repos),
restic mount + cp -r can be really slow.
 2023-05-31 22:44:36 +02:00
Michael Eischer
88a10a368f
Merge pull request #4342 from greatroar/errors 
cmd: Don't check for errno == 0
 2023-05-31 22:17:07 +02:00
Michael Eischer
9a7056a479 index: implement indexmap.grow() without random access 2023-05-30 20:13:33 +02:00
Michael Eischer
fc05e35a08 index: let indexmap.Each iterate in allocation order 
Iterating through the indexmap according to the bucket order has the
problem that all indexEntries are accessed in random order which is
rather cache inefficient.

As we already keep a list of all allocated blocks, just iterate through
it. This allows iterating through a batch of indexEntries without random
memory accesses. In addition, the packID will likely remain similar
across multiple blobs as all blobs of a pack file are added as a single
batch.
 2023-05-30 20:12:36 +02:00
Michael Eischer
f1c388c623 index: remove redundant storage of indexmap size 2023-05-30 20:11:53 +02:00
Michael Eischer
12141afbad index: Allow inlining of HAT 2023-05-30 20:11:14 +02:00
Michael Eischer
fed33295c3 index: store indexEntries in hashed array tree 
This data structure reduces the wasted memory to O(sqrt(n)). The
top-layer of the hashed array tree (HAT) also has a size of O(sqrt(n)),
which makes it cache efficient. The top-layer should be small enough to
easily fit into the CPU cache and thus only adds little overhead
compared to directly accessing an index entry via a pointer.
 2023-05-29 00:24:15 +02:00
Michael Eischer
b217f38ee7 index: Remove pointers from within indexentrys 
The indexEntry objects are now allocated in a separate array. References
to an indexEntry are now stored as array indices. This has the benefit
of allowing the garbage collector to ignore the indexEntry objects as
these do not contain pointers and are part of a single large allocation.
 2023-05-29 00:24:15 +02:00
Michael Eischer
0c1240360d index: add garbage collection benchmark 
Allocates an index and repeatedly triggers the GC.
 2023-05-29 00:23:04 +02:00
Michael Eischer
ffca602315 repository: Fix panic in benchmarkLoadIndex 2023-05-28 23:55:47 +02:00