Michael Eischer
6f53ecc1ae
adapt workers based on whether an operation is CPU or IO-bound
...
Use runtime.GOMAXPROCS(0) as worker count for CPU-bound tasks,
repo.Connections() for IO-bound task and a combination if a task can be
both. Streaming packs is treated as IO-bound as adding more worker
cannot provide a speedup.
Typical IO-bound tasks are download / uploading / deleting files.
Decoding / Encoding / Verifying are usually CPU-bound. Several tasks are
a combination of both, e.g. for combined download and decode functions.
In the latter case add both limits together. As the backends have their
own concurrency limits restic still won't download more than
repo.Connections() files in parallel, but the additional workers can
decode already downloaded data in parallel.
2022-07-03 12:19:26 +02:00
Michael Eischer
120ccc8754
repository: Rework blob saving to use an async pack uploader
...
Previously, SaveAndEncrypt would assemble blobs into packs and either
return immediately if the pack is not yet full or upload the pack file
otherwise. The upload will block the current goroutine until it
finishes.
Now, the upload is done using separate goroutines. This requires changes
to the error handling. As uploads are no longer tied to a SaveAndEncrypt
call, failed uploads are signaled using an errgroup.
To count the uploaded amount of data, the pack header overhead is no
longer returned by `packer.Finalize` but rather by
`packer.HeaderOverhead`. This helper method is necessary to continue
returning the pack header overhead directly to the responsible call to
`repository.SaveBlob`. Without the method this would not be possible,
as packs are finalized asynchronously.
2022-07-02 22:42:34 +02:00
Alexander Neumann
99634c0936
Return real size from SaveBlob
2022-07-02 18:55:12 +02:00
Michael Eischer
6fb408d90e
repository: implement pack compression
2022-04-30 11:34:10 +02:00
Alexander Neumann
db8a958991
Merge pull request #3683 from MichaelEischer/fix-golangci-lint-warnings
...
Fix golangci lint warnings
2022-03-29 11:45:10 +02:00
Michael Eischer
c60540b196
add go:build headers everywhere
2022-03-28 22:23:47 +02:00
Michael Eischer
243698680a
crypto: Use helpers for size calculations
2022-03-28 22:09:49 +02:00
Michael Eischer
27524979e8
restorer: Remove dead code
2022-02-13 11:43:09 +01:00
Michael Eischer
f40abd92fa
restorer: convert to use StreamPack
2022-02-12 20:18:25 +01:00
Michael Eischer
921e328b56
restore: Fix linting error
2021-09-19 14:41:07 +02:00
Michael Eischer
2cdc0719af
restorer: Sanitize verify errors
2021-09-19 14:01:26 +02:00
greatroar
bdcdfaf6b4
restore --verify: buffer reuse consistency and comment
2021-09-19 13:11:27 +02:00
greatroar
2b94742ca5
Replace no-op closures in restorer by nil check
2021-09-19 13:11:16 +02:00
greatroar
d357744104
Handle canceled context in restore --verify properly
2021-09-19 13:11:05 +02:00
greatroar
d4225ec803
Simplify buffer growing in Restorer.verifyFile
...
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
greatroar
de8521ae56
Report number of successes from Restorer.VerifyFiles
...
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
greatroar
bb066cf7d3
Concurrent Restorer.VerifyFiles
...
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
greatroar
556424d61b
Reuse buffer in Restorer.VerifyFiles
...
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
greatroar
973fa921cb
Test and document Restorer.VerifyFiles
2021-09-19 12:52:11 +02:00
Alexander Weiss
81876d5c1b
Simplify cache logic
2021-09-03 21:01:00 +02:00
Michael Eischer
bd316d3893
restore: Test partial pack downloads in filerestorer
2021-06-29 21:11:30 +02:00
Michael Eischer
e8bbb05328
restore: Correctly handle partial pack download errors
...
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
2021-06-29 20:54:16 +02:00
MichaelEischer
cc254dfefe
Merge pull request #3362 from greatroar/darwin-preallocate
...
Use FcntlFstore to preallocate on Mac
2021-04-10 22:47:56 +02:00
greatroar
23531be272
Use FcntlFstore to preallocate on Mac
2021-04-10 16:54:07 +02:00
Michael Eischer
f70aca6f6f
restorer: Skip preallocate test if not supported by the filesystem
2021-03-04 20:33:46 +01:00
Michael Eischer
47faf69230
restorer: Check dropped error
2021-01-31 18:06:28 +01:00
Alexander Weiss
5e22ae10f1
Add error handling for fileRestorer
2021-01-31 14:22:57 +01:00
Alexander Weiss
573221aa40
Add failing test for fileRestorer
2021-01-31 13:40:42 +01:00
Alexander Neumann
04ca69cc78
Address issues reported by golint
2021-01-30 20:45:57 +01:00
Alexander Neumann
0858fbf6aa
Add more error handling
2021-01-30 20:19:47 +01:00
Michael Eischer
69d5b4c36b
restorer: lower-case variable name
2021-01-03 13:55:59 +01:00
Alexander Weiss
34a33565c8
Fix loadBlob in filerestorer
2021-01-01 08:06:04 +01:00
Alexander Weiss
7409225fa8
Add filerestorer test where only parts of pack are used
2021-01-01 07:24:46 +01:00
Alexander Weiss
07b3f65a6f
filesrestorer: Re-use buffer
2021-01-01 07:24:46 +01:00
Alexander Weiss
3e0acf1395
restore: Don't save (part of) pack in memory
2021-01-01 07:24:46 +01:00
Alexander Neumann
36c5d39c2c
Fix issues reported by semgrep
2020-12-11 09:41:59 +01:00
Alexander Weiss
aa7a5f19c2
Use BlobHandle in index methods
2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691
Make BlobHandle substruct of Blob
2020-11-22 20:41:10 +01:00
Alexander Neumann
56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
...
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
Alexander Neumann
c193cea119
Pass Context to NewRestorer()
2020-10-10 15:24:26 +02:00
Alexander Neumann
f0d49ca600
Merge pull request #2933 from MichaelEischer/less-context-todo
...
Replace most usages of context.TODO()
2020-10-10 15:03:44 +02:00
kitone
6099f81692
Fix #1212 restore code produces inconsistent timestamps/permissions.
...
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
2020-10-10 13:46:44 +02:00
kitone
9d7f616190
Improve restorer debug log information
2020-10-10 13:46:43 +02:00
kitone
295ddb9e57
Add test case for inconsistent timestamps and permissions restoration
...
Reproduce from https://github.com/restic/restic/issues/1212
2020-10-10 13:46:43 +02:00
Michael Eischer
645a6efaf2
restorer: remove redundant type specification
2020-10-09 22:37:56 +02:00
Michael Eischer
603bb0e309
restore: Use proper context while loading snapshot
2020-10-09 22:37:56 +02:00
Michael Eischer
eba5dd831f
Fix typos reported by misspell
2020-10-06 14:55:13 +02:00
Michael Eischer
efbb850d92
Remove a few redundant type specifiers
...
This is the result of running `gofmt -s -w **/*.go`
2020-10-06 14:55:13 +02:00
greatroar
1e1a1f3078
Upgrade cespare/xxhash to 2.1.1
2020-10-05 17:25:39 +02:00
greatroar
9abef3bf1a
Move internal/fs.TestChdir to internal/test.Chdir
2020-09-17 10:43:33 +02:00
