1827b16ade
Merge pull request #3519 from greatroar/maphash
...
Replace siphash by hash/maphash
2021-09-19 19:46:03 +02:00
8d2996eaaa
Replace siphash by hash/maphash
...
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.
Benchmark results on linux/amd64, -benchtime=3s:
name old time/op new time/op delta
IndexHasUnknown-8 50.6ns ±10% 41.0ns ±19% -18.92% (p=0.000 n=9+10)
IndexHasKnown-8 52.6ns ±12% 41.5ns ±12% -21.13% (p=0.000 n=9+10)
IndexMapHash-8 3.64µs ± 1% 2.00µs ± 0% -45.09% (p=0.000 n=10+9)
IndexAlloc-8 700ms ± 1% 601ms ± 6% -14.18% (p=0.000 n=8+10)
IndexAllocParallel-8 205ms ± 5% 192ms ± 8% -6.18% (p=0.043 n=10+10)
MasterIndexAlloc-8 319ms ± 1% 279ms ± 5% -12.58% (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8 156ns ± 8% 147ns ± 6% -5.46% (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8 150ns ± 7% 142ns ± 8% -5.69% (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8 74.4ns ± 6% 72.0ns ± 9% ~ (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8 67.4ns ± 9% 65.5ns ± 7% ~ (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8 461ns ± 2% 445ns ± 2% -3.49% (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8 408ns ±11% 378ns ± 5% -7.22% (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8 479ns ± 1% 437ns ± 4% -8.82% (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8 406ns ± 8% 343ns ±15% -15.44% (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8 480ns ± 1% 455ns ± 5% -5.15% (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8 391ns ±18% 382ns ± 8% ~ (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8 71.0ns ± 8% 57.2ns ±11% -19.36% (p=0.000 n=9+10)
PackerManager-8 254ms ± 1% 254ms ± 1% ~ (p=0.285 n=15+15)
name old speed new speed delta
IndexMapHash-8 1.12GB/s ± 1% 2.05GB/s ± 0% +82.13% (p=0.000 n=10+9)
PackerManager-8 208MB/s ± 1% 207MB/s ± 1% ~ (p=0.281 n=15+15)
name old alloc/op new alloc/op delta
IndexMapHash-8 0.00B 0.00B ~ (all equal)
IndexAlloc-8 400MB ± 0% 400MB ± 0% ~ (p=1.000 n=9+10)
IndexAllocParallel-8 401MB ± 0% 401MB ± 0% +0.00% (p=0.000 n=10+10)
MasterIndexAlloc-8 258MB ± 0% 262MB ± 0% +1.42% (p=0.000 n=9+10)
PackerManager-8 73.1kB ± 0% 73.1kB ± 0% ~ (p=0.382 n=13+13)
name old allocs/op new allocs/op delta
IndexMapHash-8 0.00 0.00 ~ (all equal)
IndexAlloc-8 907k ± 0% 907k ± 0% -0.00% (p=0.000 n=10+10)
IndexAllocParallel-8 907k ± 0% 907k ± 0% +0.00% (p=0.009 n=10+10)
MasterIndexAlloc-8 327k ± 0% 317k ± 0% -3.06% (p=0.000 n=10+10)
PackerManager-8 744 ± 0% 744 ± 0% ~ (all equal)
2021-09-19 16:05:18 +02:00
58efe21eca
Merge pull request #3264 from amozoss/upstream-master
...
Refactor backup progress
2021-09-19 14:54:42 +02:00
921e328b56
restore: Fix linting error
2021-09-19 14:41:07 +02:00
2cdc0719af
restorer: Sanitize verify errors
2021-09-19 14:01:26 +02:00
bdcdfaf6b4
restore --verify: buffer reuse consistency and comment
2021-09-19 13:11:27 +02:00
2b94742ca5
Replace no-op closures in restorer by nil check
2021-09-19 13:11:16 +02:00
d357744104
Handle canceled context in restore --verify properly
2021-09-19 13:11:05 +02:00
d4225ec803
Simplify buffer growing in Restorer.verifyFile
...
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
de8521ae56
Report number of successes from Restorer.VerifyFiles
...
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
bb066cf7d3
Concurrent Restorer.VerifyFiles
...
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
556424d61b
Reuse buffer in Restorer.VerifyFiles
...
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
973fa921cb
Test and document Restorer.VerifyFiles
2021-09-19 12:52:11 +02:00
fbb0e6499a
ui: consolidate backup ui in ui/backup package
2021-09-12 16:20:15 +02:00
cccb0d4064
ui: assert that ProgressPrinter interface is implemented
2021-09-12 15:26:40 +02:00
d62bfed65d
ui: move SetDryRun to ProgressReporter
2021-09-12 15:25:58 +02:00
77b129ec74
ui: remove unused Summary field
2021-09-12 15:25:58 +02:00
3024239e40
ui/json: restore summary output
2021-09-12 15:25:58 +02:00
5ccf583b8a
ui: restore a few comments
2021-09-12 15:25:58 +02:00
80cbaf6d38
ui: Simplify ReportTotal methods
2021-09-12 15:25:58 +02:00
448419990c
Refactor backup progress
...
Move the shared logic into the progress
Allows logic to be shared with forth coming restore status
2021-09-12 15:25:58 +02:00
fa5ca8af81
Merge pull request #2856 from aawsome/remove-readahead
...
Simplify cache logic
2021-09-04 20:25:49 +02:00
b45d88e124
Merge pull request #3496 from juergenhoetzel/imporove-snapshot-filter-warning-message
...
Improve snapshot filter warning message
2021-09-03 23:42:27 +02:00
a29777f467
Merge pull request #3501 from greatroar/printprogress
...
Streamline progress printing in cmd/restic
2021-09-03 23:34:36 +02:00
81876d5c1b
Simplify cache logic
2021-09-03 21:01:00 +02:00
5aaa3e93c1
internal/ui/termstatus: Optimize and publish Truncate
...
name old time/op new time/op delta
TruncateASCII-8 347ns ± 1% 69ns ± 1% -80.02% (p=0.000 n=9+10)
TruncateUnicode-8 447ns ± 3% 348ns ± 1% -22.04% (p=0.000 n=10+10)
2021-09-03 11:48:22 +02:00
ec2e3b260e
Merge pull request #3499 from greatroar/wrappedconn-pointer
...
rclone: Return one fewer value from run
2021-08-31 21:48:36 +02:00
950b818274
rclone: Return one fewer value from run
2021-08-26 18:12:08 +02:00
defe19fdf6
Quote snapshot prefix in error string
2021-08-25 16:11:28 +02:00
68370feeee
backends: Remove TestSaveFilenames test
...
Filenames are expected to match the sha256 sum of the file content. This
rule is now enforced by the rest server thus making this test useless.
2021-08-15 18:24:16 +02:00
574c83e47f
rest: Fix test to use paths which are the sha256 sum of the data
2021-08-15 18:19:43 +02:00
e6a5801155
rest: Fix test backend url
...
The rest config normally uses prepareURL to sanitize URLs and ensures
that the URL ends with a slash. However, the test used an URL without a
trailing slash, which after the rest server changes causes test
failures.
2021-08-15 18:16:17 +02:00
f4c5dec05d
backend: test that a wrong hash fails an upload
2021-08-04 22:17:46 +02:00
7c1903e1ee
panic if hash returns an error
...
Add a sanity check that the interface contract is honoured.
2021-08-04 22:17:46 +02:00
51b7e3119b
mem: calculate md5 content hash for uploads
...
The mem backend is primarily used for testing. This ensures that the
upload hash calculation gets appropriate test coverage.
2021-08-04 22:17:46 +02:00
a009b39e4c
gs/swift: calculate md5 content hash for upload
2021-08-04 22:17:46 +02:00
1d3e99f475
azure: check upload using md5 content hash
...
For files below 256MB this uses the md5 hash calculated while assembling
the pack file. For larger files the hash for each 100MB part is
calculated on the fly. That hash is also reused as temporary filename.
As restic only uploads encrypted data which includes among others a
random initialization vector, the file hash shouldn't be susceptible to
md5 collision attacks (even though the algorithm is broken).
2021-08-04 22:17:46 +02:00
9aa2eff384
Add plumbing to calculate backend specific file hash for upload
...
This enables the backends to request the calculation of a
backend-specific hash. For the currently supported backends this will
always be MD5. The hash calculation happens as early as possible, for
pack files this is during assembly of the pack file. That way the hash
would even capture corruptions of the temporary pack file on disk.
2021-08-04 22:17:46 +02:00
ee2f14eaf0
s3: enable content hash calculation for uploads
2021-08-04 22:12:12 +02:00
6586e90acf
Modernize internal/cache error handling
2021-08-04 22:02:42 +02:00
ea04f40eb3
Save cached files to a temporary location first
2021-08-04 22:02:42 +02:00
f9b6f8fd45
Replace duplicate type checking in cache with a function
2021-08-04 22:02:42 +02:00
81e2499d19
Sync directory to get durable writes in local backend
2021-08-04 21:51:53 +02:00
195a5cf996
Save files under temporary name in local backend
...
Fixes #3435 .
2021-08-04 21:51:53 +02:00
38a8a48a25
Simplify dry run backend
2021-08-04 21:19:29 +02:00
77bf148460
backup: add --dry-run/-n flag to show what would happen.
...
This can be used to check how large a backup is or validate exclusions.
It does not actually write any data to the underlying backend. This is
implemented as a simple overlay backend that accepts writes without
forwarding them, passes through reads, and generally does the minimal
necessary to pretend that progress is actually happening.
Fixes #1542
Example usage:
$ restic -vv --dry-run . | grep add
new /changelog/unreleased/issue-1542, saved in 0.000s (350 B added)
modified /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added)
modified /cmd/restic/global.go, saved in 0.000s (0 B added)
new /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added)
new /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added)
modified /internal/backend/test/tests.go, saved in 0.000s (0 B added)
modified /internal/repository/repository.go, saved in 0.000s (20.707 KiB added)
modified /internal/ui/backup.go, saved in 0.000s (9.110 KiB added)
modified /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added)
modified /restic, saved in 0.131s (25.542 MiB added)
Would add to the repo: 25.892 MiB
2021-08-04 21:19:29 +02:00
533ac4fd95
Merge pull request #3467 from greatroar/wrappedconn-pointer
...
Use rclone.wrappedConn by pointer
2021-08-04 21:18:53 +02:00
326fefcd80
Allow --tag and --keep-tag to match untagged snapshots
2021-08-02 23:06:20 +02:00
fa3eed1998
Use rclone.wrappedConn by pointer
...
This shaves a kilobyte off the Linux binary by not generating a
non-pointer interface implementation.
2021-08-01 09:11:50 +02:00
5571c3f7fd
Merge pull request #3453 from MichaelEischer/http2-zero-length-workaround
...
rest: Workaround Http2 zero-length reply bug
2021-07-31 20:30:06 +02:00
MichaelEischer
greatroar
greatroar
Dan Willoughby
Alexander Weiss
Juergen Hoetzel
Ryan Hitchman
David le Blanc