Jason Lenz
2b1932a258
Report symlink sizes from FUSE mount for snapshot dir
...
Fixes #3667 .
2022-03-17 22:21:47 -05:00
greatroar
fdc738fb70
Report symlink sizes from FUSE mount
...
Fixes #3667 .
2022-03-13 16:48:35 +01:00
Michael Eischer
8ae4d86a84
rename snapshot timestamp filter variable
2022-02-05 22:42:38 +01:00
Aneesh Agrawal
058dfc20da
Avoid choosing parent snapshot newer than time of current snapshot
...
Currently, `restic backup` (if a `--parent` is not provided)
will choose the most recent matching snapshot as the parent snapshot.
This makes sense in the usual case,
where we tag the snapshot-being-created with the current time.
However, this doesn't make sense if the user has passed `--time`
and is currently creating a snapshot older than the latest snapshot.
Instead, choose the most recent snapshot
which is not newer than the snapshot-being-created's timestamp,
to avoid any time travel.
Impetus for this change:
I'm using restic for the first time!
I have a number of existing BTRFS snapshots
I am backing up via restic to serve as my initial set of backups.
I initially `restic backup`'d the most recent snapshot to test,
then started backing up each of the other snapshots.
I noticed in `restic cat snapshot <id>` output
that all the remaining snapshots have the most recent as the parent.
2022-01-23 23:55:00 -05:00
rawtaz
7f6fc78f95
Merge pull request #3544 from restic/fix-b2-delete-retry
...
b2: Successful delete if file does not exist
2022-01-13 23:28:30 +01:00
Charlie Jiang
a5b0e0bef4
fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever
...
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2021-12-28 13:14:46 +08:00
Michael Eischer
50053a85d3
rest: Adjust http2 missing eof test to golang >= 1.17.3, >= 1.16.10
...
The missing eof with http2 when a response included a content-length
header but no data, has been fixed in golang 1.17.3/1.16.10. Therefore
just drop the canary test and schedule it for removal once go 1.18 is
required as minimum version by restic.
2021-11-13 21:57:30 +01:00
MichaelEischer
f1cfb97237
Merge pull request #3514 from phcreery/rclone_timeout
...
rclone: extend timeout from 60s to 240s
2021-11-07 18:10:34 +01:00
MichaelEischer
cb81ee9396
Merge pull request #3474 from kitone/fix-issue-3382
...
Honor RESTIC_CACHE_DIR environment variable
2021-11-07 17:57:54 +01:00
Michael Eischer
b0e64deb27
rclone: Fix timeout calculation
2021-11-07 17:49:33 +01:00
phcreery
43d173b042
rclone: add timeout option and documentation
2021-11-07 17:49:21 +01:00
Michael Eischer
15d7313387
local: Fix fd leak when encountering files directly inside data/
2021-11-06 19:44:57 +01:00
MichaelEischer
78c7dd53ef
Merge pull request #3526 from greatroar/dump-refactor
...
Refactor internal/dump + concurrent load/write
2021-11-05 22:38:39 +01:00
kitone
1b23675f21
cache --cleanup should handle directories created by restic check.
...
Because there is no guarantee that a cleanup of these directories will occur
after the "restic check", we extend the behavior to detect and manage these
specific cache directories and allow their cleanup too.
2021-11-04 15:10:38 +01:00
greatroar
c71729dfc4
Refactor internal/dump + concurrent load/write
...
Package internal/dump has been reworked so its API consists of a single
type Dumper that handles tar and zip formats. Tree loading and node
writing happen concurrently.
2021-11-01 23:01:55 +01:00
Alexander Neumann
fb5d9345a7
Merge pull request #3510 from MichaelEischer/fix-archiver-early-on-abort
...
archiver: Fix TestArchiverAbortEarlyOnError test
2021-10-16 15:37:41 +02:00
kitone
95eb859b54
Honor RESTIC_CACHE_DIR environment variable
...
Fix #3382 : restic check doesn't obey the RESTIC_CACHE_DIR environment variable
2021-10-10 16:00:02 +02:00
Michael Eischer
257740b0cc
b2: Successful delete if file does not exist
...
When deleting a file, B2 sometimes returns a "500 Service Unavailable"
error but nevertheless correctly deletes the file. Due to retries in
the B2 library blazer, we sometimes also see a "400 File not present"
error. The retries of restic for the delete request then fail with
"404 File with such name does not exist.".
As we have to rely on request retries in a distributed system to handle
temporary errors, also consider a delete request to be successful if the
file is reported as not existing. This should be safe as B2 claims to
provide a strongly consistent bucket listing and thus a missing file
shouldn't mysteriously show up again later on.
2021-10-09 23:51:12 +02:00
MichaelEischer
f2a3b3b4a1
Merge pull request #3537 from greatroar/dump-lru
...
Check cap instead of len in bloblru
2021-10-09 20:00:27 +02:00
Garry McNulty
708d7a2574
s3: Add warning if key ID or secret is empty
...
Also add debug message if no credential types are available.
Closes #2388
2021-10-06 23:13:40 +01:00
greatroar
634a9c162d
Check cap instead of len in bloblru
...
restic dump uses bloblru.Cache to keep buffers alive, but also reuses
evicted buffers. That means large buffers may be used to store small
blobs, causing the cache to think it's using less memory than it
actually does.
2021-10-03 09:34:17 +02:00
greatroar
c892c0bab9
internal/restic: Don't allocate in Tree.Insert
...
name old time/op new time/op delta
BuildTree-8 34.6µs ± 4% 7.0µs ± 3% -79.68% (p=0.000 n=18+19)
name old alloc/op new alloc/op delta
BuildTree-8 34.0kB ± 0% 0.9kB ± 0% -97.37% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
BuildTree-8 108 ± 0% 1 ± 0% -99.07% (p=0.000 n=20+20)
2021-09-26 18:08:48 +02:00
MichaelEischer
78dac2fd48
Merge pull request #3523 from greatroar/swift-v2
...
Upgrade ncw/swift to v2
2021-09-24 21:41:36 +02:00
Michael Eischer
5ea8bba1a1
swift: restore context err check for list()
2021-09-24 21:19:46 +02:00
MichaelEischer
a5e103a212
Merge pull request #3522 from greatroar/dump-lru
...
Use LRU cache in restic dump
2021-09-24 20:33:58 +02:00
greatroar
e7ec0453b1
Upgrade ncw/swift to v2
2021-09-24 19:08:37 +02:00
greatroar
fe04d024c7
Use LRU cache in restic dump
2021-09-24 15:45:08 +02:00
Uli Martens
718966a81a
Move Blobcache into dedicated internal package
2021-09-24 15:45:00 +02:00
greatroar
4f33eca634
Remove unused Writer arg to internal/dump.writeDump
2021-09-24 15:40:42 +02:00
Sam Lucidi
897d8e662c
Add --insecure-tls flag to disable SSL cert verification
...
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2021-09-21 10:52:40 -04:00
MichaelEischer
1827b16ade
Merge pull request #3519 from greatroar/maphash
...
Replace siphash by hash/maphash
2021-09-19 19:46:03 +02:00
greatroar
8d2996eaaa
Replace siphash by hash/maphash
...
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.
Benchmark results on linux/amd64, -benchtime=3s:
name old time/op new time/op delta
IndexHasUnknown-8 50.6ns ±10% 41.0ns ±19% -18.92% (p=0.000 n=9+10)
IndexHasKnown-8 52.6ns ±12% 41.5ns ±12% -21.13% (p=0.000 n=9+10)
IndexMapHash-8 3.64µs ± 1% 2.00µs ± 0% -45.09% (p=0.000 n=10+9)
IndexAlloc-8 700ms ± 1% 601ms ± 6% -14.18% (p=0.000 n=8+10)
IndexAllocParallel-8 205ms ± 5% 192ms ± 8% -6.18% (p=0.043 n=10+10)
MasterIndexAlloc-8 319ms ± 1% 279ms ± 5% -12.58% (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8 156ns ± 8% 147ns ± 6% -5.46% (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8 150ns ± 7% 142ns ± 8% -5.69% (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8 74.4ns ± 6% 72.0ns ± 9% ~ (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8 67.4ns ± 9% 65.5ns ± 7% ~ (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8 461ns ± 2% 445ns ± 2% -3.49% (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8 408ns ±11% 378ns ± 5% -7.22% (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8 479ns ± 1% 437ns ± 4% -8.82% (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8 406ns ± 8% 343ns ±15% -15.44% (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8 480ns ± 1% 455ns ± 5% -5.15% (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8 391ns ±18% 382ns ± 8% ~ (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8 71.0ns ± 8% 57.2ns ±11% -19.36% (p=0.000 n=9+10)
PackerManager-8 254ms ± 1% 254ms ± 1% ~ (p=0.285 n=15+15)
name old speed new speed delta
IndexMapHash-8 1.12GB/s ± 1% 2.05GB/s ± 0% +82.13% (p=0.000 n=10+9)
PackerManager-8 208MB/s ± 1% 207MB/s ± 1% ~ (p=0.281 n=15+15)
name old alloc/op new alloc/op delta
IndexMapHash-8 0.00B 0.00B ~ (all equal)
IndexAlloc-8 400MB ± 0% 400MB ± 0% ~ (p=1.000 n=9+10)
IndexAllocParallel-8 401MB ± 0% 401MB ± 0% +0.00% (p=0.000 n=10+10)
MasterIndexAlloc-8 258MB ± 0% 262MB ± 0% +1.42% (p=0.000 n=9+10)
PackerManager-8 73.1kB ± 0% 73.1kB ± 0% ~ (p=0.382 n=13+13)
name old allocs/op new allocs/op delta
IndexMapHash-8 0.00 0.00 ~ (all equal)
IndexAlloc-8 907k ± 0% 907k ± 0% -0.00% (p=0.000 n=10+10)
IndexAllocParallel-8 907k ± 0% 907k ± 0% +0.00% (p=0.009 n=10+10)
MasterIndexAlloc-8 327k ± 0% 317k ± 0% -3.06% (p=0.000 n=10+10)
PackerManager-8 744 ± 0% 744 ± 0% ~ (all equal)
2021-09-19 16:05:18 +02:00
MichaelEischer
58efe21eca
Merge pull request #3264 from amozoss/upstream-master
...
Refactor backup progress
2021-09-19 14:54:42 +02:00
Michael Eischer
921e328b56
restore: Fix linting error
2021-09-19 14:41:07 +02:00
Michael Eischer
2cdc0719af
restorer: Sanitize verify errors
2021-09-19 14:01:26 +02:00
greatroar
bdcdfaf6b4
restore --verify: buffer reuse consistency and comment
2021-09-19 13:11:27 +02:00
greatroar
2b94742ca5
Replace no-op closures in restorer by nil check
2021-09-19 13:11:16 +02:00
greatroar
d357744104
Handle canceled context in restore --verify properly
2021-09-19 13:11:05 +02:00
greatroar
d4225ec803
Simplify buffer growing in Restorer.verifyFile
...
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
greatroar
de8521ae56
Report number of successes from Restorer.VerifyFiles
...
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
greatroar
bb066cf7d3
Concurrent Restorer.VerifyFiles
...
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
greatroar
556424d61b
Reuse buffer in Restorer.VerifyFiles
...
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
greatroar
973fa921cb
Test and document Restorer.VerifyFiles
2021-09-19 12:52:11 +02:00
Michael Eischer
e0d615c264
archiver: Fix TestArchiverAbortEarlyOnError test
...
This can be caused when the test has uploaded four blobs, then queues
two blobs for upload which are delayed. Then a seventh file can be
opened which lead to a test failure.
2021-09-12 22:17:17 +02:00
Michael Eischer
fbb0e6499a
ui: consolidate backup ui in ui/backup package
2021-09-12 16:20:15 +02:00
Michael Eischer
cccb0d4064
ui: assert that ProgressPrinter interface is implemented
2021-09-12 15:26:40 +02:00
Michael Eischer
d62bfed65d
ui: move SetDryRun to ProgressReporter
2021-09-12 15:25:58 +02:00
Michael Eischer
77b129ec74
ui: remove unused Summary field
2021-09-12 15:25:58 +02:00
Michael Eischer
3024239e40
ui/json: restore summary output
2021-09-12 15:25:58 +02:00
Michael Eischer
5ccf583b8a
ui: restore a few comments
2021-09-12 15:25:58 +02:00