octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2025-01-25 16:18:34 +00:00
Code Issues Releases Wiki Activity
5,535 Commits 7 Branches 33 Tags
Commit Graph

1047 Commits

Author SHA1 Message Date
Alexander Neumann
f0d49ca600
Merge pull request #2933 from MichaelEischer/less-context-todo 
Replace most usages of context.TODO()
 2020-10-10 15:03:44 +02:00
kitone
6099f81692 Fix #1212 restore code produces inconsistent timestamps/permissions. 
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
 2020-10-10 13:46:44 +02:00
kitone
9d7f616190 Improve restorer debug log information 2020-10-10 13:46:43 +02:00
kitone
295ddb9e57 Add test case for inconsistent timestamps and permissions restoration 
Reproduce from https://github.com/restic/restic/issues/1212
 2020-10-10 13:46:43 +02:00
Michael Eischer
e638b46a13 Embed context into ReaderAt 
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
 2020-10-09 22:39:07 +02:00
Michael Eischer
d6cfe857b7 pass proper context into MasterIndex.RebuildIndex 2020-10-09 22:39:07 +02:00
Michael Eischer
37a5e2d681 rest: use global context on repository creation 2020-10-09 22:39:06 +02:00
Michael Eischer
4b0fcaed45 unlock: use proper context for locks cleanup 
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
 2020-10-09 22:37:56 +02:00
Michael Eischer
645a6efaf2 restorer: remove redundant type specification 2020-10-09 22:37:56 +02:00
Michael Eischer
a449450021 init: pass proper context to master key generation 
This is no change in behavior as a canceled context did later on cause
the config file creation to fail. Therefore this change just lets the
repository initialization fail a bit earlier.
 2020-10-09 22:37:56 +02:00
Michael Eischer
603bb0e309 restore: Use proper context while loading snapshot 2020-10-09 22:37:56 +02:00
Michael Eischer
c458e114d4 pass context to Find / FindSnapshot 
This allows proper interruption of restic while it searches for
snapshots or key files.
 2020-10-09 22:37:56 +02:00
Michael Eischer
45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
Michael Eischer
307a6ba3a3 Upgrade minio sdk to v7 
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
 2020-10-09 22:37:24 +02:00
Michael Eischer
50da20d93d Warn if backup failed to read tree blob 2020-10-09 22:36:27 +02:00
Alexander Neumann
5fd3dbccb7
Merge pull request #2963 from MichaelEischer/fix-status-deadlock 
backup: Fix possible deadlock of scanner goroutine
 2020-10-09 21:35:50 +02:00
Alexander Neumann
35655a481b
Merge pull request #2931 from MichaelEischer/reduce-debug-overhead 
Reduce debug log overhead
 2020-10-09 21:35:11 +02:00
Alexander Neumann
30cb553c8d
Merge pull request #2932 from MichaelEischer/proper-rclone-create 
Call rclone.Create to create a new repository for the rclone backend
 2020-10-09 21:29:15 +02:00
MichaelEischer
356ac404cd
Merge pull request #2996 from greatroar/setconsolecursorposition 
Use windows.SetConsoleCursorPosition in ui/termstatus
 2020-10-09 18:19:02 +02:00
Michael Eischer
88c8e903d2 filter: Fix glob matching on absolute path marker on windows 
A pattern part containing "/" is used to mark a path or a pattern as
absolute. However, on Windows the path separator is "\" such that glob
patterns like "?" could match the marker. The code now explicitly skips
the marker when the pattern does not represent an absolute path.
 2020-10-09 16:11:05 +02:00
greatroar
740758a5fa Optimize filter pattern matching 
By replacing "**" with "", checking for this special path component can
be reduced to a length-zero check.

name                          old time/op    new time/op    delta
FilterLines-8                   44.7ms ± 5%    44.9ms ± 5%     ~     (p=0.631 n=10+10)
FilterPatterns/Relative-8       13.6ms ± 4%    13.4ms ± 5%     ~     (p=0.165 n=10+10)
FilterPatterns/Absolute-8       10.9ms ± 5%    10.7ms ± 4%     ~     (p=0.052 n=10+10)
FilterPatterns/Wildcard-8       53.7ms ± 5%    50.4ms ± 5%   -6.00%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-8     128ms ± 2%      95ms ± 1%  -25.54%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=1.000 n=9+8)
FilterPatterns/Absolute-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.903 n=9+8)
FilterPatterns/Wildcard-8       19.7MB ± 0%    19.7MB ± 0%   -0.00%  (p=0.022 n=10+9)
FilterPatterns/ManyNoMatch-8    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-8        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-8     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-09 15:51:14 +02:00
MichaelEischer
862ee4b2c9
Merge pull request #2970 from labkode/fopenskip 
Skip fopen for file change check to avoid network penalty
 2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador
958dc6aafc Skip fopen for file change check to avoid network penalty 2020-10-09 10:53:28 +02:00
MichaelEischer
88cc444779
Merge pull request #2934 from MichaelEischer/upgrade-backoff 
Upgrade github.com/cenkalti/backoff module
 2020-10-08 20:33:30 +02:00
Michael Eischer
b79f18209f Upgrade github.com/cenkalti/backoff module 
We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats
an operation endlessly when a retry count of 0 is specified. This
required a few fixes for the tests.
 2020-10-07 22:04:59 +02:00
Michael Eischer
8388e67c4b filter: cleanup path separator conversion 2020-10-07 21:14:07 +02:00
Michael Eischer
0acc3c5923 filter: special case patterns without globbing characters 
In case a part of a path is a simple string, we can just check for
equality without complex parsing in filepath.Match.

name                          old time/op    new time/op    delta
FilterLines-4                   34.8ms ±17%    41.2ms ±23%  +18.36%  (p=0.000 n=10+10)
FilterPatterns/Relative-4       21.7ms ± 6%    12.1ms ±23%  -44.46%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       10.0ms ± 5%     9.1ms ±11%   -9.80%  (p=0.006 n=10+9)
FilterPatterns/Wildcard-4       47.0ms ± 7%    42.2ms ± 5%  -10.19%  (p=0.000 n=9+10)
FilterPatterns/ManyNoMatch-4     190ms ± 1%     131ms ±20%  -31.47%  (p=0.000 n=8+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.870 n=9+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.145 n=10+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    19.7MB ± 0%  +37.91%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.421 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:55:43 +02:00
Michael Eischer
54a124de3b filter: explicitly test separate ListWithChild function 2020-10-07 20:47:52 +02:00
Michael Eischer
bcc3bddcf4 filter: only check whether a child path could match when necessary 
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.3ms ± 9%    21.7ms ± 6%   -6.68%  (p=0.004 n=10+10)
FilterPatterns/Absolute-4       13.9ms ± 7%    10.0ms ± 5%  -27.61%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4       51.4ms ± 7%    47.0ms ± 7%   -8.51%  (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4     551ms ± 9%     190ms ± 1%  -65.41%  (p=0.000 n=10+8)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.665 n=10+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.480 n=9+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    14.3MB ± 0%     ~     (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:47:52 +02:00
Michael Eischer
17c53efb0d filter: Optimize double wildcard expansion 
This only allocates a single slice to expand the double wildcard and
only copies the pattern prefix once.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       22.7ms ± 5%    23.3ms ± 9%     ~     (p=0.353 n=10+10)
FilterPatterns/Absolute-4       14.2ms ±13%    13.9ms ± 7%     ~     (p=0.853 n=10+10)
FilterPatterns/Wildcard-4        266ms ±16%      51ms ± 7%  -80.67%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4     554ms ± 6%     551ms ± 9%     ~     (p=0.436 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.349 n=10+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.073 n=10+9)
FilterPatterns/Wildcard-4        141MB ± 0%      14MB ± 0%  -89.89%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     0.09M ± 0%  -94.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:47:48 +02:00
Michael Eischer
7959796269 filter: Special case for absolute paths 
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.6ms ±20%    22.7ms ± 5%     ~     (p=0.684 n=10+10)
FilterPatterns/Absolute-4       32.3ms ± 8%    14.2ms ±13%  -56.01%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        334ms ±17%     266ms ±16%  -20.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     709ms ± 7%     554ms ± 6%  -21.89%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%   +0.00%  (p=0.046 n=9+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.464 n=10+10)
FilterPatterns/Wildcard-4        141MB ± 0%     141MB ± 0%     ~     (p=0.163 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     1.63M ± 0%     ~     (p=0.072 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:47:29 +02:00
Michael Eischer
375c2a56de filter: Parse filter patterns only once 
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       30.3ms ±10%    23.6ms ±20%  -22.12%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       49.0ms ± 3%    32.3ms ± 8%  -33.94%  (p=0.000 n=8+10)
FilterPatterns/Wildcard-4        345ms ± 9%     334ms ±17%     ~     (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4     3.93s ± 2%     0.71s ± 7%  -81.98%  (p=0.000 n=9+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       4.63MB ± 0%    3.57MB ± 0%  -22.98%  (p=0.000 n=9+9)
FilterPatterns/Absolute-4       8.54MB ± 0%    3.57MB ± 0%  -58.20%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        146MB ± 0%     141MB ± 0%   -2.93%  (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4     907MB ± 0%       4MB ± 0%  -99.61%  (p=0.000 n=9+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        66.6k ± 0%     22.2k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4        88.7k ± 0%     22.2k ± 0%  -75.00%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.70M ± 0%     1.63M ± 0%   -3.92%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     4.46M ± 0%     0.02M ± 0%  -99.50%  (p=0.000 n=10+10)
 2020-10-07 20:47:27 +02:00
Michael Eischer
b8eacd1364 filter: Reduce redundant path and pattern splitting 
A single call to filter.List will split the path only once and also
split each search pattern only once and use it for both match and
childMatch.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       62.1ms ±15%    30.3ms ±10%  -51.22%  (p=0.000 n=9+10)
FilterPatterns/Absolute-4        111ms ±10%      49ms ± 3%  -56.08%  (p=0.000 n=10+8)
FilterPatterns/Wildcard-4        393ms ±15%     345ms ± 9%  -12.30%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     10.0s ± 3%      3.9s ± 2%  -60.53%  (p=0.000 n=10+9)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       16.4MB ± 0%     4.6MB ± 0%  -71.76%  (p=0.000 n=10+9)
FilterPatterns/Absolute-4       31.4MB ± 0%     8.5MB ± 0%  -72.77%  (p=0.000 n=9+10)
FilterPatterns/Wildcard-4        168MB ± 0%     146MB ± 0%  -13.19%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.23GB ± 0%    0.91GB ± 0%  -71.96%  (p=0.000 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4         178k ± 0%       67k ± 0%  -62.50%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4         266k ± 0%       89k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.87M ± 0%     1.70M ± 0%   -9.47%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     17.7M ± 0%      4.5M ± 0%  -74.87%  (p=0.000 n=9+10)
 2020-10-07 18:13:19 +02:00
Michael Eischer
e73c281142 filter: Benchmark absolute paths, wildcards and long filter lists 2020-10-07 17:54:36 +02:00
Michael Eischer
fdd3b14db3 filter: test some corner cases 2020-10-07 17:09:44 +02:00
Michael Eischer
f4282aa6fd local: mark repository files as read-only 
This is intended to prevent accidental modifications of data files.
Marking the files as read-only was accidentally removed in #1258.
 2020-10-07 12:29:37 +02:00
Michael Eischer
40ee17167e local: Ignore permission errors on chmod call in Save/Remove operation 
The file is already created with the proper permissions, thus the chmod
call is not critical. However, some file systems have don't allow
modifications of the file permissions. Similarly the chmod call in the Remove
operation should not prevent it from working.
 2020-10-07 12:29:37 +02:00
greatroar
f3e933f0c1 Use windows.SetConsoleCursorPosition in ui/termstatus 2020-10-07 11:18:54 +02:00
MichaelEischer
09d39e260d
Merge pull request #2980 from greatroar/error-wrapping 
Remove repetitive error wrapping from internal/cache
 2020-10-06 15:19:00 +02:00
Michael Eischer
1579d2a8ec Remove some unused assignments 2020-10-06 14:55:13 +02:00
Michael Eischer
eba5dd831f Fix typos reported by misspell 2020-10-06 14:55:13 +02:00
Michael Eischer
9ffb698c8d Modernize context import in internal/fuse 
Migrated using `go fix`.
 2020-10-06 14:55:13 +02:00
Michael Eischer
efbb850d92 Remove a few redundant type specifiers 
This is the result of running `gofmt -s -w **/*.go`
 2020-10-06 14:55:13 +02:00
greatroar
673dda77c0 Less repetitive error messages in internal/cache 
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
 2020-10-05 20:28:54 +02:00
MichaelEischer
1a490acd67
Merge pull request #2982 from greatroar/archiver-error-handling 
Check error in archiver before calling Select
 2020-10-05 19:19:47 +02:00
MichaelEischer
c84643c6a9
Merge pull request #2981 from greatroar/mkdircachedir 
Simplify cache directory creation
 2020-10-05 18:15:23 +02:00
greatroar
1e1a1f3078 Upgrade cespare/xxhash to 2.1.1 2020-10-05 17:25:39 +02:00
greatroar
c4e2203e45 Check error in archiver before calling Select 
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
 2020-10-05 11:11:04 +02:00
greatroar
7d0fa1a686 Simplify cache directory creation 2020-10-05 10:46:45 +02:00
Ingo Gottwald
8b8e230771 Swap deprecated GCS lib with replacement 2020-10-03 18:55:56 +02:00