eleith
a24e986b2b
do not require gs bucket permissions to init repository
...
a gs service account may only have object permissions on an existing
bucket but no bucket create/get permissions.
these service accounts currently are blocked from initialization a
restic repository because restic can not determine if the bucket exists.
this PR updates the logic to assume the bucket exists when the bucket
attribute request results in a permissions denied error.
this way, restic can still initialize a repository if the service
account does have object permissions
fixes: https://github.com/restic/restic/issues/3100
2020-11-18 06:14:11 -08:00
rawtaz
1b6e8c888f
Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param
...
Remove `--drive-use-trash=false` from rclone param
2020-11-17 11:21:20 +01:00
Michael Eischer
1f43cac12d
check: Only track data blobs when unused blobs should be reported
...
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00
Michael Eischer
6da66c15d8
check: Simplify referenced blob tracking
...
The result is identical as long as the context in not canceled. However,
in that case the result is incomplete anyways.
2020-11-15 18:42:55 +01:00
Michael Eischer
3500f9490c
check: Simplify blob status tracking
...
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
2020-11-15 18:42:42 +01:00
Michael Eischer
b8c7543a55
check: Merge 'size could not be found' and 'not found in index' errors
...
By construction these two errors always show up in pairs: 'size could
not be found' is printed when the blob is not found in the repository
index. That blob is also part of the `blobs` array. Later on, check
iterates over that array and checks whether the blob is marked as
existing. Which cannot be the case as that mark is generated by
iterating over the repository index.
The merged warning no longer reports the blob index within a file. That
information could also be derived by printing the affected tree using
`cat` and searching for the blob.
2020-11-15 18:41:50 +01:00
Alexander Neumann
3c0c0c132b
Merge pull request #3006 from aawsome/new-rebuild-index
...
Reimplement rebuild-index and remove /internal/index
2020-11-15 17:48:43 +01:00
Alexander Neumann
9968220652
Merge pull request #2850 from greatroar/packer-malloc
...
Decrease allocation rate in internal/pack
2020-11-15 17:19:49 +01:00
greatroar
ab2b7d7f9a
Decrease allocation rate in internal/pack
...
internal/repository benchmark results:
name old time/op new time/op delta
PackerManager-8 179ms ± 1% 181ms ± 1% +0.78% (p=0.009 n=10+10)
name old speed new speed delta
PackerManager-8 294MB/s ± 1% 292MB/s ± 1% -0.77% (p=0.009 n=10+10)
name old alloc/op new alloc/op delta
PackerManager-8 91.3kB ± 0% 72.2kB ± 0% -20.92% (p=0.000 n=9+7)
name old allocs/op new allocs/op delta
PackerManager-8 1.38k ± 0% 0.76k ± 0% -45.20% (p=0.000 n=10+7)
2020-11-15 16:51:47 +01:00
greatroar
9a8a2cae4c
Move pack testing logic to test file
2020-11-15 16:45:49 +01:00
Alexander Weiss
9607cad267
Remove internal/index
2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a
Reimplement rebuild-index
2020-11-15 07:05:09 +01:00
Alexander Weiss
187c8fb259
Parallelize MasterIndex.Save()
2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5
Add extraObsolete to MasterIndex.Save
2020-11-15 07:05:09 +01:00
Alexander Weiss
5898cb341f
Use CreateIndexFromPacks() in test
2020-11-15 07:05:05 +01:00
Alexander Weiss
43732bb885
Add CreateIndexFromPacks()
2020-11-15 07:04:51 +01:00
greatroar
c45f8ee075
Replace restic.statT interface by concrete types
...
name old time/op new time/op delta
NodeFillUser-8 1.81µs ± 9% 1.50µs ± 5% -17.07% (p=0.000 n=19+20)
NodeFromFileInfo-8 1.76µs ± 4% 1.49µs ± 6% -15.63% (p=0.000 n=20+19)
name old alloc/op new alloc/op delta
NodeFillUser-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
NodeFromFileInfo-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
NodeFillUser-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
NodeFromFileInfo-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
MichaelEischer
3601a9b6cd
Merge pull request #2690 from SkYNewZ/master
...
Fix #2688 : Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
MichaelEischer
333c5a19d4
Merge pull request #3082 from aawsome/check-sizes
...
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
Quentin Lemaire
ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list
2020-11-14 15:48:56 +00:00
Alexander Weiss
17bb77b1f9
check: Also check blob length and offset
2020-11-14 00:42:49 +01:00
Alexander Weiss
80dcfca191
check: Check sizes computed from index and pack header
2020-11-14 00:42:49 +01:00
tofran
94a154c7ca
Remove --drive-use-trash=false
from rclone param
...
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
2020-11-13 22:58:48 +00:00
Alexander Neumann
04d1983800
Merge pull request #3090 from fgma/vss-fix-386
...
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-13 21:30:34 +01:00
Alexander Neumann
8ad9f88993
helpers: Improve error message
2020-11-12 20:38:31 +01:00
fgma
f9223cd827
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-12 19:31:00 +01:00
rawtaz
dfb9326b1b
Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag
...
Extended option to select V1 API for ListObjects on S3 backend
2020-11-11 20:52:44 +01:00
Alexander Neumann
4a0b7328ec
s3: Remove dots for config description
2020-11-11 20:20:35 +01:00
greatroar
63e32c44c0
Improve error reporting from restic diff
...
Instead of a stacktrace, restic diff 111 222 now reports:
Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
Nick Douma
829959390a
Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1
2020-11-11 11:54:38 +01:00
Nick Douma
ccd55d529d
Add s3.list-objects-v1 extended option and default to false
2020-11-11 11:54:36 +01:00
Nick Douma
4ddcc17135
Add support for boolean extended options
2020-11-11 11:54:27 +01:00
MichaelEischer
407843c5f9
Merge pull request #3034 from hoyho/gh_master
...
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
MichaelEischer
46d31ab86d
Merge pull request #3058 from greatroar/counter
...
Replace restic.Progress with new progress.Counter (fixes two race conditions)
2020-11-09 22:19:09 +01:00
Alexander Neumann
c986823d3f
Merge pull request #3048 from aawsome/check-pack-index
...
check: check index for packs that are read
2020-11-09 20:12:32 +01:00
Alexander Weiss
239931578c
check: check index for packs that are read
2020-11-09 17:28:14 +01:00
hoyho
9df52327cc
bugfix: omit ENOTDATA for extended attributes
...
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
greatroar
21b787a4d1
Stop Counters where they're constructed and started
2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2
Replace restic.Progress with new progress.Counter
...
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
Alexander Neumann
3bf43d7951
Merge pull request #2475 from restic/use-github-actions
...
Use GitHub Actions for CI
2020-11-08 17:30:12 +01:00
Alexander Neumann
5cf42884c8
Run tests on GitHub Actions
2020-11-08 16:04:57 +01:00
Alexander Weiss
826cfa0533
fix context in archiver tests
2020-11-08 08:24:24 +01:00
Alexander Weiss
fef408a8bd
Return context error in mem backend
2020-11-08 00:05:53 +01:00
greatroar
a2d4209322
Don't recurse in local backend's List if not required
...
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.
Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9 ).
2020-11-07 08:54:13 +01:00
Alexander Weiss
fd33030556
Use in-memory index to rebuild index in prune
2020-11-06 20:23:30 +01:00
Alexander Weiss
38cc4393f6
Add Masterindex.Save(); Add Index.Packs()
2020-11-06 20:23:30 +01:00
Alexander Weiss
aaf1c44362
Fix #3062
2020-11-05 17:05:42 +01:00
Ivan Andreev
ab2790d9de
Fix http2 stream reset between restic and rest backends #3014
2020-11-05 15:57:40 +03:00
Nick Craig-Wood
86b5d8ffaa
s3: add bucket-lookup parameter to select path or dns style bucket lookup
...
This is to enable restic working with Alibaba cloud
Fixes #2528
2020-11-05 12:20:10 +01:00
Alexander Neumann
ae5302c7a8
Add comment that keepBlobs is modified
2020-11-05 10:33:38 +01:00
Alexander Neumann
866a52ad4e
Remove unneeded seek
...
The file returned from DownloadAndHash() is already seeked to the start
of the file.
2020-11-05 10:31:49 +01:00
Alexander Neumann
a4507610a0
Fix typo
2020-11-05 10:31:49 +01:00
Alexander Neumann
7def2d8ea7
Use a non-constant seed
2020-11-05 10:31:49 +01:00
Alexander Neumann
ee0112ab3b
Clarify message about expected error
2020-11-05 10:31:49 +01:00
Michael Eischer
b373f164fe
prune: Parallelize repack command
2020-11-05 10:31:49 +01:00
Alexander Neumann
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command
...
Reimplementation of prune
2020-11-05 10:12:19 +01:00
fgma
916b2d303b
vss: fix potential crash (not reachable in restic) ( #3045 )
...
HasSufficientPrivilegesForVSS() now returns an error
2020-11-04 22:14:18 +01:00
Alexander Neumann
a06f5c28c0
Merge pull request #3055 from greatroar/cleanup
...
Don't separately allocate sync.{Mutex,Once} if not necessary
2020-11-04 11:11:21 +01:00
greatroar
c2f3eee5af
Don't separately allocate sync.{Mutex,Once} if not necessary
...
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
2020-11-04 09:55:43 +01:00
Alexander Weiss
b2f5381737
Make realistic forget --prune --dryrun
2020-11-03 16:42:21 +01:00
Alexander Weiss
ce7d613749
Add Blob.Handle()
2020-11-03 16:42:21 +01:00
Alexander Weiss
581d90cf91
Make some pack parameters public
2020-11-03 16:42:21 +01:00
Alexander Neumann
9a88fb253b
Merge pull request #3051 from greatroar/sanitize-env
...
Sanitize environment before starting backend processes (rclone, ssh)
2020-11-02 21:18:57 +01:00
Alexander Neumann
f14436953a
Merge pull request #3050 from greatroar/widechars
...
Fix string truncation in ui/termstatus
2020-11-02 16:58:51 +01:00
greatroar
11fbaaae9a
Sanitize environment before starting backend processes (rclone, ssh)
...
The restic security model includes full trust of the local machine, so
this should not fix any actual security problems, but it's better to be
safe than sorry.
Fixes #2192 .
2020-11-02 16:41:23 +01:00
greatroar
3ed84ff0c6
Fix string truncation in ui/termstatus
...
Fixes #3046 .
2020-11-02 12:50:49 +01:00
Alexander Neumann
8e965ed4eb
Merge pull request #3039 from MichaelEischer/tar-proper-dirs
...
Properly dump directories to tar
2020-11-02 12:30:26 +01:00
Alexander Neumann
5f0fa2129e
Improve readability
...
It's time to use a switch statement.
2020-11-02 11:24:26 +01:00
Alexander Neumann
6509c207f4
Merge pull request #2997 from MichaelEischer/faster-excludes
...
Speedup exclude/include checking
2020-11-02 11:07:21 +01:00
Alexander Neumann
445b845267
Merge pull request #2978 from MichaelEischer/warn-tree-error
...
Warn if backup failed to read tree blob
2020-11-02 10:14:12 +01:00
Alexander Neumann
3ff37215df
Merge pull request #2935 from MichaelEischer/upgrade-minio
...
Upgrade minio SDK to version 7
2020-11-02 09:09:10 +01:00
Sergio Rubio
e708628cfd
Remove unused function
...
Not currently used, and it'd need to be added to the MasterIndex interface first.
2020-10-28 13:24:49 +01:00
MichaelEischer
bb4b3481a6
Merge pull request #3031 from greatroar/debug-no-seek
...
Allow debug logging to pipes and terminals
2020-10-25 17:59:06 +01:00
MichaelEischer
ad3a52e6f0
Merge pull request #3026 from greatroar/refactor-ui
...
internal/ui refactoring
2020-10-25 17:51:39 +01:00
Michael Eischer
8d7d6ad2d5
dump: include username in tar
2020-10-24 22:42:01 +02:00
Michael Eischer
fe09e6f865
dump: test proper permissions and directory name
2020-10-24 22:42:01 +02:00
Michael Eischer
1e3c9a2c11
dump: Fix file permission to tar mapping
...
The file permissions included a go specific directory bit which
accidentially forced the usage of the GNU header format. This leads
to problems with 7zip on Windows or when extended attributes are
used.
2020-10-24 22:42:01 +02:00
Michael Eischer
e21dcb0eea
dump: Additional ACL tests
2020-10-24 22:41:01 +02:00
MichaelEischer
31b8d7a639
Merge pull request #2274 from fgma/master
...
Support for Volume Shadow Copy Service (VSS) on windows
2020-10-24 15:43:34 +02:00
fgma
5695f9ebd2
vss: Implement VSS support for Windows
...
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.
Snapshotting volumes also covers mountpoints but skips UNC paths.
2020-10-24 11:35:57 +02:00
greatroar
63be3704d9
Allow debug logging to pipes and terminals
2020-10-19 22:30:30 +02:00
greatroar
35419de232
Simplify ui.StdioWrapper.Write
...
Instead of looping to find line breaks, make it look for the last one.
2020-10-17 20:30:46 +02:00
greatroar
863a590a81
Refactor termstatus.Term.{Print,Error} methods
2020-10-17 20:30:46 +02:00
greatroar
7c0b6a82db
Remove unused public method ui.linesWriter.Flush
2020-10-17 20:30:24 +02:00
MichaelEischer
96a912b65a
Merge pull request #3025 from aawsome/fix-index-ids
...
Fix setting of ID in DecodeIndex
2020-10-17 09:54:45 +02:00
Alexander Weiss
b44ecde8b0
Fix setting of ID in DecodeIndex
2020-10-17 09:12:58 +02:00
MichaelEischer
39fe1e96fe
Merge pull request #3018 from greatroar/background-check
...
Fix IsProcessBackground on Linux with stdin redirection
2020-10-16 22:50:19 +02:00
MichaelEischer
4ba237bb93
Merge pull request #3019 from greatroar/refactor-decodeindex
...
Refactor index decoding
2020-10-15 23:22:33 +02:00
greatroar
b27375f5ce
defer close(ch) outside repository.RunWorkers
2020-10-14 15:50:16 +02:00
greatroar
720e0ee0c7
if cond { return true }; return false => return cond
2020-10-13 20:56:43 +02:00
greatroar
27db3ec262
Refactor index decoding
...
Decoding old-format indices no longer requires loading and decrypting
twice.
2020-10-13 20:47:50 +02:00
greatroar
f80b07b2c8
Fix IsProcessBackground on Linux with stdin redirection
...
The previous implementation assumed that stdin was a terminal.
It now checks the terminal's fd.
2020-10-13 13:12:02 +02:00
MichaelEischer
41a45ae908
Merge pull request #3016 from greatroar/uid-gid-lookup
...
Simplify/optimize cached UID/GID lookups
2020-10-12 23:24:55 +02:00
Alexander Neumann
56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
...
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
greatroar
f0cd16e5ea
Cache uint32-typed ids in lookup{Username,Group}
...
NodeFillUser-8 1.92µs ± 5% 1.75µs ± 3% -8.89% (p=0.000 n=10+10)
NodeFromFileInfo-8 1.89µs ± 7% 1.76µs ± 4% -6.69% (p=0.001 n=10+10)
name old alloc/op new alloc/op delta
NodeFillUser-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10)
NodeFromFileInfo-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
NodeFillUser-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)
NodeFromFileInfo-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)
2020-10-12 15:08:15 +02:00
greatroar
a03fe4562e
Remove unused error return in lookup{Username,Group}
2020-10-12 14:48:23 +02:00
Alexander Neumann
c193cea119
Pass Context to NewRestorer()
2020-10-10 15:24:26 +02:00
Alexander Neumann
f0d49ca600
Merge pull request #2933 from MichaelEischer/less-context-todo
...
Replace most usages of context.TODO()
2020-10-10 15:03:44 +02:00
kitone
6099f81692
Fix #1212 restore code produces inconsistent timestamps/permissions.
...
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
2020-10-10 13:46:44 +02:00
kitone
9d7f616190
Improve restorer debug log information
2020-10-10 13:46:43 +02:00
kitone
295ddb9e57
Add test case for inconsistent timestamps and permissions restoration
...
Reproduce from https://github.com/restic/restic/issues/1212
2020-10-10 13:46:43 +02:00
Michael Eischer
e638b46a13
Embed context into ReaderAt
...
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
2020-10-09 22:39:07 +02:00
Michael Eischer
d6cfe857b7
pass proper context into MasterIndex.RebuildIndex
2020-10-09 22:39:07 +02:00
Michael Eischer
37a5e2d681
rest: use global context on repository creation
2020-10-09 22:39:06 +02:00
Michael Eischer
4b0fcaed45
unlock: use proper context for locks cleanup
...
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
2020-10-09 22:37:56 +02:00
Michael Eischer
645a6efaf2
restorer: remove redundant type specification
2020-10-09 22:37:56 +02:00
Michael Eischer
a449450021
init: pass proper context to master key generation
...
This is no change in behavior as a canceled context did later on cause
the config file creation to fail. Therefore this change just lets the
repository initialization fail a bit earlier.
2020-10-09 22:37:56 +02:00
Michael Eischer
603bb0e309
restore: Use proper context while loading snapshot
2020-10-09 22:37:56 +02:00
Michael Eischer
c458e114d4
pass context to Find / FindSnapshot
...
This allows proper interruption of restic while it searches for
snapshots or key files.
2020-10-09 22:37:56 +02:00
Michael Eischer
45e9a55c62
Wire context into backend layout detection
2020-10-09 22:37:24 +02:00
Michael Eischer
307a6ba3a3
Upgrade minio sdk to v7
...
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
2020-10-09 22:37:24 +02:00
Michael Eischer
50da20d93d
Warn if backup failed to read tree blob
2020-10-09 22:36:27 +02:00
Alexander Neumann
5fd3dbccb7
Merge pull request #2963 from MichaelEischer/fix-status-deadlock
...
backup: Fix possible deadlock of scanner goroutine
2020-10-09 21:35:50 +02:00
Alexander Neumann
35655a481b
Merge pull request #2931 from MichaelEischer/reduce-debug-overhead
...
Reduce debug log overhead
2020-10-09 21:35:11 +02:00
Alexander Neumann
30cb553c8d
Merge pull request #2932 from MichaelEischer/proper-rclone-create
...
Call rclone.Create to create a new repository for the rclone backend
2020-10-09 21:29:15 +02:00
MichaelEischer
356ac404cd
Merge pull request #2996 from greatroar/setconsolecursorposition
...
Use windows.SetConsoleCursorPosition in ui/termstatus
2020-10-09 18:19:02 +02:00
Michael Eischer
88c8e903d2
filter: Fix glob matching on absolute path marker on windows
...
A pattern part containing "/" is used to mark a path or a pattern as
absolute. However, on Windows the path separator is "\" such that glob
patterns like "?" could match the marker. The code now explicitly skips
the marker when the pattern does not represent an absolute path.
2020-10-09 16:11:05 +02:00
greatroar
740758a5fa
Optimize filter pattern matching
...
By replacing "**" with "", checking for this special path component can
be reduced to a length-zero check.
name old time/op new time/op delta
FilterLines-8 44.7ms ± 5% 44.9ms ± 5% ~ (p=0.631 n=10+10)
FilterPatterns/Relative-8 13.6ms ± 4% 13.4ms ± 5% ~ (p=0.165 n=10+10)
FilterPatterns/Absolute-8 10.9ms ± 5% 10.7ms ± 4% ~ (p=0.052 n=10+10)
FilterPatterns/Wildcard-8 53.7ms ± 5% 50.4ms ± 5% -6.00% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-8 128ms ± 2% 95ms ± 1% -25.54% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=1.000 n=9+8)
FilterPatterns/Absolute-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.903 n=9+8)
FilterPatterns/Wildcard-8 19.7MB ± 0% 19.7MB ± 0% -0.00% (p=0.022 n=10+9)
FilterPatterns/ManyNoMatch-8 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-8 88.7k ± 0% 88.7k ± 0% ~ (all equal)
FilterPatterns/ManyNoMatch-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-09 15:51:14 +02:00
MichaelEischer
862ee4b2c9
Merge pull request #2970 from labkode/fopenskip
...
Skip fopen for file change check to avoid network penalty
2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador
958dc6aafc
Skip fopen for file change check to avoid network penalty
2020-10-09 10:53:28 +02:00
MichaelEischer
88cc444779
Merge pull request #2934 from MichaelEischer/upgrade-backoff
...
Upgrade github.com/cenkalti/backoff module
2020-10-08 20:33:30 +02:00
Michael Eischer
b79f18209f
Upgrade github.com/cenkalti/backoff module
...
We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats
an operation endlessly when a retry count of 0 is specified. This
required a few fixes for the tests.
2020-10-07 22:04:59 +02:00
Michael Eischer
8388e67c4b
filter: cleanup path separator conversion
2020-10-07 21:14:07 +02:00
Michael Eischer
0acc3c5923
filter: special case patterns without globbing characters
...
In case a part of a path is a simple string, we can just check for
equality without complex parsing in filepath.Match.
name old time/op new time/op delta
FilterLines-4 34.8ms ±17% 41.2ms ±23% +18.36% (p=0.000 n=10+10)
FilterPatterns/Relative-4 21.7ms ± 6% 12.1ms ±23% -44.46% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 10.0ms ± 5% 9.1ms ±11% -9.80% (p=0.006 n=10+9)
FilterPatterns/Wildcard-4 47.0ms ± 7% 42.2ms ± 5% -10.19% (p=0.000 n=9+10)
FilterPatterns/ManyNoMatch-4 190ms ± 1% 131ms ±20% -31.47% (p=0.000 n=8+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.870 n=9+9)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.145 n=10+10)
FilterPatterns/Wildcard-4 14.3MB ± 0% 19.7MB ± 0% +37.91% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.421 n=10+9)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:55:43 +02:00
Michael Eischer
54a124de3b
filter: explicitly test separate ListWithChild function
2020-10-07 20:47:52 +02:00
Michael Eischer
bcc3bddcf4
filter: only check whether a child path could match when necessary
...
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.
name old time/op new time/op delta
FilterPatterns/Relative-4 23.3ms ± 9% 21.7ms ± 6% -6.68% (p=0.004 n=10+10)
FilterPatterns/Absolute-4 13.9ms ± 7% 10.0ms ± 5% -27.61% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 51.4ms ± 7% 47.0ms ± 7% -8.51% (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4 551ms ± 9% 190ms ± 1% -65.41% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.665 n=10+9)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.480 n=9+10)
FilterPatterns/Wildcard-4 14.3MB ± 0% 14.3MB ± 0% ~ (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:47:52 +02:00
Michael Eischer
17c53efb0d
filter: Optimize double wildcard expansion
...
This only allocates a single slice to expand the double wildcard and
only copies the pattern prefix once.
name old time/op new time/op delta
FilterPatterns/Relative-4 22.7ms ± 5% 23.3ms ± 9% ~ (p=0.353 n=10+10)
FilterPatterns/Absolute-4 14.2ms ±13% 13.9ms ± 7% ~ (p=0.853 n=10+10)
FilterPatterns/Wildcard-4 266ms ±16% 51ms ± 7% -80.67% (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4 554ms ± 6% 551ms ± 9% ~ (p=0.436 n=10+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.349 n=10+10)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.073 n=10+9)
FilterPatterns/Wildcard-4 141MB ± 0% 14MB ± 0% -89.89% (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 1.63M ± 0% 0.09M ± 0% -94.56% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:47:48 +02:00
Michael Eischer
7959796269
filter: Special case for absolute paths
...
name old time/op new time/op delta
FilterPatterns/Relative-4 23.6ms ±20% 22.7ms ± 5% ~ (p=0.684 n=10+10)
FilterPatterns/Absolute-4 32.3ms ± 8% 14.2ms ±13% -56.01% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 334ms ±17% 266ms ±16% -20.56% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 709ms ± 7% 554ms ± 6% -21.89% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% +0.00% (p=0.046 n=9+10)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.464 n=10+10)
FilterPatterns/Wildcard-4 141MB ± 0% 141MB ± 0% ~ (p=0.163 n=9+10)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 1.63M ± 0% 1.63M ± 0% ~ (p=0.072 n=10+10)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:47:29 +02:00
Michael Eischer
375c2a56de
filter: Parse filter patterns only once
...
name old time/op new time/op delta
FilterPatterns/Relative-4 30.3ms ±10% 23.6ms ±20% -22.12% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 49.0ms ± 3% 32.3ms ± 8% -33.94% (p=0.000 n=8+10)
FilterPatterns/Wildcard-4 345ms ± 9% 334ms ±17% ~ (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4 3.93s ± 2% 0.71s ± 7% -81.98% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 4.63MB ± 0% 3.57MB ± 0% -22.98% (p=0.000 n=9+9)
FilterPatterns/Absolute-4 8.54MB ± 0% 3.57MB ± 0% -58.20% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 146MB ± 0% 141MB ± 0% -2.93% (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4 907MB ± 0% 4MB ± 0% -99.61% (p=0.000 n=9+9)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 66.6k ± 0% 22.2k ± 0% -66.67% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 88.7k ± 0% 22.2k ± 0% -75.00% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 1.70M ± 0% 1.63M ± 0% -3.92% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 4.46M ± 0% 0.02M ± 0% -99.50% (p=0.000 n=10+10)
2020-10-07 20:47:27 +02:00
Michael Eischer
b8eacd1364
filter: Reduce redundant path and pattern splitting
...
A single call to filter.List will split the path only once and also
split each search pattern only once and use it for both match and
childMatch.
name old time/op new time/op delta
FilterPatterns/Relative-4 62.1ms ±15% 30.3ms ±10% -51.22% (p=0.000 n=9+10)
FilterPatterns/Absolute-4 111ms ±10% 49ms ± 3% -56.08% (p=0.000 n=10+8)
FilterPatterns/Wildcard-4 393ms ±15% 345ms ± 9% -12.30% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 10.0s ± 3% 3.9s ± 2% -60.53% (p=0.000 n=10+9)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 16.4MB ± 0% 4.6MB ± 0% -71.76% (p=0.000 n=10+9)
FilterPatterns/Absolute-4 31.4MB ± 0% 8.5MB ± 0% -72.77% (p=0.000 n=9+10)
FilterPatterns/Wildcard-4 168MB ± 0% 146MB ± 0% -13.19% (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4 3.23GB ± 0% 0.91GB ± 0% -71.96% (p=0.000 n=10+9)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 178k ± 0% 67k ± 0% -62.50% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 266k ± 0% 89k ± 0% -66.67% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 1.87M ± 0% 1.70M ± 0% -9.47% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 17.7M ± 0% 4.5M ± 0% -74.87% (p=0.000 n=9+10)
2020-10-07 18:13:19 +02:00
Michael Eischer
e73c281142
filter: Benchmark absolute paths, wildcards and long filter lists
2020-10-07 17:54:36 +02:00
Michael Eischer
fdd3b14db3
filter: test some corner cases
2020-10-07 17:09:44 +02:00
Michael Eischer
f4282aa6fd
local: mark repository files as read-only
...
This is intended to prevent accidental modifications of data files.
Marking the files as read-only was accidentally removed in #1258 .
2020-10-07 12:29:37 +02:00
Michael Eischer
40ee17167e
local: Ignore permission errors on chmod call in Save/Remove operation
...
The file is already created with the proper permissions, thus the chmod
call is not critical. However, some file systems have don't allow
modifications of the file permissions. Similarly the chmod call in the Remove
operation should not prevent it from working.
2020-10-07 12:29:37 +02:00
greatroar
f3e933f0c1
Use windows.SetConsoleCursorPosition in ui/termstatus
2020-10-07 11:18:54 +02:00
MichaelEischer
09d39e260d
Merge pull request #2980 from greatroar/error-wrapping
...
Remove repetitive error wrapping from internal/cache
2020-10-06 15:19:00 +02:00
Michael Eischer
1579d2a8ec
Remove some unused assignments
2020-10-06 14:55:13 +02:00
Michael Eischer
eba5dd831f
Fix typos reported by misspell
2020-10-06 14:55:13 +02:00
Michael Eischer
9ffb698c8d
Modernize context import in internal/fuse
...
Migrated using `go fix`.
2020-10-06 14:55:13 +02:00
Michael Eischer
efbb850d92
Remove a few redundant type specifiers
...
This is the result of running `gofmt -s -w **/*.go`
2020-10-06 14:55:13 +02:00
greatroar
673dda77c0
Less repetitive error messages in internal/cache
...
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
2020-10-05 20:28:54 +02:00
MichaelEischer
1a490acd67
Merge pull request #2982 from greatroar/archiver-error-handling
...
Check error in archiver before calling Select
2020-10-05 19:19:47 +02:00
MichaelEischer
c84643c6a9
Merge pull request #2981 from greatroar/mkdircachedir
...
Simplify cache directory creation
2020-10-05 18:15:23 +02:00
greatroar
1e1a1f3078
Upgrade cespare/xxhash to 2.1.1
2020-10-05 17:25:39 +02:00
greatroar
c4e2203e45
Check error in archiver before calling Select
...
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
2020-10-05 11:11:04 +02:00
greatroar
7d0fa1a686
Simplify cache directory creation
2020-10-05 10:46:45 +02:00
Ingo Gottwald
8b8e230771
Swap deprecated GCS lib with replacement
2020-10-03 18:55:56 +02:00
Ingo Gottwald
00cedd22aa
Replace deprecated method in gs backend
2020-10-01 10:02:42 +02:00
Michael Eischer
d44df9d00d
backup: Always remove the status lines once a backup ends
2020-09-30 23:13:10 +02:00
Michael Eischer
8d0ba55ecd
backup: Fix possible deadlock of scanner goroutine
...
When the backup is interrupted for some reason while the scanner is
still active this could lead to a deadlock. Interruptions are triggered
by canceling the context object used by both the backup progress UI and
the scanner. It is possible that a context is canceled between the
respective check in the scanner and it calling the `ReportTotal` method
of the UI. The latter method sends a message to the UI goroutine.
However, a canceled context will also stop that goroutine, which can
cause the channel send operation to block indefinitely.
This is resolved by adding a `closed` channel which is closed once the
UI goroutine is stopped and serves as an escape hatch for reported UI
updates.
This change covers not just the ReportTotal method but all potentially
affected methods of the progress UI implementation.
2020-09-30 23:13:10 +02:00
MichaelEischer
48d0ab5276
Merge pull request #2959 from restic/keep-text
...
backup: Correct keep policy text
2020-09-30 17:57:00 +02:00
MichaelEischer
fd02407863
Merge pull request #2849 from classmarkets/gcs-access-token
...
gs: support authentication with access token
2020-09-30 17:42:56 +02:00
Leo R. Lundgren
028c9a5343
backup: Correct keep policy text
...
Makes the following corrections to the "Applying Policy:" output:
- keep the last 1 snapshots snapshots => keep 1 latest snapshots
- keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots
2020-09-28 14:26:53 +02:00
MichaelEischer
14b312f00d
Merge pull request #2658 from creativeprojects/issue-2241
...
Don't echo authentication passwords (rest backend)
2020-09-22 22:17:53 +02:00
Fred
206cadfab4
Hide password from repository URLs
2020-09-22 22:00:51 +02:00
MichaelEischer
4875f7b659
Merge pull request #2614 from greatroar/simplify-fs
...
Simplify internal/fs
2020-09-21 22:05:10 +02:00
greatroar
0d65b78168
Simplify os.ModeType|os.ModeCharDevice => os.ModeType
...
Since Go 1.12, ModeCharDevice is included in ModeType:
golang/go@a2a3dd00c9
2020-09-21 14:21:32 +02:00
greatroar
95ebba85ff
Remove stray Printf from internal/fs
2020-09-21 14:21:32 +02:00
greatroar
59b343a9bf
Remove OS-specific versions of fs.MkdirAll
...
Go has supported Windows paths correctly since 1.11, see
https://github.com/golang/go/issues/10900 and the commit referenced
there.
2020-09-21 14:21:32 +02:00
greatroar
1557c58eef
Fix and simplify fs.Reader
...
fakeDir.{Readdir,Readdirnames} weren't handling the case n == 0
correctly. fakeFileInfo.sys is always nil, so omit the field.
2020-09-21 14:21:32 +02:00
greatroar
c504aa505c
Remove unused fs.Rename
2020-09-21 14:21:32 +02:00
greatroar
1b20f6beec
Remove io.Writer from fs.File
...
It was only used in a single test, which now uses plain *os.File instead.
2020-09-21 14:21:32 +02:00
Michael Eischer
8c36317b71
rclone: use configured number of connections during create
2020-09-19 19:11:43 +02:00
Michael Eischer
60a5c35de9
rclone: close connection to rclone if open fails
2020-09-19 19:11:43 +02:00
Michael Eischer
07f4e7d10b
Only log HTTP requests when a debug log is configured
...
The logged HTTP requests are only visible when a debug log is
configured.
2020-09-19 17:07:53 +02:00
Michael Eischer
f003410402
init: Add --copy-chunker-params
option
...
This allows creating multiple repositories with identical chunker
parameters which is required for working deduplication when copying
snapshots between different repositories.
2020-09-19 16:53:05 +02:00
Michael Eischer
9ad8250a78
Reduce overhead of debug calls if no log is enabled
...
In case no debug log is configured, then calls to debug.Log only incur
the costs to check a single boolean flag making the call really cheap.
2020-09-19 16:45:01 +02:00
greatroar
cb3f531050
Make Cache.Path private
...
It's only used inside the package.
2020-09-18 11:17:29 +02:00
greatroar
23fcbb275a
Remove restic.Cache interface
...
It was used in one code path, which then asserted its concrete type as
*cache.Cache.
Privatised some of the interface methods.
2020-09-18 10:48:13 +02:00
greatroar
6e3215a80d
Remove unused error type from internal/cache
2020-09-18 10:38:14 +02:00
greatroar
9abef3bf1a
Move internal/fs.TestChdir to internal/test.Chdir
2020-09-17 10:43:33 +02:00
Alexander Neumann
0c48e515f0
Merge pull request #2630 from MichaelEischer/fix-staticcheck
...
Fix lots of small issues reported by staticcheck
2020-09-13 16:19:22 +02:00
Alexander Neumann
97950ab81a
options: Fix test for Go >= 1.15
2020-09-12 17:36:44 +02:00
MichaelEischer
88664ba222
Merge pull request #2893 from MichaelEischer/restore-preallocate
...
restorer: Preallocate files
2020-09-08 22:43:05 +02:00
Michael Eischer
121233e1b3
Add preallocate tests
2020-09-07 21:41:47 +02:00
Michael Eischer
8cc9514879
restorer: pre-allocate files before loading chunks
2020-09-07 21:41:47 +02:00
Michael Eischer
2e7d475029
Process packs in order of first appearance
2020-09-06 21:25:35 +02:00
Michael Eischer
c46edcd9d6
error strings should not end with punctuation
2020-09-05 10:07:17 +02:00
Michael Eischer
1ede018ea6
error variable names should start with 'Err'
2020-09-05 10:07:17 +02:00
Michael Eischer
b77e933d80
Convert underscore variable names to camelCase
2020-09-05 10:07:16 +02:00
Michael Eischer
d0329cf3eb
Adjust comments to match name of exported methods
2020-09-05 10:07:16 +02:00
Michael Eischer
dc31529fc3
Unindent else block after if block ending with a return statement
2020-09-05 10:07:16 +02:00
Michael Eischer
4784540f04
repository: Simplify worker group code
2020-09-05 10:07:16 +02:00
Michael Eischer
84ea2389ae
archiver_test: Only skip symlinks on windows
2020-09-05 10:07:16 +02:00
Michael Eischer
b4a7ce86cf
uint cannot be less than zero
2020-09-05 10:07:16 +02:00
Michael Eischer
7ee0964880
crypto: Remove unused error
2020-09-05 10:07:16 +02:00
Michael Eischer
f6f11400c2
restic: Remove unused variable value
2020-09-05 10:07:16 +02:00
Michael Eischer
e2dc5034d3
restic/file: Use FileType for all File constants
...
The data type of a variable is only carried on when using iota.
2020-09-05 10:07:16 +02:00
Michael Eischer
9a1b3cb5d9
restorer: Remove unused treeID struct member from test
2020-09-05 10:07:16 +02:00
Michael Eischer
ee05501ce7
archiver: Remove unused chmod function
2020-09-05 10:07:16 +02:00
Michael Eischer
014600bee6
archiver: Remove unused variable from test
2020-09-05 10:07:16 +02:00
Michael Eischer
d9a80e07b9
repository: Simplify index age calculation
2020-09-05 10:07:16 +02:00
Michael Eischer
460e2ffbf6
Collapse a few boolean operations
2020-09-05 10:07:14 +02:00
Michael Eischer
2f8335554c
Remove a few unused variables
2020-09-05 10:06:23 +02:00
Michael Eischer
37113282ca
Merge a few variable declaration and initializations
2020-09-05 10:05:34 +02:00
Michael Eischer
337725c354
Use sort.Strings
2020-09-05 10:05:34 +02:00
Michael Eischer
dcf9ded977
dump: Use slashes as path separators in tar on windows
...
This conversion is necessary as paths returned by filepath.Rel use the
operating-system native path separator
2020-08-30 18:26:50 +02:00
Simon Beck
fe6445e0f4
dump: Added unit tests for tar output
2020-08-30 16:00:48 +02:00
Simon Beck
80a11960dd
dump: Always dump relative paths into tarballs
...
Tarballs should only contain relative paths.
2020-08-29 21:14:34 +02:00
Simon Beck
d6f739ec22
dump: Extract tar code to internal/dump
2020-08-29 21:14:34 +02:00
Michael Eischer
b25978a53c
backup: Fix reporting of directory count in summary
...
Previously the directory stats were reported immediately after calling
`SaveDir`. However, as the latter method saves the tree asynchronously
the stats were still initialized to their nil value. The stats are now
reported via a callback similar to the one used for the fileSaver.
2020-08-27 22:43:51 +02:00
Alexander Neumann
b1b3f1ecb6
Merge pull request #2674 from MichaelEischer/prune-strict-checks
...
prune: Stricter error checks
2020-08-23 10:30:41 +02:00
Hristo Trendev
51c22f4223
local backend: ignore not supported error on sync()
...
Closes #2395
2020-08-22 01:27:07 +02:00
Michael Eischer
f4b9544ab2
prune: Add test that repack aborts on wrong blob
2020-08-16 11:34:01 +02:00
Michael Eischer
367449dede
prune: Reduce memory allocations while repacking
...
The slicing operator `slice[low:high]` default to 0 for the lower bound and
len(slice) for the upper bound when either or both are not specified.
Fix the code to use `cap(slice)` to check for the slice capacity.
2020-08-16 11:34:01 +02:00
Michael Eischer
7042bafea5
prune: Abort repacking when a pack contains a wrong blob
...
If a blob in a pack file can be decrypted successfully but contains data
that results in a different hash than stated in the header pack, then
abort repacking. As both the pack header and the blob are
cryptographically verified this either means than a malicious entity
tampered with the backup or indicates hardware problems on the client.
prune should fail with an error in both cases.
2020-08-16 11:34:01 +02:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" ( #2885 )
...
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
MichaelEischer
eca0f0ad24
Merge pull request #2863 from aawsome/index-no-duplicates
...
Don't save exact duplicates in merged index
2020-08-08 18:24:14 +02:00
Alexander Weiss
b112533812
Don't save exact duplicates when merging indexes
2020-08-05 06:32:02 +02:00
Alexander Weiss
5e63294355
Add benchmark MasterIndexAlloc
2020-08-05 06:32:02 +02:00
Michael Eischer
05116e4787
prune: Cleanup progress bar handling while repacking
2020-08-03 19:32:46 +02:00
Alexander Neumann
b7b479b668
Merge pull request #2599 from MichaelEischer/tweak-mem-usage
...
Reduce memory usage when searching for used blobs
2020-08-01 13:49:02 +02:00
Alexander Neumann
4cf9656f12
Merge pull request #2861 from MichaelEischer/fix-rclone-crash
...
rclone: Don't panic after unexpected subprocess exit
2020-08-01 13:08:50 +02:00
Alexander Neumann
2580eef2aa
Merge pull request #2318 from classmarkets/2175-named-keys
...
Allow specifying user and host when adding keys
2020-08-01 13:06:31 +02:00
Michael Eischer
a178e5628e
Remove duplicate TreeLoader interface
2020-08-01 12:29:16 +02:00
Michael Eischer
af66a62c04
FindUsedBlobs: Test that seen blobs are skipped
...
This also copies the TreeLoader interface from internal/walker to allow
stubbing the repository in the call to `FindUsedBlobs`.
2020-08-01 12:29:16 +02:00
Michael Eischer
9ea1a78bd4
FindUsedBlobs: Check for seen blobs before loading trees
...
The only effective change in behavior is that that toplevel nodes can
also be skipped.
2020-08-01 12:29:16 +02:00
Michael Eischer
184103647a
FindUsedBlobs: merge seen into blobs BlobSet
...
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.
Suggested-by: Alexander Weiss <alex@weissfam.de>
2020-08-01 12:29:16 +02:00
Michael Eischer
c81b122374
rclone: Don't panic after unexpected subprocess exit
...
As the connection to the rclone child process is now closed after an
unexpected subprocess exit, later requests will cause the http2
transport to try to reestablish a new connection. As previously this never
should have happened, the connection called panic in that case. This
panic is now replaced with a simple error message, as it no longer
indicates an internal problem.
2020-08-01 12:17:40 +02:00
Alexander Weiss
f8316948d1
Optimize FUSE - make command restic mount
faster and consume less memory
...
- Add Open() functionality to dir
- only access index for blobs when file is read
- Implement NodeOpener and put one-time file stuff there
- Add comment about locking as suggested by bazil.org/fuse
=> Thanks at Michael Eischer for suggesting the last two improvements
2020-07-28 23:01:18 +02:00
Michael Eischer
ea97ff1ba4
rclone: Skip crash test when rclone is not found
2020-07-26 12:06:18 +02:00
Michael Eischer
01b9581453
rclone: Better field names for stdio conn
2020-07-26 00:29:25 +02:00
Michael Eischer
3cd927d180
rclone: Give rclone time to finish before closing stdin pipe
...
Calling `Close()` on the rclone backend sometimes failed during test
execution with 'signal: Broken pipe'. The stdio connection closed both
the stdin and stdout file descriptors at the same moment, therefore
giving rclone no chance to properly send any final http2 data frames.
Now the stdin connection to rclone is closed first and will only be
forcefully closed after a timeout. In case rclone exits before the
timeout then the stdio connection will be closed normally.
2020-07-26 00:29:25 +02:00
greatroar
bf7b1f12ea
rclone: Add test for pipe handling when rclone exits
2020-07-26 00:29:25 +02:00
Michael Eischer
8554332894
rclone: Close rclone side of stdio_conn pipes
...
restic did not notice when the rclone subprocess exited unexpectedly.
restic manually created pipes for stdin and stdout and used these for the
connection to the rclone subprocess. The process creating a pipe gets
file descriptors for the sender and receiver side of a pipe and passes
them on to the subprocess. The expected behavior would be that reads or
writes in the parent process fail / return once the child process dies
as a pipe would now just have a reader or writer but not both.
However, this never happened as restic kept the reader and writer
file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe`
close the subprocess side of pipes once the child process was started
and close the parent process side of pipes once wait has finished. We
can't use these functions as we need access to the raw `os.File` so just
replicate that behavior.
2020-07-26 00:29:25 +02:00
greatroar
3e93b36ca4
Make rclone.New private
2020-07-26 00:28:45 +02:00
Michael Eischer
c847aace35
Rename Index interface to MasterIndex
...
The interface is now only implemented by repository.MasterIndex.
2020-07-25 21:19:46 +02:00
Alexander Weiss
9d1fb94c6c
make Lookup() return all blobs
...
+ simplify syntax
2020-07-25 21:18:34 +02:00
MichaelEischer
020cab8e08
Merge pull request #2787 from greatroar/no-blobsize-cache
...
Remove blob size cache from restic mount
2020-07-25 20:46:35 +02:00
greatroar
07da61baee
Remove blob size cache from restic mount
2020-07-25 19:39:14 +02:00
MichaelEischer
3d530dfc91
Merge pull request #2827 from aawsome/archiver-test-contents
...
Make self-healing work when backing up with parent snapshot
2020-07-25 13:13:18 +02:00
MichaelEischer
bbc960f957
Merge pull request #2635 from greatroar/optimize-sortbycached
...
Optimize sorting blobs by cache status
2020-07-25 12:35:42 +02:00
greatroar
309598c237
Simplify sortCachedPacksFirst test in internal/repository
...
The test now uses the fact that the sort is stable. It's not guaranteed
to be, but the test is cleaner and more exhaustive. sortCachedPacksFirst
no longer needs a return value.
2020-07-25 12:12:59 +02:00
greatroar
03d23e6faa
Speed up blob sorting in internal/repository
...
name old time/op new time/op delta
SortCachedPacksFirst-8 208µs ± 3% 186µs ± 3% -10.74% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
SortCachedPacksFirst-8 213kB ± 0% 139kB ± 0% -34.62% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
SortCachedPacksFirst-8 1.03k ± 0% 1.03k ± 0% -0.19% (p=0.000 n=10+10)
2020-07-25 12:12:59 +02:00
greatroar
b10acd2af7
Test and benchmark blob sorting in internal/repository
2020-07-25 12:12:58 +02:00
Alexander Weiss
9175795fdb
Check contents in archiver
...
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
2020-07-25 08:18:28 +02:00
rawtaz
5d8d70542f
Merge pull request #2852 from MichaelEischer/drop-go-1.11
...
Drop support for Go version 1.11
2020-07-25 01:17:36 +02:00
Michael Eischer
7c23381a2b
Drop support for Go version 1.11
2020-07-24 18:52:39 +02:00
MichaelEischer
34181b13a2
Merge pull request #2328 from MichaelEischer/no-repeated-checks
...
Fix duplicate tree checks within `restic check`
2020-07-22 22:08:02 +02:00
Alexander Weiss
a666a6d576
Add tests and merge indexes in index benchmarks
2020-07-22 21:54:02 +02:00
Alexander Weiss
e388d962a5
Merge final indexes together for faster index access
2020-07-22 21:54:02 +02:00
Peter Schultz
758b44b9c0
gs: support authentication with access token
...
In the Google Cloud Storage backend, support specifying access tokens
directly, as an alternative to a credentials file. This is useful when
restic is used non-interactively by some other program that is already
authenticated and eliminates the need to store long lived credentials.
The access token is specified in the GOOGLE_ACCESS_TOKEN environment
variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.
2020-07-22 16:23:03 +02:00
Alexander Weiss
3b7a3711e6
Add more realistic index benchmarks
...
+ reduce test size of BenchmarkMasterIndexLookupParallel
2020-07-21 07:18:20 +02:00
Michael Eischer
9b0e718852
checker: Test that blob types are not confused
2020-07-20 23:43:47 +02:00
MichaelEischer
82c908871d
Merge pull request #2812 from greatroar/chaining
...
Chaining hash table for repository.Index
2020-07-20 23:29:36 +02:00
Michael Eischer
ddf0b8cd0b
checker: Properly distinguish between data and tree blobs
...
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
2020-07-20 22:58:39 +02:00
Michael Eischer
2d0c138c9b
checker: Test that check only decodes trees once
...
The `DuplicateTree` flag is necessary to ensure that failures cannot be
swallowed. The old checker implementation ignores errors from LoadTree
if the corresponding tree was already checked.
2020-07-20 22:51:53 +02:00
Michael Eischer
ef325ffc02
checker: Cleanup error handling code
...
This change only moves code around but does not result in any change in
behavior.
2020-07-20 22:51:53 +02:00
Michael Eischer
7a165f32a9
checker: Traverse trees in depth-first order
...
Backups traverse the file tree in depth-first order and saves trees on
the way back up. This results in tree packs filled in a way comparable
to the reverse Polish notation. In order to check tree blobs in that
order, the treeFilter would have to delay the forwarding of tree nodes
until all children of it are processed which would complicate the
implementation.
Therefore do the next similar thing and traverse the tree in depth-first
order, but process trees already on the way down. The tree blob ids are
added in reverse order to the backlog, which is once again reverted when
removing the ids from the back of the backlog.
2020-07-20 22:51:53 +02:00
Michael Eischer
36c69e3ca7
checker: Unify blobs, processed trees and referenced blobs map
...
The blobRefs map and the processedTrees IDSet are merged to reduce the
memory usage. The blobRefs map now uses separate flags to track blob
usage as data or tree blob. This prevents skipping of trees whose
content is identical to an already processed data blob. A third flag
tracks whether a blob exists or not, which removes the need for the
blobs IDSet.
2020-07-20 22:51:47 +02:00
Michael Eischer
35d8413639
checker: Remove dead index map
2020-07-20 22:37:31 +02:00
Michael Eischer
c66a0e408c
checker: Reduce cost of debug log
...
Avoid duplicate allocation of the Subtree list.
2020-07-20 22:37:31 +02:00
Michael Eischer
70f4c014ef
checker: Decode identical tree nodes only once
...
Even though the checkTreeWorker skips already processed chunks,
filterTrees did queue the same tree blob on every occurence. This
becomes a serious performance bottleneck for larger number of snapshots
that cover mostly the same directories. Therefore decode a tree blob
exactly once.
2020-07-20 22:37:31 +02:00
Michael Eischer
f0d8710611
Add benchmark for checker scaling with snapshot count
2020-07-20 22:37:31 +02:00
Alexander Neumann
8074879c5f
Remove 'go generate'
2020-07-19 17:28:42 +02:00
greatroar
7bda28f31f
Chaining hash table for repository.Index
...
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.
Benchmark results on linux/amd64, -benchtime=3s -count=20:
name old time/op new time/op delta
PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.231 n=20+20)
DecodeIndex-8 4.54s ± 0% 4.30s ± 0% -5.20% (p=0.000 n=18+17)
DecodeIndexParallel-8 4.54s ± 0% 4.30s ± 0% -5.22% (p=0.000 n=19+18)
IndexHasUnknown-8 44.4ns ± 5% 50.5ns ±11% +13.82% (p=0.000 n=19+17)
IndexHasKnown-8 48.3ns ± 0% 51.5ns ±12% +6.68% (p=0.001 n=16+20)
IndexAlloc-8 758ms ± 1% 616ms ± 1% -18.69% (p=0.000 n=19+19)
IndexAllocParallel-8 234ms ± 3% 204ms ± 2% -12.60% (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8 122ns ± 0% 145ns ± 9% +18.44% (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8 369ns ± 2% 429ns ± 8% +16.27% (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8 68.4ns ± 5% 74.9ns ±13% +9.47% (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8 315ns ± 3% 369ns ±11% +17.14% (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8 743ns ± 1% 816ns ± 2% +9.87% (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8 238ns ± 1% 260ns ± 2% +9.14% (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8 1.01µs ± 3% 1.11µs ± 2% +9.79% (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8 222ns ± 0% 269ns ± 2% +20.83% (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8 1.06µs ± 2% 1.19µs ± 2% +12.95% (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8 413ns ± 1% 530ns ± 1% +28.19% (p=0.000 n=18+20)
SaveAndEncrypt-8 30.2ms ± 1% 30.4ms ± 0% +0.71% (p=0.000 n=19+19)
LoadTree-8 540µs ± 1% 576µs ± 1% +6.73% (p=0.000 n=20+20)
LoadBlob-8 5.64ms ± 0% 5.64ms ± 0% ~ (p=0.883 n=18+17)
LoadAndDecrypt-8 5.93ms ± 0% 5.95ms ± 1% ~ (p=0.247 n=20+19)
LoadIndex-8 25.1ms ± 0% 24.5ms ± 1% -2.54% (p=0.000 n=18+17)
name old speed new speed delta
PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.229 n=20+20)
SaveAndEncrypt-8 139MB/s ± 1% 138MB/s ± 0% -0.71% (p=0.000 n=19+19)
LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% ~ (p=0.890 n=18+17)
LoadAndDecrypt-8 169MB/s ± 0% 168MB/s ± 1% ~ (p=0.227 n=20+19)
name old alloc/op new alloc/op delta
PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.772 n=12+19)
IndexAlloc-8 786MB ± 0% 400MB ± 0% -49.04% (p=0.000 n=20+18)
IndexAllocParallel-8 786MB ± 0% 401MB ± 0% -49.04% (p=0.000 n=19+15)
SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% +0.00% (p=0.000 n=19+19)
name old allocs/op new allocs/op delta
PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal)
IndexAlloc-8 977k ± 0% 907k ± 0% -7.18% (p=0.000 n=20+20)
IndexAllocParallel-8 977k ± 0% 907k ± 0% -7.17% (p=0.000 n=19+15)
SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)
2020-07-19 13:58:22 +02:00
greatroar
255ba83c4b
Parallel index benchmarks + benchmark optimizations
...
createRandomIndex was using the global RNG, which locks on every call
It was also using twice as many random numbers as necessary and doing
a float division in every iteration of the inner loop.
BenchmarkDecodeIndex was using too short an input, especially for a
parallel version. (It may now be using one that is a bit large.)
Results on linux/amd64, -benchtime=3s -count=20:
name old time/op new time/op delta
PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.165 n=20+20)
DecodeIndex-8 13.6µs ± 2% 4539886.8µs ± 0% +33293901.38% (p=0.000 n=20+18)
IndexHasUnknown-8 44.4ns ± 7% 44.4ns ± 5% ~ (p=0.873 n=20+19)
IndexHasKnown-8 49.2ns ± 3% 48.3ns ± 0% -1.86% (p=0.000 n=20+16)
IndexAlloc-8 802ms ± 1% 758ms ± 1% -5.51% (p=0.000 n=20+19)
MasterIndexLookupSingleIndex-8 124ns ± 1% 122ns ± 0% -1.41% (p=0.000 n=20+14)
MasterIndexLookupMultipleIndex-8 373ns ± 2% 369ns ± 2% -1.13% (p=0.001 n=20+20)
MasterIndexLookupSingleIndexUnknown-8 67.8ns ± 3% 68.4ns ± 5% ~ (p=0.753 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8 316ns ± 3% 315ns ± 3% ~ (p=0.846 n=20+20)
SaveAndEncrypt-8 30.5ms ± 1% 30.2ms ± 1% -1.09% (p=0.000 n=19+19)
LoadTree-8 527µs ± 1% 540µs ± 1% +2.37% (p=0.000 n=19+20)
LoadBlob-8 5.65ms ± 0% 5.64ms ± 0% -0.21% (p=0.000 n=19+18)
LoadAndDecrypt-8 7.07ms ± 2% 5.93ms ± 0% -16.15% (p=0.000 n=19+20)
LoadIndex-8 32.1ms ± 2% 25.1ms ± 0% -21.64% (p=0.000 n=20+18)
name old speed new speed delta
PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.159 n=20+20)
SaveAndEncrypt-8 138MB/s ± 1% 139MB/s ± 1% +1.10% (p=0.000 n=19+19)
LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% +0.21% (p=0.000 n=19+18)
LoadAndDecrypt-8 141MB/s ± 2% 169MB/s ± 0% +19.24% (p=0.000 n=19+20)
name old alloc/op new alloc/op delta
PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.826 n=19+12)
IndexAlloc-8 786MB ± 0% 786MB ± 0% +0.01% (p=0.000 n=20+20)
SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% -0.00% (p=0.012 n=20+19)
name old allocs/op new allocs/op delta
PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal)
IndexAlloc-8 977k ± 0% 977k ± 0% +0.01% (p=0.022 n=20+20)
SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)
2020-07-19 13:58:05 +02:00
Lars Lehtonen
9ac90cf5cd
internal/fuse: fix dropped test error
2020-07-12 21:42:31 -07:00
greatroar
58719e1f47
Replace mount's per-file cache by a global LRU cache
2020-07-12 18:27:16 +02:00
greatroar
d42c169458
Fix quadratic file reading in restic mount
2020-07-12 18:27:16 +02:00
greatroar
02bec13ef2
Fix repository_test.BenchmarkSaveAndEncrypt
...
The benchmark was actually testing the speed of index lookups.
name old time/op new time/op delta
SaveAndEncrypt-8 101ns ± 2% 31505824ns ± 1% +31311591.31% (p=0.000 n=10+10)
name old speed new speed delta
SaveAndEncrypt-8 41.7TB/s ± 2% 0.0TB/s ± 1% -100.00% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
SaveAndEncrypt-8 1.00B ± 0% 20989508.40B ± 0% +2098950740.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
SaveAndEncrypt-8 0.00 123.00 ± 0% +Inf% (p=0.000 n=10+9)
(The actual speed is ca. 131MiB/s.)
2020-07-05 17:41:42 +02:00
MichaelEischer
212607dc8a
Merge pull request #2760 from greatroar/backend-benchmark
...
Fix backend benchmarks + a micro-optimization
2020-06-17 23:17:05 +02:00
greatroar
190d8e2f51
Flatten backend.LimitedReadCloser structure
...
This inlines the io.LimitedReader into the LimitedReadCloser body to
achieve fewer allocations. Results on linux/amd64:
name old time/op new time/op delta
Backend/BenchmarkLoadPartialFile-8 412µs ± 4% 413µs ± 4% ~ (p=0.634 n=17+17)
Backend/BenchmarkLoadPartialFileOffset-8 455µs ±13% 441µs ±10% ~ (p=0.072 n=20+18)
name old speed new speed delta
Backend/BenchmarkLoadPartialFile-8 10.2GB/s ± 3% 10.2GB/s ± 4% ~ (p=0.817 n=16+17)
Backend/BenchmarkLoadPartialFileOffset-8 9.25GB/s ±12% 9.54GB/s ± 9% ~ (p=0.072 n=20+18)
name old alloc/op new alloc/op delta
Backend/BenchmarkLoadPartialFile-8 888B ± 0% 872B ± 0% -1.80% (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8 888B ± 0% 872B ± 0% -1.80% (p=0.000 n=15+15)
name old allocs/op new allocs/op delta
Backend/BenchmarkLoadPartialFile-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=15+15)
2020-06-17 13:11:45 +02:00
greatroar
f4cd2a7120
Make backend benchmarks fairer by removing checks
...
Checking whether the right data is returned takes up half the time in
some benchmarks. Results for local backend benchmarks on linux/amd64:
name old time/op new time/op delta
Backend/BenchmarkLoadFile-8 4.89ms ± 0% 2.72ms ± 1% -44.26% (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8 936µs ± 6% 439µs ±15% -53.07% (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8 940µs ± 1% 456µs ±10% -51.50% (p=0.008 n=5+5)
Backend/BenchmarkSave-8 23.9ms ±14% 24.8ms ±41% ~ (p=0.690 n=5+5)
name old speed new speed delta
Backend/BenchmarkLoadFile-8 3.43GB/s ± 0% 6.16GB/s ± 1% +79.40% (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8 4.48GB/s ± 6% 9.63GB/s ±14% +114.78% (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8 4.46GB/s ± 1% 9.22GB/s ±10% +106.74% (p=0.008 n=5+5)
Backend/BenchmarkSave-8 706MB/s ±13% 698MB/s ±31% ~ (p=0.690 n=5+5)
2020-06-17 13:11:45 +02:00
Alexander Weiss
1361341c58
don't save duplicate packIDs when using internal/repository/Index.Store
2020-06-14 07:56:24 +02:00
Alexander Weiss
ce4a2f4ca6
save packIDs and duplicates separately
...
A side remark to the definition of Index.blob:
Another possibility would have been to use:
blob map[restic.BlobHandle]*indexEntry
This would have led to the following sizes:
key: 32 + 1 = 33 bytes
value: 8 bytes
indexEntry: 8 + 4 + 4 = 16 bytes
each packID: 32 bytes
To save N index entries, we would therefore have needed:
N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes
More precicely, using a pointer instead of a direct entry is the better memory choice if:
OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1)
Under the assumption of OF=1.5, this means using pointers would have been the better choice
if sizeof(indexEntry) > 24 bytes.
2020-06-14 07:56:21 +02:00
Alexander Weiss
cf979e2b81
make offset and length uint32
2020-06-14 07:50:19 +02:00
Michael Eischer
d92e2c5769
simplify index code
2020-06-14 07:50:19 +02:00
Alexander Weiss
7419844885
add changelog, benchmark, memory calculation
2020-06-14 07:50:15 +02:00
Alexander Weiss
d3c59d18e5
Fix inconsistency of saving/loading config file
...
Fix saving/loading config file: Always set ID to a zero ID.
2020-06-13 16:30:23 +02:00
MichaelEischer
dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256
...
Use Minio's optimized SHA-256
2020-06-12 23:32:58 +02:00
MichaelEischer
6896c6449b
Merge pull request #2779 from greatroar/archiver-comment
...
Fix up comment on archiver.BlobSaver.Save
2020-06-12 23:04:21 +02:00
MichaelEischer
735a8074d5
Merge pull request #2773 from aawsome/index-uploads+knownblobs
...
Fix non-intuitive repo behavior
2020-06-12 22:41:04 +02:00
Alexander Weiss
70347e95d5
disable index uploads for prune command
...
+ modifications of changelog
2020-06-12 09:24:38 +02:00
greatroar
0fa3091c78
Fix up comment on archiver.BlobSaver.Save
2020-06-11 13:40:30 +02:00
Alexander Weiss
91906911b0
Fix non-intuitive repository behavior
...
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
-> also cleans up pending index entries when they are saved in the index
-> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
-> removes the need of an index uploader
-> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
2020-06-11 13:05:23 +02:00
MichaelEischer
6856d1e422
Merge pull request #2749 from aawsome/fix-fullindex
...
Change condition for full index
2020-06-10 20:40:19 +02:00
Alexander Weiss
8c1261ff02
changed condition for full index
2020-06-07 22:00:49 +02:00
MichaelEischer
84475aa3a8
Merge pull request #2730 from greatroar/mount-build-tags
...
Simplify build tags for restic mount
2020-05-23 20:33:43 +02:00
Martin Michlmayr
5cc1760fdf
Fix typos
2020-05-16 14:05:26 +08:00
greatroar
649cbec6c5
Simplify build tags for restic mount
...
This command can only be built on Darwin, FreeBSD and Linux
(and if we upgrade bazil.org/fuse, only FreeBSD and Linux:
https://github.com/bazil/fuse/issues/224 ).
Listing the few supported operating systems explicitly here makes
porting restic to new platforms easier.
2020-05-12 11:30:41 +02:00
greatroar
9f7cd69f13
Move Index.FindBlob to tests
2020-04-29 10:57:01 +02:00
greatroar
f97a680887
Fix repository benchmarks
...
BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of
their time measuring SHA-256 performance in their checks.
2020-04-28 07:57:29 +02:00
greatroar
42a3db05b0
Use Minio's optimized SHA-256
...
internal/repository benchmarks on an Intel i7-3770k:
name old speed new speed delta
PackerManager-8 209MB/s ± 1% 291MB/s ± 1% +38.94% (p=0.008 n=5+5)
SaveAndEncrypt-8 112MB/s ± 1% 135MB/s ± 1% +20.25% (p=0.008 n=5+5)
2020-04-28 07:57:18 +02:00
greatroar
e7d7b85d59
Merge Repository.{LoadBlob,loadBlob}
...
Pushing the allocation logic down into the former loadBlob body means
that fewer allocations have to be performed:
name old time/op new time/op delta
LoadTree-8 478µs ± 1% 481µs ± 2% ~ (p=0.315 n=9+10)
LoadBlob-8 11.6ms ± 1% 11.6ms ± 2% ~ (p=0.393 n=10+10)
LoadAndDecrypt-8 13.3ms ± 3% 13.3ms ± 3% ~ (p=0.905 n=10+9)
LoadIndex-8 33.6ms ± 2% 33.2ms ± 1% -1.15% (p=0.028 n=10+9)
name old alloc/op new alloc/op delta
LoadTree-8 41.2kB ± 0% 41.1kB ± 0% -0.23% (p=0.000 n=10+10)
LoadBlob-8 2.28kB ± 0% 2.18kB ± 0% -4.21% (p=0.000 n=10+10)
LoadAndDecrypt-8 2.10MB ± 0% 2.10MB ± 0% ~ (all equal)
LoadIndex-8 5.22MB ± 0% 5.22MB ± 0% ~ (p=0.631 n=10+10)
name old allocs/op new allocs/op delta
LoadTree-8 652 ± 0% 651 ± 0% -0.15% (p=0.000 n=10+10)
LoadBlob-8 24.0 ± 0% 23.0 ± 0% -4.17% (p=0.000 n=10+10)
LoadAndDecrypt-8 30.0 ± 0% 30.0 ± 0% ~ (all equal)
LoadIndex-8 30.2k ± 0% 30.2k ± 0% ~ (p=0.610 n=10+10)
name old speed new speed delta
LoadBlob-8 86.4MB/s ± 1% 85.9MB/s ± 2% ~ (p=0.393 n=10+10)
LoadAndDecrypt-8 75.4MB/s ± 3% 75.4MB/s ± 3% ~ (p=0.858 n=10+9)
2020-04-23 10:04:20 +02:00
greatroar
be5a0ff59f
Centralize buffer allocation and size checking in Repository.LoadBlob
...
Benchmark results for internal/repository:
name old time/op new time/op delta
LoadTree-8 479µs ± 2% 478µs ± 1% ~ (p=0.780 n=10+9)
LoadBlob-8 11.6ms ± 2% 11.6ms ± 1% ~ (p=0.631 n=10+10)
LoadAndDecrypt-8 13.2ms ± 2% 13.3ms ± 3% ~ (p=0.631 n=10+10)
name old alloc/op new alloc/op delta
LoadTree-8 41.2kB ± 0% 41.2kB ± 0% ~ (all equal)
LoadBlob-8 2.28kB ± 0% 2.28kB ± 0% ~ (all equal)
LoadAndDecrypt-8 2.10MB ± 0% 2.10MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
LoadTree-8 652 ± 0% 652 ± 0% ~ (all equal)
LoadBlob-8 24.0 ± 0% 24.0 ± 0% ~ (all equal)
LoadAndDecrypt-8 30.0 ± 0% 30.0 ± 0% ~ (all equal)
name old speed new speed delta
LoadBlob-8 86.2MB/s ± 2% 86.4MB/s ± 1% ~ (p=0.594 n=10+10)
LoadAndDecrypt-8 75.7MB/s ± 2% 75.4MB/s ± 3% ~ (p=0.617 n=10+10)
2020-04-23 10:04:20 +02:00
MichaelEischer
4f00564574
Merge pull request #2621 from greatroar/fixes
...
Some small fixes
2020-04-18 18:07:08 +02:00
MichaelEischer
f77477129f
Merge pull request #2584 from greatroar/mount-cache-uid-gid
...
Cache uid and gid for top directories in internal/fuse
2020-04-18 17:45:14 +02:00
greatroar
2e31120f89
Remove unused argument to restic.fakeFile
2020-04-18 17:40:13 +02:00
greatroar
8fb2c0d3c1
Typo in crypto test name
2020-04-18 17:39:06 +02:00
greatroar
072cf7b02d
Fix debug messages in internal/fuse
2020-04-18 17:39:06 +02:00
greatroar
df66daa5c9
Fix context usage in backend tests
...
Found by go vet. This is also the only complaint is has.
2020-04-18 17:39:06 +02:00
MichaelEischer
16710454f4
Merge pull request #2628 from MichaelEischer/one-element-pack-lists
...
cache: Don't sort one element pack lists
2020-04-18 17:09:06 +02:00
MichaelEischer
7910ff4c0e
Merge pull request #2648 from nairb774/iowritestring
...
termstatus: Use io.WriteString to output messages.
2020-04-18 13:48:42 +02:00
MichaelEischer
c4da9d1e90
Merge pull request #2638 from greatroar/no-close-in-packer
...
Don't Close in Packer.Finalize
2020-04-18 13:07:21 +02:00
MichaelEischer
a1352906e2
Merge pull request #2622 from greatroar/optimize-packer-manager
...
Fix PackerManager benchmark and optimize hashing.Writer
2020-04-18 12:46:34 +02:00
MichaelEischer
b7c0d4d8bf
Merge pull request #2644 from greatroar/signal-notify-buffered
...
Make all signal.Notify channels buffered
2020-04-18 11:31:02 +02:00
Michael Eischer
bdf7ba20cb
archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError
...
The Save methods of the BlobSaver, FileSaver and TreeSaver return early
on when the archiver is stopped due to an error. For that they select on
both the tomb.Dying() and context.Done() channels, which can lead to a
race condition when the tomb is killed due to an error: The tomb first
closes its Dying channel before canceling all child contexts.
Archiver.SaveDir only aborts its execution once the context was
canceled. When the tomb killing is paused between closing its Dying
channel and canceling the child contexts, this lets the
FileSaver/TreeSaver.Save methods return immediately, however, ScanDir
still reads further files causing the test case to fail.
As a killed tomb always cancels all child contexts and as the Savers
always use a context bound to the tomb, it is sufficient to just use
context.Done() as escape hatch in the Save functions. This fixes the
mismatch between SaveDir and Save.
Adjust the tests to use contexts bound to the tomb for all interactions
with the Savers.
2020-04-13 18:23:17 +02:00
Brian Atkinson
b8da7b1f4d
termstatus: Use io.WriteString to output messages.
...
The previous implementation was repeating the implementation that is
found inside of io.WriteString. Simplify by making use of the stdlib's
implementation.
2020-03-26 14:55:00 -07:00
Peter Schultz
90fc639a67
Allow specifying user and host when adding keys
...
The username and hostname for new keys can be specified with the new
--user and --host flags, respectively. The flags are used only by the
`key add` command and are otherwise ignored.
This allows adding keys with for a desired user and host without having
to run restic as that particular user on that particular host, making
automated key management easier.
Co-authored-by: James TD Smith <ahktenzero@mohorovi.cc>
2020-03-23 13:11:10 +01:00
rawtaz
9efbe98879
Merge pull request #2623 from alrs/internal-restic-err-before-close
...
internal/restic: close os.File after checking for error
2020-03-18 22:23:20 +01:00
rawtaz
7d9300efca
Merge pull request #2637 from greatroar/unused
...
Remove Go 1.5 compatibility code
2020-03-18 22:22:15 +01:00
greatroar
47d4d5bf1b
Make all signal.Notify channels buffered
2020-03-12 20:59:39 +01:00
greatroar
74a64c47e4
Move testing logic to test file in internal/pack
2020-03-09 14:32:28 +01:00
greatroar
a23e9c86ba
Remove closing logic from Packer.Finalize
...
The method only ever receives *hashing.Writers, which don't implement
io.Closer. These come from packerManager.findPacker and have their
actual writers closed in Repository.savePacker. Moving the closing logic
to hashing.Writer results in "file already closed" errors.
2020-03-09 14:31:45 +01:00
greatroar
4de12bf593
Remove restic.RandReader
...
math/rand.Rand has implemented Reader since Go 1.6. The repacking tests
are not deterministic, but they weren't before, either.
2020-03-09 10:00:28 +01:00
greatroar
8cf3bb8737
Revert "Put host last in SSH command line"
...
This reverts commit e1969d1e33
.
2020-03-08 16:45:33 +01:00
Michael Eischer
b46cc6d57e
repository: Don't sort one element pack lists
...
When loading a blob, restic first looks up pack files containing the
blob. To avoid unnecessary work an already cached pack file is preferred.
However, if there is only a single pack file to choose from (which is
the normal case) sorting the one-element list won't change anything.
Therefore avoid the unnecessary cache check in that case.
2020-03-07 10:26:06 +01:00
Lars Lehtonen
4a2156d3f0
internal/restic: close os.File after checking for error
2020-03-05 16:22:46 -08:00
greatroar
41fee11f66
Micro-optimization for hashing.Writer/PackerManager
...
name old time/op new time/op delta
PackerManager-8 247ms ± 1% 246ms ± 1% -0.43% (p=0.001 n=18+18)
name old speed new speed delta
PackerManager-8 213MB/s ± 1% 214MB/s ± 1% +0.43% (p=0.001 n=18+18)
name old alloc/op new alloc/op delta
PackerManager-8 92.2kB ± 0% 91.5kB ± 0% -0.82% (p=0.000 n=19+20)
name old allocs/op new allocs/op delta
PackerManager-8 1.43k ± 0% 1.41k ± 0% -1.67% (p=0.000 n=20+20)
2020-03-05 22:30:04 +01:00
greatroar
b592614061
Improve PackerManager benchmark
...
The previous benchmark spent much of its time allocating RNGs and
generating too many random numbers. It now spends 90% of its time
hashing and half of the rest writing to files.
name old time/op new time/op delta
PackerManager-8 319ms ± 1% 247ms ± 1% -22.48% (p=0.000 n=20+18)
name old speed new speed delta
PackerManager-8 143MB/s ± 1% 213MB/s ± 1% +48.63% (p=0.000 n=10+18)
name old alloc/op new alloc/op delta
PackerManager-8 635kB ± 0% 92kB ± 0% -85.48% (p=0.000 n=10+19)
name old allocs/op new allocs/op delta
PackerManager-8 1.64k ± 0% 1.43k ± 0% -12.76% (p=0.000 n=10+20)
2020-03-05 22:30:03 +01:00
greatroar
b7c3039eb2
Remove Go 1.5 compatibility code from PackerManager benchmark
...
This alone is enough to speed up the benchmark by ~10%.
2020-03-05 22:29:06 +01:00
Alexander Neumann
52abec967f
Merge pull request #2605 from middelink/fix-2604
...
Fix running tests on a SELinux enabled system
2020-03-01 20:25:28 +01:00
Pauline Middelink
2828a9c2b0
Fix running tests on a SELinux enabled system
...
Archivers TestMetadataChanged incorrectly clears the Extended Attributes
from the expected metadata of the temporary file. This is incorrect as on
SELinux enabled filesystem, as the kernel will automaticly add a SElinux
label. However, since ExtendedAttributes{} != ExtendedAttributes{nil} we
still need to clear them if there are no attributes found.
2020-03-01 20:23:22 +01:00
Alexander Neumann
95da6c1c1d
Merge pull request #2589 from greatroar/no-stable-sort
...
Replace sort.Stable by sort.Strings
2020-03-01 19:40:28 +01:00
Alexander Neumann
0c03a80fc4
Merge pull request #2592 from greatroar/sftp-ipv6
...
Support IPv6 in SFTP backend
2020-03-01 19:38:44 +01:00
Alexander Neumann
c50f91b7f9
Merge pull request #2602 from greatroar/no-bufpool
...
Remove sync.Pool from internal/repository
2020-03-01 10:50:57 +01:00
Alexander Neumann
e851d29565
Merge pull request #2608 from greatroar/simplify-termstatus
...
Simplify termstatus
2020-03-01 10:48:29 +01:00
Alexander Neumann
b67b7ebfe6
Merge pull request #2583 from greatroar/unused
...
Remove some unused or duplicated code
2020-03-01 10:46:17 +01:00
greatroar
751eba0e68
Remove unnecessary pipe checks in termstatus
...
canUpdateStatus has already determined that the file descriptor is not a
pipe.
2020-02-29 18:03:49 +01:00
greatroar
7447c44484
Use golang.org/x/sys/windows in termstatus
...
Some functionality is missing, but at least the types are all defined.
Replaced short, word, dword by their Go names to match the x/sys
convention.
2020-02-29 18:03:49 +01:00
greatroar
c8a672fa29
Remove code copy-pasted from x/crypto/ssh/terminal
2020-02-29 18:03:49 +01:00
greatroar
863ba76494
Drop mattn/go-isatty in favor of crypto/ssh/terminal
2020-02-29 18:03:47 +01:00
greatroar
8526cc6647
Remove sync.Pool from internal/repository
...
The pool was used improperly, causing more allocations to be
performed than without it.
name old time/op new time/op delta
SaveAndEncrypt-8 36.8ms ± 2% 36.9ms ± 2% ~ (p=0.218 n=10+10)
name old speed new speed delta
SaveAndEncrypt-8 114MB/s ± 2% 114MB/s ± 2% ~ (p=0.218 n=10+10)
name old alloc/op new alloc/op delta
SaveAndEncrypt-8 21.1MB ± 0% 21.0MB ± 0% -0.44% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
SaveAndEncrypt-8 79.0 ± 0% 77.0 ± 0% -2.53% (p=0.000 n=10+10)
2020-02-29 17:54:46 +01:00
greatroar
5cd0bce452
Honor RESTIC_CACHE_DIR on Mac and Windows
2020-02-28 15:44:32 +01:00
rawtaz
58bd165253
Merge pull request #2581 from aawsome/multiple-hostnames
...
Allow multiple hostnames tags
2020-02-27 08:35:23 +01:00
rawtaz
65d3fb6b33
Merge pull request #2603 from greatroar/restorer-waitgroup
...
Fix unsafe sync.WaitGroup usage in restorer.fileRestorer
2020-02-27 00:30:59 +01:00
greatroar
de5516a90e
Fix sync.WaitGroup usage in restorer.fileRestorer
2020-02-27 00:07:49 +01:00
greatroar
4f6fd9fb98
Remove remnant of Go 1.9 compatibility code from tests
2020-02-26 22:23:38 +01:00
Alexander Weiss
9a9101d144
Support specifying multiple host flags for various commands
...
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`,
`stats` and `tag` commands will now take into account multiple
`--host` and `-H` flags.
2020-02-26 22:17:59 +01:00
Alexander Neumann
99fd80a585
Remove all workarounds for Go < 1.11
2020-02-26 20:35:13 +01:00
Igor Fedorenko
f17ffa0283
restorer: Allow writing target file blobs out of order
...
Much simpler implementation that guarantees each required pack
is downloaded only once (and hence does not need to manage
pack cache). Also improves large file restore performance.
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2020-02-26 16:14:45 +01:00
greatroar
5e2afd91e7
Assert that archiver.Tree implements fmt.Stringer
2020-02-26 11:05:38 +01:00
greatroar
79b882e901
Merge duplicated readdir functionality
...
internal/archiver.readdir and internal/fs.ReadDir were unused.
internal/fs.ReadDirNames and internal/archiver.readdirnames were doing
nearly the same thing, except one sorted its output and opened with
fs.O_NOFOLLOW. Both were only used in internal/archiver.
2020-02-26 11:05:38 +01:00
greatroar
1b502fa9ef
Cache uid and gid for top directories in internal/fuse
2020-02-24 10:46:09 +01:00
greatroar
e1969d1e33
Put host last in SSH command line
...
This is how the SSH manpage says the command line should look, and the
"--" prevents mistakes in hostnames from being interpreted as options.
2020-02-19 15:53:20 +01:00
greatroar
6ac6bca7a1
Support IPv6 in SFTP backend
...
The previous code was doing its own hostname:port splitting, which
caused IPv6 addresses to be misinterpreted.
2020-02-19 15:42:12 +01:00
greatroar
3a6feb0596
Replace sort.Stable by sort.Strings
...
Calling the slow, O(n lg² n) sort.Stable is equivalent to sort.Strings
for a slice of unique strings.
2020-02-18 19:41:06 +01:00
greatroar
2f8aa2ce30
Remove unused fs.FS from archiver.FileSaver
2020-02-18 10:39:14 +01:00
Michael Eischer
e1f722d266
archiver: Fix flaky TestArchiverAbortEarlyOnError
...
Each of the random test files was split into the same five blobs. The
test fails once the fifth blob is passed on to `SaveBlob`. That is for
certain interleavings of goroutine execution it would be possible for
the test to trigger the testErr just after storing the first file.
The fixed test uses a different file content for each of the nine files
and fails after writing the fourth blob. The file content is also small
enough to ensure that for each file only a single blob is saved. This
guarantees that the test cannot fail before reading the first four
files. FileReadConcurrency = 2 allows up to two files queued for
processing. Therefore the test can at most open the sixth file before it
has to save the fourth file / blob which triggers the testErr.
2020-02-14 23:16:13 +01:00
Michael Eischer
42a3292bcf
Better name for jsonstatus package
...
internal/ui/jsonstatus and termstatus sound similar but are not related
in any way. Instead `internal/ui/backup` and `internal/ui/jsonstatus/status`
are the counterparts. Rename the latter to `internal/ui/json/backup` to
make this clear.
2020-02-13 21:14:20 +01:00
Michael Eischer
ef70a2fcb3
Fix mangled JSON output by backup command
...
jsonstatus wrote the JSON output without synchronization to the
stdio_wrapper which caused mangling between different status lines.
Use the Print and Error methods of termstatus instead which use a
central goroutine to synchronize output.
2020-02-13 21:14:20 +01:00
rawtaz
680a14afa1
Merge pull request #2530 from restic/fix-sftp-mkdirall
...
sftp: Use MkdirAll provided by the client
2020-02-13 01:12:59 +01:00
rawtaz
d8da9c4401
Merge pull request #2577 from alrs/fix-internal-errs
...
internal: Fix code and test dropped errors
2020-02-12 23:41:54 +01:00
rawtaz
299f5971f2
Merge pull request #2560 from brualan/master
...
Two small improvements to code quality
2020-02-12 22:40:20 +01:00
Lars Lehtonen
72734d59b5
internal/archiver: fix dropped error
2020-02-12 13:37:37 -08:00
Lars Lehtonen
3ed54e762e
internal/backend/sftp: fix dropped test error
2020-02-12 13:36:21 -08:00
Lars Lehtonen
fea835b4e2
internal/repository: fix dropped test error
2020-02-12 13:33:54 -08:00
Lars Lehtonen
16b321b140
internal/restic: fix dropped test error
2020-02-12 13:32:45 -08:00
Michael Eischer
760863e7f9
archiver: Fix TestRacyFileSwap on windows
2020-02-11 21:09:47 +01:00