octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-12-23 11:28:54 +00:00
Code Issues Releases Wiki Activity
6,338 Commits 7 Branches 33 Tags 66 MiB
22147e1e02
Commit Graph

1431 Commits

Author SHA1 Message Date
greatroar
0666c4d244 Make restic.{lchown,mknod} regular functions 
This way, they can be inlined and dead code can be removed on Windows.
Also fixed some comments.
 2021-05-27 22:51:40 +02:00
rawtaz
fdbd65485e
Merge pull request #3402 from MichaelEischer/misc-fixes 
Various small code cleanups
 2021-05-24 11:30:31 +02:00
Alexander Neumann
4cabad8c34
Merge pull request #3325 from MichaelEischer/fix-mintty-output 
Fix windows terminal output for mintty
 2021-05-18 09:29:24 +02:00
Michael Eischer
cf92c58460 Properly wrap errors in readerat helper 2021-05-17 21:08:23 +02:00
Michael Eischer
75c990504d azure/gs: Fix default value in connections help text 2021-05-17 20:56:51 +02:00
Michael Eischer
55bea6e7a6 filter: Fix crash for '**' pattern 2021-05-14 23:50:31 +02:00
MichaelEischer
6d8ceefd67
Merge pull request #3373 from greatroar/simplify-limiter 
Simplify internal/limiter
 2021-05-14 20:27:31 +02:00
MichaelEischer
64b00d28b1
Merge pull request #3345 from greatroar/sftp-enospc 
Check for ENOSPC and remove broken files in SFTP
 2021-05-13 20:09:38 +02:00
greatroar
ae170e2b38 Simplify internal/limiter 2021-04-24 11:54:43 +02:00
Michael Eischer
7cb8ea69ba Add test to mintty pipe detection 2021-04-11 20:02:09 +02:00
Michael Eischer
80564a9bc9 Properly detect mintty output redirection 
mintty on windows always uses pipes to connect stdout between processes
and for the terminal output. The previous implementation always assumed
that stdout connected to a pipe means that stdout is displayed on a
mintty terminal. However, this detection breaks when using pipes to
connect processes and for powershell which uses pipes when redirecting
to a file.

Now the pipe filename is queried and matched against the pattern used by
msys / cygwin when connected to the terminal. In all other cases assume
that a pipe is just a regular pipe.
 2021-04-11 20:02:09 +02:00
Michael Eischer
5e6af77b7a Unify interactive terminal detection code 
Previously the progress bar / status update interval used
stdoutIsTerminal to determine whether it is possible to update the
progress bar or not. However, its implementation differed from the
detection within the backup command which included additional checks to
detect the presence of mintty on Windows. mintty behaves like a terminal
but uses pipes for communication.

This adds stdoutCanUpdateStatus() which calls the same terminal detection
code used by backup. This ensures that all commands consistently switch
between interactive and non-interactive terminal mode.

stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus()
does so. This is required to properly handle the special case of mintty.
 2021-04-11 20:02:09 +02:00
MichaelEischer
cc254dfefe
Merge pull request #3362 from greatroar/darwin-preallocate 
Use FcntlFstore to preallocate on Mac
 2021-04-10 22:47:56 +02:00
greatroar
23531be272 Use FcntlFstore to preallocate on Mac 2021-04-10 16:54:07 +02:00
rawtaz
74c0607c92
Merge pull request #3319 from MichaelEischer/skip-prealloc-test 
restorer: Skip preallocate test if not supported by the filesystem
 2021-04-07 18:59:06 +02:00
greatroar
dc88ca79b6 Handle lack of space and remove broken files in SFTP backend 2021-03-27 15:02:14 +01:00
Alexander Neumann
88a23521dd
Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check 
s3: Fix sanity check
 2021-03-11 13:13:46 +01:00
greatroar
187a77fb27 Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist 2021-03-10 21:05:24 +01:00
Michael Eischer
2a9f0f19b6 s3: Fix sanity check 
The sanity check shouldn't replace the error message if there is already
one.
 2021-03-08 20:23:57 +01:00
Michael Eischer
f70aca6f6f restorer: Skip preallocate test if not supported by the filesystem 2021-03-04 20:33:46 +01:00
Michael Eischer
a293fd9aef local: Ignore files in intermediate folders 
For example the data folder of a repository might contain hidden files
which caused list operations to fail.
 2021-02-27 13:47:55 +01:00
Alexander Neumann
9985368d46
Merge pull request #3255 from MichaelEischer/restorer-check-error 
restorer: Check dropped error
 2021-02-03 16:10:46 +01:00
Alexander Neumann
2dd592a06c
Merge pull request #3254 from lorenz/allow-http2 
Allow HTTP/2
 2021-02-02 21:08:20 +01:00
Michael Eischer
47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
Alexander Weiss
5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss
573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
Lorenz Brun
5427119205 Allow HTTP/2 2021-01-31 02:44:30 +01:00
Michael Eischer
e0867c9682 backend: try to cleanup test leftovers 2021-01-30 21:23:20 +01:00
Michael Eischer
f740b2fb23 mem: check upload length before storing upload 2021-01-30 21:23:20 +01:00
Alexander Neumann
0e5f2fff71
Merge pull request #3243 from restic/fix-scanner-overlap 
backup: Fix total size for overlapping targets
 2021-01-30 21:17:21 +01:00
Alexander Neumann
04ca69cc78 Address issues reported by golint 2021-01-30 20:45:57 +01:00
Alexander Neumann
f867e65bcd Fix issues reported by staticcheck 2021-01-30 20:43:53 +01:00
Alexander Neumann
0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
Alexander Neumann
aef3658a5f Address review comments 2021-01-30 20:02:37 +01:00
Alexander Neumann
200f09522d Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann
3c753c071c errcheck: More error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann
16313bfcc9 errcheck: Add error check for MergeFinalIndexes() 2021-01-30 20:02:37 +01:00
Alexander Neumann
75f53955ee errcheck: Add error checks 
Most added checks are straight forward.
 2021-01-30 20:02:37 +01:00
Michael Eischer
a53778cd83 rest: handle dropped error in save operation 2021-01-30 19:25:04 +01:00
Michael Eischer
8a486eafed gs: Don't drop error when finishing upload 
The error returned when finishing the upload of an object was dropped.
This could cause silent upload failures and thus data loss in certain
cases. When a MD5 hash for the uploaded blob is specified, a wrong
hash/damaged upload would return its error via the Close() whose error
was dropped.
 2021-01-30 13:31:32 +01:00
Alexander Neumann
cdd704920d azure: Pass data length to Azure libray 
The azureAdapter was used directly without a pointer, but the Len()
method was only defined with a pointer receiver (which means Len() is
not present on a azureAdapter{}, only on a pointer to it).
 2021-01-29 21:08:41 +01:00
Alexander Neumann
bbdf18c4a2
Merge pull request #3176 from MichaelEischer/backend-content-length 
Pass upload size to backends and sanity check it
 2021-01-29 20:33:44 +01:00
Michael Eischer
1f583b3d8e backend: test that incomplete uploads fail 2021-01-29 13:51:53 +01:00
Michael Eischer
c73316a111 backends: add sanity check for the uploaded file size 
Bugs in the error handling while uploading a file to the backend could
cause incomplete files, e.g. https://github.com/golang/go/issues/42400
which could affect the local backend.

Proactively add sanity checks which will treat an upload as failed if
the reported upload size does not match the actual file size.
 2021-01-29 13:51:51 +01:00
Michael Eischer
4526d5d197 swift: explicitly pass upload size to library 
This allows properly setting the content-length which could help the
server-side to detect incomplete uploads.
 2021-01-29 13:50:46 +01:00
Michael Eischer
dca9b6f5db azure: explicitly pass upload size 
Previously the fallback from the azure library was to read the whole
blob into memory and use that to determine the upload size.
 2021-01-29 13:50:46 +01:00
Alexander Neumann
5c617859ab backup/scanner: Fix total size for overlapping targets 
Before, the scanner would could files twice if they were included in the
list of backup targets twice, e.g. `restic backup foo foo/bar` would
could the file `foo/bar` twice.

This commit uses the tree structure from the archiver to run the
scanner, so both parts see the same files.
 2021-01-29 11:31:36 +01:00
Alexander Neumann
81211750ba archiver/tree: Introduce functions Leaf() and NodeNames() 2021-01-29 11:11:28 +01:00
greatroar
6bd8a2faaa backup: Add --ignore-ctime option and document change detection 2021-01-28 23:42:10 +01:00
MichaelEischer
43cb26010a
Merge pull request #3242 from greatroar/fprintln 
internal/ui/termstatus: Use Fprintln to get a newline
 2021-01-28 20:34:06 +01:00
Alexander Neumann
4d40c70214
Merge pull request #3211 from MichaelEischer/sftp-speedup 
Speed-up caching/pack download via SFTP
 2021-01-28 14:16:55 +01:00
Alexander Neumann
6aa7e9f9c6
Merge pull request #3174 from MichaelEischer/parallelize-lock-loading 
Parallelize lock file loading
 2021-01-28 13:52:12 +01:00
Alexander Neumann
bdfedf1f5b
Merge pull request #3173 from MichaelEischer/unify-index-loading 
Unify index loading
 2021-01-28 13:50:42 +01:00
greatroar
b9cfe6f68a internal/ui/termstatus: Use Fprintln to get a newline 2021-01-28 13:30:10 +01:00
Michael Eischer
68608a89ad restic: add comment about StreamTrees shutdown 2021-01-28 11:10:50 +01:00
Michael Eischer
ddb7697d29 restic: Test progress reporting of StreamTrees 2021-01-28 11:10:50 +01:00
Michael Eischer
313ad0e32f progress/counter: Fix test for final report call 2021-01-28 11:10:50 +01:00
Michael Eischer
e2b0072441 check: add progress bar to the tree structure check 2021-01-28 11:10:50 +01:00
Michael Eischer
505f8a2229 progress/counter: Support updating the progress bar maximum 2021-01-28 11:10:47 +01:00
Michael Eischer
eda8c67616 restic: let FindUsedBlobs handle multiple snapshots at once 2021-01-28 11:08:43 +01:00
Michael Eischer
258ce0c1e5 parallel: report progress for StreamTrees 
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
 2021-01-28 11:08:43 +01:00
Michael Eischer
f2a1b125cb restic: Actually parallelize FindUsedBlobs 2021-01-28 11:08:43 +01:00
Michael Eischer
6e03f80ca2 check: Split the parallelized tree loader into a reusable component 
The actual code change is minimal
 2021-01-28 11:08:43 +01:00
Michael Eischer
1d7bb01a6b check: Cleanup tree loading and switch to use errgroup 
The helper methods are now wired up in the Structure method.
 2021-01-28 11:08:43 +01:00
Alexander Neumann
a4689eb3b9
Merge pull request #3199 from MichaelEischer/non-interactive-counter 
Don't print progress on non-interactive terminals
 2021-01-28 10:53:38 +01:00
Alexander Neumann
c5a66e9181 ui: Simlify channel receive 2021-01-28 10:42:02 +01:00
Michael Eischer
678e75e1c2 sftp: enforce use of optimized upload method 
ReadFrom was already used by Save before, this just ensures that this
won't accidentally change in the future.
 2021-01-03 22:23:53 +01:00
Michael Eischer
6b5b29dbee limiter: add unit tests 2021-01-03 22:23:53 +01:00
Michael Eischer
f35f2c48cd limiter: support WriteTo in LimitBackend for read rate limiting 2021-01-03 22:23:53 +01:00
Michael Eischer
bcb852a8d0 hashing: support WriteTo in the reader 2021-01-03 22:23:53 +01:00
Michael Eischer
69d5b4c36b restorer: lower-case variable name 2021-01-03 13:55:59 +01:00
Alexander Weiss
34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss
7409225fa8 Add filerestorer test where only parts of pack are used 2021-01-01 07:24:46 +01:00
Alexander Weiss
07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss
3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Alexander Neumann
8b84c96d9d
Merge pull request #3204 from MichaelEischer/archiver-tomb-race 
archiver: fix race condition during worker startup
 2020-12-30 20:04:20 +01:00
Michael Eischer
debc4a3a99 archiver: fix race condition during worker startup 
When the tomb is created with a canceled context, then the workers
started via `t.Go` exist nearly immediately. Once for the first time all
started goroutines have been stopped, it is not allowed to issue further
calls to `t.Go`. This is a problem when the started goroutines exit
immediately, as for example the first goroutine might already have
stopped before starting the second one, which is not allowed as once the
first goroutines has stopped no goroutines were running.

To fix this race condition the startup and main task of the archiver now
also run within a `t.Go` function. This also allows unifying the error
handling as it is no longer necessary to distinguish between errors
returned by the workers or the saveTree processing. The tomb now just
returns the first error encountered, which should also be the most
descriptive one.
 2020-12-30 17:31:22 +01:00
MichaelEischer
31e56f1ad5
Merge pull request #3197 from SkYNewZ/fix/3183 
Fix tag handling for multiple tag lists
 2020-12-29 18:44:38 +01:00
Alexander Neumann
b2efa0af39
Merge pull request #3164 from MichaelEischer/improve-context-cancel 
Improve context cancel handling in archiver and backends
 2020-12-29 17:03:42 +01:00
Michael Eischer
cff4955a48 ui: remove dead struct member 2020-12-29 16:32:18 +01:00
Michael Eischer
92da5168e1 ui: force backup progress update on signal 2020-12-29 16:32:18 +01:00
Michael Eischer
34afc93ddc ui/progress: extract signal handling into own package 2020-12-29 16:32:18 +01:00
Michael Eischer
023eea6463 ui: don't shorten non-interactive progress output 2020-12-29 16:32:18 +01:00
Michael Eischer
684600cf42 ui: update status for the backup command on non-interactive terminals 
Allow the backup command to print status on non-interactive terminals.
The output is disabled by setting a MinUpdatePause == 0.
 2020-12-29 16:03:43 +01:00
Michael Eischer
13ce981794 ui: cleanup backup status shutdown 2020-12-29 16:03:43 +01:00
Michael Eischer
c2ef049f1b ui/progress: don't print progress on non-interactive terminals 
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058.
 2020-12-29 16:03:43 +01:00
Quentin Lemaire
334d8ce724
feat(tags): Create Flatten() method 2020-12-29 10:59:46 +01:00
greatroar
3b09ae9074 AIX port 2020-12-29 01:35:01 +01:00
greatroar
18531e3d6f Portability fixes to internal/restic 
syscall.Mknod is not available on AIX.
 2020-12-29 01:35:01 +01:00
Michael Eischer
d0ca8fb0b8 backend: test that a canceled context prevents RetryBackend operations 2020-12-28 21:06:47 +01:00
Michael Eischer
08b7f2b58d archiver: test that context canceled error is not dropped 2020-12-28 21:06:47 +01:00
Michael Eischer
e483b63c40 retrybackend: Fail operations when context is already canceled 
Depending on the used backend, operations started with a canceled
context may fail or not. For example the local backend still works in
large parts when called with a canceled context. Backends transfering
data via http don't work. It is also not possible to retry failed
operations in that state as the RetryBackend will abort with a 'context
canceled' error.

Ensure uniform behavior of all backends by checking for a canceled
context by checking for a canceled context as a first step in the
RetryBackend. This ensures uniform behavior across all backends, as
backends are always wrapped in a RetryBackend.
 2020-12-28 21:06:47 +01:00
Michael Eischer
fc60b560ba archiver: Let saveTree report a canceled context as an error 
If the context was canceled then saveTree might receive a treeID or not
depending on the timing. This could cause saveTree to incorrectly return
a nil treeID as valid. Fix this always returning an error when the
context was canceled in the meantime.
 2020-12-28 21:06:47 +01:00
Michael Eischer
736e964317 archiver: Don't loose error if background context is canceled 
A canceled background context lets the blob/tree/fileSavers exit
without reporting an error. The error handling previously replaced
a 'context canceled' error received by the main backup method with
the error reported by the savers. However, in case of a canceled
background context that error is nil, causing restic to loose the
error and save a snapshot with a nil tree.
 2020-12-28 21:06:47 +01:00
DRON-666
332b1896d1 Some options fixes 
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
 2020-12-23 23:26:04 +03:00
MichaelEischer
cb6b0f6255
Merge pull request #3181 from tWido/feature-no-retry-permission 
Don't retry when "Permission denied" occurs in local backend
 2020-12-23 20:24:29 +01:00
MichaelEischer
1e73aac610
Merge pull request #3179 from aawsome/check-filesize 
check: Remove filesize counter
 2020-12-23 20:22:02 +01:00
Alexander Weiss
2a1add7538 check: remove file size counter 2020-12-23 02:34:31 +01:00
tWido
7dab113035 Don't retry when "Permission denied" occurs in local backend 2020-12-22 23:41:12 +01:00
Michael Eischer
de99207046 repository: tweak comment for packs method 2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Michael Eischer
b9f5d3fe13 repository: Add test for ForAllIndexes 2020-12-22 22:36:18 +01:00
Michael Eischer
a12c5f1d37 repository: move otherwise unused LoadIndex to tests 2020-12-22 22:36:18 +01:00
Michael Eischer
24474a36f4 repository: deduplicate index loading implementation 2020-12-22 22:36:18 +01:00
Michael Eischer
96904f8972 check: extract parallel index loading 2020-12-22 22:36:18 +01:00
Alexander Neumann
6c514adb8a ui/progress: Use mutex instead of atomic 
The counter value needs to be aligned to 64 bit in memory for the
atomic functions to work on some platform (such as 32 bit ARM).

The atomic package says in its documentation:

> These functions require great care to be used correctly. Except for
> special, low-level applications, synchronization is better done with
> channels or the facilities of the sync package.

This commit replaces the atomic functions with a simple sync.Mutex, so
we don't have to care about alignment.
 2020-12-22 21:03:27 +01:00
DRON-666
83b10dbb12 Deduplicate dumper closing logic 2020-12-19 02:42:46 +03:00
DRON-666
e136dd8696 Deduplicate test code 2020-12-19 02:06:54 +03:00
DRON-666
33adb58817 Minor fixes and linter suggestions 2020-12-19 02:04:17 +03:00
DRON-666
89ab6d557e dump: Add zip dumper 2020-12-19 01:09:47 +03:00
DRON-666
0256f95994 dump: Split tar and walk logic 2020-12-19 01:09:47 +03:00
Michael Eischer
34b6130a0e restic: parallelize lock file loading 2020-12-18 20:46:16 +01:00
greatroar
66d904c905 Make invalid handles permanent errors 2020-12-17 12:47:53 +01:00
greatroar
746dbda413 Mark "ssh exited" errors in SFTP as permanent 2020-12-17 12:43:09 +01:00
greatroar
f7784bddb3 Don't retry when "no space left on device" in local backend 
Also adds relevant documentation to the restic.Backend interface.
 2020-12-17 12:43:09 +01:00
Alexander Neumann
e96677cafb
Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables 
swift: Add support for id based keystone v3 auth parameters
 2020-12-12 16:27:38 +01:00
Michael Eischer
1d69341e88 swift: Add support for id based keystone v3 auth parameters 
This adds support for the following environment variables, which were
previously missing:

OS_USER_ID            User ID for keystone v3 authentication
OS_USER_DOMAIN_ID     User domain ID for keystone v3 authentication
OS_PROJECT_DOMAIN_ID  Project domain ID for keystone v3 authentication
OS_TRUST_ID           Trust ID for keystone v3 authentication
 2020-12-11 19:22:34 +01:00
Alexander Neumann
36c5d39c2c Fix issues reported by semgrep 2020-12-11 09:41:59 +01:00
Alexander Neumann
7facc8ccc1
Merge pull request #2505 from aawsome/fix-repo-configfile 
Fix repo configfile
 2020-12-07 07:52:37 +01:00
Alexander Neumann
ba31c6fdaa
Merge pull request #3150 from MichaelEischer/fix-windows-redir-output 
termstatus: Fix canUpdateStatus detection for redirected output on windows
 2020-12-06 21:17:27 +01:00
Alexander Neumann
b58799d83a
Merge pull request #3152 from MichaelEischer/fix-backup-background-hang 
backup: Fix shutdown hang when running in the background on linux
 2020-12-06 21:16:00 +01:00
Alexander Neumann
0d5b764f90
Merge pull request #3130 from aawsome/snapshots-parallel 
Make loading snapshots parallel
 2020-12-06 21:08:18 +01:00
Michael Eischer
b48f579530 termstatus: Fix canUpdateStatus detection for redirected output 
The canUpdateStatus check was simplified in #2608, but it accidentally flipped
the condition. The correct check is as follows: If the output is a pipe then
restic probably runs in mintty/cygwin. In that case it's possible to
update the output status. In all other cases it isn't.

This commit inverts to condition again to offer the previous and correct
behavior.
 2020-12-06 19:02:42 +01:00
Michael Eischer
401ef92c5f backup: Fix shutdown hang when running in the background 
On shutdown the backup commands waits for the terminal output goroutine
to stop. However while running in the background the goroutine ignored
the canceled context.
 2020-12-06 18:53:41 +01:00
Alexander Weiss
e329623771 Remove LoadAllSnapshots 2020-12-06 05:22:27 +01:00
Alexander Weiss
26f85779be Parallelize ForAllSnapshots 2020-12-06 05:09:58 +01:00
Alexander Weiss
5b9ee56335 Add ForAllSnapshots 2020-12-06 05:04:21 +01:00
Michael Eischer
d05c88a5d6 fuse: Properly set uid/gid for directories 
In #2584 this was changed to use the uid/gid of the root node. This
would be okay for the top-level directory of a snapshot, however, this
change also applied to normal directories within a snapshot. This
change reverts the problematic part and adds a test that directory
attributes are represented correctly.
 2020-11-30 23:41:49 +01:00
MichaelEischer
f2959127b6
Merge pull request #3065 from greatroar/local-subdirs 
Don't recurse in local backend's List if not required
 2020-11-29 19:03:59 +01:00
Alexander Weiss
aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss
e3013271a6 Harmonize naming 2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Weiss
a851c53cbe Use PackSize in checker 2020-11-21 22:13:54 +01:00
Alexander Weiss
ce5d630681 Add MasterIndex.PackSize() 2020-11-21 22:13:54 +01:00
Alexander Weiss
c3ddde9e7d Return hdrSize in ListPack 2020-11-21 22:13:54 +01:00
greatroar
8e213e82fc backend/local: replace fs.Walk with custom walker 
This code is more strict in what it expects to find in the backend:
depending on the layout, either a directory full of files or a directory
full of such directories.
 2020-11-19 16:46:42 +01:00
Alexander Neumann
75eff92b56
Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init 
do not require gs bucket permissions to init repository
 2020-11-18 16:53:45 +01:00
eleith
a24e986b2b do not require gs bucket permissions to init repository 
a gs service account may only have object permissions on an existing
bucket but no bucket create/get permissions.

these service accounts currently are blocked from initialization a
restic repository because restic can not determine if the bucket exists.

this PR updates the logic to assume the bucket exists when the bucket
attribute request results in a permissions denied error.

this way, restic can still initialize a repository if the service
account does have object permissions

fixes: https://github.com/restic/restic/issues/3100
 2020-11-18 06:14:11 -08:00
rawtaz
1b6e8c888f
Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param 
Remove `--drive-use-trash=false` from rclone param
 2020-11-17 11:21:20 +01:00
Michael Eischer
1f43cac12d check: Only track data blobs when unused blobs should be reported 
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
 2020-11-15 18:43:07 +01:00
Michael Eischer
6da66c15d8 check: Simplify referenced blob tracking 
The result is identical as long as the context in not canceled. However,
in that case the result is incomplete anyways.
 2020-11-15 18:42:55 +01:00
Michael Eischer
3500f9490c check: Simplify blob status tracking 
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
 2020-11-15 18:42:42 +01:00
Michael Eischer
b8c7543a55 check: Merge 'size could not be found' and 'not found in index' errors 
By construction these two errors always show up in pairs: 'size could
not be found' is printed when the blob is not found in the repository
index. That blob is also part of the `blobs` array. Later on, check
iterates over that array and checks whether the blob is marked as
existing. Which cannot be the case as that mark is generated by
iterating over the repository index.

The merged warning no longer reports the blob index within a file. That
information could also be derived by printing the affected tree using
`cat` and searching for the blob.
 2020-11-15 18:41:50 +01:00
Alexander Neumann
3c0c0c132b
Merge pull request #3006 from aawsome/new-rebuild-index 
Reimplement rebuild-index and remove /internal/index
 2020-11-15 17:48:43 +01:00
Alexander Neumann
9968220652
Merge pull request #2850 from greatroar/packer-malloc 
Decrease allocation rate in internal/pack
 2020-11-15 17:19:49 +01:00
greatroar
ab2b7d7f9a Decrease allocation rate in internal/pack 
internal/repository benchmark results:

name             old time/op    new time/op    delta
PackerManager-8     179ms ± 1%     181ms ± 1%   +0.78%  (p=0.009 n=10+10)

name             old speed      new speed      delta
PackerManager-8   294MB/s ± 1%   292MB/s ± 1%   -0.77%  (p=0.009 n=10+10)

name             old alloc/op   new alloc/op   delta
PackerManager-8    91.3kB ± 0%    72.2kB ± 0%  -20.92%  (p=0.000 n=9+7)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.38k ± 0%     0.76k ± 0%  -45.20%  (p=0.000 n=10+7)
 2020-11-15 16:51:47 +01:00
greatroar
9a8a2cae4c Move pack testing logic to test file 2020-11-15 16:45:49 +01:00
Alexander Weiss
9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a Reimplement rebuild-index 2020-11-15 07:05:09 +01:00
Alexander Weiss
187c8fb259 Parallelize MasterIndex.Save() 2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5 Add extraObsolete to MasterIndex.Save 2020-11-15 07:05:09 +01:00
Alexander Weiss
5898cb341f Use CreateIndexFromPacks() in test 2020-11-15 07:05:05 +01:00
Alexander Weiss
43732bb885 Add CreateIndexFromPacks() 2020-11-15 07:04:51 +01:00
greatroar
c45f8ee075 Replace restic.statT interface by concrete types 
name                old time/op    new time/op    delta
NodeFillUser-8        1.81µs ± 9%    1.50µs ± 5%  -17.07%  (p=0.000 n=19+20)
NodeFromFileInfo-8    1.76µs ± 4%    1.49µs ± 6%  -15.63%  (p=0.000 n=20+19)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)
NodeFromFileInfo-8      496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
NodeFromFileInfo-8      3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
 2020-11-14 23:23:26 +01:00
MichaelEischer
3601a9b6cd
Merge pull request #2690 from SkYNewZ/master 
Fix #2688: Handle comma-separated list tags when using backup command
 2020-11-14 23:19:42 +01:00
MichaelEischer
333c5a19d4
Merge pull request #3082 from aawsome/check-sizes 
Check: check sizes of packs from index and packheader
 2020-11-14 22:37:42 +01:00
Quentin Lemaire
ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list 2020-11-14 15:48:56 +00:00
Alexander Weiss
17bb77b1f9 check: Also check blob length and offset 2020-11-14 00:42:49 +01:00
Alexander Weiss
80dcfca191 check: Check sizes computed from index and pack header 2020-11-14 00:42:49 +01:00
tofran
94a154c7ca Remove --drive-use-trash=false from rclone param 
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
 2020-11-13 22:58:48 +00:00
Alexander Neumann
04d1983800
Merge pull request #3090 from fgma/vss-fix-386 
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
 2020-11-13 21:30:34 +01:00
Alexander Neumann
8ad9f88993 helpers: Improve error message 2020-11-12 20:38:31 +01:00
fgma
f9223cd827 vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386 2020-11-12 19:31:00 +01:00
rawtaz
dfb9326b1b
Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag 
Extended option to select V1 API for ListObjects on S3 backend
 2020-11-11 20:52:44 +01:00
Alexander Neumann
4a0b7328ec s3: Remove dots for config description 2020-11-11 20:20:35 +01:00
greatroar
63e32c44c0 Improve error reporting from restic diff 
Instead of a stacktrace, restic diff 111 222 now reports:

	Fatal: no matching ID found for prefix "111"
 2020-11-11 16:40:40 +01:00
Nick Douma
829959390a Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1 2020-11-11 11:54:38 +01:00
Nick Douma
ccd55d529d Add s3.list-objects-v1 extended option and default to false 2020-11-11 11:54:36 +01:00
Nick Douma
4ddcc17135 Add support for boolean extended options 2020-11-11 11:54:27 +01:00
MichaelEischer
407843c5f9
Merge pull request #3034 from hoyho/gh_master 
bugfix: omit ENOTDATA for extended attributes
 2020-11-09 22:38:49 +01:00
MichaelEischer
46d31ab86d
Merge pull request #3058 from greatroar/counter 
Replace restic.Progress with new progress.Counter (fixes two race conditions)
 2020-11-09 22:19:09 +01:00
Alexander Neumann
c986823d3f
Merge pull request #3048 from aawsome/check-pack-index 
check: check index for packs that are read
 2020-11-09 20:12:32 +01:00
Alexander Weiss
239931578c check: check index for packs that are read 2020-11-09 17:28:14 +01:00
hoyho
9df52327cc bugfix: omit ENOTDATA for extended attributes 
Signed-off-by: hoyho <luohaihao@gmail.com>
 2020-11-10 00:20:34 +08:00
greatroar
21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter 
This fixes two race conditions while cleaning up the code.
 2020-11-09 12:12:35 +01:00
Alexander Neumann
3bf43d7951
Merge pull request #2475 from restic/use-github-actions 
Use GitHub Actions for CI
 2020-11-08 17:30:12 +01:00
Alexander Neumann
5cf42884c8 Run tests on GitHub Actions 2020-11-08 16:04:57 +01:00
Alexander Weiss
826cfa0533 fix context in archiver tests 2020-11-08 08:24:24 +01:00
Alexander Weiss
fef408a8bd Return context error in mem backend 2020-11-08 00:05:53 +01:00
greatroar
a2d4209322 Don't recurse in local backend's List if not required 
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.

Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9).
 2020-11-07 08:54:13 +01:00
Alexander Weiss
fd33030556 Use in-memory index to rebuild index in prune 2020-11-06 20:23:30 +01:00
Alexander Weiss
38cc4393f6 Add Masterindex.Save(); Add Index.Packs() 2020-11-06 20:23:30 +01:00
Alexander Weiss
aaf1c44362 Fix #3062 2020-11-05 17:05:42 +01:00
Ivan Andreev
ab2790d9de Fix http2 stream reset between restic and rest backends #3014 2020-11-05 15:57:40 +03:00
Nick Craig-Wood
86b5d8ffaa s3: add bucket-lookup parameter to select path or dns style bucket lookup 
This is to enable restic working with Alibaba cloud

Fixes #2528
 2020-11-05 12:20:10 +01:00
Alexander Neumann
ae5302c7a8 Add comment that keepBlobs is modified 2020-11-05 10:33:38 +01:00
Alexander Neumann
866a52ad4e Remove unneeded seek 
The file returned from DownloadAndHash() is already seeked to the start
of the file.
 2020-11-05 10:31:49 +01:00
Alexander Neumann
a4507610a0 Fix typo 2020-11-05 10:31:49 +01:00
Alexander Neumann
7def2d8ea7 Use a non-constant seed 2020-11-05 10:31:49 +01:00
Alexander Neumann
ee0112ab3b Clarify message about expected error 2020-11-05 10:31:49 +01:00
Michael Eischer
b373f164fe prune: Parallelize repack command 2020-11-05 10:31:49 +01:00
Alexander Neumann
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command 
Reimplementation of prune
 2020-11-05 10:12:19 +01:00
fgma
916b2d303b
vss: fix potential crash (not reachable in restic) (#3045) 
HasSufficientPrivilegesForVSS() now returns an error
 2020-11-04 22:14:18 +01:00
Alexander Neumann
a06f5c28c0
Merge pull request #3055 from greatroar/cleanup 
Don't separately allocate sync.{Mutex,Once} if not necessary
 2020-11-04 11:11:21 +01:00
greatroar
c2f3eee5af Don't separately allocate sync.{Mutex,Once} if not necessary 
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
 2020-11-04 09:55:43 +01:00
Alexander Weiss
b2f5381737 Make realistic forget --prune --dryrun 2020-11-03 16:42:21 +01:00
Alexander Weiss
ce7d613749 Add Blob.Handle() 2020-11-03 16:42:21 +01:00
Alexander Weiss
581d90cf91 Make some pack parameters public 2020-11-03 16:42:21 +01:00
Alexander Neumann
9a88fb253b
Merge pull request #3051 from greatroar/sanitize-env 
Sanitize environment before starting backend processes (rclone, ssh)
 2020-11-02 21:18:57 +01:00
Alexander Neumann
f14436953a
Merge pull request #3050 from greatroar/widechars 
Fix string truncation in ui/termstatus
 2020-11-02 16:58:51 +01:00
greatroar
11fbaaae9a Sanitize environment before starting backend processes (rclone, ssh) 
The restic security model includes full trust of the local machine, so
this should not fix any actual security problems, but it's better to be
safe than sorry.

Fixes #2192.
 2020-11-02 16:41:23 +01:00
greatroar
3ed84ff0c6 Fix string truncation in ui/termstatus 
Fixes #3046.
 2020-11-02 12:50:49 +01:00
Alexander Neumann
8e965ed4eb
Merge pull request #3039 from MichaelEischer/tar-proper-dirs 
Properly dump directories to tar
 2020-11-02 12:30:26 +01:00
Alexander Neumann
5f0fa2129e Improve readability 
It's time to use a switch statement.
 2020-11-02 11:24:26 +01:00
Alexander Neumann
6509c207f4
Merge pull request #2997 from MichaelEischer/faster-excludes 
Speedup exclude/include checking
 2020-11-02 11:07:21 +01:00
Alexander Neumann
445b845267
Merge pull request #2978 from MichaelEischer/warn-tree-error 
Warn if backup failed to read tree blob
 2020-11-02 10:14:12 +01:00
Alexander Neumann
3ff37215df
Merge pull request #2935 from MichaelEischer/upgrade-minio 
Upgrade minio SDK to version 7
 2020-11-02 09:09:10 +01:00
Sergio Rubio
e708628cfd
Remove unused function 
Not currently used, and it'd need to be added to the MasterIndex interface first.
 2020-10-28 13:24:49 +01:00
MichaelEischer
bb4b3481a6
Merge pull request #3031 from greatroar/debug-no-seek 
Allow debug logging to pipes and terminals
 2020-10-25 17:59:06 +01:00
MichaelEischer
ad3a52e6f0
Merge pull request #3026 from greatroar/refactor-ui 
internal/ui refactoring
 2020-10-25 17:51:39 +01:00
Michael Eischer
8d7d6ad2d5 dump: include username in tar 2020-10-24 22:42:01 +02:00
Michael Eischer
fe09e6f865 dump: test proper permissions and directory name 2020-10-24 22:42:01 +02:00
Michael Eischer
1e3c9a2c11 dump: Fix file permission to tar mapping 
The file permissions included a go specific directory bit which
accidentially forced the usage of the GNU header format. This leads
to problems with 7zip on Windows or when extended attributes are
used.
 2020-10-24 22:42:01 +02:00
Michael Eischer
e21dcb0eea dump: Additional ACL tests 2020-10-24 22:41:01 +02:00
MichaelEischer
31b8d7a639
Merge pull request #2274 from fgma/master 
Support for Volume Shadow Copy Service (VSS) on windows
 2020-10-24 15:43:34 +02:00
fgma
5695f9ebd2 vss: Implement VSS support for Windows 
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.

Snapshotting volumes also covers mountpoints but skips UNC paths.
 2020-10-24 11:35:57 +02:00
greatroar
63be3704d9 Allow debug logging to pipes and terminals 2020-10-19 22:30:30 +02:00
greatroar
35419de232 Simplify ui.StdioWrapper.Write 
Instead of looping to find line breaks, make it look for the last one.
 2020-10-17 20:30:46 +02:00
greatroar
863a590a81 Refactor termstatus.Term.{Print,Error} methods 2020-10-17 20:30:46 +02:00
greatroar
7c0b6a82db Remove unused public method ui.linesWriter.Flush 2020-10-17 20:30:24 +02:00
MichaelEischer
96a912b65a
Merge pull request #3025 from aawsome/fix-index-ids 
Fix setting of ID in DecodeIndex
 2020-10-17 09:54:45 +02:00
Alexander Weiss
b44ecde8b0 Fix setting of ID in DecodeIndex 2020-10-17 09:12:58 +02:00
MichaelEischer
39fe1e96fe
Merge pull request #3018 from greatroar/background-check 
Fix IsProcessBackground on Linux with stdin redirection
 2020-10-16 22:50:19 +02:00
MichaelEischer
4ba237bb93
Merge pull request #3019 from greatroar/refactor-decodeindex 
Refactor index decoding
 2020-10-15 23:22:33 +02:00
greatroar
b27375f5ce defer close(ch) outside repository.RunWorkers 2020-10-14 15:50:16 +02:00
greatroar
720e0ee0c7 if cond { return true }; return false => return cond 2020-10-13 20:56:43 +02:00
greatroar
27db3ec262 Refactor index decoding 
Decoding old-format indices no longer requires loading and decrypting
twice.
 2020-10-13 20:47:50 +02:00
greatroar
f80b07b2c8 Fix IsProcessBackground on Linux with stdin redirection 
The previous implementation assumed that stdin was a terminal.
It now checks the terminal's fd.
 2020-10-13 13:12:02 +02:00
MichaelEischer
41a45ae908
Merge pull request #3016 from greatroar/uid-gid-lookup 
Simplify/optimize cached UID/GID lookups
 2020-10-12 23:24:55 +02:00
Alexander Neumann
56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings 
Fix some goreport warnings
 2020-10-12 20:44:56 +02:00
greatroar
f0cd16e5ea Cache uint32-typed ids in lookup{Username,Group} 
NodeFillUser-8        1.92µs ± 5%    1.75µs ± 3%   -8.89%  (p=0.000 n=10+10)
NodeFromFileInfo-8    1.89µs ± 7%    1.76µs ± 4%   -6.69%  (p=0.001 n=10+10)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)
NodeFromFileInfo-8      504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
NodeFromFileInfo-8      5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
 2020-10-12 15:08:15 +02:00
greatroar
a03fe4562e Remove unused error return in lookup{Username,Group} 2020-10-12 14:48:23 +02:00
Alexander Neumann
c193cea119 Pass Context to NewRestorer() 2020-10-10 15:24:26 +02:00
Alexander Neumann
f0d49ca600
Merge pull request #2933 from MichaelEischer/less-context-todo 
Replace most usages of context.TODO()
 2020-10-10 15:03:44 +02:00
kitone
6099f81692 Fix #1212 restore code produces inconsistent timestamps/permissions. 
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
 2020-10-10 13:46:44 +02:00
kitone
9d7f616190 Improve restorer debug log information 2020-10-10 13:46:43 +02:00
kitone
295ddb9e57 Add test case for inconsistent timestamps and permissions restoration 
Reproduce from https://github.com/restic/restic/issues/1212
 2020-10-10 13:46:43 +02:00
Michael Eischer
e638b46a13 Embed context into ReaderAt 
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
 2020-10-09 22:39:07 +02:00
Michael Eischer
d6cfe857b7 pass proper context into MasterIndex.RebuildIndex 2020-10-09 22:39:07 +02:00
Michael Eischer
37a5e2d681 rest: use global context on repository creation 2020-10-09 22:39:06 +02:00
Michael Eischer
4b0fcaed45 unlock: use proper context for locks cleanup 
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
 2020-10-09 22:37:56 +02:00
Michael Eischer
645a6efaf2 restorer: remove redundant type specification 2020-10-09 22:37:56 +02:00
Michael Eischer
a449450021 init: pass proper context to master key generation 
This is no change in behavior as a canceled context did later on cause
the config file creation to fail. Therefore this change just lets the
repository initialization fail a bit earlier.
 2020-10-09 22:37:56 +02:00
Michael Eischer
603bb0e309 restore: Use proper context while loading snapshot 2020-10-09 22:37:56 +02:00
Michael Eischer
c458e114d4 pass context to Find / FindSnapshot 
This allows proper interruption of restic while it searches for
snapshots or key files.
 2020-10-09 22:37:56 +02:00
Michael Eischer
45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
Michael Eischer
307a6ba3a3 Upgrade minio sdk to v7 
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
 2020-10-09 22:37:24 +02:00
Michael Eischer
50da20d93d Warn if backup failed to read tree blob 2020-10-09 22:36:27 +02:00
Alexander Neumann
5fd3dbccb7
Merge pull request #2963 from MichaelEischer/fix-status-deadlock 
backup: Fix possible deadlock of scanner goroutine
 2020-10-09 21:35:50 +02:00
Alexander Neumann
35655a481b
Merge pull request #2931 from MichaelEischer/reduce-debug-overhead 
Reduce debug log overhead
 2020-10-09 21:35:11 +02:00
Alexander Neumann
30cb553c8d
Merge pull request #2932 from MichaelEischer/proper-rclone-create 
Call rclone.Create to create a new repository for the rclone backend
 2020-10-09 21:29:15 +02:00
MichaelEischer
356ac404cd
Merge pull request #2996 from greatroar/setconsolecursorposition 
Use windows.SetConsoleCursorPosition in ui/termstatus
 2020-10-09 18:19:02 +02:00
Michael Eischer
88c8e903d2 filter: Fix glob matching on absolute path marker on windows 
A pattern part containing "/" is used to mark a path or a pattern as
absolute. However, on Windows the path separator is "\" such that glob
patterns like "?" could match the marker. The code now explicitly skips
the marker when the pattern does not represent an absolute path.
 2020-10-09 16:11:05 +02:00
greatroar
740758a5fa Optimize filter pattern matching 
By replacing "**" with "", checking for this special path component can
be reduced to a length-zero check.

name                          old time/op    new time/op    delta
FilterLines-8                   44.7ms ± 5%    44.9ms ± 5%     ~     (p=0.631 n=10+10)
FilterPatterns/Relative-8       13.6ms ± 4%    13.4ms ± 5%     ~     (p=0.165 n=10+10)
FilterPatterns/Absolute-8       10.9ms ± 5%    10.7ms ± 4%     ~     (p=0.052 n=10+10)
FilterPatterns/Wildcard-8       53.7ms ± 5%    50.4ms ± 5%   -6.00%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-8     128ms ± 2%      95ms ± 1%  -25.54%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=1.000 n=9+8)
FilterPatterns/Absolute-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.903 n=9+8)
FilterPatterns/Wildcard-8       19.7MB ± 0%    19.7MB ± 0%   -0.00%  (p=0.022 n=10+9)
FilterPatterns/ManyNoMatch-8    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-8        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-8     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-09 15:51:14 +02:00
MichaelEischer
862ee4b2c9
Merge pull request #2970 from labkode/fopenskip 
Skip fopen for file change check to avoid network penalty
 2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador
958dc6aafc Skip fopen for file change check to avoid network penalty 2020-10-09 10:53:28 +02:00
MichaelEischer
88cc444779
Merge pull request #2934 from MichaelEischer/upgrade-backoff 
Upgrade github.com/cenkalti/backoff module
 2020-10-08 20:33:30 +02:00
Michael Eischer
b79f18209f Upgrade github.com/cenkalti/backoff module 
We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats
an operation endlessly when a retry count of 0 is specified. This
required a few fixes for the tests.
 2020-10-07 22:04:59 +02:00
Michael Eischer
8388e67c4b filter: cleanup path separator conversion 2020-10-07 21:14:07 +02:00
Michael Eischer
0acc3c5923 filter: special case patterns without globbing characters 
In case a part of a path is a simple string, we can just check for
equality without complex parsing in filepath.Match.

name                          old time/op    new time/op    delta
FilterLines-4                   34.8ms ±17%    41.2ms ±23%  +18.36%  (p=0.000 n=10+10)
FilterPatterns/Relative-4       21.7ms ± 6%    12.1ms ±23%  -44.46%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       10.0ms ± 5%     9.1ms ±11%   -9.80%  (p=0.006 n=10+9)
FilterPatterns/Wildcard-4       47.0ms ± 7%    42.2ms ± 5%  -10.19%  (p=0.000 n=9+10)
FilterPatterns/ManyNoMatch-4     190ms ± 1%     131ms ±20%  -31.47%  (p=0.000 n=8+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.870 n=9+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.145 n=10+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    19.7MB ± 0%  +37.91%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.421 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:55:43 +02:00
Michael Eischer
54a124de3b filter: explicitly test separate ListWithChild function 2020-10-07 20:47:52 +02:00
Michael Eischer
bcc3bddcf4 filter: only check whether a child path could match when necessary 
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.3ms ± 9%    21.7ms ± 6%   -6.68%  (p=0.004 n=10+10)
FilterPatterns/Absolute-4       13.9ms ± 7%    10.0ms ± 5%  -27.61%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4       51.4ms ± 7%    47.0ms ± 7%   -8.51%  (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4     551ms ± 9%     190ms ± 1%  -65.41%  (p=0.000 n=10+8)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.665 n=10+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.480 n=9+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    14.3MB ± 0%     ~     (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:47:52 +02:00
Michael Eischer
17c53efb0d filter: Optimize double wildcard expansion 
This only allocates a single slice to expand the double wildcard and
only copies the pattern prefix once.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       22.7ms ± 5%    23.3ms ± 9%     ~     (p=0.353 n=10+10)
FilterPatterns/Absolute-4       14.2ms ±13%    13.9ms ± 7%     ~     (p=0.853 n=10+10)
FilterPatterns/Wildcard-4        266ms ±16%      51ms ± 7%  -80.67%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4     554ms ± 6%     551ms ± 9%     ~     (p=0.436 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.349 n=10+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.073 n=10+9)
FilterPatterns/Wildcard-4        141MB ± 0%      14MB ± 0%  -89.89%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     0.09M ± 0%  -94.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:47:48 +02:00
Michael Eischer
7959796269 filter: Special case for absolute paths 
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.6ms ±20%    22.7ms ± 5%     ~     (p=0.684 n=10+10)
FilterPatterns/Absolute-4       32.3ms ± 8%    14.2ms ±13%  -56.01%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        334ms ±17%     266ms ±16%  -20.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     709ms ± 7%     554ms ± 6%  -21.89%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%   +0.00%  (p=0.046 n=9+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.464 n=10+10)
FilterPatterns/Wildcard-4        141MB ± 0%     141MB ± 0%     ~     (p=0.163 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     1.63M ± 0%     ~     (p=0.072 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:47:29 +02:00
Michael Eischer
375c2a56de filter: Parse filter patterns only once 
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       30.3ms ±10%    23.6ms ±20%  -22.12%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       49.0ms ± 3%    32.3ms ± 8%  -33.94%  (p=0.000 n=8+10)
FilterPatterns/Wildcard-4        345ms ± 9%     334ms ±17%     ~     (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4     3.93s ± 2%     0.71s ± 7%  -81.98%  (p=0.000 n=9+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       4.63MB ± 0%    3.57MB ± 0%  -22.98%  (p=0.000 n=9+9)
FilterPatterns/Absolute-4       8.54MB ± 0%    3.57MB ± 0%  -58.20%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        146MB ± 0%     141MB ± 0%   -2.93%  (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4     907MB ± 0%       4MB ± 0%  -99.61%  (p=0.000 n=9+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        66.6k ± 0%     22.2k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4        88.7k ± 0%     22.2k ± 0%  -75.00%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.70M ± 0%     1.63M ± 0%   -3.92%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     4.46M ± 0%     0.02M ± 0%  -99.50%  (p=0.000 n=10+10)
 2020-10-07 20:47:27 +02:00
Michael Eischer
b8eacd1364 filter: Reduce redundant path and pattern splitting 
A single call to filter.List will split the path only once and also
split each search pattern only once and use it for both match and
childMatch.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       62.1ms ±15%    30.3ms ±10%  -51.22%  (p=0.000 n=9+10)
FilterPatterns/Absolute-4        111ms ±10%      49ms ± 3%  -56.08%  (p=0.000 n=10+8)
FilterPatterns/Wildcard-4        393ms ±15%     345ms ± 9%  -12.30%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     10.0s ± 3%      3.9s ± 2%  -60.53%  (p=0.000 n=10+9)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       16.4MB ± 0%     4.6MB ± 0%  -71.76%  (p=0.000 n=10+9)
FilterPatterns/Absolute-4       31.4MB ± 0%     8.5MB ± 0%  -72.77%  (p=0.000 n=9+10)
FilterPatterns/Wildcard-4        168MB ± 0%     146MB ± 0%  -13.19%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.23GB ± 0%    0.91GB ± 0%  -71.96%  (p=0.000 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4         178k ± 0%       67k ± 0%  -62.50%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4         266k ± 0%       89k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.87M ± 0%     1.70M ± 0%   -9.47%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     17.7M ± 0%      4.5M ± 0%  -74.87%  (p=0.000 n=9+10)
 2020-10-07 18:13:19 +02:00
Michael Eischer
e73c281142 filter: Benchmark absolute paths, wildcards and long filter lists 2020-10-07 17:54:36 +02:00
Michael Eischer
fdd3b14db3 filter: test some corner cases 2020-10-07 17:09:44 +02:00
Michael Eischer
f4282aa6fd local: mark repository files as read-only 
This is intended to prevent accidental modifications of data files.
Marking the files as read-only was accidentally removed in #1258.
 2020-10-07 12:29:37 +02:00
Michael Eischer
40ee17167e local: Ignore permission errors on chmod call in Save/Remove operation 
The file is already created with the proper permissions, thus the chmod
call is not critical. However, some file systems have don't allow
modifications of the file permissions. Similarly the chmod call in the Remove
operation should not prevent it from working.
 2020-10-07 12:29:37 +02:00
greatroar
f3e933f0c1 Use windows.SetConsoleCursorPosition in ui/termstatus 2020-10-07 11:18:54 +02:00
MichaelEischer
09d39e260d
Merge pull request #2980 from greatroar/error-wrapping 
Remove repetitive error wrapping from internal/cache
 2020-10-06 15:19:00 +02:00
Michael Eischer
1579d2a8ec Remove some unused assignments 2020-10-06 14:55:13 +02:00
Michael Eischer
eba5dd831f Fix typos reported by misspell 2020-10-06 14:55:13 +02:00
Michael Eischer
9ffb698c8d Modernize context import in internal/fuse 
Migrated using `go fix`.
 2020-10-06 14:55:13 +02:00
Michael Eischer
efbb850d92 Remove a few redundant type specifiers 
This is the result of running `gofmt -s -w **/*.go`
 2020-10-06 14:55:13 +02:00
greatroar
673dda77c0 Less repetitive error messages in internal/cache 
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
 2020-10-05 20:28:54 +02:00
MichaelEischer
1a490acd67
Merge pull request #2982 from greatroar/archiver-error-handling 
Check error in archiver before calling Select
 2020-10-05 19:19:47 +02:00
MichaelEischer
c84643c6a9
Merge pull request #2981 from greatroar/mkdircachedir 
Simplify cache directory creation
 2020-10-05 18:15:23 +02:00
greatroar
1e1a1f3078 Upgrade cespare/xxhash to 2.1.1 2020-10-05 17:25:39 +02:00
greatroar
c4e2203e45 Check error in archiver before calling Select 
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
 2020-10-05 11:11:04 +02:00
greatroar
7d0fa1a686 Simplify cache directory creation 2020-10-05 10:46:45 +02:00
Ingo Gottwald
8b8e230771 Swap deprecated GCS lib with replacement 2020-10-03 18:55:56 +02:00
Ingo Gottwald
00cedd22aa Replace deprecated method in gs backend 2020-10-01 10:02:42 +02:00
Michael Eischer
d44df9d00d backup: Always remove the status lines once a backup ends 2020-09-30 23:13:10 +02:00
Michael Eischer
8d0ba55ecd backup: Fix possible deadlock of scanner goroutine 
When the backup is interrupted for some reason while the scanner is
still active this could lead to a deadlock. Interruptions are triggered
by canceling the context object used by both the backup progress UI and
the scanner. It is possible that a context is canceled between the
respective check in the scanner and it calling the `ReportTotal` method
of the UI. The latter method sends a message to the UI goroutine.
However, a canceled context will also stop that goroutine, which can
cause the channel send operation to block indefinitely.

This is resolved by adding a `closed` channel which is closed once the
UI goroutine is stopped and serves as an escape hatch for reported UI
updates.

This change covers not just the ReportTotal method but all potentially
affected methods of the progress UI implementation.
 2020-09-30 23:13:10 +02:00
MichaelEischer
48d0ab5276
Merge pull request #2959 from restic/keep-text 
backup: Correct keep policy text
 2020-09-30 17:57:00 +02:00
MichaelEischer
fd02407863
Merge pull request #2849 from classmarkets/gcs-access-token 
gs: support authentication with access token
 2020-09-30 17:42:56 +02:00
Leo R. Lundgren
028c9a5343 backup: Correct keep policy text 
Makes the following corrections to the "Applying Policy:" output:

- keep the last 1 snapshots snapshots => keep 1 latest snapshots
- keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots
 2020-09-28 14:26:53 +02:00
MichaelEischer
14b312f00d
Merge pull request #2658 from creativeprojects/issue-2241 
Don't echo authentication passwords (rest backend)
 2020-09-22 22:17:53 +02:00
Fred
206cadfab4 Hide password from repository URLs 2020-09-22 22:00:51 +02:00
MichaelEischer
4875f7b659
Merge pull request #2614 from greatroar/simplify-fs 
Simplify internal/fs
 2020-09-21 22:05:10 +02:00
greatroar
0d65b78168 Simplify os.ModeType|os.ModeCharDevice => os.ModeType 
Since Go 1.12, ModeCharDevice is included in ModeType:
golang/go@a2a3dd00c9
 2020-09-21 14:21:32 +02:00
greatroar
95ebba85ff Remove stray Printf from internal/fs 2020-09-21 14:21:32 +02:00
greatroar
59b343a9bf Remove OS-specific versions of fs.MkdirAll 
Go has supported Windows paths correctly since 1.11, see
https://github.com/golang/go/issues/10900 and the commit referenced
there.
 2020-09-21 14:21:32 +02:00
greatroar
1557c58eef Fix and simplify fs.Reader 
fakeDir.{Readdir,Readdirnames} weren't handling the case n == 0
correctly. fakeFileInfo.sys is always nil, so omit the field.
 2020-09-21 14:21:32 +02:00
greatroar
c504aa505c Remove unused fs.Rename 2020-09-21 14:21:32 +02:00
greatroar
1b20f6beec Remove io.Writer from fs.File 
It was only used in a single test, which now uses plain *os.File instead.
 2020-09-21 14:21:32 +02:00
Michael Eischer
8c36317b71 rclone: use configured number of connections during create 2020-09-19 19:11:43 +02:00
Michael Eischer
60a5c35de9 rclone: close connection to rclone if open fails 2020-09-19 19:11:43 +02:00
Michael Eischer
07f4e7d10b Only log HTTP requests when a debug log is configured 
The logged HTTP requests are only visible when a debug log is
configured.
 2020-09-19 17:07:53 +02:00
Michael Eischer
f003410402 init: Add --copy-chunker-params option 
This allows creating multiple repositories with identical chunker
parameters which is required for working deduplication when copying
snapshots between different repositories.
 2020-09-19 16:53:05 +02:00
Michael Eischer
9ad8250a78 Reduce overhead of debug calls if no log is enabled 
In case no debug log is configured, then calls to debug.Log only incur
the costs to check a single boolean flag making the call really cheap.
 2020-09-19 16:45:01 +02:00
greatroar
cb3f531050 Make Cache.Path private 
It's only used inside the package.
 2020-09-18 11:17:29 +02:00
greatroar
23fcbb275a Remove restic.Cache interface 
It was used in one code path, which then asserted its concrete type as
*cache.Cache.

Privatised some of the interface methods.
 2020-09-18 10:48:13 +02:00
greatroar
6e3215a80d Remove unused error type from internal/cache 2020-09-18 10:38:14 +02:00
greatroar
9abef3bf1a Move internal/fs.TestChdir to internal/test.Chdir 2020-09-17 10:43:33 +02:00
Alexander Neumann
0c48e515f0
Merge pull request #2630 from MichaelEischer/fix-staticcheck 
Fix lots of small issues reported by staticcheck
 2020-09-13 16:19:22 +02:00
Alexander Neumann
97950ab81a options: Fix test for Go >= 1.15 2020-09-12 17:36:44 +02:00
MichaelEischer
88664ba222
Merge pull request #2893 from MichaelEischer/restore-preallocate 
restorer: Preallocate files
 2020-09-08 22:43:05 +02:00
Michael Eischer
121233e1b3 Add preallocate tests 2020-09-07 21:41:47 +02:00
Michael Eischer
8cc9514879 restorer: pre-allocate files before loading chunks 2020-09-07 21:41:47 +02:00
Michael Eischer
2e7d475029 Process packs in order of first appearance 2020-09-06 21:25:35 +02:00
Michael Eischer
c46edcd9d6 error strings should not end with punctuation 2020-09-05 10:07:17 +02:00
Michael Eischer
1ede018ea6 error variable names should start with 'Err' 2020-09-05 10:07:17 +02:00
Michael Eischer
b77e933d80 Convert underscore variable names to camelCase 2020-09-05 10:07:16 +02:00
Michael Eischer
d0329cf3eb Adjust comments to match name of exported methods 2020-09-05 10:07:16 +02:00
Michael Eischer
dc31529fc3 Unindent else block after if block ending with a return statement 2020-09-05 10:07:16 +02:00
Michael Eischer
4784540f04 repository: Simplify worker group code 2020-09-05 10:07:16 +02:00
Michael Eischer
84ea2389ae archiver_test: Only skip symlinks on windows 2020-09-05 10:07:16 +02:00
Michael Eischer
b4a7ce86cf uint cannot be less than zero 2020-09-05 10:07:16 +02:00
Michael Eischer
7ee0964880 crypto: Remove unused error 2020-09-05 10:07:16 +02:00
Michael Eischer
f6f11400c2 restic: Remove unused variable value 2020-09-05 10:07:16 +02:00
Michael Eischer
e2dc5034d3 restic/file: Use FileType for all File constants 
The data type of a variable is only carried on when using iota.
 2020-09-05 10:07:16 +02:00
Michael Eischer
9a1b3cb5d9 restorer: Remove unused treeID struct member from test 2020-09-05 10:07:16 +02:00
Michael Eischer
ee05501ce7 archiver: Remove unused chmod function 2020-09-05 10:07:16 +02:00
Michael Eischer
014600bee6 archiver: Remove unused variable from test 2020-09-05 10:07:16 +02:00
Michael Eischer
d9a80e07b9 repository: Simplify index age calculation 2020-09-05 10:07:16 +02:00
Michael Eischer
460e2ffbf6 Collapse a few boolean operations 2020-09-05 10:07:14 +02:00
Michael Eischer
2f8335554c Remove a few unused variables 2020-09-05 10:06:23 +02:00
Michael Eischer
37113282ca Merge a few variable declaration and initializations 2020-09-05 10:05:34 +02:00
Michael Eischer
337725c354 Use sort.Strings 2020-09-05 10:05:34 +02:00
Michael Eischer
dcf9ded977 dump: Use slashes as path separators in tar on windows 
This conversion is necessary as paths returned by filepath.Rel use the
operating-system native path separator
 2020-08-30 18:26:50 +02:00
Simon Beck
fe6445e0f4 dump: Added unit tests for tar output 2020-08-30 16:00:48 +02:00
Simon Beck
80a11960dd dump: Always dump relative paths into tarballs 
Tarballs should only contain relative paths.
 2020-08-29 21:14:34 +02:00
Simon Beck
d6f739ec22 dump: Extract tar code to internal/dump 2020-08-29 21:14:34 +02:00
Michael Eischer
b25978a53c backup: Fix reporting of directory count in summary 
Previously the directory stats were reported immediately after calling
`SaveDir`. However, as the latter method saves the tree asynchronously
the stats were still initialized to their nil value. The stats are now
reported via a callback similar to the one used for the fileSaver.
 2020-08-27 22:43:51 +02:00
Alexander Neumann
b1b3f1ecb6
Merge pull request #2674 from MichaelEischer/prune-strict-checks 
prune: Stricter error checks
 2020-08-23 10:30:41 +02:00
Hristo Trendev
51c22f4223 local backend: ignore not supported error on sync() 
Closes #2395
 2020-08-22 01:27:07 +02:00
Michael Eischer
f4b9544ab2 prune: Add test that repack aborts on wrong blob 2020-08-16 11:34:01 +02:00
Michael Eischer
367449dede prune: Reduce memory allocations while repacking 
The slicing operator `slice[low:high]` default to 0 for the lower bound and
len(slice) for the upper bound when either or both are not specified.
Fix the code to use `cap(slice)` to check for the slice capacity.
 2020-08-16 11:34:01 +02:00
Michael Eischer
7042bafea5 prune: Abort repacking when a pack contains a wrong blob 
If a blob in a pack file can be decrypted successfully but contains data
that results in a different hash than stated in the header pack, then
abort repacking. As both the pack header and the blob are
cryptographically verified this either means than a malicious entity
tampered with the backup or indicates hardware problems on the client.
prune should fail with an error in both cases.
 2020-08-16 11:34:01 +02:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" (#2885) 
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
 2020-08-16 11:16:38 +02:00
MichaelEischer
eca0f0ad24
Merge pull request #2863 from aawsome/index-no-duplicates 
Don't save exact duplicates in merged index
 2020-08-08 18:24:14 +02:00
Alexander Weiss
b112533812 Don't save exact duplicates when merging indexes 2020-08-05 06:32:02 +02:00
Alexander Weiss
5e63294355 Add benchmark MasterIndexAlloc 2020-08-05 06:32:02 +02:00
Michael Eischer
05116e4787 prune: Cleanup progress bar handling while repacking 2020-08-03 19:32:46 +02:00
Alexander Neumann
b7b479b668
Merge pull request #2599 from MichaelEischer/tweak-mem-usage 
Reduce memory usage when searching for used blobs
 2020-08-01 13:49:02 +02:00
Alexander Neumann
4cf9656f12
Merge pull request #2861 from MichaelEischer/fix-rclone-crash 
rclone: Don't panic after unexpected subprocess exit
 2020-08-01 13:08:50 +02:00
Alexander Neumann
2580eef2aa
Merge pull request #2318 from classmarkets/2175-named-keys 
Allow specifying user and host when adding keys
 2020-08-01 13:06:31 +02:00
Michael Eischer
a178e5628e Remove duplicate TreeLoader interface 2020-08-01 12:29:16 +02:00
Michael Eischer
af66a62c04 FindUsedBlobs: Test that seen blobs are skipped 
This also copies the TreeLoader interface from internal/walker to allow
stubbing the repository in the call to `FindUsedBlobs`.
 2020-08-01 12:29:16 +02:00
Michael Eischer
9ea1a78bd4 FindUsedBlobs: Check for seen blobs before loading trees 
The only effective change in behavior is that that toplevel nodes can
also be skipped.
 2020-08-01 12:29:16 +02:00
Michael Eischer
184103647a FindUsedBlobs: merge seen into blobs BlobSet 
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.

Suggested-by: Alexander Weiss <alex@weissfam.de>
 2020-08-01 12:29:16 +02:00
Michael Eischer
c81b122374 rclone: Don't panic after unexpected subprocess exit 
As the connection to the rclone child process is now closed after an
unexpected subprocess exit, later requests will cause the http2
transport to try to reestablish a new connection. As previously this never
should have happened, the connection called panic in that case. This
panic is now replaced with a simple error message, as it no longer
indicates an internal problem.
 2020-08-01 12:17:40 +02:00
Alexander Weiss
f8316948d1 Optimize FUSE - make command restic mount faster and consume less memory 
- Add Open() functionality to dir
- only access index for blobs when file is read
- Implement NodeOpener and put one-time file stuff there
- Add comment about locking as suggested by bazil.org/fuse

=> Thanks at Michael Eischer for suggesting the last two improvements
 2020-07-28 23:01:18 +02:00
Michael Eischer
ea97ff1ba4 rclone: Skip crash test when rclone is not found 2020-07-26 12:06:18 +02:00
Michael Eischer
01b9581453 rclone: Better field names for stdio conn 2020-07-26 00:29:25 +02:00
Michael Eischer
3cd927d180 rclone: Give rclone time to finish before closing stdin pipe 
Calling `Close()` on the rclone backend sometimes failed during test
execution with 'signal: Broken pipe'. The stdio connection closed both
the stdin and stdout file descriptors at the same moment, therefore
giving rclone no chance to properly send any final http2 data frames.

Now the stdin connection to rclone is closed first and will only be
forcefully closed after a timeout. In case rclone exits before the
timeout then the stdio connection will be closed normally.
 2020-07-26 00:29:25 +02:00
greatroar
bf7b1f12ea rclone: Add test for pipe handling when rclone exits 2020-07-26 00:29:25 +02:00
Michael Eischer
8554332894 rclone: Close rclone side of stdio_conn pipes 
restic did not notice when the rclone subprocess exited unexpectedly.

restic manually created pipes for stdin and stdout and used these for the
connection to the rclone subprocess. The process creating a pipe gets
file descriptors for the sender and receiver side of a pipe and passes
them on to the subprocess. The expected behavior would be that reads or
writes in the parent process fail / return once the child process dies
as a pipe would now just have a reader or writer but not both.

However, this never happened as restic kept the reader and writer
file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe`
close the subprocess side of pipes once the child process was started
and close the parent process side of pipes once wait has finished. We
can't use these functions as we need access to the raw `os.File` so just
replicate that behavior.
 2020-07-26 00:29:25 +02:00
greatroar
3e93b36ca4 Make rclone.New private 2020-07-26 00:28:45 +02:00
Michael Eischer
c847aace35 Rename Index interface to MasterIndex 
The interface is now only implemented by repository.MasterIndex.
 2020-07-25 21:19:46 +02:00
Alexander Weiss
9d1fb94c6c make Lookup() return all blobs 
+ simplify syntax
 2020-07-25 21:18:34 +02:00
MichaelEischer
020cab8e08
Merge pull request #2787 from greatroar/no-blobsize-cache 
Remove blob size cache from restic mount
 2020-07-25 20:46:35 +02:00
greatroar
07da61baee Remove blob size cache from restic mount 2020-07-25 19:39:14 +02:00
MichaelEischer
3d530dfc91
Merge pull request #2827 from aawsome/archiver-test-contents 
Make self-healing work when backing up with parent snapshot
 2020-07-25 13:13:18 +02:00
MichaelEischer
bbc960f957
Merge pull request #2635 from greatroar/optimize-sortbycached 
Optimize sorting blobs by cache status
 2020-07-25 12:35:42 +02:00
greatroar
309598c237 Simplify sortCachedPacksFirst test in internal/repository 
The test now uses the fact that the sort is stable. It's not guaranteed
to be, but the test is cleaner and more exhaustive. sortCachedPacksFirst
no longer needs a return value.
 2020-07-25 12:12:59 +02:00
greatroar
03d23e6faa Speed up blob sorting in internal/repository 
name                    old time/op    new time/op    delta
SortCachedPacksFirst-8     208µs ± 3%     186µs ± 3%  -10.74%  (p=0.000 n=10+8)

name                    old alloc/op   new alloc/op   delta
SortCachedPacksFirst-8     213kB ± 0%     139kB ± 0%  -34.62%  (p=0.000 n=10+10)

name                    old allocs/op  new allocs/op  delta
SortCachedPacksFirst-8     1.03k ± 0%     1.03k ± 0%   -0.19%  (p=0.000 n=10+10)
 2020-07-25 12:12:59 +02:00
greatroar
b10acd2af7 Test and benchmark blob sorting in internal/repository 2020-07-25 12:12:58 +02:00
Alexander Weiss
9175795fdb Check contents in archiver 
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
 2020-07-25 08:18:28 +02:00
rawtaz
5d8d70542f
Merge pull request #2852 from MichaelEischer/drop-go-1.11 
Drop support for Go version 1.11
 2020-07-25 01:17:36 +02:00
Michael Eischer
7c23381a2b Drop support for Go version 1.11 2020-07-24 18:52:39 +02:00
MichaelEischer
34181b13a2
Merge pull request #2328 from MichaelEischer/no-repeated-checks 
Fix duplicate tree checks within `restic check`
 2020-07-22 22:08:02 +02:00
Alexander Weiss
a666a6d576 Add tests and merge indexes in index benchmarks 2020-07-22 21:54:02 +02:00
Alexander Weiss
e388d962a5 Merge final indexes together for faster index access 2020-07-22 21:54:02 +02:00
Peter Schultz
758b44b9c0 gs: support authentication with access token 
In the Google Cloud Storage backend, support specifying access tokens
directly, as an alternative to a credentials file. This is useful when
restic is used non-interactively by some other program that is already
authenticated and eliminates the need to store long lived credentials.

The access token is specified in the GOOGLE_ACCESS_TOKEN environment
variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.
 2020-07-22 16:23:03 +02:00
Alexander Weiss
3b7a3711e6 Add more realistic index benchmarks 
+ reduce test size of BenchmarkMasterIndexLookupParallel
 2020-07-21 07:18:20 +02:00
Michael Eischer
9b0e718852 checker: Test that blob types are not confused 2020-07-20 23:43:47 +02:00
MichaelEischer
82c908871d
Merge pull request #2812 from greatroar/chaining 
Chaining hash table for repository.Index
 2020-07-20 23:29:36 +02:00
Michael Eischer
ddf0b8cd0b checker: Properly distinguish between data and tree blobs 
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
 2020-07-20 22:58:39 +02:00
Michael Eischer
2d0c138c9b checker: Test that check only decodes trees once 
The `DuplicateTree` flag is necessary to ensure that failures cannot be
swallowed. The old checker implementation ignores errors from LoadTree
if the corresponding tree was already checked.
 2020-07-20 22:51:53 +02:00
Michael Eischer
ef325ffc02 checker: Cleanup error handling code 
This change only moves code around but does not result in any change in
behavior.
 2020-07-20 22:51:53 +02:00
Michael Eischer
7a165f32a9 checker: Traverse trees in depth-first order 
Backups traverse the file tree in depth-first order and saves trees on
the way back up. This results in tree packs filled in a way comparable
to the reverse Polish notation.  In order to check tree blobs in that
order, the treeFilter would have to delay the forwarding of tree nodes
until all children of it are processed which would complicate the
implementation.

Therefore do the next similar thing and traverse the tree in depth-first
order, but process trees already on the way down. The tree blob ids are
added in reverse order to the backlog, which is once again reverted when
removing the ids from the back of the backlog.
 2020-07-20 22:51:53 +02:00
Michael Eischer
36c69e3ca7 checker: Unify blobs, processed trees and referenced blobs map 
The blobRefs map and the processedTrees IDSet are merged to reduce the
memory usage. The blobRefs map now uses separate flags to track blob
usage as data or tree blob. This prevents skipping of trees whose
content is identical to an already processed data blob. A third flag
tracks whether a blob exists or not, which removes the need for the
blobs IDSet.
 2020-07-20 22:51:47 +02:00
Michael Eischer
35d8413639 checker: Remove dead index map 2020-07-20 22:37:31 +02:00
Michael Eischer
c66a0e408c checker: Reduce cost of debug log 
Avoid duplicate allocation of the Subtree list.
 2020-07-20 22:37:31 +02:00
Michael Eischer
70f4c014ef checker: Decode identical tree nodes only once 
Even though the checkTreeWorker skips already processed chunks,
filterTrees did queue the same tree blob on every occurence. This
becomes a serious performance bottleneck for larger number of snapshots
that cover mostly the same directories. Therefore decode a tree blob
exactly once.
 2020-07-20 22:37:31 +02:00
Michael Eischer
f0d8710611 Add benchmark for checker scaling with snapshot count 2020-07-20 22:37:31 +02:00
Alexander Neumann
8074879c5f Remove 'go generate' 2020-07-19 17:28:42 +02:00
greatroar
7bda28f31f Chaining hash table for repository.Index 
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.

Benchmark results on linux/amd64, -benchtime=3s -count=20:

name                                            old time/op    new time/op    delta
PackerManager-8                                    178ms ± 0%     178ms ± 0%     ~     (p=0.231 n=20+20)
DecodeIndex-8                                      4.54s ± 0%     4.30s ± 0%   -5.20%  (p=0.000 n=18+17)
DecodeIndexParallel-8                              4.54s ± 0%     4.30s ± 0%   -5.22%  (p=0.000 n=19+18)
IndexHasUnknown-8                                 44.4ns ± 5%    50.5ns ±11%  +13.82%  (p=0.000 n=19+17)
IndexHasKnown-8                                   48.3ns ± 0%    51.5ns ±12%   +6.68%  (p=0.001 n=16+20)
IndexAlloc-8                                       758ms ± 1%     616ms ± 1%  -18.69%  (p=0.000 n=19+19)
IndexAllocParallel-8                               234ms ± 3%     204ms ± 2%  -12.60%  (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8                     122ns ± 0%     145ns ± 9%  +18.44%  (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8                   369ns ± 2%     429ns ± 8%  +16.27%  (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8             68.4ns ± 5%    74.9ns ±13%   +9.47%  (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8            315ns ± 3%     369ns ±11%  +17.14%  (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8        743ns ± 1%     816ns ± 2%   +9.87%  (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8      238ns ± 1%     260ns ± 2%   +9.14%  (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8      1.01µs ± 3%    1.11µs ± 2%   +9.79%  (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8     222ns ± 0%     269ns ± 2%  +20.83%  (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8      1.06µs ± 2%    1.19µs ± 2%  +12.95%  (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8     413ns ± 1%     530ns ± 1%  +28.19%  (p=0.000 n=18+20)
SaveAndEncrypt-8                                  30.2ms ± 1%    30.4ms ± 0%   +0.71%  (p=0.000 n=19+19)
LoadTree-8                                         540µs ± 1%     576µs ± 1%   +6.73%  (p=0.000 n=20+20)
LoadBlob-8                                        5.64ms ± 0%    5.64ms ± 0%     ~     (p=0.883 n=18+17)
LoadAndDecrypt-8                                  5.93ms ± 0%    5.95ms ± 1%     ~     (p=0.247 n=20+19)
LoadIndex-8                                       25.1ms ± 0%    24.5ms ± 1%   -2.54%  (p=0.000 n=18+17)

name                                            old speed      new speed      delta
PackerManager-8                                  296MB/s ± 0%   296MB/s ± 0%     ~     (p=0.229 n=20+20)
SaveAndEncrypt-8                                 139MB/s ± 1%   138MB/s ± 0%   -0.71%  (p=0.000 n=19+19)
LoadBlob-8                                       177MB/s ± 0%   177MB/s ± 0%     ~     (p=0.890 n=18+17)
LoadAndDecrypt-8                                 169MB/s ± 0%   168MB/s ± 1%     ~     (p=0.227 n=20+19)

name                                            old alloc/op   new alloc/op   delta
PackerManager-8                                   91.8kB ± 0%    91.8kB ± 0%     ~     (p=0.772 n=12+19)
IndexAlloc-8                                       786MB ± 0%     400MB ± 0%  -49.04%  (p=0.000 n=20+18)
IndexAllocParallel-8                               786MB ± 0%     401MB ± 0%  -49.04%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                  21.0MB ± 0%    21.0MB ± 0%   +0.00%  (p=0.000 n=19+19)

name                                            old allocs/op  new allocs/op  delta
PackerManager-8                                    1.41k ± 0%     1.41k ± 0%     ~     (all equal)
IndexAlloc-8                                        977k ± 0%      907k ± 0%   -7.18%  (p=0.000 n=20+20)
IndexAllocParallel-8                                977k ± 0%      907k ± 0%   -7.17%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                    73.0 ± 0%      73.0 ± 0%     ~     (all equal)
 2020-07-19 13:58:22 +02:00
greatroar
255ba83c4b Parallel index benchmarks + benchmark optimizations 
createRandomIndex was using the global RNG, which locks on every call
It was also using twice as many random numbers as necessary and doing
a float division in every iteration of the inner loop.

BenchmarkDecodeIndex was using too short an input, especially for a
parallel version. (It may now be using one that is a bit large.)

Results on linux/amd64, -benchtime=3s -count=20:

name                                     old time/op    new time/op       delta
PackerManager-8                             178ms ± 0%        178ms ± 0%           ~     (p=0.165 n=20+20)
DecodeIndex-8                              13.6µs ± 2%  4539886.8µs ± 0%  +33293901.38%  (p=0.000 n=20+18)
IndexHasUnknown-8                          44.4ns ± 7%       44.4ns ± 5%           ~     (p=0.873 n=20+19)
IndexHasKnown-8                            49.2ns ± 3%       48.3ns ± 0%         -1.86%  (p=0.000 n=20+16)
IndexAlloc-8                                802ms ± 1%        758ms ± 1%         -5.51%  (p=0.000 n=20+19)
MasterIndexLookupSingleIndex-8              124ns ± 1%        122ns ± 0%         -1.41%  (p=0.000 n=20+14)
MasterIndexLookupMultipleIndex-8            373ns ± 2%        369ns ± 2%         -1.13%  (p=0.001 n=20+20)
MasterIndexLookupSingleIndexUnknown-8      67.8ns ± 3%       68.4ns ± 5%           ~     (p=0.753 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8     316ns ± 3%        315ns ± 3%           ~     (p=0.846 n=20+20)
SaveAndEncrypt-8                           30.5ms ± 1%       30.2ms ± 1%         -1.09%  (p=0.000 n=19+19)
LoadTree-8                                  527µs ± 1%        540µs ± 1%         +2.37%  (p=0.000 n=19+20)
LoadBlob-8                                 5.65ms ± 0%       5.64ms ± 0%         -0.21%  (p=0.000 n=19+18)
LoadAndDecrypt-8                           7.07ms ± 2%       5.93ms ± 0%        -16.15%  (p=0.000 n=19+20)
LoadIndex-8                                32.1ms ± 2%       25.1ms ± 0%        -21.64%  (p=0.000 n=20+18)

name                                     old speed      new speed         delta
PackerManager-8                           296MB/s ± 0%      296MB/s ± 0%           ~     (p=0.159 n=20+20)
SaveAndEncrypt-8                          138MB/s ± 1%      139MB/s ± 1%         +1.10%  (p=0.000 n=19+19)
LoadBlob-8                                177MB/s ± 0%      177MB/s ± 0%         +0.21%  (p=0.000 n=19+18)
LoadAndDecrypt-8                          141MB/s ± 2%      169MB/s ± 0%        +19.24%  (p=0.000 n=19+20)

name                                     old alloc/op   new alloc/op      delta
PackerManager-8                            91.8kB ± 0%       91.8kB ± 0%           ~     (p=0.826 n=19+12)
IndexAlloc-8                                786MB ± 0%        786MB ± 0%         +0.01%  (p=0.000 n=20+20)
SaveAndEncrypt-8                           21.0MB ± 0%       21.0MB ± 0%         -0.00%  (p=0.012 n=20+19)

name                                     old allocs/op  new allocs/op     delta
PackerManager-8                             1.41k ± 0%        1.41k ± 0%           ~     (all equal)
IndexAlloc-8                                 977k ± 0%         977k ± 0%         +0.01%  (p=0.022 n=20+20)
SaveAndEncrypt-8                             73.0 ± 0%         73.0 ± 0%           ~     (all equal)
 2020-07-19 13:58:05 +02:00
Lars Lehtonen
9ac90cf5cd
internal/fuse: fix dropped test error 2020-07-12 21:42:31 -07:00
greatroar
58719e1f47 Replace mount's per-file cache by a global LRU cache 2020-07-12 18:27:16 +02:00
greatroar
d42c169458 Fix quadratic file reading in restic mount 2020-07-12 18:27:16 +02:00
greatroar
02bec13ef2 Fix repository_test.BenchmarkSaveAndEncrypt 
The benchmark was actually testing the speed of index lookups.

name              old time/op    new time/op        delta
SaveAndEncrypt-8     101ns ± 2%    31505824ns ± 1%    +31311591.31%  (p=0.000 n=10+10)

name              old speed      new speed          delta
SaveAndEncrypt-8  41.7TB/s ± 2%       0.0TB/s ± 1%         -100.00%  (p=0.000 n=10+10)

name              old alloc/op   new alloc/op       delta
SaveAndEncrypt-8     1.00B ± 0%  20989508.40B ± 0%  +2098950740.00%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op      delta
SaveAndEncrypt-8      0.00             123.00 ± 0%            +Inf%  (p=0.000 n=10+9)

(The actual speed is ca. 131MiB/s.)
 2020-07-05 17:41:42 +02:00
MichaelEischer
212607dc8a
Merge pull request #2760 from greatroar/backend-benchmark 
Fix backend benchmarks + a micro-optimization
 2020-06-17 23:17:05 +02:00
greatroar
190d8e2f51 Flatten backend.LimitedReadCloser structure 
This inlines the io.LimitedReader into the LimitedReadCloser body to
achieve fewer allocations. Results on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadPartialFile-8           412µs ± 4%     413µs ± 4%    ~     (p=0.634 n=17+17)
Backend/BenchmarkLoadPartialFileOffset-8     455µs ±13%     441µs ±10%    ~     (p=0.072 n=20+18)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadPartialFile-8        10.2GB/s ± 3%  10.2GB/s ± 4%    ~     (p=0.817 n=16+17)
Backend/BenchmarkLoadPartialFileOffset-8  9.25GB/s ±12%  9.54GB/s ± 9%    ~     (p=0.072 n=20+18)

name                                      old alloc/op   new alloc/op   delta
Backend/BenchmarkLoadPartialFile-8            888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)

name                                      old allocs/op  new allocs/op  delta
Backend/BenchmarkLoadPartialFile-8            18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
 2020-06-17 13:11:45 +02:00
greatroar
f4cd2a7120 Make backend benchmarks fairer by removing checks 
Checking whether the right data is returned takes up half the time in
some benchmarks. Results for local backend benchmarks on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadFile-8                 4.89ms ± 0%    2.72ms ± 1%   -44.26%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8           936µs ± 6%     439µs ±15%   -53.07%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8     940µs ± 1%     456µs ±10%   -51.50%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                     23.9ms ±14%    24.8ms ±41%      ~     (p=0.690 n=5+5)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadFile-8               3.43GB/s ± 0%  6.16GB/s ± 1%   +79.40%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8        4.48GB/s ± 6%  9.63GB/s ±14%  +114.78%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8  4.46GB/s ± 1%  9.22GB/s ±10%  +106.74%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                    706MB/s ±13%   698MB/s ±31%      ~     (p=0.690 n=5+5)
 2020-06-17 13:11:45 +02:00
Alexander Weiss
1361341c58 don't save duplicate packIDs when using internal/repository/Index.Store 2020-06-14 07:56:24 +02:00
Alexander Weiss
ce4a2f4ca6 save packIDs and duplicates separately 
A side remark to the definition of Index.blob:

Another possibility would have been to use:
blob       map[restic.BlobHandle]*indexEntry

This would have led to the following sizes:
key: 32 + 1 = 33 bytes
value: 8 bytes
indexEntry:  8 + 4 + 4 = 16 bytes
each packID: 32 bytes

To save N index entries, we would therefore have needed:
N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes

More precicely, using a pointer instead of a direct entry is the better memory choice if:
OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1)
Under the assumption of OF=1.5, this means using pointers would have been the better choice
if sizeof(indexEntry) > 24 bytes.
 2020-06-14 07:56:21 +02:00
Alexander Weiss
cf979e2b81 make offset and length uint32 2020-06-14 07:50:19 +02:00
Michael Eischer
d92e2c5769 simplify index code 2020-06-14 07:50:19 +02:00
Alexander Weiss
7419844885 add changelog, benchmark, memory calculation 2020-06-14 07:50:15 +02:00
Alexander Weiss
d3c59d18e5 Fix inconsistency of saving/loading config file 
Fix saving/loading config file: Always set ID to a zero ID.
 2020-06-13 16:30:23 +02:00
MichaelEischer
dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256 
Use Minio's optimized SHA-256
 2020-06-12 23:32:58 +02:00
MichaelEischer
6896c6449b
Merge pull request #2779 from greatroar/archiver-comment 
Fix up comment on archiver.BlobSaver.Save
 2020-06-12 23:04:21 +02:00
MichaelEischer
735a8074d5
Merge pull request #2773 from aawsome/index-uploads+knownblobs 
Fix non-intuitive repo behavior
 2020-06-12 22:41:04 +02:00
Alexander Weiss
70347e95d5 disable index uploads for prune command 
+ modifications of changelog
 2020-06-12 09:24:38 +02:00
greatroar
0fa3091c78 Fix up comment on archiver.BlobSaver.Save 2020-06-11 13:40:30 +02:00
Alexander Weiss
91906911b0 Fix non-intuitive repository behavior 
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
 2020-06-11 13:05:23 +02:00
MichaelEischer
6856d1e422
Merge pull request #2749 from aawsome/fix-fullindex 
Change condition for full index
 2020-06-10 20:40:19 +02:00
Alexander Weiss
8c1261ff02 changed condition for full index 2020-06-07 22:00:49 +02:00
MichaelEischer
84475aa3a8
Merge pull request #2730 from greatroar/mount-build-tags 
Simplify build tags for restic mount
 2020-05-23 20:33:43 +02:00
Martin Michlmayr
5cc1760fdf
Fix typos 2020-05-16 14:05:26 +08:00
greatroar
649cbec6c5 Simplify build tags for restic mount 
This command can only be built on Darwin, FreeBSD and Linux
(and if we upgrade bazil.org/fuse, only FreeBSD and Linux:
https://github.com/bazil/fuse/issues/224).

Listing the few supported operating systems explicitly here makes
porting restic to new platforms easier.
 2020-05-12 11:30:41 +02:00
greatroar
9f7cd69f13 Move Index.FindBlob to tests 2020-04-29 10:57:01 +02:00
greatroar
f97a680887 Fix repository benchmarks 
BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of
their time measuring SHA-256 performance in their checks.
 2020-04-28 07:57:29 +02:00
greatroar
42a3db05b0 Use Minio's optimized SHA-256 
internal/repository benchmarks on an Intel i7-3770k:

name               old speed      new speed       delta
PackerManager-8     209MB/s ± 1%    291MB/s ± 1%  +38.94%  (p=0.008 n=5+5)
SaveAndEncrypt-8    112MB/s ± 1%    135MB/s ± 1%  +20.25%  (p=0.008 n=5+5)
 2020-04-28 07:57:18 +02:00
greatroar
e7d7b85d59 Merge Repository.{LoadBlob,loadBlob} 
Pushing the allocation logic down into the former loadBlob body means
that fewer allocations have to be performed:

name              old time/op    new time/op    delta
LoadTree-8           478µs ± 1%     481µs ± 2%    ~     (p=0.315 n=9+10)
LoadBlob-8          11.6ms ± 1%    11.6ms ± 2%    ~     (p=0.393 n=10+10)
LoadAndDecrypt-8    13.3ms ± 3%    13.3ms ± 3%    ~     (p=0.905 n=10+9)
LoadIndex-8         33.6ms ± 2%    33.2ms ± 1%  -1.15%  (p=0.028 n=10+9)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.1kB ± 0%  -0.23%  (p=0.000 n=10+10)
LoadBlob-8          2.28kB ± 0%    2.18kB ± 0%  -4.21%  (p=0.000 n=10+10)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%    ~     (all equal)
LoadIndex-8         5.22MB ± 0%    5.22MB ± 0%    ~     (p=0.631 n=10+10)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       651 ± 0%  -0.15%  (p=0.000 n=10+10)
LoadBlob-8            24.0 ± 0%      23.0 ± 0%  -4.17%  (p=0.000 n=10+10)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%    ~     (all equal)
LoadIndex-8          30.2k ± 0%     30.2k ± 0%    ~     (p=0.610 n=10+10)

name              old speed      new speed      delta
LoadBlob-8        86.4MB/s ± 1%  85.9MB/s ± 2%    ~     (p=0.393 n=10+10)
LoadAndDecrypt-8  75.4MB/s ± 3%  75.4MB/s ± 3%    ~     (p=0.858 n=10+9)
 2020-04-23 10:04:20 +02:00
greatroar
be5a0ff59f Centralize buffer allocation and size checking in Repository.LoadBlob 
Benchmark results for internal/repository:

name              old time/op    new time/op    delta
LoadTree-8           479µs ± 2%     478µs ± 1%   ~     (p=0.780 n=10+9)
LoadBlob-8          11.6ms ± 2%    11.6ms ± 1%   ~     (p=0.631 n=10+10)
LoadAndDecrypt-8    13.2ms ± 2%    13.3ms ± 3%   ~     (p=0.631 n=10+10)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.2kB ± 0%   ~     (all equal)
LoadBlob-8          2.28kB ± 0%    2.28kB ± 0%   ~     (all equal)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%   ~     (all equal)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       652 ± 0%   ~     (all equal)
LoadBlob-8            24.0 ± 0%      24.0 ± 0%   ~     (all equal)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%   ~     (all equal)

name              old speed      new speed      delta
LoadBlob-8        86.2MB/s ± 2%  86.4MB/s ± 1%   ~     (p=0.594 n=10+10)
LoadAndDecrypt-8  75.7MB/s ± 2%  75.4MB/s ± 3%   ~     (p=0.617 n=10+10)
 2020-04-23 10:04:20 +02:00
MichaelEischer
4f00564574
Merge pull request #2621 from greatroar/fixes 
Some small fixes
 2020-04-18 18:07:08 +02:00
MichaelEischer
f77477129f
Merge pull request #2584 from greatroar/mount-cache-uid-gid 
Cache uid and gid for top directories in internal/fuse
 2020-04-18 17:45:14 +02:00
greatroar
2e31120f89 Remove unused argument to restic.fakeFile 2020-04-18 17:40:13 +02:00
greatroar
8fb2c0d3c1 Typo in crypto test name 2020-04-18 17:39:06 +02:00
greatroar
072cf7b02d Fix debug messages in internal/fuse 2020-04-18 17:39:06 +02:00
greatroar
df66daa5c9 Fix context usage in backend tests 
Found by go vet. This is also the only complaint is has.
 2020-04-18 17:39:06 +02:00
MichaelEischer
16710454f4
Merge pull request #2628 from MichaelEischer/one-element-pack-lists 
cache: Don't sort one element pack lists
 2020-04-18 17:09:06 +02:00
MichaelEischer
7910ff4c0e
Merge pull request #2648 from nairb774/iowritestring 
termstatus: Use io.WriteString to output messages.
 2020-04-18 13:48:42 +02:00
MichaelEischer
c4da9d1e90
Merge pull request #2638 from greatroar/no-close-in-packer 
Don't Close in Packer.Finalize
 2020-04-18 13:07:21 +02:00
MichaelEischer
a1352906e2
Merge pull request #2622 from greatroar/optimize-packer-manager 
Fix PackerManager benchmark and optimize hashing.Writer
 2020-04-18 12:46:34 +02:00
MichaelEischer
b7c0d4d8bf
Merge pull request #2644 from greatroar/signal-notify-buffered 
Make all signal.Notify channels buffered
 2020-04-18 11:31:02 +02:00
Michael Eischer
bdf7ba20cb archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError 
The Save methods of the BlobSaver, FileSaver and TreeSaver return early
on when the archiver is stopped due to an error. For that they select on
both the tomb.Dying() and context.Done() channels, which can lead to a
race condition when the tomb is killed due to an error: The tomb first
closes its Dying channel before canceling all child contexts.
Archiver.SaveDir only aborts its execution once the context was
canceled. When the tomb killing is paused between closing its Dying
channel and canceling the child contexts, this lets the
FileSaver/TreeSaver.Save methods return immediately, however, ScanDir
still reads further files causing the test case to fail.

As a killed tomb always cancels all child contexts and as the Savers
always use a context bound to the tomb, it is sufficient to just use
context.Done() as escape hatch in the Save functions. This fixes the
mismatch between SaveDir and Save.

Adjust the tests to use contexts bound to the tomb for all interactions
with the Savers.
 2020-04-13 18:23:17 +02:00
Brian Atkinson
b8da7b1f4d termstatus: Use io.WriteString to output messages. 
The previous implementation was repeating the implementation that is
found inside of io.WriteString. Simplify by making use of the stdlib's
implementation.
 2020-03-26 14:55:00 -07:00
Peter Schultz
90fc639a67 Allow specifying user and host when adding keys 
The username and hostname for new keys can be specified with the new
--user and --host flags, respectively. The flags are used only by the
`key add` command and are otherwise ignored.

This allows adding keys with for a desired user and host without having
to run restic as that particular user on that particular host, making
automated key management easier.

Co-authored-by: James TD Smith <ahktenzero@mohorovi.cc>
 2020-03-23 13:11:10 +01:00
rawtaz
9efbe98879
Merge pull request #2623 from alrs/internal-restic-err-before-close 
internal/restic: close os.File after checking for error
 2020-03-18 22:23:20 +01:00
rawtaz
7d9300efca
Merge pull request #2637 from greatroar/unused 
Remove Go 1.5 compatibility code
 2020-03-18 22:22:15 +01:00
greatroar
47d4d5bf1b Make all signal.Notify channels buffered 2020-03-12 20:59:39 +01:00
greatroar
74a64c47e4 Move testing logic to test file in internal/pack 2020-03-09 14:32:28 +01:00
greatroar
a23e9c86ba Remove closing logic from Packer.Finalize 
The method only ever receives *hashing.Writers, which don't implement
io.Closer. These come from packerManager.findPacker and have their
actual writers closed in Repository.savePacker. Moving the closing logic
to hashing.Writer results in "file already closed" errors.
 2020-03-09 14:31:45 +01:00
greatroar
4de12bf593 Remove restic.RandReader 
math/rand.Rand has implemented Reader since Go 1.6. The repacking tests
are not deterministic, but they weren't before, either.
 2020-03-09 10:00:28 +01:00
greatroar
8cf3bb8737 Revert "Put host last in SSH command line" 
This reverts commit e1969d1e33.
 2020-03-08 16:45:33 +01:00
Michael Eischer
b46cc6d57e repository: Don't sort one element pack lists 
When loading a blob, restic first looks up pack files containing the
blob. To avoid unnecessary work an already cached pack file is preferred.
However, if there is only a single pack file to choose from (which is
the normal case) sorting the one-element list won't change anything.
Therefore avoid the unnecessary cache check in that case.
 2020-03-07 10:26:06 +01:00
Lars Lehtonen
4a2156d3f0
internal/restic: close os.File after checking for error 2020-03-05 16:22:46 -08:00
greatroar
41fee11f66 Micro-optimization for hashing.Writer/PackerManager 
name             old time/op    new time/op    delta
PackerManager-8     247ms ± 1%     246ms ± 1%  -0.43%  (p=0.001 n=18+18)

name             old speed      new speed      delta
PackerManager-8   213MB/s ± 1%   214MB/s ± 1%  +0.43%  (p=0.001 n=18+18)

name             old alloc/op   new alloc/op   delta
PackerManager-8    92.2kB ± 0%    91.5kB ± 0%  -0.82%  (p=0.000 n=19+20)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.43k ± 0%     1.41k ± 0%  -1.67%  (p=0.000 n=20+20)
 2020-03-05 22:30:04 +01:00
greatroar
b592614061 Improve PackerManager benchmark 
The previous benchmark spent much of its time allocating RNGs and
generating too many random numbers. It now spends 90% of its time
hashing and half of the rest writing to files.

name             old time/op    new time/op    delta
PackerManager-8     319ms ± 1%     247ms ± 1%  -22.48%  (p=0.000 n=20+18)

name             old speed      new speed      delta
PackerManager-8   143MB/s ± 1%   213MB/s ± 1%  +48.63%  (p=0.000 n=10+18)

name             old alloc/op   new alloc/op   delta
PackerManager-8     635kB ± 0%      92kB ± 0%  -85.48%  (p=0.000 n=10+19)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.64k ± 0%     1.43k ± 0%  -12.76%  (p=0.000 n=10+20)
 2020-03-05 22:30:03 +01:00
greatroar
b7c3039eb2 Remove Go 1.5 compatibility code from PackerManager benchmark 
This alone is enough to speed up the benchmark by ~10%.
 2020-03-05 22:29:06 +01:00
Alexander Neumann
52abec967f
Merge pull request #2605 from middelink/fix-2604 
Fix running tests on a SELinux enabled system
 2020-03-01 20:25:28 +01:00