Michael Eischer
185a55026b
rest: workaround for HTTP2 zero-length replies bug
...
The golang http client does not return an error when a HTTP2 reply
includes a non-zero content length but does not return any data at all.
This scenario can occur e.g. when using rclone when a file stored in a
backend seems to be accessible but then fails to download.
2021-07-10 16:59:01 +02:00
Michael Eischer
bd316d3893
restore: Test partial pack downloads in filerestorer
2021-06-29 21:11:30 +02:00
Michael Eischer
e8bbb05328
restore: Correctly handle partial pack download errors
...
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
2021-06-29 20:54:16 +02:00
MichaelEischer
c1eb7ac1a1
Merge pull request #3420 from greatroar/local-errs
...
Modernize error handling in local backend
2021-06-20 14:20:40 +02:00
greatroar
e5f0f67ba0
Modernize error handling in local backend
...
* Stop prepending the operation name: it's already part of os.PathError,
leading to repetitive errors like "Chmod: chmod /foo/bar: operation not
permitted".
* Use errors.Is to check for specific errors.
2021-06-18 11:13:27 +02:00
MichaelEischer
a476752962
Merge pull request #3421 from greatroar/s3-fileinfo
...
Return s3.fileInfos by pointer
2021-06-12 18:55:18 +02:00
MichaelEischer
e8d20ea32c
Merge pull request #3409 from greatroar/lchown-mknod
...
Make restic.{lchown,mknod} regular functions
2021-06-12 18:22:38 +02:00
greatroar
0d4f16b6ba
Return s3.fileInfos by pointer
...
Since the fileInfos are returned in a []interface, they're already
allocated on the heap. Making them pointers explicitly means the
compiler doesn't need to generate fileInfo and *fileInfo versions of the
methods on this type. The binary becomes about 7KiB smaller on
Linux/amd64.
2021-06-07 19:48:43 +02:00
greatroar
0666c4d244
Make restic.{lchown,mknod} regular functions
...
This way, they can be inlined and dead code can be removed on Windows.
Also fixed some comments.
2021-05-27 22:51:40 +02:00
rawtaz
fdbd65485e
Merge pull request #3402 from MichaelEischer/misc-fixes
...
Various small code cleanups
2021-05-24 11:30:31 +02:00
Alexander Neumann
4cabad8c34
Merge pull request #3325 from MichaelEischer/fix-mintty-output
...
Fix windows terminal output for mintty
2021-05-18 09:29:24 +02:00
Michael Eischer
cf92c58460
Properly wrap errors in readerat helper
2021-05-17 21:08:23 +02:00
Michael Eischer
75c990504d
azure/gs: Fix default value in connections help text
2021-05-17 20:56:51 +02:00
Michael Eischer
55bea6e7a6
filter: Fix crash for '**' pattern
2021-05-14 23:50:31 +02:00
MichaelEischer
6d8ceefd67
Merge pull request #3373 from greatroar/simplify-limiter
...
Simplify internal/limiter
2021-05-14 20:27:31 +02:00
MichaelEischer
64b00d28b1
Merge pull request #3345 from greatroar/sftp-enospc
...
Check for ENOSPC and remove broken files in SFTP
2021-05-13 20:09:38 +02:00
greatroar
ae170e2b38
Simplify internal/limiter
2021-04-24 11:54:43 +02:00
Michael Eischer
7cb8ea69ba
Add test to mintty pipe detection
2021-04-11 20:02:09 +02:00
Michael Eischer
80564a9bc9
Properly detect mintty output redirection
...
mintty on windows always uses pipes to connect stdout between processes
and for the terminal output. The previous implementation always assumed
that stdout connected to a pipe means that stdout is displayed on a
mintty terminal. However, this detection breaks when using pipes to
connect processes and for powershell which uses pipes when redirecting
to a file.
Now the pipe filename is queried and matched against the pattern used by
msys / cygwin when connected to the terminal. In all other cases assume
that a pipe is just a regular pipe.
2021-04-11 20:02:09 +02:00
Michael Eischer
5e6af77b7a
Unify interactive terminal detection code
...
Previously the progress bar / status update interval used
stdoutIsTerminal to determine whether it is possible to update the
progress bar or not. However, its implementation differed from the
detection within the backup command which included additional checks to
detect the presence of mintty on Windows. mintty behaves like a terminal
but uses pipes for communication.
This adds stdoutCanUpdateStatus() which calls the same terminal detection
code used by backup. This ensures that all commands consistently switch
between interactive and non-interactive terminal mode.
stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus()
does so. This is required to properly handle the special case of mintty.
2021-04-11 20:02:09 +02:00
MichaelEischer
cc254dfefe
Merge pull request #3362 from greatroar/darwin-preallocate
...
Use FcntlFstore to preallocate on Mac
2021-04-10 22:47:56 +02:00
greatroar
23531be272
Use FcntlFstore to preallocate on Mac
2021-04-10 16:54:07 +02:00
rawtaz
74c0607c92
Merge pull request #3319 from MichaelEischer/skip-prealloc-test
...
restorer: Skip preallocate test if not supported by the filesystem
2021-04-07 18:59:06 +02:00
greatroar
dc88ca79b6
Handle lack of space and remove broken files in SFTP backend
2021-03-27 15:02:14 +01:00
Alexander Neumann
88a23521dd
Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check
...
s3: Fix sanity check
2021-03-11 13:13:46 +01:00
greatroar
187a77fb27
Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist
2021-03-10 21:05:24 +01:00
Michael Eischer
2a9f0f19b6
s3: Fix sanity check
...
The sanity check shouldn't replace the error message if there is already
one.
2021-03-08 20:23:57 +01:00
Michael Eischer
f70aca6f6f
restorer: Skip preallocate test if not supported by the filesystem
2021-03-04 20:33:46 +01:00
Michael Eischer
a293fd9aef
local: Ignore files in intermediate folders
...
For example the data folder of a repository might contain hidden files
which caused list operations to fail.
2021-02-27 13:47:55 +01:00
Alexander Neumann
9985368d46
Merge pull request #3255 from MichaelEischer/restorer-check-error
...
restorer: Check dropped error
2021-02-03 16:10:46 +01:00
Alexander Neumann
2dd592a06c
Merge pull request #3254 from lorenz/allow-http2
...
Allow HTTP/2
2021-02-02 21:08:20 +01:00
Michael Eischer
47faf69230
restorer: Check dropped error
2021-01-31 18:06:28 +01:00
Alexander Weiss
5e22ae10f1
Add error handling for fileRestorer
2021-01-31 14:22:57 +01:00
Alexander Weiss
573221aa40
Add failing test for fileRestorer
2021-01-31 13:40:42 +01:00
Lorenz Brun
5427119205
Allow HTTP/2
2021-01-31 02:44:30 +01:00
Michael Eischer
e0867c9682
backend: try to cleanup test leftovers
2021-01-30 21:23:20 +01:00
Michael Eischer
f740b2fb23
mem: check upload length before storing upload
2021-01-30 21:23:20 +01:00
Alexander Neumann
0e5f2fff71
Merge pull request #3243 from restic/fix-scanner-overlap
...
backup: Fix total size for overlapping targets
2021-01-30 21:17:21 +01:00
Alexander Neumann
04ca69cc78
Address issues reported by golint
2021-01-30 20:45:57 +01:00
Alexander Neumann
f867e65bcd
Fix issues reported by staticcheck
2021-01-30 20:43:53 +01:00
Alexander Neumann
0858fbf6aa
Add more error handling
2021-01-30 20:19:47 +01:00
Alexander Neumann
aef3658a5f
Address review comments
2021-01-30 20:02:37 +01:00
Alexander Neumann
200f09522d
Add more error checks
2021-01-30 20:02:37 +01:00
Alexander Neumann
3c753c071c
errcheck: More error handling
2021-01-30 20:02:37 +01:00
Alexander Neumann
16313bfcc9
errcheck: Add error check for MergeFinalIndexes()
2021-01-30 20:02:37 +01:00
Alexander Neumann
75f53955ee
errcheck: Add error checks
...
Most added checks are straight forward.
2021-01-30 20:02:37 +01:00
Michael Eischer
a53778cd83
rest: handle dropped error in save operation
2021-01-30 19:25:04 +01:00
Michael Eischer
8a486eafed
gs: Don't drop error when finishing upload
...
The error returned when finishing the upload of an object was dropped.
This could cause silent upload failures and thus data loss in certain
cases. When a MD5 hash for the uploaded blob is specified, a wrong
hash/damaged upload would return its error via the Close() whose error
was dropped.
2021-01-30 13:31:32 +01:00
Alexander Neumann
cdd704920d
azure: Pass data length to Azure libray
...
The azureAdapter was used directly without a pointer, but the Len()
method was only defined with a pointer receiver (which means Len() is
not present on a azureAdapter{}, only on a pointer to it).
2021-01-29 21:08:41 +01:00
Alexander Neumann
bbdf18c4a2
Merge pull request #3176 from MichaelEischer/backend-content-length
...
Pass upload size to backends and sanity check it
2021-01-29 20:33:44 +01:00
Michael Eischer
1f583b3d8e
backend: test that incomplete uploads fail
2021-01-29 13:51:53 +01:00
Michael Eischer
c73316a111
backends: add sanity check for the uploaded file size
...
Bugs in the error handling while uploading a file to the backend could
cause incomplete files, e.g. https://github.com/golang/go/issues/42400
which could affect the local backend.
Proactively add sanity checks which will treat an upload as failed if
the reported upload size does not match the actual file size.
2021-01-29 13:51:51 +01:00
Michael Eischer
4526d5d197
swift: explicitly pass upload size to library
...
This allows properly setting the content-length which could help the
server-side to detect incomplete uploads.
2021-01-29 13:50:46 +01:00
Michael Eischer
dca9b6f5db
azure: explicitly pass upload size
...
Previously the fallback from the azure library was to read the whole
blob into memory and use that to determine the upload size.
2021-01-29 13:50:46 +01:00
Alexander Neumann
5c617859ab
backup/scanner: Fix total size for overlapping targets
...
Before, the scanner would could files twice if they were included in the
list of backup targets twice, e.g. `restic backup foo foo/bar` would
could the file `foo/bar` twice.
This commit uses the tree structure from the archiver to run the
scanner, so both parts see the same files.
2021-01-29 11:31:36 +01:00
Alexander Neumann
81211750ba
archiver/tree: Introduce functions Leaf() and NodeNames()
2021-01-29 11:11:28 +01:00
greatroar
6bd8a2faaa
backup: Add --ignore-ctime option and document change detection
2021-01-28 23:42:10 +01:00
MichaelEischer
43cb26010a
Merge pull request #3242 from greatroar/fprintln
...
internal/ui/termstatus: Use Fprintln to get a newline
2021-01-28 20:34:06 +01:00
Alexander Neumann
4d40c70214
Merge pull request #3211 from MichaelEischer/sftp-speedup
...
Speed-up caching/pack download via SFTP
2021-01-28 14:16:55 +01:00
Alexander Neumann
6aa7e9f9c6
Merge pull request #3174 from MichaelEischer/parallelize-lock-loading
...
Parallelize lock file loading
2021-01-28 13:52:12 +01:00
Alexander Neumann
bdfedf1f5b
Merge pull request #3173 from MichaelEischer/unify-index-loading
...
Unify index loading
2021-01-28 13:50:42 +01:00
greatroar
b9cfe6f68a
internal/ui/termstatus: Use Fprintln to get a newline
2021-01-28 13:30:10 +01:00
Michael Eischer
68608a89ad
restic: add comment about StreamTrees shutdown
2021-01-28 11:10:50 +01:00
Michael Eischer
ddb7697d29
restic: Test progress reporting of StreamTrees
2021-01-28 11:10:50 +01:00
Michael Eischer
313ad0e32f
progress/counter: Fix test for final report call
2021-01-28 11:10:50 +01:00
Michael Eischer
e2b0072441
check: add progress bar to the tree structure check
2021-01-28 11:10:50 +01:00
Michael Eischer
505f8a2229
progress/counter: Support updating the progress bar maximum
2021-01-28 11:10:47 +01:00
Michael Eischer
eda8c67616
restic: let FindUsedBlobs handle multiple snapshots at once
2021-01-28 11:08:43 +01:00
Michael Eischer
258ce0c1e5
parallel: report progress for StreamTrees
...
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
2021-01-28 11:08:43 +01:00
Michael Eischer
f2a1b125cb
restic: Actually parallelize FindUsedBlobs
2021-01-28 11:08:43 +01:00
Michael Eischer
6e03f80ca2
check: Split the parallelized tree loader into a reusable component
...
The actual code change is minimal
2021-01-28 11:08:43 +01:00
Michael Eischer
1d7bb01a6b
check: Cleanup tree loading and switch to use errgroup
...
The helper methods are now wired up in the Structure method.
2021-01-28 11:08:43 +01:00
Alexander Neumann
a4689eb3b9
Merge pull request #3199 from MichaelEischer/non-interactive-counter
...
Don't print progress on non-interactive terminals
2021-01-28 10:53:38 +01:00
Alexander Neumann
c5a66e9181
ui: Simlify channel receive
2021-01-28 10:42:02 +01:00
Michael Eischer
678e75e1c2
sftp: enforce use of optimized upload method
...
ReadFrom was already used by Save before, this just ensures that this
won't accidentally change in the future.
2021-01-03 22:23:53 +01:00
Michael Eischer
6b5b29dbee
limiter: add unit tests
2021-01-03 22:23:53 +01:00
Michael Eischer
f35f2c48cd
limiter: support WriteTo in LimitBackend for read rate limiting
2021-01-03 22:23:53 +01:00
Michael Eischer
bcb852a8d0
hashing: support WriteTo in the reader
2021-01-03 22:23:53 +01:00
Michael Eischer
69d5b4c36b
restorer: lower-case variable name
2021-01-03 13:55:59 +01:00
Alexander Weiss
34a33565c8
Fix loadBlob in filerestorer
2021-01-01 08:06:04 +01:00
Alexander Weiss
7409225fa8
Add filerestorer test where only parts of pack are used
2021-01-01 07:24:46 +01:00
Alexander Weiss
07b3f65a6f
filesrestorer: Re-use buffer
2021-01-01 07:24:46 +01:00
Alexander Weiss
3e0acf1395
restore: Don't save (part of) pack in memory
2021-01-01 07:24:46 +01:00
Alexander Neumann
8b84c96d9d
Merge pull request #3204 from MichaelEischer/archiver-tomb-race
...
archiver: fix race condition during worker startup
2020-12-30 20:04:20 +01:00
Michael Eischer
debc4a3a99
archiver: fix race condition during worker startup
...
When the tomb is created with a canceled context, then the workers
started via `t.Go` exist nearly immediately. Once for the first time all
started goroutines have been stopped, it is not allowed to issue further
calls to `t.Go`. This is a problem when the started goroutines exit
immediately, as for example the first goroutine might already have
stopped before starting the second one, which is not allowed as once the
first goroutines has stopped no goroutines were running.
To fix this race condition the startup and main task of the archiver now
also run within a `t.Go` function. This also allows unifying the error
handling as it is no longer necessary to distinguish between errors
returned by the workers or the saveTree processing. The tomb now just
returns the first error encountered, which should also be the most
descriptive one.
2020-12-30 17:31:22 +01:00
MichaelEischer
31e56f1ad5
Merge pull request #3197 from SkYNewZ/fix/3183
...
Fix tag handling for multiple tag lists
2020-12-29 18:44:38 +01:00
Alexander Neumann
b2efa0af39
Merge pull request #3164 from MichaelEischer/improve-context-cancel
...
Improve context cancel handling in archiver and backends
2020-12-29 17:03:42 +01:00
Michael Eischer
cff4955a48
ui: remove dead struct member
2020-12-29 16:32:18 +01:00
Michael Eischer
92da5168e1
ui: force backup progress update on signal
2020-12-29 16:32:18 +01:00
Michael Eischer
34afc93ddc
ui/progress: extract signal handling into own package
2020-12-29 16:32:18 +01:00
Michael Eischer
023eea6463
ui: don't shorten non-interactive progress output
2020-12-29 16:32:18 +01:00
Michael Eischer
684600cf42
ui: update status for the backup command on non-interactive terminals
...
Allow the backup command to print status on non-interactive terminals.
The output is disabled by setting a MinUpdatePause == 0.
2020-12-29 16:03:43 +01:00
Michael Eischer
13ce981794
ui: cleanup backup status shutdown
2020-12-29 16:03:43 +01:00
Michael Eischer
c2ef049f1b
ui/progress: don't print progress on non-interactive terminals
...
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058 .
2020-12-29 16:03:43 +01:00
Quentin Lemaire
334d8ce724
feat(tags): Create Flatten() method
2020-12-29 10:59:46 +01:00
greatroar
3b09ae9074
AIX port
2020-12-29 01:35:01 +01:00
greatroar
18531e3d6f
Portability fixes to internal/restic
...
syscall.Mknod is not available on AIX.
2020-12-29 01:35:01 +01:00
Michael Eischer
d0ca8fb0b8
backend: test that a canceled context prevents RetryBackend operations
2020-12-28 21:06:47 +01:00
Michael Eischer
08b7f2b58d
archiver: test that context canceled error is not dropped
2020-12-28 21:06:47 +01:00
Michael Eischer
e483b63c40
retrybackend: Fail operations when context is already canceled
...
Depending on the used backend, operations started with a canceled
context may fail or not. For example the local backend still works in
large parts when called with a canceled context. Backends transfering
data via http don't work. It is also not possible to retry failed
operations in that state as the RetryBackend will abort with a 'context
canceled' error.
Ensure uniform behavior of all backends by checking for a canceled
context by checking for a canceled context as a first step in the
RetryBackend. This ensures uniform behavior across all backends, as
backends are always wrapped in a RetryBackend.
2020-12-28 21:06:47 +01:00
Michael Eischer
fc60b560ba
archiver: Let saveTree report a canceled context as an error
...
If the context was canceled then saveTree might receive a treeID or not
depending on the timing. This could cause saveTree to incorrectly return
a nil treeID as valid. Fix this always returning an error when the
context was canceled in the meantime.
2020-12-28 21:06:47 +01:00
Michael Eischer
736e964317
archiver: Don't loose error if background context is canceled
...
A canceled background context lets the blob/tree/fileSavers exit
without reporting an error. The error handling previously replaced
a 'context canceled' error received by the main backup method with
the error reported by the savers. However, in case of a canceled
background context that error is nil, causing restic to loose the
error and save a snapshot with a nil tree.
2020-12-28 21:06:47 +01:00
DRON-666
332b1896d1
Some options
fixes
...
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
2020-12-23 23:26:04 +03:00
MichaelEischer
cb6b0f6255
Merge pull request #3181 from tWido/feature-no-retry-permission
...
Don't retry when "Permission denied" occurs in local backend
2020-12-23 20:24:29 +01:00
MichaelEischer
1e73aac610
Merge pull request #3179 from aawsome/check-filesize
...
check: Remove filesize counter
2020-12-23 20:22:02 +01:00
Alexander Weiss
2a1add7538
check: remove file size counter
2020-12-23 02:34:31 +01:00
tWido
7dab113035
Don't retry when "Permission denied" occurs in local backend
2020-12-22 23:41:12 +01:00
Michael Eischer
de99207046
repository: tweak comment for packs method
2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e
Count packs directly in RebuildIndexFiles
2020-12-22 23:01:58 +01:00
Michael Eischer
b9f5d3fe13
repository: Add test for ForAllIndexes
2020-12-22 22:36:18 +01:00
Michael Eischer
a12c5f1d37
repository: move otherwise unused LoadIndex to tests
2020-12-22 22:36:18 +01:00
Michael Eischer
24474a36f4
repository: deduplicate index loading implementation
2020-12-22 22:36:18 +01:00
Michael Eischer
96904f8972
check: extract parallel index loading
2020-12-22 22:36:18 +01:00
Alexander Neumann
6c514adb8a
ui/progress: Use mutex instead of atomic
...
The counter value needs to be aligned to 64 bit in memory for the
atomic functions to work on some platform (such as 32 bit ARM).
The atomic package says in its documentation:
> These functions require great care to be used correctly. Except for
> special, low-level applications, synchronization is better done with
> channels or the facilities of the sync package.
This commit replaces the atomic functions with a simple sync.Mutex, so
we don't have to care about alignment.
2020-12-22 21:03:27 +01:00
DRON-666
83b10dbb12
Deduplicate dumper closing logic
2020-12-19 02:42:46 +03:00
DRON-666
e136dd8696
Deduplicate test code
2020-12-19 02:06:54 +03:00
DRON-666
33adb58817
Minor fixes and linter suggestions
2020-12-19 02:04:17 +03:00
DRON-666
89ab6d557e
dump: Add zip dumper
2020-12-19 01:09:47 +03:00
DRON-666
0256f95994
dump: Split tar and walk logic
2020-12-19 01:09:47 +03:00
Michael Eischer
34b6130a0e
restic: parallelize lock file loading
2020-12-18 20:46:16 +01:00
greatroar
66d904c905
Make invalid handles permanent errors
2020-12-17 12:47:53 +01:00
greatroar
746dbda413
Mark "ssh exited" errors in SFTP as permanent
2020-12-17 12:43:09 +01:00
greatroar
f7784bddb3
Don't retry when "no space left on device" in local backend
...
Also adds relevant documentation to the restic.Backend interface.
2020-12-17 12:43:09 +01:00
Alexander Neumann
e96677cafb
Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables
...
swift: Add support for id based keystone v3 auth parameters
2020-12-12 16:27:38 +01:00
Michael Eischer
1d69341e88
swift: Add support for id based keystone v3 auth parameters
...
This adds support for the following environment variables, which were
previously missing:
OS_USER_ID User ID for keystone v3 authentication
OS_USER_DOMAIN_ID User domain ID for keystone v3 authentication
OS_PROJECT_DOMAIN_ID Project domain ID for keystone v3 authentication
OS_TRUST_ID Trust ID for keystone v3 authentication
2020-12-11 19:22:34 +01:00
Alexander Neumann
36c5d39c2c
Fix issues reported by semgrep
2020-12-11 09:41:59 +01:00
Alexander Neumann
7facc8ccc1
Merge pull request #2505 from aawsome/fix-repo-configfile
...
Fix repo configfile
2020-12-07 07:52:37 +01:00
Alexander Neumann
ba31c6fdaa
Merge pull request #3150 from MichaelEischer/fix-windows-redir-output
...
termstatus: Fix canUpdateStatus detection for redirected output on windows
2020-12-06 21:17:27 +01:00
Alexander Neumann
b58799d83a
Merge pull request #3152 from MichaelEischer/fix-backup-background-hang
...
backup: Fix shutdown hang when running in the background on linux
2020-12-06 21:16:00 +01:00
Alexander Neumann
0d5b764f90
Merge pull request #3130 from aawsome/snapshots-parallel
...
Make loading snapshots parallel
2020-12-06 21:08:18 +01:00
Michael Eischer
b48f579530
termstatus: Fix canUpdateStatus detection for redirected output
...
The canUpdateStatus check was simplified in #2608 , but it accidentally flipped
the condition. The correct check is as follows: If the output is a pipe then
restic probably runs in mintty/cygwin. In that case it's possible to
update the output status. In all other cases it isn't.
This commit inverts to condition again to offer the previous and correct
behavior.
2020-12-06 19:02:42 +01:00
Michael Eischer
401ef92c5f
backup: Fix shutdown hang when running in the background
...
On shutdown the backup commands waits for the terminal output goroutine
to stop. However while running in the background the goroutine ignored
the canceled context.
2020-12-06 18:53:41 +01:00
Alexander Weiss
e329623771
Remove LoadAllSnapshots
2020-12-06 05:22:27 +01:00
Alexander Weiss
26f85779be
Parallelize ForAllSnapshots
2020-12-06 05:09:58 +01:00
Alexander Weiss
5b9ee56335
Add ForAllSnapshots
2020-12-06 05:04:21 +01:00
Michael Eischer
d05c88a5d6
fuse: Properly set uid/gid for directories
...
In #2584 this was changed to use the uid/gid of the root node. This
would be okay for the top-level directory of a snapshot, however, this
change also applied to normal directories within a snapshot. This
change reverts the problematic part and adds a test that directory
attributes are represented correctly.
2020-11-30 23:41:49 +01:00
MichaelEischer
f2959127b6
Merge pull request #3065 from greatroar/local-subdirs
...
Don't recurse in local backend's List if not required
2020-11-29 19:03:59 +01:00
Alexander Weiss
aa7a5f19c2
Use BlobHandle in index methods
2020-11-22 20:41:12 +01:00
Alexander Weiss
e3013271a6
Harmonize naming
2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691
Make BlobHandle substruct of Blob
2020-11-22 20:41:10 +01:00
Alexander Weiss
a851c53cbe
Use PackSize in checker
2020-11-21 22:13:54 +01:00
Alexander Weiss
ce5d630681
Add MasterIndex.PackSize()
2020-11-21 22:13:54 +01:00
Alexander Weiss
c3ddde9e7d
Return hdrSize in ListPack
2020-11-21 22:13:54 +01:00
greatroar
8e213e82fc
backend/local: replace fs.Walk with custom walker
...
This code is more strict in what it expects to find in the backend:
depending on the layout, either a directory full of files or a directory
full of such directories.
2020-11-19 16:46:42 +01:00
Alexander Neumann
75eff92b56
Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init
...
do not require gs bucket permissions to init repository
2020-11-18 16:53:45 +01:00
eleith
a24e986b2b
do not require gs bucket permissions to init repository
...
a gs service account may only have object permissions on an existing
bucket but no bucket create/get permissions.
these service accounts currently are blocked from initialization a
restic repository because restic can not determine if the bucket exists.
this PR updates the logic to assume the bucket exists when the bucket
attribute request results in a permissions denied error.
this way, restic can still initialize a repository if the service
account does have object permissions
fixes: https://github.com/restic/restic/issues/3100
2020-11-18 06:14:11 -08:00
rawtaz
1b6e8c888f
Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param
...
Remove `--drive-use-trash=false` from rclone param
2020-11-17 11:21:20 +01:00
Michael Eischer
1f43cac12d
check: Only track data blobs when unused blobs should be reported
...
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00
Michael Eischer
6da66c15d8
check: Simplify referenced blob tracking
...
The result is identical as long as the context in not canceled. However,
in that case the result is incomplete anyways.
2020-11-15 18:42:55 +01:00
Michael Eischer
3500f9490c
check: Simplify blob status tracking
...
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
2020-11-15 18:42:42 +01:00
Michael Eischer
b8c7543a55
check: Merge 'size could not be found' and 'not found in index' errors
...
By construction these two errors always show up in pairs: 'size could
not be found' is printed when the blob is not found in the repository
index. That blob is also part of the `blobs` array. Later on, check
iterates over that array and checks whether the blob is marked as
existing. Which cannot be the case as that mark is generated by
iterating over the repository index.
The merged warning no longer reports the blob index within a file. That
information could also be derived by printing the affected tree using
`cat` and searching for the blob.
2020-11-15 18:41:50 +01:00
Alexander Neumann
3c0c0c132b
Merge pull request #3006 from aawsome/new-rebuild-index
...
Reimplement rebuild-index and remove /internal/index
2020-11-15 17:48:43 +01:00
Alexander Neumann
9968220652
Merge pull request #2850 from greatroar/packer-malloc
...
Decrease allocation rate in internal/pack
2020-11-15 17:19:49 +01:00
greatroar
ab2b7d7f9a
Decrease allocation rate in internal/pack
...
internal/repository benchmark results:
name old time/op new time/op delta
PackerManager-8 179ms ± 1% 181ms ± 1% +0.78% (p=0.009 n=10+10)
name old speed new speed delta
PackerManager-8 294MB/s ± 1% 292MB/s ± 1% -0.77% (p=0.009 n=10+10)
name old alloc/op new alloc/op delta
PackerManager-8 91.3kB ± 0% 72.2kB ± 0% -20.92% (p=0.000 n=9+7)
name old allocs/op new allocs/op delta
PackerManager-8 1.38k ± 0% 0.76k ± 0% -45.20% (p=0.000 n=10+7)
2020-11-15 16:51:47 +01:00
greatroar
9a8a2cae4c
Move pack testing logic to test file
2020-11-15 16:45:49 +01:00
Alexander Weiss
9607cad267
Remove internal/index
2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a
Reimplement rebuild-index
2020-11-15 07:05:09 +01:00
Alexander Weiss
187c8fb259
Parallelize MasterIndex.Save()
2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5
Add extraObsolete to MasterIndex.Save
2020-11-15 07:05:09 +01:00
Alexander Weiss
5898cb341f
Use CreateIndexFromPacks() in test
2020-11-15 07:05:05 +01:00
Alexander Weiss
43732bb885
Add CreateIndexFromPacks()
2020-11-15 07:04:51 +01:00
greatroar
c45f8ee075
Replace restic.statT interface by concrete types
...
name old time/op new time/op delta
NodeFillUser-8 1.81µs ± 9% 1.50µs ± 5% -17.07% (p=0.000 n=19+20)
NodeFromFileInfo-8 1.76µs ± 4% 1.49µs ± 6% -15.63% (p=0.000 n=20+19)
name old alloc/op new alloc/op delta
NodeFillUser-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
NodeFromFileInfo-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
NodeFillUser-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
NodeFromFileInfo-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
MichaelEischer
3601a9b6cd
Merge pull request #2690 from SkYNewZ/master
...
Fix #2688 : Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
MichaelEischer
333c5a19d4
Merge pull request #3082 from aawsome/check-sizes
...
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
Quentin Lemaire
ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list
2020-11-14 15:48:56 +00:00
Alexander Weiss
17bb77b1f9
check: Also check blob length and offset
2020-11-14 00:42:49 +01:00
Alexander Weiss
80dcfca191
check: Check sizes computed from index and pack header
2020-11-14 00:42:49 +01:00
tofran
94a154c7ca
Remove --drive-use-trash=false
from rclone param
...
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
2020-11-13 22:58:48 +00:00
Alexander Neumann
04d1983800
Merge pull request #3090 from fgma/vss-fix-386
...
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-13 21:30:34 +01:00
Alexander Neumann
8ad9f88993
helpers: Improve error message
2020-11-12 20:38:31 +01:00
fgma
f9223cd827
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-12 19:31:00 +01:00
rawtaz
dfb9326b1b
Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag
...
Extended option to select V1 API for ListObjects on S3 backend
2020-11-11 20:52:44 +01:00
Alexander Neumann
4a0b7328ec
s3: Remove dots for config description
2020-11-11 20:20:35 +01:00
greatroar
63e32c44c0
Improve error reporting from restic diff
...
Instead of a stacktrace, restic diff 111 222 now reports:
Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
Nick Douma
829959390a
Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1
2020-11-11 11:54:38 +01:00
Nick Douma
ccd55d529d
Add s3.list-objects-v1 extended option and default to false
2020-11-11 11:54:36 +01:00
Nick Douma
4ddcc17135
Add support for boolean extended options
2020-11-11 11:54:27 +01:00
MichaelEischer
407843c5f9
Merge pull request #3034 from hoyho/gh_master
...
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
MichaelEischer
46d31ab86d
Merge pull request #3058 from greatroar/counter
...
Replace restic.Progress with new progress.Counter (fixes two race conditions)
2020-11-09 22:19:09 +01:00
Alexander Neumann
c986823d3f
Merge pull request #3048 from aawsome/check-pack-index
...
check: check index for packs that are read
2020-11-09 20:12:32 +01:00
Alexander Weiss
239931578c
check: check index for packs that are read
2020-11-09 17:28:14 +01:00
hoyho
9df52327cc
bugfix: omit ENOTDATA for extended attributes
...
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
greatroar
21b787a4d1
Stop Counters where they're constructed and started
2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2
Replace restic.Progress with new progress.Counter
...
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
Alexander Neumann
3bf43d7951
Merge pull request #2475 from restic/use-github-actions
...
Use GitHub Actions for CI
2020-11-08 17:30:12 +01:00
Alexander Neumann
5cf42884c8
Run tests on GitHub Actions
2020-11-08 16:04:57 +01:00
Alexander Weiss
826cfa0533
fix context in archiver tests
2020-11-08 08:24:24 +01:00
Alexander Weiss
fef408a8bd
Return context error in mem backend
2020-11-08 00:05:53 +01:00
greatroar
a2d4209322
Don't recurse in local backend's List if not required
...
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.
Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9 ).
2020-11-07 08:54:13 +01:00
Alexander Weiss
fd33030556
Use in-memory index to rebuild index in prune
2020-11-06 20:23:30 +01:00
Alexander Weiss
38cc4393f6
Add Masterindex.Save(); Add Index.Packs()
2020-11-06 20:23:30 +01:00
Alexander Weiss
aaf1c44362
Fix #3062
2020-11-05 17:05:42 +01:00
Ivan Andreev
ab2790d9de
Fix http2 stream reset between restic and rest backends #3014
2020-11-05 15:57:40 +03:00
Nick Craig-Wood
86b5d8ffaa
s3: add bucket-lookup parameter to select path or dns style bucket lookup
...
This is to enable restic working with Alibaba cloud
Fixes #2528
2020-11-05 12:20:10 +01:00
Alexander Neumann
ae5302c7a8
Add comment that keepBlobs is modified
2020-11-05 10:33:38 +01:00
Alexander Neumann
866a52ad4e
Remove unneeded seek
...
The file returned from DownloadAndHash() is already seeked to the start
of the file.
2020-11-05 10:31:49 +01:00
Alexander Neumann
a4507610a0
Fix typo
2020-11-05 10:31:49 +01:00
Alexander Neumann
7def2d8ea7
Use a non-constant seed
2020-11-05 10:31:49 +01:00
Alexander Neumann
ee0112ab3b
Clarify message about expected error
2020-11-05 10:31:49 +01:00
Michael Eischer
b373f164fe
prune: Parallelize repack command
2020-11-05 10:31:49 +01:00
Alexander Neumann
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command
...
Reimplementation of prune
2020-11-05 10:12:19 +01:00
fgma
916b2d303b
vss: fix potential crash (not reachable in restic) ( #3045 )
...
HasSufficientPrivilegesForVSS() now returns an error
2020-11-04 22:14:18 +01:00
Alexander Neumann
a06f5c28c0
Merge pull request #3055 from greatroar/cleanup
...
Don't separately allocate sync.{Mutex,Once} if not necessary
2020-11-04 11:11:21 +01:00
greatroar
c2f3eee5af
Don't separately allocate sync.{Mutex,Once} if not necessary
...
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
2020-11-04 09:55:43 +01:00
Alexander Weiss
b2f5381737
Make realistic forget --prune --dryrun
2020-11-03 16:42:21 +01:00
Alexander Weiss
ce7d613749
Add Blob.Handle()
2020-11-03 16:42:21 +01:00
Alexander Weiss
581d90cf91
Make some pack parameters public
2020-11-03 16:42:21 +01:00
Alexander Neumann
9a88fb253b
Merge pull request #3051 from greatroar/sanitize-env
...
Sanitize environment before starting backend processes (rclone, ssh)
2020-11-02 21:18:57 +01:00
Alexander Neumann
f14436953a
Merge pull request #3050 from greatroar/widechars
...
Fix string truncation in ui/termstatus
2020-11-02 16:58:51 +01:00
greatroar
11fbaaae9a
Sanitize environment before starting backend processes (rclone, ssh)
...
The restic security model includes full trust of the local machine, so
this should not fix any actual security problems, but it's better to be
safe than sorry.
Fixes #2192 .
2020-11-02 16:41:23 +01:00
greatroar
3ed84ff0c6
Fix string truncation in ui/termstatus
...
Fixes #3046 .
2020-11-02 12:50:49 +01:00
Alexander Neumann
8e965ed4eb
Merge pull request #3039 from MichaelEischer/tar-proper-dirs
...
Properly dump directories to tar
2020-11-02 12:30:26 +01:00
Alexander Neumann
5f0fa2129e
Improve readability
...
It's time to use a switch statement.
2020-11-02 11:24:26 +01:00
Alexander Neumann
6509c207f4
Merge pull request #2997 from MichaelEischer/faster-excludes
...
Speedup exclude/include checking
2020-11-02 11:07:21 +01:00
Alexander Neumann
445b845267
Merge pull request #2978 from MichaelEischer/warn-tree-error
...
Warn if backup failed to read tree blob
2020-11-02 10:14:12 +01:00
Alexander Neumann
3ff37215df
Merge pull request #2935 from MichaelEischer/upgrade-minio
...
Upgrade minio SDK to version 7
2020-11-02 09:09:10 +01:00
Sergio Rubio
e708628cfd
Remove unused function
...
Not currently used, and it'd need to be added to the MasterIndex interface first.
2020-10-28 13:24:49 +01:00
MichaelEischer
bb4b3481a6
Merge pull request #3031 from greatroar/debug-no-seek
...
Allow debug logging to pipes and terminals
2020-10-25 17:59:06 +01:00
MichaelEischer
ad3a52e6f0
Merge pull request #3026 from greatroar/refactor-ui
...
internal/ui refactoring
2020-10-25 17:51:39 +01:00
Michael Eischer
8d7d6ad2d5
dump: include username in tar
2020-10-24 22:42:01 +02:00
Michael Eischer
fe09e6f865
dump: test proper permissions and directory name
2020-10-24 22:42:01 +02:00
Michael Eischer
1e3c9a2c11
dump: Fix file permission to tar mapping
...
The file permissions included a go specific directory bit which
accidentially forced the usage of the GNU header format. This leads
to problems with 7zip on Windows or when extended attributes are
used.
2020-10-24 22:42:01 +02:00
Michael Eischer
e21dcb0eea
dump: Additional ACL tests
2020-10-24 22:41:01 +02:00
MichaelEischer
31b8d7a639
Merge pull request #2274 from fgma/master
...
Support for Volume Shadow Copy Service (VSS) on windows
2020-10-24 15:43:34 +02:00
fgma
5695f9ebd2
vss: Implement VSS support for Windows
...
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.
Snapshotting volumes also covers mountpoints but skips UNC paths.
2020-10-24 11:35:57 +02:00
greatroar
63be3704d9
Allow debug logging to pipes and terminals
2020-10-19 22:30:30 +02:00
greatroar
35419de232
Simplify ui.StdioWrapper.Write
...
Instead of looping to find line breaks, make it look for the last one.
2020-10-17 20:30:46 +02:00
greatroar
863a590a81
Refactor termstatus.Term.{Print,Error} methods
2020-10-17 20:30:46 +02:00
greatroar
7c0b6a82db
Remove unused public method ui.linesWriter.Flush
2020-10-17 20:30:24 +02:00
MichaelEischer
96a912b65a
Merge pull request #3025 from aawsome/fix-index-ids
...
Fix setting of ID in DecodeIndex
2020-10-17 09:54:45 +02:00
Alexander Weiss
b44ecde8b0
Fix setting of ID in DecodeIndex
2020-10-17 09:12:58 +02:00
MichaelEischer
39fe1e96fe
Merge pull request #3018 from greatroar/background-check
...
Fix IsProcessBackground on Linux with stdin redirection
2020-10-16 22:50:19 +02:00
MichaelEischer
4ba237bb93
Merge pull request #3019 from greatroar/refactor-decodeindex
...
Refactor index decoding
2020-10-15 23:22:33 +02:00
greatroar
b27375f5ce
defer close(ch) outside repository.RunWorkers
2020-10-14 15:50:16 +02:00
greatroar
720e0ee0c7
if cond { return true }; return false => return cond
2020-10-13 20:56:43 +02:00
greatroar
27db3ec262
Refactor index decoding
...
Decoding old-format indices no longer requires loading and decrypting
twice.
2020-10-13 20:47:50 +02:00
greatroar
f80b07b2c8
Fix IsProcessBackground on Linux with stdin redirection
...
The previous implementation assumed that stdin was a terminal.
It now checks the terminal's fd.
2020-10-13 13:12:02 +02:00
MichaelEischer
41a45ae908
Merge pull request #3016 from greatroar/uid-gid-lookup
...
Simplify/optimize cached UID/GID lookups
2020-10-12 23:24:55 +02:00
Alexander Neumann
56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
...
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
greatroar
f0cd16e5ea
Cache uint32-typed ids in lookup{Username,Group}
...
NodeFillUser-8 1.92µs ± 5% 1.75µs ± 3% -8.89% (p=0.000 n=10+10)
NodeFromFileInfo-8 1.89µs ± 7% 1.76µs ± 4% -6.69% (p=0.001 n=10+10)
name old alloc/op new alloc/op delta
NodeFillUser-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10)
NodeFromFileInfo-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
NodeFillUser-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)
NodeFromFileInfo-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)
2020-10-12 15:08:15 +02:00
greatroar
a03fe4562e
Remove unused error return in lookup{Username,Group}
2020-10-12 14:48:23 +02:00
Alexander Neumann
c193cea119
Pass Context to NewRestorer()
2020-10-10 15:24:26 +02:00
Alexander Neumann
f0d49ca600
Merge pull request #2933 from MichaelEischer/less-context-todo
...
Replace most usages of context.TODO()
2020-10-10 15:03:44 +02:00
kitone
6099f81692
Fix #1212 restore code produces inconsistent timestamps/permissions.
...
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
2020-10-10 13:46:44 +02:00
kitone
9d7f616190
Improve restorer debug log information
2020-10-10 13:46:43 +02:00
kitone
295ddb9e57
Add test case for inconsistent timestamps and permissions restoration
...
Reproduce from https://github.com/restic/restic/issues/1212
2020-10-10 13:46:43 +02:00
Michael Eischer
e638b46a13
Embed context into ReaderAt
...
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
2020-10-09 22:39:07 +02:00
Michael Eischer
d6cfe857b7
pass proper context into MasterIndex.RebuildIndex
2020-10-09 22:39:07 +02:00
Michael Eischer
37a5e2d681
rest: use global context on repository creation
2020-10-09 22:39:06 +02:00
Michael Eischer
4b0fcaed45
unlock: use proper context for locks cleanup
...
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
2020-10-09 22:37:56 +02:00