bd316d3893
restore: Test partial pack downloads in filerestorer
2021-06-29 21:11:30 +02:00
e8bbb05328
restore: Correctly handle partial pack download errors
...
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
2021-06-29 20:54:16 +02:00
c1eb7ac1a1
Merge pull request #3420 from greatroar/local-errs
...
Modernize error handling in local backend
2021-06-20 14:20:40 +02:00
e5f0f67ba0
Modernize error handling in local backend
...
* Stop prepending the operation name: it's already part of os.PathError,
leading to repetitive errors like "Chmod: chmod /foo/bar: operation not
permitted".
* Use errors.Is to check for specific errors.
2021-06-18 11:13:27 +02:00
a476752962
Merge pull request #3421 from greatroar/s3-fileinfo
...
Return s3.fileInfos by pointer
2021-06-12 18:55:18 +02:00
e8d20ea32c
Merge pull request #3409 from greatroar/lchown-mknod
...
Make restic.{lchown,mknod} regular functions
2021-06-12 18:22:38 +02:00
0d4f16b6ba
Return s3.fileInfos by pointer
...
Since the fileInfos are returned in a []interface, they're already
allocated on the heap. Making them pointers explicitly means the
compiler doesn't need to generate fileInfo and *fileInfo versions of the
methods on this type. The binary becomes about 7KiB smaller on
Linux/amd64.
2021-06-07 19:48:43 +02:00
0666c4d244
Make restic.{lchown,mknod} regular functions
...
This way, they can be inlined and dead code can be removed on Windows.
Also fixed some comments.
2021-05-27 22:51:40 +02:00
fdbd65485e
Merge pull request #3402 from MichaelEischer/misc-fixes
...
Various small code cleanups
2021-05-24 11:30:31 +02:00
4cabad8c34
Merge pull request #3325 from MichaelEischer/fix-mintty-output
...
Fix windows terminal output for mintty
2021-05-18 09:29:24 +02:00
cf92c58460
Properly wrap errors in readerat helper
2021-05-17 21:08:23 +02:00
75c990504d
azure/gs: Fix default value in connections help text
2021-05-17 20:56:51 +02:00
55bea6e7a6
filter: Fix crash for '**' pattern
2021-05-14 23:50:31 +02:00
6d8ceefd67
Merge pull request #3373 from greatroar/simplify-limiter
...
Simplify internal/limiter
2021-05-14 20:27:31 +02:00
64b00d28b1
Merge pull request #3345 from greatroar/sftp-enospc
...
Check for ENOSPC and remove broken files in SFTP
2021-05-13 20:09:38 +02:00
ae170e2b38
Simplify internal/limiter
2021-04-24 11:54:43 +02:00
7cb8ea69ba
Add test to mintty pipe detection
2021-04-11 20:02:09 +02:00
80564a9bc9
Properly detect mintty output redirection
...
mintty on windows always uses pipes to connect stdout between processes
and for the terminal output. The previous implementation always assumed
that stdout connected to a pipe means that stdout is displayed on a
mintty terminal. However, this detection breaks when using pipes to
connect processes and for powershell which uses pipes when redirecting
to a file.
Now the pipe filename is queried and matched against the pattern used by
msys / cygwin when connected to the terminal. In all other cases assume
that a pipe is just a regular pipe.
2021-04-11 20:02:09 +02:00
5e6af77b7a
Unify interactive terminal detection code
...
Previously the progress bar / status update interval used
stdoutIsTerminal to determine whether it is possible to update the
progress bar or not. However, its implementation differed from the
detection within the backup command which included additional checks to
detect the presence of mintty on Windows. mintty behaves like a terminal
but uses pipes for communication.
This adds stdoutCanUpdateStatus() which calls the same terminal detection
code used by backup. This ensures that all commands consistently switch
between interactive and non-interactive terminal mode.
stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus()
does so. This is required to properly handle the special case of mintty.
2021-04-11 20:02:09 +02:00
cc254dfefe
Merge pull request #3362 from greatroar/darwin-preallocate
...
Use FcntlFstore to preallocate on Mac
2021-04-10 22:47:56 +02:00
23531be272
Use FcntlFstore to preallocate on Mac
2021-04-10 16:54:07 +02:00
74c0607c92
Merge pull request #3319 from MichaelEischer/skip-prealloc-test
...
restorer: Skip preallocate test if not supported by the filesystem
2021-04-07 18:59:06 +02:00
dc88ca79b6
Handle lack of space and remove broken files in SFTP backend
2021-03-27 15:02:14 +01:00
88a23521dd
Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check
...
s3: Fix sanity check
2021-03-11 13:13:46 +01:00
187a77fb27
Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist
2021-03-10 21:05:24 +01:00
2a9f0f19b6
s3: Fix sanity check
...
The sanity check shouldn't replace the error message if there is already
one.
2021-03-08 20:23:57 +01:00
f70aca6f6f
restorer: Skip preallocate test if not supported by the filesystem
2021-03-04 20:33:46 +01:00
a293fd9aef
local: Ignore files in intermediate folders
...
For example the data folder of a repository might contain hidden files
which caused list operations to fail.
2021-02-27 13:47:55 +01:00
9985368d46
Merge pull request #3255 from MichaelEischer/restorer-check-error
...
restorer: Check dropped error
2021-02-03 16:10:46 +01:00
2dd592a06c
Merge pull request #3254 from lorenz/allow-http2
...
Allow HTTP/2
2021-02-02 21:08:20 +01:00
47faf69230
restorer: Check dropped error
2021-01-31 18:06:28 +01:00
5e22ae10f1
Add error handling for fileRestorer
2021-01-31 14:22:57 +01:00
573221aa40
Add failing test for fileRestorer
2021-01-31 13:40:42 +01:00
5427119205
Allow HTTP/2
2021-01-31 02:44:30 +01:00
e0867c9682
backend: try to cleanup test leftovers
2021-01-30 21:23:20 +01:00
f740b2fb23
mem: check upload length before storing upload
2021-01-30 21:23:20 +01:00
0e5f2fff71
Merge pull request #3243 from restic/fix-scanner-overlap
...
backup: Fix total size for overlapping targets
2021-01-30 21:17:21 +01:00
04ca69cc78
Address issues reported by golint
2021-01-30 20:45:57 +01:00
f867e65bcd
Fix issues reported by staticcheck
2021-01-30 20:43:53 +01:00
0858fbf6aa
Add more error handling
2021-01-30 20:19:47 +01:00
aef3658a5f
Address review comments
2021-01-30 20:02:37 +01:00
200f09522d
Add more error checks
2021-01-30 20:02:37 +01:00
3c753c071c
errcheck: More error handling
2021-01-30 20:02:37 +01:00
16313bfcc9
errcheck: Add error check for MergeFinalIndexes()
2021-01-30 20:02:37 +01:00
75f53955ee
errcheck: Add error checks
...
Most added checks are straight forward.
2021-01-30 20:02:37 +01:00
a53778cd83
rest: handle dropped error in save operation
2021-01-30 19:25:04 +01:00
8a486eafed
gs: Don't drop error when finishing upload
...
The error returned when finishing the upload of an object was dropped.
This could cause silent upload failures and thus data loss in certain
cases. When a MD5 hash for the uploaded blob is specified, a wrong
hash/damaged upload would return its error via the Close() whose error
was dropped.
2021-01-30 13:31:32 +01:00
cdd704920d
azure: Pass data length to Azure libray
...
The azureAdapter was used directly without a pointer, but the Len()
method was only defined with a pointer receiver (which means Len() is
not present on a azureAdapter{}, only on a pointer to it).
2021-01-29 21:08:41 +01:00
bbdf18c4a2
Merge pull request #3176 from MichaelEischer/backend-content-length
...
Pass upload size to backends and sanity check it
2021-01-29 20:33:44 +01:00
1f583b3d8e
backend: test that incomplete uploads fail
2021-01-29 13:51:53 +01:00
c73316a111
backends: add sanity check for the uploaded file size
...
Bugs in the error handling while uploading a file to the backend could
cause incomplete files, e.g. https://github.com/golang/go/issues/42400
which could affect the local backend.
Proactively add sanity checks which will treat an upload as failed if
the reported upload size does not match the actual file size.
2021-01-29 13:51:51 +01:00
4526d5d197
swift: explicitly pass upload size to library
...
This allows properly setting the content-length which could help the
server-side to detect incomplete uploads.
2021-01-29 13:50:46 +01:00
dca9b6f5db
azure: explicitly pass upload size
...
Previously the fallback from the azure library was to read the whole
blob into memory and use that to determine the upload size.
2021-01-29 13:50:46 +01:00
5c617859ab
backup/scanner: Fix total size for overlapping targets
...
Before, the scanner would could files twice if they were included in the
list of backup targets twice, e.g. `restic backup foo foo/bar` would
could the file `foo/bar` twice.
This commit uses the tree structure from the archiver to run the
scanner, so both parts see the same files.
2021-01-29 11:31:36 +01:00
81211750ba
archiver/tree: Introduce functions Leaf() and NodeNames()
2021-01-29 11:11:28 +01:00
6bd8a2faaa
backup: Add --ignore-ctime option and document change detection
2021-01-28 23:42:10 +01:00
43cb26010a
Merge pull request #3242 from greatroar/fprintln
...
internal/ui/termstatus: Use Fprintln to get a newline
2021-01-28 20:34:06 +01:00
4d40c70214
Merge pull request #3211 from MichaelEischer/sftp-speedup
...
Speed-up caching/pack download via SFTP
2021-01-28 14:16:55 +01:00
6aa7e9f9c6
Merge pull request #3174 from MichaelEischer/parallelize-lock-loading
...
Parallelize lock file loading
2021-01-28 13:52:12 +01:00
bdfedf1f5b
Merge pull request #3173 from MichaelEischer/unify-index-loading
...
Unify index loading
2021-01-28 13:50:42 +01:00
b9cfe6f68a
internal/ui/termstatus: Use Fprintln to get a newline
2021-01-28 13:30:10 +01:00
68608a89ad
restic: add comment about StreamTrees shutdown
2021-01-28 11:10:50 +01:00
ddb7697d29
restic: Test progress reporting of StreamTrees
2021-01-28 11:10:50 +01:00
313ad0e32f
progress/counter: Fix test for final report call
2021-01-28 11:10:50 +01:00
e2b0072441
check: add progress bar to the tree structure check
2021-01-28 11:10:50 +01:00
505f8a2229
progress/counter: Support updating the progress bar maximum
2021-01-28 11:10:47 +01:00
eda8c67616
restic: let FindUsedBlobs handle multiple snapshots at once
2021-01-28 11:08:43 +01:00
258ce0c1e5
parallel: report progress for StreamTrees
...
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
2021-01-28 11:08:43 +01:00
f2a1b125cb
restic: Actually parallelize FindUsedBlobs
2021-01-28 11:08:43 +01:00
6e03f80ca2
check: Split the parallelized tree loader into a reusable component
...
The actual code change is minimal
2021-01-28 11:08:43 +01:00
1d7bb01a6b
check: Cleanup tree loading and switch to use errgroup
...
The helper methods are now wired up in the Structure method.
2021-01-28 11:08:43 +01:00
a4689eb3b9
Merge pull request #3199 from MichaelEischer/non-interactive-counter
...
Don't print progress on non-interactive terminals
2021-01-28 10:53:38 +01:00
c5a66e9181
ui: Simlify channel receive
2021-01-28 10:42:02 +01:00
678e75e1c2
sftp: enforce use of optimized upload method
...
ReadFrom was already used by Save before, this just ensures that this
won't accidentally change in the future.
2021-01-03 22:23:53 +01:00
6b5b29dbee
limiter: add unit tests
2021-01-03 22:23:53 +01:00
f35f2c48cd
limiter: support WriteTo in LimitBackend for read rate limiting
2021-01-03 22:23:53 +01:00
bcb852a8d0
hashing: support WriteTo in the reader
2021-01-03 22:23:53 +01:00
69d5b4c36b
restorer: lower-case variable name
2021-01-03 13:55:59 +01:00
34a33565c8
Fix loadBlob in filerestorer
2021-01-01 08:06:04 +01:00
7409225fa8
Add filerestorer test where only parts of pack are used
2021-01-01 07:24:46 +01:00
07b3f65a6f
filesrestorer: Re-use buffer
2021-01-01 07:24:46 +01:00
3e0acf1395
restore: Don't save (part of) pack in memory
2021-01-01 07:24:46 +01:00
8b84c96d9d
Merge pull request #3204 from MichaelEischer/archiver-tomb-race
...
archiver: fix race condition during worker startup
2020-12-30 20:04:20 +01:00
debc4a3a99
archiver: fix race condition during worker startup
...
When the tomb is created with a canceled context, then the workers
started via `t.Go` exist nearly immediately. Once for the first time all
started goroutines have been stopped, it is not allowed to issue further
calls to `t.Go`. This is a problem when the started goroutines exit
immediately, as for example the first goroutine might already have
stopped before starting the second one, which is not allowed as once the
first goroutines has stopped no goroutines were running.
To fix this race condition the startup and main task of the archiver now
also run within a `t.Go` function. This also allows unifying the error
handling as it is no longer necessary to distinguish between errors
returned by the workers or the saveTree processing. The tomb now just
returns the first error encountered, which should also be the most
descriptive one.
2020-12-30 17:31:22 +01:00
31e56f1ad5
Merge pull request #3197 from SkYNewZ/fix/3183
...
Fix tag handling for multiple tag lists
2020-12-29 18:44:38 +01:00
b2efa0af39
Merge pull request #3164 from MichaelEischer/improve-context-cancel
...
Improve context cancel handling in archiver and backends
2020-12-29 17:03:42 +01:00
cff4955a48
ui: remove dead struct member
2020-12-29 16:32:18 +01:00
92da5168e1
ui: force backup progress update on signal
2020-12-29 16:32:18 +01:00
34afc93ddc
ui/progress: extract signal handling into own package
2020-12-29 16:32:18 +01:00
023eea6463
ui: don't shorten non-interactive progress output
2020-12-29 16:32:18 +01:00
684600cf42
ui: update status for the backup command on non-interactive terminals
...
Allow the backup command to print status on non-interactive terminals.
The output is disabled by setting a MinUpdatePause == 0.
2020-12-29 16:03:43 +01:00
13ce981794
ui: cleanup backup status shutdown
2020-12-29 16:03:43 +01:00
c2ef049f1b
ui/progress: don't print progress on non-interactive terminals
...
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058 .
2020-12-29 16:03:43 +01:00
334d8ce724
feat(tags): Create Flatten() method
2020-12-29 10:59:46 +01:00
3b09ae9074
AIX port
2020-12-29 01:35:01 +01:00
18531e3d6f
Portability fixes to internal/restic
...
syscall.Mknod is not available on AIX.
2020-12-29 01:35:01 +01:00
d0ca8fb0b8
backend: test that a canceled context prevents RetryBackend operations
2020-12-28 21:06:47 +01:00
08b7f2b58d
archiver: test that context canceled error is not dropped
2020-12-28 21:06:47 +01:00
e483b63c40
retrybackend: Fail operations when context is already canceled
...
Depending on the used backend, operations started with a canceled
context may fail or not. For example the local backend still works in
large parts when called with a canceled context. Backends transfering
data via http don't work. It is also not possible to retry failed
operations in that state as the RetryBackend will abort with a 'context
canceled' error.
Ensure uniform behavior of all backends by checking for a canceled
context by checking for a canceled context as a first step in the
RetryBackend. This ensures uniform behavior across all backends, as
backends are always wrapped in a RetryBackend.
2020-12-28 21:06:47 +01:00
fc60b560ba
archiver: Let saveTree report a canceled context as an error
...
If the context was canceled then saveTree might receive a treeID or not
depending on the timing. This could cause saveTree to incorrectly return
a nil treeID as valid. Fix this always returning an error when the
context was canceled in the meantime.
2020-12-28 21:06:47 +01:00
736e964317
archiver: Don't loose error if background context is canceled
...
A canceled background context lets the blob/tree/fileSavers exit
without reporting an error. The error handling previously replaced
a 'context canceled' error received by the main backup method with
the error reported by the savers. However, in case of a canceled
background context that error is nil, causing restic to loose the
error and save a snapshot with a nil tree.
2020-12-28 21:06:47 +01:00
332b1896d1
Some `options` fixes
...
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
2020-12-23 23:26:04 +03:00
cb6b0f6255
Merge pull request #3181 from tWido/feature-no-retry-permission
...
Don't retry when "Permission denied" occurs in local backend
2020-12-23 20:24:29 +01:00
1e73aac610
Merge pull request #3179 from aawsome/check-filesize
...
check: Remove filesize counter
2020-12-23 20:22:02 +01:00
2a1add7538
check: remove file size counter
2020-12-23 02:34:31 +01:00
7dab113035
Don't retry when "Permission denied" occurs in local backend
2020-12-22 23:41:12 +01:00
de99207046
repository: tweak comment for packs method
2020-12-22 23:01:58 +01:00
68b74e359e
Count packs directly in RebuildIndexFiles
2020-12-22 23:01:58 +01:00
b9f5d3fe13
repository: Add test for ForAllIndexes
2020-12-22 22:36:18 +01:00
a12c5f1d37
repository: move otherwise unused LoadIndex to tests
2020-12-22 22:36:18 +01:00
24474a36f4
repository: deduplicate index loading implementation
2020-12-22 22:36:18 +01:00
96904f8972
check: extract parallel index loading
2020-12-22 22:36:18 +01:00
6c514adb8a
ui/progress: Use mutex instead of atomic
...
The counter value needs to be aligned to 64 bit in memory for the
atomic functions to work on some platform (such as 32 bit ARM).
The atomic package says in its documentation:
> These functions require great care to be used correctly. Except for
> special, low-level applications, synchronization is better done with
> channels or the facilities of the sync package.
This commit replaces the atomic functions with a simple sync.Mutex, so
we don't have to care about alignment.
2020-12-22 21:03:27 +01:00
83b10dbb12
Deduplicate dumper closing logic
2020-12-19 02:42:46 +03:00
e136dd8696
Deduplicate test code
2020-12-19 02:06:54 +03:00
33adb58817
Minor fixes and linter suggestions
2020-12-19 02:04:17 +03:00
89ab6d557e
dump: Add zip dumper
2020-12-19 01:09:47 +03:00
0256f95994
dump: Split tar and walk logic
2020-12-19 01:09:47 +03:00
34b6130a0e
restic: parallelize lock file loading
2020-12-18 20:46:16 +01:00
66d904c905
Make invalid handles permanent errors
2020-12-17 12:47:53 +01:00
746dbda413
Mark "ssh exited" errors in SFTP as permanent
2020-12-17 12:43:09 +01:00
f7784bddb3
Don't retry when "no space left on device" in local backend
...
Also adds relevant documentation to the restic.Backend interface.
2020-12-17 12:43:09 +01:00
e96677cafb
Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables
...
swift: Add support for id based keystone v3 auth parameters
2020-12-12 16:27:38 +01:00
1d69341e88
swift: Add support for id based keystone v3 auth parameters
...
This adds support for the following environment variables, which were
previously missing:
OS_USER_ID User ID for keystone v3 authentication
OS_USER_DOMAIN_ID User domain ID for keystone v3 authentication
OS_PROJECT_DOMAIN_ID Project domain ID for keystone v3 authentication
OS_TRUST_ID Trust ID for keystone v3 authentication
2020-12-11 19:22:34 +01:00
36c5d39c2c
Fix issues reported by semgrep
2020-12-11 09:41:59 +01:00
7facc8ccc1
Merge pull request #2505 from aawsome/fix-repo-configfile
...
Fix repo configfile
2020-12-07 07:52:37 +01:00
ba31c6fdaa
Merge pull request #3150 from MichaelEischer/fix-windows-redir-output
...
termstatus: Fix canUpdateStatus detection for redirected output on windows
2020-12-06 21:17:27 +01:00
b58799d83a
Merge pull request #3152 from MichaelEischer/fix-backup-background-hang
...
backup: Fix shutdown hang when running in the background on linux
2020-12-06 21:16:00 +01:00
0d5b764f90
Merge pull request #3130 from aawsome/snapshots-parallel
...
Make loading snapshots parallel
2020-12-06 21:08:18 +01:00
b48f579530
termstatus: Fix canUpdateStatus detection for redirected output
...
The canUpdateStatus check was simplified in #2608 , but it accidentally flipped
the condition. The correct check is as follows: If the output is a pipe then
restic probably runs in mintty/cygwin. In that case it's possible to
update the output status. In all other cases it isn't.
This commit inverts to condition again to offer the previous and correct
behavior.
2020-12-06 19:02:42 +01:00
401ef92c5f
backup: Fix shutdown hang when running in the background
...
On shutdown the backup commands waits for the terminal output goroutine
to stop. However while running in the background the goroutine ignored
the canceled context.
2020-12-06 18:53:41 +01:00
e329623771
Remove LoadAllSnapshots
2020-12-06 05:22:27 +01:00
26f85779be
Parallelize ForAllSnapshots
2020-12-06 05:09:58 +01:00
5b9ee56335
Add ForAllSnapshots
2020-12-06 05:04:21 +01:00
d05c88a5d6
fuse: Properly set uid/gid for directories
...
In #2584 this was changed to use the uid/gid of the root node. This
would be okay for the top-level directory of a snapshot, however, this
change also applied to normal directories within a snapshot. This
change reverts the problematic part and adds a test that directory
attributes are represented correctly.
2020-11-30 23:41:49 +01:00
f2959127b6
Merge pull request #3065 from greatroar/local-subdirs
...
Don't recurse in local backend's List if not required
2020-11-29 19:03:59 +01:00
aa7a5f19c2
Use BlobHandle in index methods
2020-11-22 20:41:12 +01:00
e3013271a6
Harmonize naming
2020-11-22 20:41:12 +01:00
92bd448691
Make BlobHandle substruct of Blob
2020-11-22 20:41:10 +01:00
a851c53cbe
Use PackSize in checker
2020-11-21 22:13:54 +01:00
ce5d630681
Add MasterIndex.PackSize()
2020-11-21 22:13:54 +01:00
c3ddde9e7d
Return hdrSize in ListPack
2020-11-21 22:13:54 +01:00
8e213e82fc
backend/local: replace fs.Walk with custom walker
...
This code is more strict in what it expects to find in the backend:
depending on the layout, either a directory full of files or a directory
full of such directories.
2020-11-19 16:46:42 +01:00
75eff92b56
Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init
...
do not require gs bucket permissions to init repository
2020-11-18 16:53:45 +01:00
a24e986b2b
do not require gs bucket permissions to init repository
...
a gs service account may only have object permissions on an existing
bucket but no bucket create/get permissions.
these service accounts currently are blocked from initialization a
restic repository because restic can not determine if the bucket exists.
this PR updates the logic to assume the bucket exists when the bucket
attribute request results in a permissions denied error.
this way, restic can still initialize a repository if the service
account does have object permissions
fixes: https://github.com/restic/restic/issues/3100
2020-11-18 06:14:11 -08:00
1b6e8c888f
Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param
...
Remove `--drive-use-trash=false` from rclone param
2020-11-17 11:21:20 +01:00
1f43cac12d
check: Only track data blobs when unused blobs should be reported
...
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00
6da66c15d8
check: Simplify referenced blob tracking
...
The result is identical as long as the context in not canceled. However,
in that case the result is incomplete anyways.
2020-11-15 18:42:55 +01:00
3500f9490c
check: Simplify blob status tracking
...
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
2020-11-15 18:42:42 +01:00
b8c7543a55
check: Merge 'size could not be found' and 'not found in index' errors
...
By construction these two errors always show up in pairs: 'size could
not be found' is printed when the blob is not found in the repository
index. That blob is also part of the `blobs` array. Later on, check
iterates over that array and checks whether the blob is marked as
existing. Which cannot be the case as that mark is generated by
iterating over the repository index.
The merged warning no longer reports the blob index within a file. That
information could also be derived by printing the affected tree using
`cat` and searching for the blob.
2020-11-15 18:41:50 +01:00
3c0c0c132b
Merge pull request #3006 from aawsome/new-rebuild-index
...
Reimplement rebuild-index and remove /internal/index
2020-11-15 17:48:43 +01:00
9968220652
Merge pull request #2850 from greatroar/packer-malloc
...
Decrease allocation rate in internal/pack
2020-11-15 17:19:49 +01:00
ab2b7d7f9a
Decrease allocation rate in internal/pack
...
internal/repository benchmark results:
name old time/op new time/op delta
PackerManager-8 179ms ± 1% 181ms ± 1% +0.78% (p=0.009 n=10+10)
name old speed new speed delta
PackerManager-8 294MB/s ± 1% 292MB/s ± 1% -0.77% (p=0.009 n=10+10)
name old alloc/op new alloc/op delta
PackerManager-8 91.3kB ± 0% 72.2kB ± 0% -20.92% (p=0.000 n=9+7)
name old allocs/op new allocs/op delta
PackerManager-8 1.38k ± 0% 0.76k ± 0% -45.20% (p=0.000 n=10+7)
2020-11-15 16:51:47 +01:00
9a8a2cae4c
Move pack testing logic to test file
2020-11-15 16:45:49 +01:00
9607cad267
Remove internal/index
2020-11-15 07:05:09 +01:00
30b6a0878a
Reimplement rebuild-index
2020-11-15 07:05:09 +01:00
187c8fb259
Parallelize MasterIndex.Save()
2020-11-15 07:05:09 +01:00
1ec628ddf5
Add extraObsolete to MasterIndex.Save
2020-11-15 07:05:09 +01:00
5898cb341f
Use CreateIndexFromPacks() in test
2020-11-15 07:05:05 +01:00
43732bb885
Add CreateIndexFromPacks()
2020-11-15 07:04:51 +01:00
c45f8ee075
Replace restic.statT interface by concrete types
...
name old time/op new time/op delta
NodeFillUser-8 1.81µs ± 9% 1.50µs ± 5% -17.07% (p=0.000 n=19+20)
NodeFromFileInfo-8 1.76µs ± 4% 1.49µs ± 6% -15.63% (p=0.000 n=20+19)
name old alloc/op new alloc/op delta
NodeFillUser-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
NodeFromFileInfo-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
NodeFillUser-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
NodeFromFileInfo-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
3601a9b6cd
Merge pull request #2690 from SkYNewZ/master
...
Fix #2688 : Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
333c5a19d4
Merge pull request #3082 from aawsome/check-sizes
...
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list
2020-11-14 15:48:56 +00:00
17bb77b1f9
check: Also check blob length and offset
2020-11-14 00:42:49 +01:00
80dcfca191
check: Check sizes computed from index and pack header
2020-11-14 00:42:49 +01:00
94a154c7ca
Remove `--drive-use-trash=false` from rclone param
...
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
2020-11-13 22:58:48 +00:00
04d1983800
Merge pull request #3090 from fgma/vss-fix-386
...
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-13 21:30:34 +01:00
8ad9f88993
helpers: Improve error message
2020-11-12 20:38:31 +01:00
f9223cd827
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-12 19:31:00 +01:00
dfb9326b1b
Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag
...
Extended option to select V1 API for ListObjects on S3 backend
2020-11-11 20:52:44 +01:00
4a0b7328ec
s3: Remove dots for config description
2020-11-11 20:20:35 +01:00
63e32c44c0
Improve error reporting from restic diff
...
Instead of a stacktrace, restic diff 111 222 now reports:
Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
829959390a
Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1
2020-11-11 11:54:38 +01:00
ccd55d529d
Add s3.list-objects-v1 extended option and default to false
2020-11-11 11:54:36 +01:00
4ddcc17135
Add support for boolean extended options
2020-11-11 11:54:27 +01:00
407843c5f9
Merge pull request #3034 from hoyho/gh_master
...
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
46d31ab86d
Merge pull request #3058 from greatroar/counter
...
Replace restic.Progress with new progress.Counter (fixes two race conditions)
2020-11-09 22:19:09 +01:00
c986823d3f
Merge pull request #3048 from aawsome/check-pack-index
...
check: check index for packs that are read
2020-11-09 20:12:32 +01:00
239931578c
check: check index for packs that are read
2020-11-09 17:28:14 +01:00
9df52327cc
bugfix: omit ENOTDATA for extended attributes
...
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
21b787a4d1
Stop Counters where they're constructed and started
2020-11-09 13:03:31 +01:00
ddca699cd2
Replace restic.Progress with new progress.Counter
...
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
3bf43d7951
Merge pull request #2475 from restic/use-github-actions
...
Use GitHub Actions for CI
2020-11-08 17:30:12 +01:00
5cf42884c8
Run tests on GitHub Actions
2020-11-08 16:04:57 +01:00
826cfa0533
fix context in archiver tests
2020-11-08 08:24:24 +01:00
fef408a8bd
Return context error in mem backend
2020-11-08 00:05:53 +01:00
a2d4209322
Don't recurse in local backend's List if not required
...
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.
Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9 ).
2020-11-07 08:54:13 +01:00
fd33030556
Use in-memory index to rebuild index in prune
2020-11-06 20:23:30 +01:00
38cc4393f6
Add Masterindex.Save(); Add Index.Packs()
2020-11-06 20:23:30 +01:00
aaf1c44362
Fix #3062
2020-11-05 17:05:42 +01:00
ab2790d9de
Fix http2 stream reset between restic and rest backends #3014
2020-11-05 15:57:40 +03:00
86b5d8ffaa
s3: add bucket-lookup parameter to select path or dns style bucket lookup
...
This is to enable restic working with Alibaba cloud
Fixes #2528
2020-11-05 12:20:10 +01:00
ae5302c7a8
Add comment that keepBlobs is modified
2020-11-05 10:33:38 +01:00
866a52ad4e
Remove unneeded seek
...
The file returned from DownloadAndHash() is already seeked to the start
of the file.
2020-11-05 10:31:49 +01:00
a4507610a0
Fix typo
2020-11-05 10:31:49 +01:00
7def2d8ea7
Use a non-constant seed
2020-11-05 10:31:49 +01:00
ee0112ab3b
Clarify message about expected error
2020-11-05 10:31:49 +01:00
b373f164fe
prune: Parallelize repack command
2020-11-05 10:31:49 +01:00
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command
...
Reimplementation of prune
2020-11-05 10:12:19 +01:00
916b2d303b
vss: fix potential crash (not reachable in restic) ( #3045 )
...
HasSufficientPrivilegesForVSS() now returns an error
2020-11-04 22:14:18 +01:00
a06f5c28c0
Merge pull request #3055 from greatroar/cleanup
...
Don't separately allocate sync.{Mutex,Once} if not necessary
2020-11-04 11:11:21 +01:00
c2f3eee5af
Don't separately allocate sync.{Mutex,Once} if not necessary
...
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
2020-11-04 09:55:43 +01:00
b2f5381737
Make realistic forget --prune --dryrun
2020-11-03 16:42:21 +01:00
ce7d613749
Add Blob.Handle()
2020-11-03 16:42:21 +01:00
581d90cf91
Make some pack parameters public
2020-11-03 16:42:21 +01:00
9a88fb253b
Merge pull request #3051 from greatroar/sanitize-env
...
Sanitize environment before starting backend processes (rclone, ssh)
2020-11-02 21:18:57 +01:00
f14436953a
Merge pull request #3050 from greatroar/widechars
...
Fix string truncation in ui/termstatus
2020-11-02 16:58:51 +01:00
11fbaaae9a
Sanitize environment before starting backend processes (rclone, ssh)
...
The restic security model includes full trust of the local machine, so
this should not fix any actual security problems, but it's better to be
safe than sorry.
Fixes #2192 .
2020-11-02 16:41:23 +01:00
3ed84ff0c6
Fix string truncation in ui/termstatus
...
Fixes #3046 .
2020-11-02 12:50:49 +01:00
8e965ed4eb
Merge pull request #3039 from MichaelEischer/tar-proper-dirs
...
Properly dump directories to tar
2020-11-02 12:30:26 +01:00
5f0fa2129e
Improve readability
...
It's time to use a switch statement.
2020-11-02 11:24:26 +01:00
6509c207f4
Merge pull request #2997 from MichaelEischer/faster-excludes
...
Speedup exclude/include checking
2020-11-02 11:07:21 +01:00
445b845267
Merge pull request #2978 from MichaelEischer/warn-tree-error
...
Warn if backup failed to read tree blob
2020-11-02 10:14:12 +01:00
3ff37215df
Merge pull request #2935 from MichaelEischer/upgrade-minio
...
Upgrade minio SDK to version 7
2020-11-02 09:09:10 +01:00
e708628cfd
Remove unused function
...
Not currently used, and it'd need to be added to the MasterIndex interface first.
2020-10-28 13:24:49 +01:00
bb4b3481a6
Merge pull request #3031 from greatroar/debug-no-seek
...
Allow debug logging to pipes and terminals
2020-10-25 17:59:06 +01:00
ad3a52e6f0
Merge pull request #3026 from greatroar/refactor-ui
...
internal/ui refactoring
2020-10-25 17:51:39 +01:00
8d7d6ad2d5
dump: include username in tar
2020-10-24 22:42:01 +02:00
fe09e6f865
dump: test proper permissions and directory name
2020-10-24 22:42:01 +02:00
1e3c9a2c11
dump: Fix file permission to tar mapping
...
The file permissions included a go specific directory bit which
accidentially forced the usage of the GNU header format. This leads
to problems with 7zip on Windows or when extended attributes are
used.
2020-10-24 22:42:01 +02:00
e21dcb0eea
dump: Additional ACL tests
2020-10-24 22:41:01 +02:00
31b8d7a639
Merge pull request #2274 from fgma/master
...
Support for Volume Shadow Copy Service (VSS) on windows
2020-10-24 15:43:34 +02:00
5695f9ebd2
vss: Implement VSS support for Windows
...
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.
Snapshotting volumes also covers mountpoints but skips UNC paths.
2020-10-24 11:35:57 +02:00
63be3704d9
Allow debug logging to pipes and terminals
2020-10-19 22:30:30 +02:00
35419de232
Simplify ui.StdioWrapper.Write
...
Instead of looping to find line breaks, make it look for the last one.
2020-10-17 20:30:46 +02:00
863a590a81
Refactor termstatus.Term.{Print,Error} methods
2020-10-17 20:30:46 +02:00
7c0b6a82db
Remove unused public method ui.linesWriter.Flush
2020-10-17 20:30:24 +02:00
96a912b65a
Merge pull request #3025 from aawsome/fix-index-ids
...
Fix setting of ID in DecodeIndex
2020-10-17 09:54:45 +02:00
b44ecde8b0
Fix setting of ID in DecodeIndex
2020-10-17 09:12:58 +02:00
39fe1e96fe
Merge pull request #3018 from greatroar/background-check
...
Fix IsProcessBackground on Linux with stdin redirection
2020-10-16 22:50:19 +02:00
4ba237bb93
Merge pull request #3019 from greatroar/refactor-decodeindex
...
Refactor index decoding
2020-10-15 23:22:33 +02:00
b27375f5ce
defer close(ch) outside repository.RunWorkers
2020-10-14 15:50:16 +02:00
720e0ee0c7
if cond { return true }; return false => return cond
2020-10-13 20:56:43 +02:00
27db3ec262
Refactor index decoding
...
Decoding old-format indices no longer requires loading and decrypting
twice.
2020-10-13 20:47:50 +02:00
f80b07b2c8
Fix IsProcessBackground on Linux with stdin redirection
...
The previous implementation assumed that stdin was a terminal.
It now checks the terminal's fd.
2020-10-13 13:12:02 +02:00
41a45ae908
Merge pull request #3016 from greatroar/uid-gid-lookup
...
Simplify/optimize cached UID/GID lookups
2020-10-12 23:24:55 +02:00
56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
...
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
f0cd16e5ea
Cache uint32-typed ids in lookup{Username,Group}
...
NodeFillUser-8 1.92µs ± 5% 1.75µs ± 3% -8.89% (p=0.000 n=10+10)
NodeFromFileInfo-8 1.89µs ± 7% 1.76µs ± 4% -6.69% (p=0.001 n=10+10)
name old alloc/op new alloc/op delta
NodeFillUser-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10)
NodeFromFileInfo-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
NodeFillUser-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)
NodeFromFileInfo-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)
2020-10-12 15:08:15 +02:00
a03fe4562e
Remove unused error return in lookup{Username,Group}
2020-10-12 14:48:23 +02:00
c193cea119
Pass Context to NewRestorer()
2020-10-10 15:24:26 +02:00
f0d49ca600
Merge pull request #2933 from MichaelEischer/less-context-todo
...
Replace most usages of context.TODO()
2020-10-10 15:03:44 +02:00
6099f81692
Fix #1212 restore code produces inconsistent timestamps/permissions.
...
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
2020-10-10 13:46:44 +02:00
9d7f616190
Improve restorer debug log information
2020-10-10 13:46:43 +02:00
295ddb9e57
Add test case for inconsistent timestamps and permissions restoration
...
Reproduce from https://github.com/restic/restic/issues/1212
2020-10-10 13:46:43 +02:00
e638b46a13
Embed context into ReaderAt
...
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
2020-10-09 22:39:07 +02:00
d6cfe857b7
pass proper context into MasterIndex.RebuildIndex
2020-10-09 22:39:07 +02:00
37a5e2d681
rest: use global context on repository creation
2020-10-09 22:39:06 +02:00
4b0fcaed45
unlock: use proper context for locks cleanup
...
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
2020-10-09 22:37:56 +02:00
645a6efaf2
restorer: remove redundant type specification
2020-10-09 22:37:56 +02:00
Michael Eischer
greatroar
rawtaz
Alexander Neumann
Alexander Weiss
Lorenz Brun
Alexander Neumann
Quentin Lemaire
DRON-666
tWido
eleith
tofran
fgma
Nick Douma
hoyho
Ivan Andreev
Nick Craig-Wood
fgma
Sergio Rubio
kitone