Michael Eischer
a60ee9b764
retry: limit retries based on elapsed time not count
...
Depending on how long an operation takes to fail, the total retry
duration can currently vary between 1.5 and 15 minutes. In particular
for temporarily interrupted network connections, the former timeout is
too short. Thus always use a limit of 15 minutes.
2024-05-24 20:24:02 +02:00
Michael Eischer
a3633cad9e
retry: explicitly log failed requests
...
This simplifies finding the request in the log output that cause an
operation to fail.
2024-05-24 20:24:02 +02:00
greatroar
66d03c797e
ui, termstatus: Move WrapStdio
...
Saves some imports. ui still needs to import ui/termstatus from
message.go.
2024-05-21 11:25:16 +02:00
greatroar
0b56214473
ui: Simplify stdio wrapper
...
The StdioWrapper type is really just a pair of io.WriteClosers, so
remove it in favor of a function that returns two of those. Test
coverage increases because the removed code was not tested.
2024-05-21 11:23:32 +02:00
greatroar
7f439a9c34
ui: Inline lineWriter into StdioWrapper
2024-05-21 11:00:09 +02:00
Michael Eischer
c56ecec9aa
azure: deduplicate cli and default credentials case
2024-05-18 22:15:54 +02:00
Maik Riechert
355f520936
Azure: add option to force use of CLI credential
2024-05-18 22:15:54 +02:00
Michael Eischer
223aa22cb0
replace some uses of restic.Repository with finegrained interfaces
2024-05-18 21:42:51 +02:00
Michael Eischer
291c9677de
restic/repository: remove Backend() method
2024-05-18 21:42:51 +02:00
Michael Eischer
673496b091
repository: clean cache between CheckPack retries
...
The cache cleanup pattern is also used in ListPack etc.
2024-05-18 21:42:51 +02:00
Michael Eischer
3d2410ed50
Replace some repo.RemoveUnpacked usages
...
These will eventually be blocked as they do not delete Snapshots.
2024-05-18 21:42:51 +02:00
Michael Eischer
d2c26e33f3
repository: remove further usages of repo.Backend()
2024-05-18 21:42:51 +02:00
Michael Eischer
8a425c2f0a
remove usages of repo.Backend() from tests
2024-05-18 21:42:51 +02:00
Michael Eischer
aa4647f773
repository: unexport PackBlobIterator
2024-05-18 21:42:51 +02:00
Michael Eischer
94e863885c
check: move verification of individual pack file to repository
2024-05-18 21:42:50 +02:00
Michael Eischer
e40943a75d
restic: remove backend usage from lock test
2024-05-18 21:38:31 +02:00
Michael Eischer
67e2ba0d40
repository: Lock requires *repository.Repository
...
This allows the Lock function to access the backend, even once the
Backend method is removed from the interface.
2024-05-18 21:38:31 +02:00
Michael Eischer
d8b184b3d3
repository: convert test helper to return *repository.Repository
2024-05-18 21:38:31 +02:00
Michael Eischer
a1ca5e15c4
migrations: add temporary hack for s3_layout
...
The migration will be removed after the next restic release anyways.
Thus, there's no need for a clean implementation.
2024-05-18 21:38:31 +02:00
Michael Eischer
34d90aecf9
migrations: move logic of upgrade_repo_v2 to repository package
...
The migration modifies repository internals and thus should live within
the repository package.
2024-05-18 21:38:31 +02:00
Michael Eischer
ab9077bc13
replace usages of backend.Remove() with repository.RemoveUnpacked()
...
RemoveUnpacked will eventually block removal of all filetypes other than
snapshots. However, getting there requires a major refactor to provide
some components with privileged access.
2024-05-18 21:38:31 +02:00
Michael Eischer
0c1ba6d95d
backend: remove unused Location method
2024-05-18 21:38:31 +02:00
Michael Eischer
74d90653e0
check: use ReadFull to load pack header in checkPack
...
This ensures that the pack header is actually read completely.
Previously, for a truncated file it was possible to only read a part of
the header, as backend.Load(...) is not guaranteed to return as many
bytes as requested by the length parameter.
2024-05-18 21:28:54 +02:00
Michael Eischer
8f8d872a68
fix compatibility with go 1.19
2024-05-18 21:28:54 +02:00
Michael Eischer
ff0744b3af
check: test checkPack retries
2024-05-18 21:28:54 +02:00
Michael Eischer
987c3b250c
repository: test retries of ListPack
2024-05-18 21:28:54 +02:00
Michael Eischer
bf16096771
repository: test LoadBlob retries
2024-05-18 21:28:54 +02:00
Michael Eischer
4f45668b7c
repository: rework and extend LoadRaw tests
2024-05-18 21:28:54 +02:00
Michael Eischer
ac805d6838
cache: cleanup debug logs
2024-05-18 21:28:54 +02:00
Michael Eischer
5214af88e2
cache: test forget behavior
2024-05-18 21:28:54 +02:00
Michael Eischer
3ff063e913
check: verify pack a second time if broken
2024-05-18 21:28:54 +02:00
Michael Eischer
385cee09dc
repository: fix caching of tree packs in LoadBlobsFromPack
2024-05-18 21:28:54 +02:00
Michael Eischer
e734746f75
cache: forget cached file at most once
...
This is inspired by the circuit breaker pattern used for distributed
systems. If too many requests fails, then it is better to immediately
fail new requests for a limited time to give the backend time to
recover.
By only forgetting a file in the cache at most once, we can ensure that
a broken file is only retrieved once again from the backend. If the file
stored there is broken, previously it would be cached and deleted
continuously. Now, it is retrieved only once again, all later requests
just use the cached copy and either succeed or fail immediately.
2024-05-18 21:28:54 +02:00
Michael Eischer
97a307df1a
cache: Always use cached file if it exists
...
A file is always cached whole. Thus, any out of bounds access will also
fail when directed at the backend. To handle case in which the cached
file is broken, then caller must call Cache.Forget(h) for the file in
question.
2024-05-18 21:28:54 +02:00
Michael Eischer
433a6aad29
repository: remove redundant blob loading fallback from RepairPacks
...
LoadBlobsFromPack already implements the same fallback behavior.
2024-05-18 21:28:54 +02:00
Michael Eischer
e401af07b2
check: fix error message formatting
2024-05-18 21:28:54 +02:00
Michael Eischer
7017adb7e9
repository: retry failed ListPack once
2024-05-18 21:28:54 +02:00
Michael Eischer
e33ce7f408
repository: retry failed LoadBlob once
2024-05-18 21:28:54 +02:00
Michael Eischer
2ace242f36
repository: make reloading broken files explicit
2024-05-18 21:28:54 +02:00
Michael Eischer
e9390352a7
cache: code cleanups
2024-05-18 21:26:00 +02:00
Michael Eischer
503c8140b1
repository: unify blob decoding code
2024-05-18 21:26:00 +02:00
Michael Eischer
6563f1d2ca
repository: remove redundant debug log
2024-05-18 21:26:00 +02:00
Michael Eischer
021fb49559
repository: Implement repository.LoadUnpacked using LoadRaw
...
Both functions were using a similar implementation.
2024-05-18 21:26:00 +02:00
Michael Eischer
779c8d3527
debug/repair packs/upgrade repo v2: use repository.LoadRaw
...
This replaces calling the low-level backend.Load() method.
2024-05-18 21:26:00 +02:00
Michael Eischer
1d6d3656b0
repository: move backend.LoadAll to repository.LoadRaw
...
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.
The removed cache backend test will be added again later on.
2024-05-18 21:26:00 +02:00
Michael Eischer
47232bf8b0
backend: move LimitReadCloser to util package
...
The helper is only intended for usage by backend implementations.
2024-05-18 21:26:00 +02:00
Michael Eischer
53d15bcd1b
retry: add circuit breaker to load method
...
If a file exhausts its retry attempts, then it is likely not accessible
the next time. Thus, immediately fail all load calls for that file to
avoid useless retries.
2024-05-18 19:59:26 +02:00
Michael Eischer
394c8ca3ed
rest/rclone/s3/sftp/swift: move short file detection behind feature gate
...
These backends tend to use a large variety of server implementations.
Some of those implementations might prove problematic with the new
checks.
2024-05-18 19:59:26 +02:00
Michael Eischer
6328b7e1f5
replace "too small" with "too short" in error messages
2024-05-18 19:59:26 +02:00
Michael Eischer
aeb7eb245c
retry: do not retry permanent errors
...
This is currently gated behind a feature flag as some unexpected
interactions might show up in the wild.
2024-05-18 19:59:26 +02:00
Michael Eischer
bf8cc59889
Use generic backend-error-redesign feature flag instead of http-timeouts
...
An individual flag for each change of the backend error handling would
be too finegrained. Thus, add a generic flag.
2024-05-18 19:54:52 +02:00
Michael Eischer
4740528a0b
backend: add tests for IsPermanentError
2024-05-18 19:54:52 +02:00
Michael Eischer
6a85df7297
backend: add IsPermanentError() method to interface
2024-05-18 19:54:52 +02:00
Michael Eischer
cfc420664a
mem: stricter handling of out of bounds requests
2024-05-18 19:54:52 +02:00
Michael Eischer
d40f23e716
azure/b2/gs/s3/swift: adapt cloud backend
2024-05-18 19:54:51 +02:00
Michael Eischer
e793c002ec
local: stricter handling of short files
2024-05-18 19:54:21 +02:00
Michael Eischer
b4895ebd76
rest: rework error reporting and report too short files
2024-05-18 19:54:21 +02:00
Michael Eischer
eaa3f81d6b
sftp: check for truncated files without an extra backend request
2024-05-18 19:54:21 +02:00
Michael Eischer
c6d74458ee
sftp: improve handling of too short files
2024-05-18 19:54:21 +02:00
Michael Eischer
7ed560a201
Merge pull request #4796 from MichaelEischer/parallel-dump-load
...
dump: Parallelize loading large files
2024-05-14 22:35:44 +02:00
Michael Eischer
92221c2a6d
Merge pull request #4708 from zmanda/windows-securitydesc
...
Back up and restore SecurityDescriptors on Windows
2024-05-12 14:14:39 +00:00
Michael Eischer
b5fdb1d637
Merge pull request #4782 from MichaelEischer/fix-sftp-performance
...
Fix sftp upload performance
2024-05-12 15:28:33 +02:00
Michael Eischer
ebd01a4675
backend: add tests for watchdogRoundTripper
2024-05-09 23:46:17 +02:00
Michael Eischer
8778670232
backend: cancel stuck http requests
...
requests that make no upload or download progress within a timeout are
canceled.
2024-05-09 23:46:17 +02:00
Michael Eischer
0987c731ec
backend: configure protocol-level connection health checks
...
This should detect a connection that is stuck for more than 2 minutes.
2024-05-09 23:46:17 +02:00
aneesh-n
a4fd1b91e5
Fix review comments
...
Change lowerPrivileges from bool to atomic.Bool.
Add missing cleanup from upstream go-winio.
Add handling for ERROR_NOT_ALL_ASSIGNED warning.
2024-05-06 16:54:08 -06:00
Michael Eischer
4d55a62ada
bloblru: add test for GetOrCompute
2024-05-05 12:00:25 +02:00
Michael Eischer
7cce667f92
fuse: switch to use bloblru.GetOrCompute
2024-05-05 11:38:42 +02:00
Michael Eischer
bd03af2feb
dump: add GetOrCompute to bloblru cache
2024-05-05 11:38:42 +02:00
Michael Eischer
45509eafc8
dump: load blobs of a file from repository in parallel
2024-05-05 11:38:42 +02:00
Michael Eischer
ffe5439149
Merge pull request #4605 from MichaelEischer/better-restorer-error-handling
...
Rework repository.StreamPacks & better restorer error handling
2024-05-01 16:37:41 +02:00
aneesh-n
672f6cd776
Fix review comments for privileges and security flags
2024-04-29 17:29:51 -06:00
aneesh-n
08c6945d61
Fix review comments
2024-04-29 16:21:38 -06:00
Aneesh N
3f76b902e5
Merge branch 'master' into windows-securitydesc
2024-04-29 14:40:34 -06:00
DRON-666
125dba23c5
Rearange code
2024-04-29 01:27:34 +03:00
DRON-666
7ee889bb0d
Use S_FALSE and MaxInt
2024-04-29 01:25:25 +03:00
DRON-666
90b168eb6c
isMountPointExcluded to isMountPointIncluded
2024-04-29 01:23:50 +03:00
DRON-666
24330c19a8
Use kebab case in option names
2024-04-29 01:21:33 +03:00
DRON-666
5703e5a652
Fix texts and comments
2024-04-29 01:18:46 +03:00
DRON-666
0a8f9c5d9c
vss: Add tests for "provider" option
2024-04-28 22:45:21 +03:00
DRON-666
bb0f93ef3d
vss: Add "provider" option
2024-04-28 22:45:21 +03:00
DRON-666
3bac1f0135
vss: Fix issues reported by linters
2024-04-28 22:45:21 +03:00
DRON-666
88c509e3e9
vss: Change ErrorHandler signature
...
We don't need `error` here: the only existing implementation
of `ErrorHandler` always call `Backup.Error` and all
implementations of `Backup.Error` always return nil.
2024-04-28 22:44:16 +03:00
DRON-666
9d3d915e2c
vss: Add some tests
2024-04-28 22:44:16 +03:00
DRON-666
c4f67c0064
vss: Add volume filtering
...
Add options to exclude all mountpoints and arbitrary volumes from snapshotting.
2024-04-28 22:44:15 +03:00
DRON-666
7470e5356e
vss: Add "timeout" option
...
Changing multiple "callAsyncFunctionAndWait" with fixed timeout
to calculated timeout based on deadline.
2024-04-28 22:44:15 +03:00
DRON-666
78dbc5ec58
vss: Add initial support for extended options
2024-04-28 22:44:15 +03:00
Michael Eischer
935327d480
sftp: slightly increase write concurrency
...
This should increase upload throughput for high latency links a bit.
2024-04-28 11:50:09 +02:00
Michael Eischer
669a669603
sftp: Fix upload performance issue
...
Since pkg/sftp 1.13.0 files were uploaded sequentially using 32kb chunks
instead of sending 64 chunks in parallel.
2024-04-28 11:48:26 +02:00
Michael Eischer
940a3159b5
let index.Each() and pack.Size() return error on canceled context
...
This forces a caller to actually check that the function did complete.
2024-04-22 22:39:32 +02:00
Michael Eischer
31624aeffd
Improve command shutdown on context cancellation
2024-04-22 22:31:38 +02:00
Michael Eischer
b15d867414
Merge pull request #4763 from MichaelEischer/refactor-prune
...
Refactor repair index / prune into the repository package
2024-04-22 22:24:53 +02:00
Michael Eischer
2e6c43c695
Merge pull request #4761 from MichaelEischer/fix-cache-race
...
cache: ignore ErrNotExist during cleanup of old files
2024-04-22 21:46:06 +02:00
Michael Eischer
f7632de3d6
Merge pull request #4772 from MichaelEischer/better-error-on-too-large-blob
...
repository: Better error message if blob is larger than 4GB
2024-04-22 21:45:06 +02:00
Michael Eischer
20d8eed400
repository: streamPack: separate requests for gap larger than 1MB
...
With most cloud providers, traffic is much more expensive than API
calls. Thus slightly bias streamPack towards a bit more API calls in
exchange for slightly less traffic.
2024-04-22 21:21:23 +02:00
Michael Eischer
cf700d8794
repository: streamPack: reuse zstd decoder
2024-04-22 21:21:23 +02:00
Michael Eischer
666a0b0bdb
repository: streamPack: replace streaming with chunked download
...
Due to the interface of streamPack, we cannot guarantee that operations
progress fast enough that the underlying connections remains open. This
introduces partial failures which massively complicate the error
handling.
Switch to a simpler approach that retrieves the pack in chunks of 32MB.
If a blob is larger than this limit, then it is downloaded separately.
To avoid multiple copies in memory, an auxiliary interface
`discardReader` is introduced that allows directly accessing the
downloaded byte slices, while still supporting the streaming used by the
`check` command.
2024-04-22 21:21:23 +02:00
Michael Eischer
621012dac0
repository: Add blob loading fallback to LoadBlobsFromPack
...
Try to retrieve individual blobs via LoadBlob if streaming did not work.
2024-04-21 21:35:55 +02:00
Michael Eischer
10355c3fb6
repository: Better error message if blob is larger than 4GB
2024-04-19 22:00:35 +02:00
coderwander
a82ed71de7
Fix struct names
...
Signed-off-by: coderwander <770732124@qq.com>
2024-04-18 10:02:09 +08:00
