Michael Eischer
a23cb3a428
restore: reduce memory usage
2024-06-12 22:36:52 +02:00
Michael Eischer
2b50c2606c
restorer: use options struct
2024-06-12 22:36:52 +02:00
Michael Eischer
0fcd89f892
restorer: remove special case for blobs with many occurrences
...
Loading blobs by now is no longer prone to timeouts when processing
takes a long time.
2024-06-12 22:36:52 +02:00
Michael Eischer
fd2ff464a2
restorer: remove stale comment
2024-06-12 22:36:52 +02:00
Michael Eischer
607daeed4f
restore: move nil pointer check into restoreui
2024-06-12 22:36:52 +02:00
Michael Eischer
30320a249a
restore: let filerestorer also handle empty files
...
This get's rid of the corresponding special cases.
2024-06-12 22:36:52 +02:00
Michael Eischer
c166ad7daf
restore: factor out file creation helper
2024-06-12 22:36:52 +02:00
Michael Eischer
da4512738a
Merge pull request #4853 from MichaelEischer/safer-repo-init
...
init: double check that no repository exists yet
2024-06-09 11:27:45 +02:00
Michael Eischer
8440b94159
Merge pull request #4807 from zmanda/windows-extendedattribs
...
Back up and restore Extended Attributes on Windows NTFS
2024-06-09 09:18:04 +00:00
Michael Eischer
bab760369f
repository: double check that there is not repository before init
...
Apparently, calling `Stat` on the config file can be unreliable for some
backends.
2024-06-09 00:05:32 +02:00
Michael Eischer
78485160fc
Merge pull request #4849 from fthoma/table-tcwidth
...
Use character display width for table padding
2024-06-07 19:53:14 +00:00
Michael Eischer
7cd324fe26
ui/table: avoid duplicate table cell width calculation
2024-06-07 21:45:40 +02:00
Michael Eischer
edd3e214c2
ui/table: fix width calculation of multi-line column headers
2024-06-07 21:44:49 +02:00
Michael Eischer
3f878aa8e7
Merge pull request #4845 from greatroar/errors
...
Fix error handling bug + clean up error messages
2024-06-07 17:07:07 +00:00
Michael Eischer
112de8526d
Merge pull request #4852 from MichaelEischer/master
...
crypto: cleanup ciphertext verification error
2024-06-07 19:06:08 +02:00
Michael Eischer
0a70bbcea5
Merge pull request #4844 from MichaelEischer/improve-timeout-error
...
backend: Improve timeout error message
2024-06-07 19:05:39 +02:00
aneesh-n
4f053da06a
Fix test case by correcting cleanup order
2024-06-05 23:10:29 -06:00
aneesh-n
2101dfe448
Add missing return to fix failing test
2024-06-05 22:40:21 -06:00
aneesh-n
7a48c9ebd7
Fix review comments
2024-06-05 16:06:57 -06:00
Michael Eischer
b82ecc6425
crypto: cleanup ciphertext verification error
2024-06-05 22:33:20 +02:00
Florian Thoma
e9de9684f4
Use character display width for table padding
...
Using len(...) for table cell padding produced wrong results for unicode
chracters leading to misaligned tables. Implementation changed to take
the actual terminal display width into consideration.
2024-06-05 09:33:15 +02:00
Leo R. Lundgren
b2bbbe805f
azure: Improve error message in azure.Create()
2024-06-03 23:37:17 +02:00
Michael Eischer
660679c2f6
Merge pull request #4835 from MichaelEischer/fix-list-cancel
...
backend/retry: do not log final error if context was canceled
2024-06-01 19:17:30 +02:00
Michael Eischer
db2398f35b
backend: increase request progress timeout to 5 minutes
...
Apparently, 2 minutes are too short in some cases and can result in
canceled List requests.
2024-06-01 19:01:51 +02:00
Michael Eischer
6bf3d4859f
backend: improve error on http request timeout
...
Now yields a "request timeout" error instead of "context canceled".
2024-06-01 18:52:39 +02:00
greatroar
4a874000b7
gs: Replace some errors.Wrap calls
...
The first one in Create is already a WithStack error. The rest were
referencing code that hasn't existed for quite some time. Note that
errors from Google SDKs tends to start with "google:" or "googleapi:".
Also, use restic/internal/errors.
2024-06-01 15:11:06 +02:00
Michael Eischer
38654a3bd7
backend/retry: do not log final error if context was canceled
...
Calls to `List(ctx, ...)` are usually stopped by canceling the context
once no further entries are required by the caller. Thus, don't log the
final error if the used context was canceled.
2024-05-30 18:48:52 +02:00
Michael Eischer
2280fbfd2e
Merge pull request #4810 from konidev20/fix-gh-4768-add-custom-user-agent-for-http-client
...
Allow custom User-Agent to be specified for outgoing requests
2024-05-30 16:34:52 +00:00
Srigovind Nayak
de7b418bbe
http: allow custom User-Agent for outgoing HTTP requests
2024-05-30 15:38:06 +02:00
Michael Eischer
6869bdaaa8
backup: implement --skip-if-unchanged
2024-05-30 15:14:31 +02:00
Michael Eischer
7b4f81d964
Merge pull request #4829 from greatroar/termstatus
...
termstatus: Do formatting in Message, not Terminal
2024-05-30 11:57:44 +00:00
Michael Eischer
52b3fee42b
Merge pull request #4818 from MichaelEischer/move-code
...
Move cache / index / hashing / pack packages
2024-05-30 13:54:31 +02:00
greatroar
46f04bc5f0
termstatus: Do formatting in Message, not Terminal
2024-05-27 19:05:39 +02:00
Michael Eischer
1c6067d93d
bloblru: variable name cleanup
2024-05-26 12:42:46 +02:00
Michael Eischer
21ce03cff2
bloblru: move defer outside critical section
2024-05-26 12:38:20 +02:00
Michael Eischer
0ad0b7ca7c
bloblru: fix race condition that can compute value multiple times
2024-05-26 12:37:24 +02:00
Michael Eischer
7ea508e7b8
Merge pull request #4815 from greatroar/termstatus
...
Termstatus refactoring
2024-05-25 12:08:53 +00:00
Michael Eischer
496e57f956
hashing: move to repository package
2024-05-25 13:13:03 +02:00
Michael Eischer
5e0ea8fcfa
pack: move to repository package
2024-05-25 13:13:03 +02:00
Michael Eischer
50ec408302
index: move to repository package
2024-05-25 13:13:03 +02:00
Michael Eischer
8e5d7d719c
cache: move to backend package
2024-05-24 23:04:06 +02:00
Michael Eischer
3c7b7efdc9
repository: remove prune plan parts once they are no longer necessary
2024-05-24 22:18:14 +02:00
Michael Eischer
462b82a060
index: reduce size of compressed indexes
...
use the same index size for compressed and uncompressed indexes.
Otherwise, decoding the index of a compressed repository requires
significantly more memory.
2024-05-24 22:18:14 +02:00
Michael Eischer
77873f5a9d
repository: let prune control data structure of usedBlobs set
2024-05-24 22:18:14 +02:00
Michael Eischer
2033c02b09
index: replace CountedBlobSet with AssociatedSet
2024-05-24 22:18:14 +02:00
Michael Eischer
93098e9265
prune: hide implementation details of counted blob set
2024-05-24 21:42:56 +02:00
Michael Eischer
860b595a8b
backend: increase watchdog test timeout for deflaking
2024-05-24 21:33:17 +02:00
Michael Eischer
027cc64737
repository: fix prune heuristic to allow resuming interrupted runs
...
Pack files created by interrupted prune runs, appear to consist only of
duplicate blobs on the next run. This caused the previous heuristic to
ignore those pack files. Now, a duplicate blob in a specific pack file
is also selected if that pack file only contains duplicate blobs. This
allows prune to select the already rewritten pack files.
2024-05-24 21:33:17 +02:00
Michael Eischer
e52033a8bd
index: slightly reduce Rewrite concurrency
...
The index operations are likely CPU-bounded. Thus, reduce the
concurrency accordingly.
2024-05-24 21:33:17 +02:00
Michael Eischer
57d69aa640
index: cleanup SaveIndex method
2024-05-24 21:33:17 +02:00
Michael Eischer
2ca1c37721
index: additional tests for new index save methods
2024-05-24 21:33:17 +02:00
Michael Eischer
5f7b48e65f
index: replace Save() method with Rewrite and SaveFallback
...
Rewrite implements a streaming rewrite of the index that excludes the
given packs. For this it loads all index files from the repository and
only modifies those that require changes. This will reduce the index
churn when running prune. Rewrite does not require the in-memory index
and thus can drop it to significantly reduce the memory usage.
However, `prune --unsafe-recovery` cannot use this strategy and requires
a separate method to save the whole in-memory index. This is now handled
using SaveFallback.
2024-05-24 21:33:17 +02:00
Michael Eischer
ad98fbf7dd
restic: add IDSet.Clone() method
2024-05-24 21:33:17 +02:00
Michael Eischer
72482ce5bd
index: misc cleanups
2024-05-24 21:33:17 +02:00
Michael Eischer
68fa0e0305
prune: no longer disable automatic index updates
...
this allows prune to resume an interrupted prune run.
2024-05-24 21:33:17 +02:00
Michael Eischer
9aa0c90fb2
index: remove supersedes field
...
Using the field with its current semantics is nearly impossible to get
right. Remove it as it will be replaced anyways in repository format 3.
2024-05-24 21:33:17 +02:00
Michael Eischer
76e6719f2e
repository: make CreateIndexFromPacks method private
2024-05-24 21:33:17 +02:00
Michael Eischer
04ad9f0c0c
repository: remove Packer and SavePacker from public interface
2024-05-24 21:33:17 +02:00
Michael Eischer
550d1eeac3
repository: remove SaveIndex from interface
...
The method is now only indirectly accessible via Prune or RepairIndex.
2024-05-24 21:33:17 +02:00
Michael Eischer
fb59e00614
index: rewrite MasterIndex load/save test to be independent of repository
2024-05-24 21:33:17 +02:00
Michael Eischer
447b486c20
index: deduplicate index loading of check and repository
2024-05-24 21:33:17 +02:00
Michael Eischer
6ca12c1b4a
archiver: replace most uses of restic.Repository
2024-05-24 21:33:17 +02:00
Michael Eischer
864995271e
repository: unwrap BlobHandle parameters of LookupBlob
...
The method now uses the same parameters as LookupBlobSize.
2024-05-24 21:33:17 +02:00
Michael Eischer
1266a4932f
repository: fix parameter order of LookupBlobSize
...
All methods should use blobType followed by ID.
2024-05-24 21:33:17 +02:00
Michael Eischer
e848ad651a
restic: name parameters in restic interface
2024-05-24 21:33:17 +02:00
Michael Eischer
0bb0720348
test cleanups
2024-05-24 21:33:17 +02:00
Michael Eischer
c01bcb1001
archiver: remove unused masterIndex from test
2024-05-24 21:33:17 +02:00
Michael Eischer
0aa5c53842
repository: replace HasBlob with LookupBlobSize
2024-05-24 21:33:17 +02:00
Michael Eischer
8f1e70cd9b
repository: remove clearIndex and packSize from public interface
2024-05-24 21:33:17 +02:00
Michael Eischer
4df887406f
repository: inline MasterIndex interface into Repository interface
2024-05-24 21:33:17 +02:00
Michael Eischer
3cc6827f09
forget: add feature flag for safe --keep-tags behavior
2024-05-24 20:45:33 +02:00
Michael Eischer
57f9739573
forget: Add --unsafe-allow-remove-all option
...
To prevent accidentally wiping all snapshots from a repository, that
option can only be used if either a snapshot filter or a keep policy is
specified.
Essentially, the option allows `forget --tag something
--unsafe-allow-remove-all` calls to remove all snapshots with a specific
tag.
2024-05-24 20:45:33 +02:00
Michael Eischer
c0e1f36830
forget: refuse deleting the last snapshot in a snapshot group
...
`--keep-tag invalid-tag` was previously able to wipe all snapshots in a
repository. As a user specified a `--keep-*` option this is likely
unintentional. This forbid deleting all snapshot if a `--keep-*` option
was specified to prevent data loss. (Not specifying such an option
currently also causes the command to abort)
2024-05-24 20:45:33 +02:00
Michael Eischer
d106ad6921
restic: regenerate snapshot keep policy golden test files
2024-05-24 20:45:33 +02:00
Michael Eischer
e4a48085ae
backend/retry: feature flag new retry behavior
2024-05-24 20:24:02 +02:00
Michael Eischer
b1266867d2
repository: wait max 1 minutes for lock removal if context is canceled
...
The toplevel context in restic only canceled if the user interrupts a
restic operation. If the network connection has failed this can require
waiting the full retry duration of 15 minutes which is a bad user
experience for interactive usage. Thus limit the delay to one minute in
this case.
2024-05-24 20:24:02 +02:00
Michael Eischer
98709a4372
retry: reduce total number of retries
...
Retries in restic try to solve two main problems:
- retry a temporarily failed operation
- tolerate temporary network interruptions
The first problem only requires a few retries, whereas the last one benefits
primarily from spreading the requests over a longer duration.
Increasing the default multiplier and the initial interval works for
both cases. The first few retries only take a few seconds, while later
retries quickly reach the maximum interval of one minute. This ensures
that the total number of retries issued by restic will remain at around
21 retries for a 15 minute period. As the concurrency in restic is
bounded, retries drastically reduce the number of requests sent to a
backend. This helps to prevent overloading the backend.
2024-05-24 20:24:02 +02:00
Michael Eischer
512cd6ef07
retry: ensure that there's always at least one retry
...
Previously, if an operation failed after 15 minutes, then it would never
be retried. This means that large backend requests are more unreliable
than smaller ones.
2024-05-24 20:24:02 +02:00
Michael Eischer
a60ee9b764
retry: limit retries based on elapsed time not count
...
Depending on how long an operation takes to fail, the total retry
duration can currently vary between 1.5 and 15 minutes. In particular
for temporarily interrupted network connections, the former timeout is
too short. Thus always use a limit of 15 minutes.
2024-05-24 20:24:02 +02:00
Michael Eischer
a3633cad9e
retry: explicitly log failed requests
...
This simplifies finding the request in the log output that cause an
operation to fail.
2024-05-24 20:24:02 +02:00
greatroar
66d03c797e
ui, termstatus: Move WrapStdio
...
Saves some imports. ui still needs to import ui/termstatus from
message.go.
2024-05-21 11:25:16 +02:00
greatroar
0b56214473
ui: Simplify stdio wrapper
...
The StdioWrapper type is really just a pair of io.WriteClosers, so
remove it in favor of a function that returns two of those. Test
coverage increases because the removed code was not tested.
2024-05-21 11:23:32 +02:00
greatroar
7f439a9c34
ui: Inline lineWriter into StdioWrapper
2024-05-21 11:00:09 +02:00
Michael Eischer
c56ecec9aa
azure: deduplicate cli and default credentials case
2024-05-18 22:15:54 +02:00
Maik Riechert
355f520936
Azure: add option to force use of CLI credential
2024-05-18 22:15:54 +02:00
Michael Eischer
223aa22cb0
replace some uses of restic.Repository with finegrained interfaces
2024-05-18 21:42:51 +02:00
Michael Eischer
291c9677de
restic/repository: remove Backend() method
2024-05-18 21:42:51 +02:00
Michael Eischer
673496b091
repository: clean cache between CheckPack retries
...
The cache cleanup pattern is also used in ListPack etc.
2024-05-18 21:42:51 +02:00
Michael Eischer
3d2410ed50
Replace some repo.RemoveUnpacked usages
...
These will eventually be blocked as they do not delete Snapshots.
2024-05-18 21:42:51 +02:00
Michael Eischer
d2c26e33f3
repository: remove further usages of repo.Backend()
2024-05-18 21:42:51 +02:00
Michael Eischer
8a425c2f0a
remove usages of repo.Backend() from tests
2024-05-18 21:42:51 +02:00
Michael Eischer
aa4647f773
repository: unexport PackBlobIterator
2024-05-18 21:42:51 +02:00
Michael Eischer
94e863885c
check: move verification of individual pack file to repository
2024-05-18 21:42:50 +02:00
Michael Eischer
e40943a75d
restic: remove backend usage from lock test
2024-05-18 21:38:31 +02:00
Michael Eischer
67e2ba0d40
repository: Lock requires *repository.Repository
...
This allows the Lock function to access the backend, even once the
Backend method is removed from the interface.
2024-05-18 21:38:31 +02:00
Michael Eischer
d8b184b3d3
repository: convert test helper to return *repository.Repository
2024-05-18 21:38:31 +02:00
Michael Eischer
a1ca5e15c4
migrations: add temporary hack for s3_layout
...
The migration will be removed after the next restic release anyways.
Thus, there's no need for a clean implementation.
2024-05-18 21:38:31 +02:00
Michael Eischer
34d90aecf9
migrations: move logic of upgrade_repo_v2 to repository package
...
The migration modifies repository internals and thus should live within
the repository package.
2024-05-18 21:38:31 +02:00
Michael Eischer
ab9077bc13
replace usages of backend.Remove() with repository.RemoveUnpacked()
...
RemoveUnpacked will eventually block removal of all filetypes other than
snapshots. However, getting there requires a major refactor to provide
some components with privileged access.
2024-05-18 21:38:31 +02:00
Michael Eischer
0c1ba6d95d
backend: remove unused Location method
2024-05-18 21:38:31 +02:00
