octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-23 21:27:34 +00:00
Code Issues Releases Wiki Activity
7,937 Commits 7 Branches 33 Tags 66 MiB
a60ee9b764
Commit Graph

7937 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Michael Eischer
a60ee9b764 retry: limit retries based on elapsed time not count 
Depending on how long an operation takes to fail, the total retry
duration can currently vary between 1.5 and 15 minutes. In particular
for temporarily interrupted network connections, the former timeout is
too short. Thus always use a limit of 15 minutes.
 2024-05-24 20:24:02 +02:00
Michael Eischer
a3633cad9e retry: explicitly log failed requests 
This simplifies finding the request in the log output that cause an
operation to fail.
 2024-05-24 20:24:02 +02:00
Michael Eischer
8898f61717
Merge pull request #4809 from MichaelEischer/update-changelog 
add retries for corrupted blobs to changelog
 2024-05-18 23:04:13 +02:00
Michael Eischer
5f23baabcc add retries for corrupted blobs to changelog 2024-05-18 23:03:24 +02:00
Michael Eischer
9c5bac6f25
Merge pull request #4799 from letmaik/letmaik/azure-force-cli-credential 
Azure: add option to force use of CLI credential
 2024-05-18 20:22:15 +00:00
Michael Eischer
c56ecec9aa azure: deduplicate cli and default credentials case 2024-05-18 22:15:54 +02:00
Maik Riechert
355f520936 Azure: add option to force use of CLI credential 2024-05-18 22:15:54 +02:00
Michael Eischer
1dfe1b8732
Merge pull request #4802 from MichaelEischer/backend-cleanups 
Repository: Remove Backend() method
 2024-05-18 22:02:45 +02:00
Michael Eischer
223aa22cb0 replace some uses of restic.Repository with finegrained interfaces 2024-05-18 21:42:51 +02:00
Michael Eischer
291c9677de restic/repository: remove Backend() method 2024-05-18 21:42:51 +02:00
Michael Eischer
673496b091 repository: clean cache between CheckPack retries 
The cache cleanup pattern is also used in ListPack etc.
 2024-05-18 21:42:51 +02:00
Michael Eischer
3d2410ed50 Replace some repo.RemoveUnpacked usages 
These will eventually be blocked as they do not delete Snapshots.
 2024-05-18 21:42:51 +02:00
Michael Eischer
d2c26e33f3 repository: remove further usages of repo.Backend() 2024-05-18 21:42:51 +02:00
Michael Eischer
8a425c2f0a remove usages of repo.Backend() from tests 2024-05-18 21:42:51 +02:00
Michael Eischer
aa4647f773 repository: unexport PackBlobIterator 2024-05-18 21:42:51 +02:00
Michael Eischer
94e863885c check: move verification of individual pack file to repository 2024-05-18 21:42:50 +02:00
Michael Eischer
e40943a75d restic: remove backend usage from lock test 2024-05-18 21:38:31 +02:00
Michael Eischer
67e2ba0d40 repository: Lock requires *repository.Repository 
This allows the Lock function to access the backend, even once the
Backend method is removed from the interface.
 2024-05-18 21:38:31 +02:00
Michael Eischer
d8b184b3d3 repository: convert test helper to return *repository.Repository 2024-05-18 21:38:31 +02:00
Michael Eischer
a1ca5e15c4 migrations: add temporary hack for s3_layout 
The migration will be removed after the next restic release anyways.
Thus, there's no need for a clean implementation.
 2024-05-18 21:38:31 +02:00
Michael Eischer
34d90aecf9 migrations: move logic of upgrade_repo_v2 to repository package 
The migration modifies repository internals and thus should live within
the repository package.
 2024-05-18 21:38:31 +02:00
Michael Eischer
ab9077bc13 replace usages of backend.Remove() with repository.RemoveUnpacked() 
RemoveUnpacked will eventually block removal of all filetypes other than
snapshots. However, getting there requires a major refactor to provide
some components with privileged access.
 2024-05-18 21:38:31 +02:00
Michael Eischer
8274f5b101 prune: remove Backend.IsNotExist() 
Only handling one specific error is not particularly useful.
 2024-05-18 21:38:31 +02:00
Michael Eischer
9795198189 debug: remove Backend.Stat() usage 2024-05-18 21:38:31 +02:00
Michael Eischer
0c1ba6d95d backend: remove unused Location method 2024-05-18 21:38:31 +02:00
Michael Eischer
eb6c653f89
Merge pull request #4800 from MichaelEischer/cleanup-load 
Retry loading of corrupted data from backend / cache
 2024-05-18 21:34:54 +02:00
Michael Eischer
74d90653e0 check: use ReadFull to load pack header in checkPack 
This ensures that the pack header is actually read completely.
Previously, for a truncated file it was possible to only read a part of
the header, as backend.Load(...) is not guaranteed to return as many
bytes as requested by the length parameter.
 2024-05-18 21:28:54 +02:00
Michael Eischer
8f8d872a68 fix compatibility with go 1.19 2024-05-18 21:28:54 +02:00
Michael Eischer
ff0744b3af check: test checkPack retries 2024-05-18 21:28:54 +02:00
Michael Eischer
987c3b250c repository: test retries of ListPack 2024-05-18 21:28:54 +02:00
Michael Eischer
bf16096771 repository: test LoadBlob retries 2024-05-18 21:28:54 +02:00
Michael Eischer
4f45668b7c repository: rework and extend LoadRaw tests 2024-05-18 21:28:54 +02:00
Michael Eischer
ac805d6838 cache: cleanup debug logs 2024-05-18 21:28:54 +02:00
Michael Eischer
5214af88e2 cache: test forget behavior 2024-05-18 21:28:54 +02:00
Michael Eischer
3ff063e913 check: verify pack a second time if broken 2024-05-18 21:28:54 +02:00
Michael Eischer
385cee09dc repository: fix caching of tree packs in LoadBlobsFromPack 2024-05-18 21:28:54 +02:00
Michael Eischer
e734746f75 cache: forget cached file at most once 
This is inspired by the circuit breaker pattern used for distributed
systems. If too many requests fails, then it is better to immediately
fail new requests for a limited time to give the backend time to
recover.

By only forgetting a file in the cache at most once, we can ensure that
a broken file is only retrieved once again from the backend. If the file
stored there is broken, previously it would be cached and deleted
continuously. Now, it is retrieved only once again, all later requests
just use the cached copy and either succeed or fail immediately.
 2024-05-18 21:28:54 +02:00
Michael Eischer
97a307df1a cache: Always use cached file if it exists 
A file is always cached whole. Thus, any out of bounds access will also
fail when directed at the backend. To handle case in which the cached
file is broken, then caller must call Cache.Forget(h) for the file in
question.
 2024-05-18 21:28:54 +02:00
Michael Eischer
8cce06d915 repair packs: drop experimental warning 
This warning should already have been removed once the feature flag was
dropped.
 2024-05-18 21:28:54 +02:00
Michael Eischer
433a6aad29 repository: remove redundant blob loading fallback from RepairPacks 
LoadBlobsFromPack already implements the same fallback behavior.
 2024-05-18 21:28:54 +02:00
Michael Eischer
e401af07b2 check: fix error message formatting 2024-05-18 21:28:54 +02:00
Michael Eischer
7017adb7e9 repository: retry failed ListPack once 2024-05-18 21:28:54 +02:00
Michael Eischer
e33ce7f408 repository: retry failed LoadBlob once 2024-05-18 21:28:54 +02:00
Michael Eischer
2ace242f36 repository: make reloading broken files explicit 2024-05-18 21:28:54 +02:00
Michael Eischer
e9390352a7 cache: code cleanups 2024-05-18 21:26:00 +02:00
Michael Eischer
503c8140b1 repository: unify blob decoding code 2024-05-18 21:26:00 +02:00
Michael Eischer
6563f1d2ca repository: remove redundant debug log 2024-05-18 21:26:00 +02:00
Michael Eischer
021fb49559 repository: Implement repository.LoadUnpacked using LoadRaw 
Both functions were using a similar implementation.
 2024-05-18 21:26:00 +02:00
Michael Eischer
779c8d3527 debug/repair packs/upgrade repo v2: use repository.LoadRaw 
This replaces calling the low-level backend.Load() method.
 2024-05-18 21:26:00 +02:00
Michael Eischer
1d6d3656b0 repository: move backend.LoadAll to repository.LoadRaw 
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.

The removed cache backend test will be added again later on.
 2024-05-18 21:26:00 +02:00