octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-23 13:17:42 +00:00
Code Issues Releases Wiki Activity
7,889 Commits 7 Branches 33 Tags 66 MiB
779c8d3527
Commit Graph

1988 Commits

Author SHA1 Message Date
Michael Eischer
f7632de3d6
Merge pull request #4772 from MichaelEischer/better-error-on-too-large-blob 
repository: Better error message if blob is larger than 4GB
 2024-04-22 21:45:06 +02:00
Michael Eischer
20d8eed400 repository: streamPack: separate requests for gap larger than 1MB 
With most cloud providers, traffic is much more expensive than API
calls. Thus slightly bias streamPack towards a bit more API calls in
exchange for slightly less traffic.
 2024-04-22 21:21:23 +02:00
Michael Eischer
cf700d8794 repository: streamPack: reuse zstd decoder 2024-04-22 21:21:23 +02:00
Michael Eischer
666a0b0bdb repository: streamPack: replace streaming with chunked download 
Due to the interface of streamPack, we cannot guarantee that operations
progress fast enough that the underlying connections remains open. This
introduces partial failures which massively complicate the error
handling.

Switch to a simpler approach that retrieves the pack in chunks of 32MB.
If a blob is larger than this limit, then it is downloaded separately.

To avoid multiple copies in memory, an auxiliary interface
`discardReader` is introduced that allows directly accessing the
downloaded byte slices, while still supporting the streaming used by the
`check` command.
 2024-04-22 21:21:23 +02:00
Michael Eischer
621012dac0 repository: Add blob loading fallback to LoadBlobsFromPack 
Try to retrieve individual blobs via LoadBlob if streaming did not work.
 2024-04-21 21:35:55 +02:00
Michael Eischer
10355c3fb6 repository: Better error message if blob is larger than 4GB 2024-04-19 22:00:35 +02:00
coderwander
a82ed71de7 Fix struct names 
Signed-off-by: coderwander <770732124@qq.com>
 2024-04-18 10:02:09 +08:00
Michael Eischer
09587e6c08 repository: duplicate a few blobs in prune tests 2024-04-14 13:57:19 +02:00
Michael Eischer
defd7ae729 prune/repair index: reset in-memory index after command 
The current in-memory index becomes stale after prune or repair index
have run. Thus, just drop the in-memory index altogether once these
commands have finished.
 2024-04-14 13:46:24 +02:00
Michael Eischer
038586dc9d repository: add minimal test for prune 2024-04-14 13:45:17 +02:00
Michael Eischer
d8622c86eb prune: clean up internal interface 2024-04-14 13:45:15 +02:00
Michael Eischer
8d507c1372 repository: add basic test for RepairIndex 2024-04-14 13:45:15 +02:00
Michael Eischer
310db03c0e repair index: improve log output if index cannot be deleted 
The operation will always fail with an error if an index cannot be
deleted. Thus, this change is purely cosmetic.
 2024-04-14 13:45:13 +02:00
Michael Eischer
7d1b9cde34 repository: use normal Init method in tests 2024-04-14 13:45:11 +02:00
Michael Eischer
b25fc2c89d repository: remove redundant flushes from tests 2024-04-14 13:45:10 +02:00
Michael Eischer
c65459cd8a repository: speed up tests 2024-04-14 13:45:10 +02:00
Michael Eischer
eda9f7beb4 ui/progress: add helper to print messages during tests 2024-04-14 13:45:08 +02:00
Michael Eischer
35277b7797 backend/mem: cleanup not found error message 2024-04-14 13:45:06 +02:00
Michael Eischer
7ba5e95a82 check: allow tests to only verify pack&index integrity 2024-04-14 13:45:04 +02:00
Michael Eischer
4c9a10ca37 repair packs: deduplicate index rebuild 2024-04-14 13:45:02 +02:00
Michael Eischer
85e4021619 prune: move additional option checks to repository 2024-04-14 13:44:58 +02:00
Michael Eischer
55d56db31b
Merge pull request #4743 from MichaelEischer/deprecate-s3legacy-layout 
Deprecate s3legacy layout
 2024-04-11 22:09:34 +02:00
Michael Eischer
fc3b548625 prune: move logic into repository package 2024-04-10 21:30:52 +02:00
Michael Eischer
866ddf5698 repair index: refactor code into repository package 2024-04-10 21:30:52 +02:00
Michael Eischer
591b421c4a Deprecate s3legacy layout 2024-04-10 21:27:56 +02:00
Michael Eischer
bf054c09d2 backup: Ignore xattr.list permission error for parent directories 
On FreeBSD, limited users may not be able to even list xattrs for the
parent directories above the snapshot source paths. As this can cause
the backup to fail, just ignore those errors.
 2024-04-10 20:46:15 +02:00
Michael Eischer
0747cf5319 cache: ignore ErrNotExist during cleanup of old files 
Two restic processes running concurrently can try to remove the same
files from the cache. This could cause one process to fail with an error
if the other one has already remove a file that the current process also
tries to delete.
 2024-04-10 19:25:51 +02:00
Michael Eischer
49f98f25fc
Merge pull request #4742 from MichaelEischer/consistent-rtest-import 
Use consistent alias for interal/test package
 2024-04-03 21:47:32 +02:00
Michael Eischer
6cca1d5705
Merge pull request #4655 from ae-govau/unixsocket 
Enhancement: option to send HTTP over unix socket
 2024-04-03 19:29:21 +00:00
Michael Eischer
ec2b79834a use consistent alias for interal/test package 2024-03-29 00:24:03 +01:00
Michael Eischer
5e98f1e2eb repository: fix test setup race conditions 2024-03-28 23:17:02 +01:00
Michael Eischer
dc441c57a7 repository: unify repository initialization in tests 
Tests should use a helper from internal/repository/testing.go to
construct a Repository object.
 2024-03-28 23:17:02 +01:00
Michael Eischer
3ba1fa3cee repository: remove a few global variables 2024-03-28 23:17:02 +01:00
Michael Eischer
044e8bf821 repository: parallelize lock tests 2024-03-28 23:17:02 +01:00
Michael Eischer
e8df50fa3c repository: remove global list of locks 2024-03-28 22:46:33 +01:00
Michael Eischer
cbb5f89252 lock: move code to repository package 2024-03-28 22:46:33 +01:00
Michael Eischer
7f9ad1c3db
Merge pull request #4705 from MichaelEischer/snapshot-statistics 
Store snapshot statistics & print snapshot size
 2024-03-28 22:41:45 +01:00
Michael Eischer
d705741571 backup: test that deviceID is only stored for hardlinks 2024-03-28 19:32:50 +01:00
Michael Eischer
a9b3d86c4f features: remove example feature 2024-03-28 19:12:07 +01:00
Michael Eischer
a26d6ffa72 archiver: move deviceID handling behind feature flag 2024-03-28 19:12:07 +01:00
Michael Eischer
2ba21fe72b archiver: only store deviceID for hardlinks 
The deviceID can change e.g. when backing up from filesystem snapshot.
It is only used for hardlink detection. Thus there it is not necessary
to store it for everything else.
 2024-03-28 19:12:07 +01:00
Michael Eischer
8876e3025b
Merge pull request #4724 from MichaelEischer/disable-old-index 
Deprecated legacy index format
 2024-03-28 18:24:47 +01:00
Adam Eijdenberg
6e775d3787 Enhancement: option to send HTTP over unix socket 
add tests for unix socket connection

switch HTTP rest-server test to use any free port

allow rest-server test graceful shutdown opportunity
 2024-03-28 17:41:41 +01:00
avoidalone
ac948fccda fix some typos 
Signed-off-by: avoidalone <wuguangdong@outlook.com>
 2024-03-11 14:35:12 +08:00
Michael Eischer
f8852f0eb6 repair index: fix deletion of legacy indexes 2024-03-09 18:21:22 +01:00
Michael Eischer
1a8bf358f1 index: deprecate legacy index format 2024-03-09 18:21:14 +01:00
Michael Eischer
a9b64cd7ad features: print warning for stable/depreacted feature flags 2024-03-09 17:29:52 +01:00
Michael Eischer
70839155f2 features: add tests 2024-03-09 17:29:52 +01:00
Michael Eischer
5974a79497 features: add basic feature flag implementation 2024-03-09 17:29:52 +01:00
Michael Eischer
e71660cd1e backup: rename data_added_in_repo statistic to data_added_packed 2024-02-25 20:40:52 +01:00
Aneesh Nireshwalia
062d408987
Clean up SecurityDescriptor helper 2024-02-24 14:23:04 -07:00
Aneesh Nireshwalia
5764300022
Add changelog and fix lint error 2024-02-24 13:47:49 -07:00
Aneesh Nireshwalia
90916f53de
Add test cases for security descriptors 2024-02-24 13:27:01 -07:00
Aneesh Nireshwalia
70cf8e3788
Add support for backup/restore of security descriptors 2024-02-24 13:25:28 -07:00
Aneesh Nireshwalia
e3e59fef24
Fix CombineErrors and fillExtendedAttr error handling 2024-02-24 13:22:34 -07:00
Aneesh Nireshwalia
09ce1b4e58
Create helper for SecurityDescriptor related functions 2024-02-24 13:16:25 -07:00
Michael Eischer
a8f5684f68 archiver: test statistics in snapshot 2024-02-23 22:05:15 +01:00
Michael Eischer
681395955e archiver: test backup summary calculation 2024-02-23 21:46:39 +01:00
Michael Eischer
38f91d3b5e backup: store statistics in snapshot 2024-02-23 20:27:13 +01:00
Michael Eischer
86897314d5 backup: expose data_added_in_repo in JSON output 
The value describes how much data was added after compression.
Previously, it was only available in the text output.
 2024-02-23 20:27:13 +01:00
Michael Eischer
a59f654fa6 archiver: refactor summary collection from ui into the archiver 2024-02-23 20:27:13 +01:00
Michael Eischer
8b1a85711f archiver: unexport save/saveDir/saveTree methods 2024-02-23 20:24:21 +01:00
Aneesh Nireshwalia
4bbd25a37f
Add tests for generic attribute changes 2024-02-22 17:55:50 -07:00
Aneesh Nireshwalia
d4be734c73
Handle readonly empty files in windows 2024-02-22 17:54:43 -07:00
Aneesh Nireshwalia
eeb1aa5388
Add ability to report warnings to terminal 
Report warnings to terminal when unrecognized generic attributes are found in the repository.
 2024-02-22 17:52:26 -07:00
Aneesh Nireshwalia
0962917974
Support windows metadata using generic attribs 
Add new generic_attributes attribute in Node.
Use the generic attributes to add support for creation time and file attributes like hidden, readonly, encrypted in windows. Handle permission errors for readonly files in windows.
Handle backup and restore of encrypted attributes using windows system calls.
 2024-02-22 17:31:20 -07:00
Aneesh Nireshwalia
62a8a599f1
Add optional messages for Equals helper 2024-02-22 16:58:12 -07:00
Aneesh Nireshwalia
94de87d4b7
Add CombineErrors helper function 2024-02-22 16:57:00 -07:00
Michael Eischer
4c3218ef9f repository: include packID in StreamPack for decrypt/decompress errors 2024-02-17 19:38:01 +01:00
Michael Eischer
18b0bbbf42 repository: use fmt.Errorf in StreamPacks 2024-02-17 19:37:32 +01:00
Michael Eischer
ed4a4f8748 check: exclude inaccessible files from the repair pack suggestion 2024-02-12 20:25:15 +01:00
Michael Eischer
4073299a7c check: fix missing error if blob is invalid 2024-02-12 20:20:13 +01:00
Michael Eischer
22a3cea1b3 checker: wrap all pack errors in ErrPackData 2024-02-12 20:19:32 +01:00
Michael Eischer
5b5d506472 backup: report files whose chunks failed to upload 2024-02-11 22:43:26 +01:00
Alexander Neumann
c0514dd8ba Fix linter errors (except for tests) 2024-02-10 22:58:10 +01:00
Michael Eischer
5957417b1f Apply changelog entry / documentation improvements from review 2024-02-04 18:55:41 +01:00
Michael Eischer
86b38a0b17 rename --no-verify-pack to --no-extra-verify 2024-02-04 17:01:05 +01:00
Michael Eischer
c32e5e2abb pack: verify integrity of pack file header 2024-02-04 15:31:42 +01:00
Michael Eischer
c97a271e89 repository: ask users to report corrupted data while saving blobs 2024-02-04 15:31:42 +01:00
Michael Eischer
193140525c repository: test verification of blobs/unpacked data 2024-02-04 15:31:42 +01:00
Michael Eischer
2dbb18128c repository: Allow skipping verification for tests 
Some tests have to explicitly create pack files with blobs that don't
match their ID. For those blobs the builtin verification of the
repository must be disabled.
 2024-02-03 18:22:47 +01:00
Michael Eischer
30a84e9003 backup: verify unpacked files before upload 2024-02-03 18:22:47 +01:00
Michael Eischer
c01a0c6da7 backup: verify blobs before upload 
This only covers the blobs themselves, the pack header is not verified
so far. Unpacked files are also not covered by the integrity check.
 2024-02-03 18:22:47 +01:00
Michael Eischer
16e3f79e8b repository: make repo.Options configurable for test repos 2024-02-03 18:22:47 +01:00
Michael Eischer
bb92b487f7 repository: fix repack test 2024-02-03 18:22:47 +01:00
Michael Eischer
e44e4b00a6
Merge pull request #4550 from ndecker/ls-ncdu 
Ls ncdu
 2024-01-27 12:27:35 +00:00
Michael Eischer
d4ed7c8858 walker: add tests for leaveDir 2024-01-27 13:17:33 +01:00
Michael Eischer
2c80cfa4a5 walker: fix missing leaveDir if directory is partially skipped 2024-01-27 13:17:33 +01:00
Michael Eischer
9ecbda059c walker: add callback to inform about leaving a directory 2024-01-27 13:17:32 +01:00
Michael Eischer
246559e654 check: cleanup s3 legacy detection 2024-01-27 13:02:04 +01:00
Michael Eischer
1dfd854769 lock: switch to repository.List 2024-01-27 13:02:04 +01:00
Michael Eischer
bfb56b78e1 replace some usages of restic.Repository with more specific interface 
This should eventually make it easier to test the code.
 2024-01-27 13:02:02 +01:00
Michael Eischer
3424088274
Merge pull request #4644 from MichaelEischer/refactor-repair-packs 
Refactor and test `repair packs`
 2024-01-27 13:00:51 +01:00
Michael Eischer
724ec179e3
Merge pull request #4648 from MichaelEischer/repository-removekey 
repository: Introduce RemoveKey function
 2024-01-27 13:00:42 +01:00
Michael Eischer
f0e1ad2285 fix linter warning 2024-01-27 12:51:45 +01:00
Michael Eischer
fd579421dd repository: deduplicate test 2024-01-27 12:51:45 +01:00
Michael Eischer
42c9318b9c repair pack: add tests 2024-01-27 12:51:45 +01:00
Michael Eischer
764b0bacd6 repair pack: add support for truncated files 2024-01-27 12:51:45 +01:00
Michael Eischer
7c351bc53c repair pack: reenable auto index updates 
The method is not available on the restic.Repository interface that is
used for testing. Drop the call as a small amount of additional index
writes is not a problem.
 2024-01-27 12:51:45 +01:00
Michael Eischer
feeab84204 repair pack: extract the repair logic into the repository package 
Currently, the cmd/restic package contains a significant amount of code
that modifies repository internals. This code should in the mid-term
move into the repository package.
 2024-01-27 12:51:45 +01:00