Michael Eischer
3ba1fa3cee
repository: remove a few global variables
2024-03-28 23:17:02 +01:00
Michael Eischer
044e8bf821
repository: parallelize lock tests
2024-03-28 23:17:02 +01:00
Michael Eischer
e8df50fa3c
repository: remove global list of locks
2024-03-28 22:46:33 +01:00
Michael Eischer
cbb5f89252
lock: move code to repository package
2024-03-28 22:46:33 +01:00
Michael Eischer
7f9ad1c3db
Merge pull request #4705 from MichaelEischer/snapshot-statistics
...
Store snapshot statistics & print snapshot size
2024-03-28 22:41:45 +01:00
Michael Eischer
d705741571
backup: test that deviceID is only stored for hardlinks
2024-03-28 19:32:50 +01:00
Michael Eischer
a9b3d86c4f
features: remove example feature
2024-03-28 19:12:07 +01:00
Michael Eischer
a26d6ffa72
archiver: move deviceID handling behind feature flag
2024-03-28 19:12:07 +01:00
Michael Eischer
2ba21fe72b
archiver: only store deviceID for hardlinks
...
The deviceID can change e.g. when backing up from filesystem snapshot.
It is only used for hardlink detection. Thus there it is not necessary
to store it for everything else.
2024-03-28 19:12:07 +01:00
Michael Eischer
8876e3025b
Merge pull request #4724 from MichaelEischer/disable-old-index
...
Deprecated legacy index format
2024-03-28 18:24:47 +01:00
Adam Eijdenberg
6e775d3787
Enhancement: option to send HTTP over unix socket
...
add tests for unix socket connection
switch HTTP rest-server test to use any free port
allow rest-server test graceful shutdown opportunity
2024-03-28 17:41:41 +01:00
avoidalone
ac948fccda
fix some typos
...
Signed-off-by: avoidalone <wuguangdong@outlook.com>
2024-03-11 14:35:12 +08:00
Michael Eischer
f8852f0eb6
repair index: fix deletion of legacy indexes
2024-03-09 18:21:22 +01:00
Michael Eischer
1a8bf358f1
index: deprecate legacy index format
2024-03-09 18:21:14 +01:00
Michael Eischer
a9b64cd7ad
features: print warning for stable/depreacted feature flags
2024-03-09 17:29:52 +01:00
Michael Eischer
70839155f2
features: add tests
2024-03-09 17:29:52 +01:00
Michael Eischer
5974a79497
features: add basic feature flag implementation
2024-03-09 17:29:52 +01:00
Michael Eischer
e71660cd1e
backup: rename data_added_in_repo statistic to data_added_packed
2024-02-25 20:40:52 +01:00
Michael Eischer
a8f5684f68
archiver: test statistics in snapshot
2024-02-23 22:05:15 +01:00
Michael Eischer
681395955e
archiver: test backup summary calculation
2024-02-23 21:46:39 +01:00
Michael Eischer
38f91d3b5e
backup: store statistics in snapshot
2024-02-23 20:27:13 +01:00
Michael Eischer
86897314d5
backup: expose data_added_in_repo in JSON output
...
The value describes how much data was added after compression.
Previously, it was only available in the text output.
2024-02-23 20:27:13 +01:00
Michael Eischer
a59f654fa6
archiver: refactor summary collection from ui into the archiver
2024-02-23 20:27:13 +01:00
Michael Eischer
8b1a85711f
archiver: unexport save/saveDir/saveTree methods
2024-02-23 20:24:21 +01:00
Aneesh Nireshwalia
4bbd25a37f
Add tests for generic attribute changes
2024-02-22 17:55:50 -07:00
Aneesh Nireshwalia
d4be734c73
Handle readonly empty files in windows
2024-02-22 17:54:43 -07:00
Aneesh Nireshwalia
eeb1aa5388
Add ability to report warnings to terminal
...
Report warnings to terminal when unrecognized generic attributes are found in the repository.
2024-02-22 17:52:26 -07:00
Aneesh Nireshwalia
0962917974
Support windows metadata using generic attribs
...
Add new generic_attributes attribute in Node.
Use the generic attributes to add support for creation time and file attributes like hidden, readonly, encrypted in windows. Handle permission errors for readonly files in windows.
Handle backup and restore of encrypted attributes using windows system calls.
2024-02-22 17:31:20 -07:00
Aneesh Nireshwalia
62a8a599f1
Add optional messages for Equals helper
2024-02-22 16:58:12 -07:00
Aneesh Nireshwalia
94de87d4b7
Add CombineErrors helper function
2024-02-22 16:57:00 -07:00
Michael Eischer
4c3218ef9f
repository: include packID in StreamPack for decrypt/decompress errors
2024-02-17 19:38:01 +01:00
Michael Eischer
18b0bbbf42
repository: use fmt.Errorf in StreamPacks
2024-02-17 19:37:32 +01:00
Michael Eischer
ed4a4f8748
check: exclude inaccessible files from the repair pack suggestion
2024-02-12 20:25:15 +01:00
Michael Eischer
4073299a7c
check: fix missing error if blob is invalid
2024-02-12 20:20:13 +01:00
Michael Eischer
22a3cea1b3
checker: wrap all pack errors in ErrPackData
2024-02-12 20:19:32 +01:00
Michael Eischer
5b5d506472
backup: report files whose chunks failed to upload
2024-02-11 22:43:26 +01:00
Alexander Neumann
c0514dd8ba
Fix linter errors (except for tests)
2024-02-10 22:58:10 +01:00
Michael Eischer
5957417b1f
Apply changelog entry / documentation improvements from review
2024-02-04 18:55:41 +01:00
Michael Eischer
86b38a0b17
rename --no-verify-pack to --no-extra-verify
2024-02-04 17:01:05 +01:00
Michael Eischer
c32e5e2abb
pack: verify integrity of pack file header
2024-02-04 15:31:42 +01:00
Michael Eischer
c97a271e89
repository: ask users to report corrupted data while saving blobs
2024-02-04 15:31:42 +01:00
Michael Eischer
193140525c
repository: test verification of blobs/unpacked data
2024-02-04 15:31:42 +01:00
Michael Eischer
2dbb18128c
repository: Allow skipping verification for tests
...
Some tests have to explicitly create pack files with blobs that don't
match their ID. For those blobs the builtin verification of the
repository must be disabled.
2024-02-03 18:22:47 +01:00
Michael Eischer
30a84e9003
backup: verify unpacked files before upload
2024-02-03 18:22:47 +01:00
Michael Eischer
c01a0c6da7
backup: verify blobs before upload
...
This only covers the blobs themselves, the pack header is not verified
so far. Unpacked files are also not covered by the integrity check.
2024-02-03 18:22:47 +01:00
Michael Eischer
16e3f79e8b
repository: make repo.Options configurable for test repos
2024-02-03 18:22:47 +01:00
Michael Eischer
bb92b487f7
repository: fix repack test
2024-02-03 18:22:47 +01:00
Michael Eischer
e44e4b00a6
Merge pull request #4550 from ndecker/ls-ncdu
...
Ls ncdu
2024-01-27 12:27:35 +00:00
Michael Eischer
d4ed7c8858
walker: add tests for leaveDir
2024-01-27 13:17:33 +01:00
Michael Eischer
2c80cfa4a5
walker: fix missing leaveDir if directory is partially skipped
2024-01-27 13:17:33 +01:00
Michael Eischer
9ecbda059c
walker: add callback to inform about leaving a directory
2024-01-27 13:17:32 +01:00
Michael Eischer
246559e654
check: cleanup s3 legacy detection
2024-01-27 13:02:04 +01:00
Michael Eischer
1dfd854769
lock: switch to repository.List
2024-01-27 13:02:04 +01:00
Michael Eischer
bfb56b78e1
replace some usages of restic.Repository with more specific interface
...
This should eventually make it easier to test the code.
2024-01-27 13:02:02 +01:00
Michael Eischer
3424088274
Merge pull request #4644 from MichaelEischer/refactor-repair-packs
...
Refactor and test `repair packs`
2024-01-27 13:00:51 +01:00
Michael Eischer
724ec179e3
Merge pull request #4648 from MichaelEischer/repository-removekey
...
repository: Introduce RemoveKey function
2024-01-27 13:00:42 +01:00
Michael Eischer
f0e1ad2285
fix linter warning
2024-01-27 12:51:45 +01:00
Michael Eischer
fd579421dd
repository: deduplicate test
2024-01-27 12:51:45 +01:00
Michael Eischer
42c9318b9c
repair pack: add tests
2024-01-27 12:51:45 +01:00
Michael Eischer
764b0bacd6
repair pack: add support for truncated files
2024-01-27 12:51:45 +01:00
Michael Eischer
7c351bc53c
repair pack: reenable auto index updates
...
The method is not available on the restic.Repository interface that is
used for testing. Drop the call as a small amount of additional index
writes is not a problem.
2024-01-27 12:51:45 +01:00
Michael Eischer
feeab84204
repair pack: extract the repair logic into the repository package
...
Currently, the cmd/restic package contains a significant amount of code
that modifies repository internals. This code should in the mid-term
move into the repository package.
2024-01-27 12:51:45 +01:00
Michael Eischer
cb50832d50
index: let MasterIndex.Save also delete obsolete indexes
2024-01-27 12:51:08 +01:00
Michael Eischer
bedff1ed6d
split deleteFiles into UI and logic parts
2024-01-27 12:51:08 +01:00
Michael Eischer
c13bf0b607
repository: Introduce RemoveKey function
...
This replaces directly removing keys via the backend.
2024-01-27 12:42:58 +01:00
Michael Eischer
446167ae80
Merge pull request #4643 from MichaelEischer/remove-redundant-poly1305-mask
...
Remove redundant poly1305 key masking
2024-01-23 19:46:40 +01:00
Michael Eischer
1419baf67a
Merge pull request #4645 from MichaelEischer/improve-lock-checking
...
lock: checkForOtherLocks processes each lock at most once
2024-01-23 19:46:00 +01:00
Michael Eischer
6bdca13603
ui/termstatus: simplify cleaning up on termination
...
`writeStatus` also cleans no longer used status lines.
The old code actually cleaned one line too much. However, as that line
was never used it makes no difference.
2024-01-21 21:27:27 +01:00
Michael Eischer
f1f34eb3e5
lock: checkForOtherLocks processes each lock at most once
...
If a lock could not be loaded, then restic would check all lock files
again. These repeated checks are not useful as the status of a lock file
cannot change unless its ID changes too. Thus, skip already check lock
files on retries.
2024-01-20 22:40:12 +01:00
Michael Eischer
fee83e1c09
Remove redundant poly1305 key masking
...
The implementation in crypto/poly1305 already performs the exact same
masking.
2024-01-20 12:36:59 +01:00
Michael Eischer
a763a5c67d
s3: minor cleanups for archive storage class handling
2024-01-20 11:25:28 +01:00
Vladislav Belous
8ca58b487c
S3: do not set storage class for metadata when using archive storage
2024-01-20 11:04:15 +01:00
Michael Eischer
2c310a526e
repository: Replace StreamPack function with LoadBlobsFromPack method
...
LoadBlobsFromPack is now part of the repository struct. This ensures
that users of that method don't have to deal will internals of the
repository implementation.
The filerestorer tests now also contain far fewer pack file
implementation details.
2024-01-19 21:40:43 +01:00
Michael Eischer
6b7b5c89e9
repository: prepare StreamPack refactor
2024-01-19 21:40:43 +01:00
Michael Eischer
22d0c3f8dc
check: Use PackBlobIterator instead of StreamPack
...
To only stream the content of a pack file once, check used StreamPack
with a custom pack load function. This combination was always brittle
and complicates using StreamPack everywhere else. Now that StreamPack
internally uses PackBlobIterator use that primitive instead, which is a
much better fit for what the check command requires.
2024-01-19 21:40:36 +01:00
Michael Eischer
fb422497af
repository: split StreamPack implementation
...
Move the actual decoding of the pack data into a separate iterator.
2024-01-19 21:39:55 +01:00
Michael Eischer
54c5c72e5a
Merge pull request #4616 from MichaelEischer/fix-rest-connection-close
...
rest: fix and cleanup closing of http response body
2024-01-19 21:31:35 +01:00
Michael Eischer
bd883caae1
CI: enable bodyclose linter
2024-01-19 21:17:18 +01:00
Michael Eischer
b1a8fd1d03
rest: fix and cleanup closing of http response body
...
If client.Do returns an error, then there's no body that has to be
closed. For requests for which we are not interested in the response
body, immediately drain and close the body to make sure it isn't
forgotten later on.
This change in particular adds the missing `Close()` call for the
`List()` command.
2024-01-19 21:17:17 +01:00
Michael Eischer
fdcbb53017
walker: test skipping for root node
2024-01-19 21:16:06 +01:00
Michael Eischer
0b39940fdb
walker: Remove ignoreTrees functionality
...
It was only used in two places:
- stats: apparently as a minor performance optimization, which is
unlikely to be important
- find: filtered directories would be ignored. However, this
optimization missed that it is possible that two directories have the
exact same content. Such directories would be incorrectly ignored too.
Example:
```
mkdir test test/a test/b
restic backup test
restic find latest test/b
-> incorrectly does not return anything
```
Thus, remove the functionality as it's apparently too complex to use
correctly.
2024-01-19 21:16:06 +01:00
Michael Eischer
7b2de84763
Merge pull request #4618 from MichaelEischer/workaround-rclone-list-errors
...
rclone: Workaround for incorrect "not found" errors while listing files
2024-01-09 18:28:31 +01:00
Michael Eischer
c31e9418ba
Merge pull request #4626 from MichaelEischer/reliable-large-restores
...
Improve reliability of large restores
2024-01-09 18:23:09 +01:00
Michael Eischer
2e8de9edfd
rclone: Workaround for incorrect "not found" errors while listing files
...
rclone returns a "not found" error if an internal error occurs while
listing a folder. Ignoring this error lets restic erroneously think that
there are no files, which can cause `prune` to wipe the whole
repository.
2024-01-09 18:20:16 +01:00
Michael Eischer
e78be75d1e
restore: separately restore blobs that are frequently referenced
...
Writing these blobs to their files can take a long time and consequently
cause the backend connection to time out. Avoid that by retrieving these
blobs separately.
2024-01-08 21:00:13 +01:00
Michael Eischer
2267910418
restore: split error reporting from downloadPack
2024-01-08 20:57:00 +01:00
Michael Eischer
00d18b7a88
restore: cleanup downloadPack
2024-01-08 20:53:08 +01:00
Michael Eischer
9328f34d43
restore: split downloadPack into smaller methods
2024-01-08 20:52:36 +01:00
Michael Eischer
77434c6e2b
Merge pull request #4474 from ekristen/aws-assume-role
...
Allow AWS Assume Role
2024-01-08 19:07:17 +00:00
Michael Eischer
e4a7eb09ef
Merge pull request #4624 from MichaelEischer/better-restorer-error-reporting
...
Improver restorer error reporting
2024-01-07 11:20:29 +01:00
Michael Eischer
f8b4e932ef
Merge pull request #4620 from MichaelEischer/improve-irregular-file-handling
...
Improve irregular file handling
2024-01-07 11:12:07 +01:00
Michael Eischer
dac3508170
restore: only report errors for blobs that actually failed to load
...
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
2024-01-07 10:54:56 +01:00
Michael Eischer
77b1c52673
repository: test that StreamPack only delivers blobs once
2024-01-07 10:54:53 +01:00
Michael Eischer
fe5c337ca2
repository: StreamPack delivers blobs at most once
...
If an error occurred while streaming a pack file, this could result in
passing some of the blobs multiple times to the callback function. This
significantly complicates using StreamPack correctly and is unnecessary.
Retries do not change the content of a blob and thus only deliver the
same result over and over again.
2024-01-07 10:54:49 +01:00
Michael Eischer
0360e540af
Merge pull request #4622 from MichaelEischer/fix-outdated-windows-import
...
termstatus: update import path of golang.org/x/term
2024-01-06 23:24:16 +01:00
Michael Eischer
e6dfefba13
termstatus: update import path of golang.org/x/term
2024-01-06 21:59:26 +01:00
Michael Eischer
02bc73f5eb
s3: minor code cleanups
2024-01-06 21:44:53 +01:00
Michael Eischer
20cf4777cb
s3: check for EnvAWS credentials before Static credentials
...
EnvAWS considers more environment variables, including AWS_SESSION_TOKEN
and thus should be checked first.
2024-01-06 21:43:47 +01:00
Erik Kristensen
5ffb536aae
feat: support AWS assume role
2024-01-06 21:19:58 +01:00
Michael Eischer
33b7c84a7a
deduplicate string formatting of snapshot metadata
...
This removes the spurious ")" bracket at the end and normalizes the
metadata format used by the `ls` command.
2024-01-06 20:20:51 +01:00
