2
2
mirror of https://github.com/octoleo/restic.git synced 2024-12-05 03:13:18 +00:00
Commit Graph

55 Commits

Author SHA1 Message Date
Michael Eischer
ae1cb889dd Add more checks for canceled contexts 2024-07-31 19:30:47 +02:00
Michael Eischer
0dcac90bea restorer: don't track already uptodate blobs 2024-07-21 12:03:56 +02:00
Michael Eischer
6f8e17a463 restorer: minor code cleanups 2024-07-21 12:03:56 +02:00
Michael Eischer
5d6ce59a8d restorer: also truncate files if their content is already uptodate
Files for which no blobs have to be restored, still have to be truncated
to the correct size. Take a file with content "foobar" that should be
replaced by restore with content "foo". The first three bytes are
already uptodate, such that no data has to be written. As file
truncation normally happens when writing data, a special case is
necessary.

This no blobs written special case is unified with the empty file
special case.
2024-07-21 12:03:56 +02:00
Michael Eischer
2833b2f699 restore: fix progress bar for partially up to date files 2024-07-14 11:21:11 +02:00
Michael Eischer
c77b2d5ca2 restore: avoid long cancelation delay for frequently used blobs 2024-07-14 11:21:11 +02:00
Michael Eischer
f64191da9c restore: improve reporting of cancelation errors 2024-07-14 11:21:08 +02:00
Michael Eischer
2971a769da restore: fix corrupt restore of partially up to date files 2024-07-12 22:27:00 +02:00
Michael Eischer
31887ea9aa restore: fix hang on command cancelation 2024-07-12 22:24:35 +02:00
Michael Eischer
f4b15fdd96 restore: allow deleting a directory to replace it with a file
When the `--delete` option is specified, recursively delete directories
that should be replaced with a file.
2024-07-05 22:38:39 +02:00
Michael Eischer
ae978d60cc restore: pass action enum to restore progress 2024-07-05 20:41:27 +02:00
Michael Eischer
c47bf33884 restore: print processed files at maximum verbosity 2024-07-05 20:41:27 +02:00
Michael Eischer
f1c76a8286 restore: fix corrupted sparse files 2024-06-13 21:17:30 +02:00
Michael Eischer
a66658b4c9 restore: only restore changed file parts
For files that should be overwritten, first check whether their content
matches the snapshot and only restore diverging parts.
2024-06-13 21:17:30 +02:00
Michael Eischer
a23cb3a428 restore: reduce memory usage 2024-06-12 22:36:52 +02:00
Michael Eischer
0fcd89f892 restorer: remove special case for blobs with many occurrences
Loading blobs by now is no longer prone to timeouts when processing
takes a long time.
2024-06-12 22:36:52 +02:00
Michael Eischer
fd2ff464a2 restorer: remove stale comment 2024-06-12 22:36:52 +02:00
Michael Eischer
607daeed4f restore: move nil pointer check into restoreui 2024-06-12 22:36:52 +02:00
Michael Eischer
30320a249a restore: let filerestorer also handle empty files
This get's rid of the corresponding special cases.
2024-06-12 22:36:52 +02:00
Michael Eischer
864995271e repository: unwrap BlobHandle parameters of LookupBlob
The method now uses the same parameters as LookupBlobSize.
2024-05-24 21:33:17 +02:00
Michael Eischer
2c310a526e repository: Replace StreamPack function with LoadBlobsFromPack method
LoadBlobsFromPack is now part of the repository struct. This ensures
that users of that method don't have to deal will internals of the
repository implementation.

The filerestorer tests now also contain far fewer pack file
implementation details.
2024-01-19 21:40:43 +01:00
Michael Eischer
e78be75d1e restore: separately restore blobs that are frequently referenced
Writing these blobs to their files can take a long time and consequently
cause the backend connection to time out. Avoid that by retrieving these
blobs separately.
2024-01-08 21:00:13 +01:00
Michael Eischer
2267910418 restore: split error reporting from downloadPack 2024-01-08 20:57:00 +01:00
Michael Eischer
00d18b7a88 restore: cleanup downloadPack 2024-01-08 20:53:08 +01:00
Michael Eischer
9328f34d43 restore: split downloadPack into smaller methods 2024-01-08 20:52:36 +01:00
Michael Eischer
dac3508170 restore: only report errors for blobs that actually failed to load
Previously, errors would be reported for all blobs of a packfile that
failed to stream. Now, only the not yet processed blobs are reported.
2024-01-07 10:54:56 +01:00
Michael Eischer
94cbc6392d restore: slightly reduce memory usage while restoring files
The information which target files are contained in a pack file is no
longer necessary after processing a pack.
2023-04-23 11:33:21 +02:00
Mark Herrmann
f875a8843d restore: Add progress bar
Co-authored-by: Mark Herrmann <mark.herrmann@mailbox.org>
2023-04-07 12:08:23 +02:00
Michael Eischer
c147422ba5 repository: special case SaveBlob for all zero chunks
Sparse files contain large regions containing only zero bytes. Checking
that a blob only contains zeros is possible with over 100GB/s for modern
x86 CPUs. Calculating sha256 hashes is only possible with 500MB/s (or
2GB/s using hardware acceleration). Thus we can speed up the hash
calculation for all zero blobs (which always have length
chunker.MinSize) by checking for zero bytes and then using the
precomputed hash.

The all zeros check is only performed for blobs with the minimal chunk
size, and thus should add no overhead most of the time. For chunks which
are not all zero but have the minimal chunks size, the overhead will be
below 2% based on the above performance numbers.

This allows reading sparse sections of files as fast as the kernel can
return data to us. On my system using BTRFS this resulted in about
4GB/s.
2022-09-24 21:39:39 +02:00
Michael Eischer
5b6a77058a Enable sparseness only conditionally
We can either preallocate storage for a file or sparsify it. This
detects a pack file as sparse if it contains an all zero block or
consists of only one block. As the file sparsification is just an
approximation, hide it behind a `--sparse` parameter.
2022-09-24 21:20:00 +02:00
Michael Eischer
6f53ecc1ae adapt workers based on whether an operation is CPU or IO-bound
Use runtime.GOMAXPROCS(0) as worker count for CPU-bound tasks,
repo.Connections() for IO-bound task and a combination if a task can be
both. Streaming packs is treated as IO-bound as adding more worker
cannot provide a speedup.

Typical IO-bound tasks are download / uploading / deleting files.
Decoding / Encoding / Verifying are usually CPU-bound. Several tasks are
a combination of both, e.g. for combined download and decode functions.
In the latter case add both limits together. As the backends have their
own concurrency limits restic still won't download more than
repo.Connections() files in parallel, but the additional workers can
decode already downloaded data in parallel.
2022-07-03 12:19:26 +02:00
Michael Eischer
6fb408d90e repository: implement pack compression 2022-04-30 11:34:10 +02:00
Michael Eischer
243698680a crypto: Use helpers for size calculations 2022-03-28 22:09:49 +02:00
Michael Eischer
27524979e8 restorer: Remove dead code 2022-02-13 11:43:09 +01:00
Michael Eischer
f40abd92fa restorer: convert to use StreamPack 2022-02-12 20:18:25 +01:00
Alexander Weiss
81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
Michael Eischer
e8bbb05328 restore: Correctly handle partial pack download errors
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
2021-06-29 20:54:16 +02:00
Michael Eischer
47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
Alexander Weiss
5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Michael Eischer
69d5b4c36b restorer: lower-case variable name 2021-01-03 13:55:59 +01:00
Alexander Weiss
34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss
07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss
3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Alexander Weiss
aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Michael Eischer
8cc9514879 restorer: pre-allocate files before loading chunks 2020-09-07 21:41:47 +02:00
Michael Eischer
2e7d475029 Process packs in order of first appearance 2020-09-06 21:25:35 +02:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" (#2885)
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
Alexander Weiss
9d1fb94c6c make Lookup() return all blobs
+ simplify syntax
2020-07-25 21:18:34 +02:00
greatroar
de5516a90e Fix sync.WaitGroup usage in restorer.fileRestorer 2020-02-27 00:07:49 +01:00
Igor Fedorenko
f17ffa0283 restorer: Allow writing target file blobs out of order
Much simpler implementation that guarantees each required pack
is downloaded only once (and hence does not need to manage
pack cache). Also improves large file restore performance.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2020-02-26 16:14:45 +01:00