octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-27 07:16:40 +00:00
Code Issues Releases Wiki Activity
5,407 Commits 7 Branches 33 Tags 66 MiB
54d58edacc
Commit Graph

1006 Commits

Author SHA1 Message Date
Alexander Neumann
bdfedf1f5b
Merge pull request #3173 from MichaelEischer/unify-index-loading 
Unify index loading
 2021-01-28 13:50:42 +01:00
Michael Eischer
e2b0072441 check: add progress bar to the tree structure check 2021-01-28 11:10:50 +01:00
Michael Eischer
505f8a2229 progress/counter: Support updating the progress bar maximum 2021-01-28 11:10:47 +01:00
Michael Eischer
eda8c67616 restic: let FindUsedBlobs handle multiple snapshots at once 2021-01-28 11:08:43 +01:00
Michael Eischer
258ce0c1e5 parallel: report progress for StreamTrees 
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
 2021-01-28 11:08:43 +01:00
Michael Eischer
3d6a3e2555 copy: Remove treeCloner struct 2021-01-28 11:08:43 +01:00
Michael Eischer
0caad1e890 copy: parallelize tree walk 2021-01-28 11:08:43 +01:00
Alexander Neumann
a4689eb3b9
Merge pull request #3199 from MichaelEischer/non-interactive-counter 
Don't print progress on non-interactive terminals
 2021-01-28 10:53:38 +01:00
Alexander Weiss
d7dc19a496 prune: Always repack packs containing tree blobs 2021-01-15 16:42:04 +01:00
Michael Eischer
f3442ce8a5 Test that WriteTo of a backend's Load remains accessible 2021-01-03 22:23:53 +01:00
Michael Eischer
97388b3504 Properly check that --quiet and --verbose are not combined 
If --verbose is specified once, then globalOptions.Verbose == 1.
Previously --quiet --verbose would silently ignore the --verbose flag.
 2020-12-30 21:24:18 +01:00
MichaelEischer
e1efc193e1
Merge pull request #3139 from aawsome/prune-healing 
prune: Add healing of repository in some situations
 2020-12-29 22:17:27 +01:00
Alexander Weiss
f0113139ea prune: Correct error message 2020-12-29 20:20:05 +01:00
Alexander Weiss
f6df94a50e prune: Add self-healing 
Allow prune to heal situations where blobs in the index are missing or
the corresponding packfiles are damaged if those blobs are not needed.
 2020-12-29 20:20:05 +01:00
MichaelEischer
31e56f1ad5
Merge pull request #3197 from SkYNewZ/fix/3183 
Fix tag handling for multiple tag lists
 2020-12-29 18:44:38 +01:00
greatroar
dec5008369 Warn when unlock fails instead of returning an error 
Only one caller was checking the error.
 2020-12-29 17:48:20 +01:00
Quentin Lemaire
25ecf9eafb
fix(cmd_tag): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Quentin Lemaire
e88f3fb80c
fix(cmd_backup): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Michael Eischer
05a987b07c Support values less than 1 for RESTIC_PROGRESS_FPS 
For example set the variable to 0.016666 to print the progress once per
minute.
 2020-12-29 16:32:18 +01:00
Michael Eischer
85fe5feadb Unify progress report frequency calculation 2020-12-29 16:03:43 +01:00
Michael Eischer
969141b5e9 Honor RESTIC_PROGRESS_FPS env variable on non-interactive terminals 
This makes it possible to use the environment variable to also get
regular progress updates on non-interactive terminals.
 2020-12-29 16:03:43 +01:00
Michael Eischer
c2ef049f1b ui/progress: don't print progress on non-interactive terminals 
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058.
 2020-12-29 16:03:43 +01:00
greatroar
18531e3d6f Portability fixes to internal/restic 
syscall.Mknod is not available on AIX.
 2020-12-29 01:35:01 +01:00
DRON-666
332b1896d1 Some options fixes 
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
 2020-12-23 23:26:04 +03:00
Alexander Weiss
d8d2cc6dd9 Simplify rebuild-index code 2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Michael Eischer
ccc84af73d debug/list: parallelize index loading 2020-12-22 22:36:18 +01:00
Alexander Neumann
edf89e1c74 forget: Enable --dry-run together with --prune 2020-12-22 20:58:02 +01:00
Michael Eischer
cfea79d0c5 prune: don't print stacktrace on console 2020-12-19 14:28:48 +01:00
DRON-666
2ca76afc2b dump: Add new option --archive 2020-12-19 01:09:47 +03:00
DRON-666
0256f95994 dump: Split tar and walk logic 2020-12-19 01:09:47 +03:00
Alexander Neumann
0d5b764f90
Merge pull request #3130 from aawsome/snapshots-parallel 
Make loading snapshots parallel
 2020-12-06 21:08:18 +01:00
Alexander Weiss
e329623771 Remove LoadAllSnapshots 2020-12-06 05:22:27 +01:00
Alexander Weiss
5b9ee56335 Add ForAllSnapshots 2020-12-06 05:04:21 +01:00
Alexander Weiss
83c8a9b058 Bugfix: packSizeFromList should save size from List() 2020-12-05 20:58:36 +01:00
Michael Eischer
f72f6c9c80 Fix debug build 2020-11-29 18:44:36 +01:00
MichaelEischer
52b98f7f95
Merge pull request #3017 from greatroar/files-from0 
Add backup options --files-from-verbatim and --files-from-raw
 2020-11-29 18:15:21 +01:00
Alexander Neumann
04d856e601
Merge pull request #3136 from restic/rawtaz-copy-doc 
doc: Emphasize double transfer and duplication in copy command
 2020-11-29 13:57:10 +01:00
Alexander Neumann
a7b49c4889
Merge pull request #3119 from restic/keep-mountpoints 
Keep mountpoints as empty directories for --one-file-system
 2020-11-29 11:22:12 +01:00
Leo R. Lundgren
f70b10d0ee doc: Emphasize double transfer and duplication in copy command 2020-11-28 19:49:34 +01:00
greatroar
55bf76ba0c backup: Add --files-from-{verbatim,raw} options 2020-11-28 18:22:31 +01:00
Alexander Neumann
82ae942965 backup: Keep mountpoints for --one-file-system 
When a file system is mounted at a directory, lstat() returns attributes
of the root node of the mounted file system, including the device ID of
the other file system. The previous code used when --one-file-system is
specified excluded the directory itself because of that.

This commit changes the code so that mountpoints are kept as empty
directories, its attributes set to the root note of the mounted file
system. The behavior mimics `tar`, which does the same.
 2020-11-28 17:00:31 +01:00
Alexander Neumann
f576d3d826 Add tests 2020-11-28 17:00:31 +01:00
Alexander Neumann
037f0a4c91 Refactor device ID checking 2020-11-28 17:00:31 +01:00
Alexander Neumann
a3105799c9 Print warning when unlocking the repo fails 2020-11-28 11:31:25 +01:00
Kyle Brennan
666768cd17 cat: Respect --no-lock flag 2020-11-28 11:30:02 +01:00
Alexander Weiss
aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Weiss
67c938f232 Use PackSize in prune 2020-11-21 22:13:54 +01:00
Alexander Weiss
4960b841e6 Use PackSize in rebuild-index 2020-11-21 22:13:54 +01:00
Alexander Neumann
110a32a08b
Merge pull request #3113 from aawsome/fix-prune-stats-duplicates 
Fix statistics in prune for duplicates
 2020-11-19 20:34:11 +01:00
Alexander Weiss
cb5ec7ea6b Fix statistics in prune for duplicates 
Note that this fix only solves the statistics problem, if
all duplicates are marked for repacking.

If not all duplicates are marked for repacking, we lack the
information which

The situation that not all duplicates are marked for repacking can occur
when using the `max-repack-size` option
 2020-11-18 22:30:22 +01:00
Michael Eischer
625410f003 Limit progress bar updates to once per second on non-terminal outputs 
The code accidentally checked whether stdin is a terminal instead of
stdout, the former is not relevant here as the output is printed on
stdout.
 2020-11-18 22:12:07 +01:00
Alexander Neumann
d857fb6e59 Allow using --repository-file in init 2020-11-17 20:43:46 +01:00
Michael Eischer
1f43cac12d check: Only track data blobs when unused blobs should be reported 
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
 2020-11-15 18:43:07 +01:00
Michael Eischer
3500f9490c check: Simplify blob status tracking 
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
 2020-11-15 18:42:42 +01:00
Michael Eischer
caac38ed27 check: Tweak subset percentage over 100% error message 2020-11-15 18:13:50 +01:00
fgma
8f9cea8cc0 Check data subset: check random percentage subset 2020-11-15 18:13:50 +01:00
Alexander Neumann
3c0c0c132b
Merge pull request #3006 from aawsome/new-rebuild-index 
Reimplement rebuild-index and remove /internal/index
 2020-11-15 17:48:43 +01:00
kitone
3c03b35212 Fix #1681 should not try to create the mount point if it doesn't exist, rather return an error 2020-11-15 17:09:30 +01:00
Alexander Weiss
9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a Reimplement rebuild-index 2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5 Add extraObsolete to MasterIndex.Save 2020-11-15 07:05:09 +01:00
MichaelEischer
3601a9b6cd
Merge pull request #2690 from SkYNewZ/master 
Fix #2688: Handle comma-separated list tags when using backup command
 2020-11-14 23:19:42 +01:00
Michael Eischer
fdec8051ab tags: Tweak description of --add/set/remove options 2020-11-14 22:55:30 +01:00
MichaelEischer
333c5a19d4
Merge pull request #3082 from aawsome/check-sizes 
Check: check sizes of packs from index and packheader
 2020-11-14 22:37:42 +01:00
Quentin Lemaire
a8ad6b9a4b
fix(tags): Change tags list type according to restic.TagList 2020-11-14 16:24:58 +00:00
Quentin Lemaire
b0882b3f3c
fix(snapshots): Update help message to better match the 'forget' command one 2020-11-14 15:48:57 +00:00
Quentin Lemaire
ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list 2020-11-14 15:48:56 +00:00
Alexander Weiss
80dcfca191 check: Check sizes computed from index and pack header 2020-11-14 00:42:49 +01:00
Alexander Neumann
59ea5a4208 diff: Correctly count top-level blobs 2020-11-13 21:11:21 +01:00
greatroar
63e32c44c0 Improve error reporting from restic diff 
Instead of a stacktrace, restic diff 111 222 now reports:

	Fatal: no matching ID found for prefix "111"
 2020-11-11 16:40:40 +01:00
greatroar
21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter 
This fixes two race conditions while cleaning up the code.
 2020-11-09 12:12:35 +01:00
Alexander Neumann
a8d21b5dcf prune: Warn if no cache is present 2020-11-08 20:25:35 +01:00
Alexander Neumann
823d0afd6e backup: Always print parent snapshot info 2020-11-08 20:25:35 +01:00
Alexander Neumann
47277c4b4c Add comments, clarify computation 2020-11-06 20:23:30 +01:00
Alexander Weiss
d2e53730d6 Add test that repo.List is only called once 2020-11-06 20:23:30 +01:00
Alexander Weiss
fd33030556 Use in-memory index to rebuild index in prune 2020-11-06 20:23:30 +01:00
Alexander Neumann
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command 
Reimplementation of prune
 2020-11-05 10:12:19 +01:00
Alexander Neumann
d35d279455 Set development version for 0.11.0 2020-11-05 09:41:40 +01:00
Alexander Neumann
b7e1ece1e0 Add version for 0.11.0 2020-11-05 09:41:35 +01:00
fgma
916b2d303b
vss: fix potential crash (not reachable in restic) (#3045) 
HasSufficientPrivilegesForVSS() now returns an error
 2020-11-04 22:14:18 +01:00
Alexander Neumann
1ca60bccfb Refactor condition for MaxRepackBytes 
Don't depend on the string (opts.MaxRepackSize) for the condition,
instead check if there's a (positive) limit configured.
 2020-11-03 16:42:21 +01:00
Alexander Neumann
7f86eb4ec0 Move helper function 2020-11-03 16:42:21 +01:00
Alexander Neumann
c1a3de4a6e Refactor max-unused calculation, add unlimited option 
Add a callback to the PruneOptions struct which calculates the number of
bytes allowed to be unused after prune is done. This way, the logic is
closer to the option parsing code.

Also, add an explicit option `unlimited` for the use case when storage
does not matter but bandwidth and time do. Internally, this sets the
maximum number of unused bytes to MaxUint64.

Rework the documentation slightly so that no more "packs" are
mentioned and it talks about "files" instead.

Make it clear in the documentation that the percentage given to
`--max-unused` is relative to the whole repository size after pruning is
done. If specified, it must be below 100%, otherwise the repository
would contain 100% of unused data, which is pointless.

I had a hard time coming up with the correct formula to calculate the
maximum number of unused bytes based on the number of used bytes. For a
fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of
unused bytes `x` the following holds:

      x ≤ p * (u+x)
    ⇔ x ≤ p*u + p*x
    ⇔ x - p*x ≤ p*u
    ⇔ x * (1-p) ≤ p*u
    ⇔ x ≤ p/(1-p) * u
 2020-11-03 16:42:21 +01:00
Alexander Neumann
f8c4dd7b1a Split packe rewrite logic into two case branches 
The comma is too sublte, let's split this into two separate branches.
 2020-11-03 16:42:21 +01:00
Alexander Neumann
a5b80452fe Add comment that usedBlobs is modified 2020-11-03 16:42:21 +01:00
Alexander Neumann
aff1e220f5 Split struct members, add comments 2020-11-03 16:42:21 +01:00
Alexander Neumann
095155d9ce Remove RepackSmall 2020-11-03 16:42:21 +01:00
Alexander Weiss
b2f5381737 Make realistic forget --prune --dryrun 2020-11-03 16:42:21 +01:00
Alexander Weiss
7f9a0a5907 Reimplementation of prune 2020-11-03 16:42:21 +01:00
Alexander Weiss
3b591ed987 Add Verboseff 2020-11-03 16:42:21 +01:00
Alexander Neumann
21ba15577e Return an error for invalid sizes 2020-11-03 11:53:49 +01:00
Alexander Neumann
6509c207f4
Merge pull request #2997 from MichaelEischer/faster-excludes 
Speedup exclude/include checking
 2020-11-02 11:07:21 +01:00
Alexander Neumann
445b845267
Merge pull request #2978 from MichaelEischer/warn-tree-error 
Warn if backup failed to read tree blob
 2020-11-02 10:14:12 +01:00
Alexander Neumann
3ff37215df
Merge pull request #2935 from MichaelEischer/upgrade-minio 
Upgrade minio SDK to version 7
 2020-11-02 09:09:10 +01:00
MichaelEischer
31b8d7a639
Merge pull request #2274 from fgma/master 
Support for Volume Shadow Copy Service (VSS) on windows
 2020-10-24 15:43:34 +02:00
fgma
5695f9ebd2 vss: Implement VSS support for Windows 
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.

Snapshotting volumes also covers mountpoints but skips UNC paths.
 2020-10-24 11:35:57 +02:00
greatroar
3807d13bdc restic mount: don't umount unless we actually tried to mount 
Previously, failure to open a repository would result in an "unable to
umount" message.
 2020-10-21 12:15:05 +02:00