octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-16 01:57:10 +00:00
Code Issues Releases Wiki Activity
5,466 Commits 7 Branches 33 Tags 66 MiB
5dad45f005
Commit Graph

1020 Commits

Author SHA1 Message Date
Max Stabel
f9e1fa26ff Fix missing rand seed for restic check --read-data-subset=x% 2021-01-29 23:18:35 +01:00
Alexander Weiss
e08e65dc30 prune: Simplify logic selecting packs to repack 2021-01-29 22:27:22 +01:00
Alexander Weiss
daeb4cdf8f prune: Fix statistics for --repack-cacheable-only 2021-01-29 22:27:22 +01:00
Alexander Neumann
a16ce65295
Merge pull request #3244 from MichaelEischer/better-damage-reports 
Print more details about possible repository damages
 2021-01-29 11:45:45 +01:00
rawtaz
de7e3a0648
Merge pull request #2823 from greatroar/trust-mtime 
Add --ignore-ctime flag to backup and document change detection
 2021-01-29 00:02:19 +01:00
greatroar
6bd8a2faaa backup: Add --ignore-ctime option and document change detection 2021-01-28 23:42:10 +01:00
Michael Eischer
58b5679f14 prune: reword missing blobs error 
The previous wording could be understood such that the prune run did
damage the repository.
 2021-01-28 21:48:24 +01:00
Michael Eischer
7b8886c052 prune: report missing but unneeded pack files 
This indicates a damaged repository so add some output to help with
debugging.
 2021-01-28 21:46:01 +01:00
Michael Eischer
ff95999246 rebuild-index: report added/removed/reindexed files 
This should help with investigating missing pack files.
 2021-01-28 21:46:01 +01:00
Michael Eischer
b71c52797a find: correctly expand multiple blob ids 
For example `restic find --show-pack-id --blob f78dc991 5b9e4366 ddd8c7d4`
would previously only expand one blob if all of them belong to the same
file.
 2021-01-28 21:21:54 +01:00
MichaelEischer
82140967d3
Merge pull request #3228 from aawsome/prune-all-trees 
prune: Remove all unused trees
 2021-01-28 21:04:27 +01:00
MichaelEischer
35033d9b79
Merge pull request #3177 from MichaelEischer/fix-2759 
prune: don't print stacktrace on console
 2021-01-28 20:28:45 +01:00
Alexander Neumann
4d40c70214
Merge pull request #3211 from MichaelEischer/sftp-speedup 
Speed-up caching/pack download via SFTP
 2021-01-28 14:16:55 +01:00
Alexander Neumann
44169d0dc4
Merge pull request #3205 from MichaelEischer/fix-quiet-verbose 
Properly check that --quiet and --verbose are not combined
 2021-01-28 13:53:31 +01:00
Alexander Neumann
bdfedf1f5b
Merge pull request #3173 from MichaelEischer/unify-index-loading 
Unify index loading
 2021-01-28 13:50:42 +01:00
Michael Eischer
e2b0072441 check: add progress bar to the tree structure check 2021-01-28 11:10:50 +01:00
Michael Eischer
505f8a2229 progress/counter: Support updating the progress bar maximum 2021-01-28 11:10:47 +01:00
Michael Eischer
eda8c67616 restic: let FindUsedBlobs handle multiple snapshots at once 2021-01-28 11:08:43 +01:00
Michael Eischer
258ce0c1e5 parallel: report progress for StreamTrees 
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
 2021-01-28 11:08:43 +01:00
Michael Eischer
3d6a3e2555 copy: Remove treeCloner struct 2021-01-28 11:08:43 +01:00
Michael Eischer
0caad1e890 copy: parallelize tree walk 2021-01-28 11:08:43 +01:00
Alexander Neumann
a4689eb3b9
Merge pull request #3199 from MichaelEischer/non-interactive-counter 
Don't print progress on non-interactive terminals
 2021-01-28 10:53:38 +01:00
Alexander Weiss
d7dc19a496 prune: Always repack packs containing tree blobs 2021-01-15 16:42:04 +01:00
Michael Eischer
f3442ce8a5 Test that WriteTo of a backend's Load remains accessible 2021-01-03 22:23:53 +01:00
Michael Eischer
97388b3504 Properly check that --quiet and --verbose are not combined 
If --verbose is specified once, then globalOptions.Verbose == 1.
Previously --quiet --verbose would silently ignore the --verbose flag.
 2020-12-30 21:24:18 +01:00
MichaelEischer
e1efc193e1
Merge pull request #3139 from aawsome/prune-healing 
prune: Add healing of repository in some situations
 2020-12-29 22:17:27 +01:00
Alexander Weiss
f0113139ea prune: Correct error message 2020-12-29 20:20:05 +01:00
Alexander Weiss
f6df94a50e prune: Add self-healing 
Allow prune to heal situations where blobs in the index are missing or
the corresponding packfiles are damaged if those blobs are not needed.
 2020-12-29 20:20:05 +01:00
MichaelEischer
31e56f1ad5
Merge pull request #3197 from SkYNewZ/fix/3183 
Fix tag handling for multiple tag lists
 2020-12-29 18:44:38 +01:00
greatroar
dec5008369 Warn when unlock fails instead of returning an error 
Only one caller was checking the error.
 2020-12-29 17:48:20 +01:00
Quentin Lemaire
25ecf9eafb
fix(cmd_tag): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Quentin Lemaire
e88f3fb80c
fix(cmd_backup): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Michael Eischer
05a987b07c Support values less than 1 for RESTIC_PROGRESS_FPS 
For example set the variable to 0.016666 to print the progress once per
minute.
 2020-12-29 16:32:18 +01:00
Michael Eischer
85fe5feadb Unify progress report frequency calculation 2020-12-29 16:03:43 +01:00
Michael Eischer
969141b5e9 Honor RESTIC_PROGRESS_FPS env variable on non-interactive terminals 
This makes it possible to use the environment variable to also get
regular progress updates on non-interactive terminals.
 2020-12-29 16:03:43 +01:00
Michael Eischer
c2ef049f1b ui/progress: don't print progress on non-interactive terminals 
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058.
 2020-12-29 16:03:43 +01:00
greatroar
18531e3d6f Portability fixes to internal/restic 
syscall.Mknod is not available on AIX.
 2020-12-29 01:35:01 +01:00
DRON-666
332b1896d1 Some options fixes 
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
 2020-12-23 23:26:04 +03:00
Alexander Weiss
d8d2cc6dd9 Simplify rebuild-index code 2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Michael Eischer
ccc84af73d debug/list: parallelize index loading 2020-12-22 22:36:18 +01:00
Alexander Neumann
edf89e1c74 forget: Enable --dry-run together with --prune 2020-12-22 20:58:02 +01:00
Michael Eischer
cfea79d0c5 prune: don't print stacktrace on console 2020-12-19 14:28:48 +01:00
DRON-666
2ca76afc2b dump: Add new option --archive 2020-12-19 01:09:47 +03:00
DRON-666
0256f95994 dump: Split tar and walk logic 2020-12-19 01:09:47 +03:00
Alexander Neumann
0d5b764f90
Merge pull request #3130 from aawsome/snapshots-parallel 
Make loading snapshots parallel
 2020-12-06 21:08:18 +01:00
Alexander Weiss
e329623771 Remove LoadAllSnapshots 2020-12-06 05:22:27 +01:00
Alexander Weiss
5b9ee56335 Add ForAllSnapshots 2020-12-06 05:04:21 +01:00
Alexander Weiss
83c8a9b058 Bugfix: packSizeFromList should save size from List() 2020-12-05 20:58:36 +01:00
Michael Eischer
f72f6c9c80 Fix debug build 2020-11-29 18:44:36 +01:00
MichaelEischer
52b98f7f95
Merge pull request #3017 from greatroar/files-from0 
Add backup options --files-from-verbatim and --files-from-raw
 2020-11-29 18:15:21 +01:00
Alexander Neumann
04d856e601
Merge pull request #3136 from restic/rawtaz-copy-doc 
doc: Emphasize double transfer and duplication in copy command
 2020-11-29 13:57:10 +01:00
Alexander Neumann
a7b49c4889
Merge pull request #3119 from restic/keep-mountpoints 
Keep mountpoints as empty directories for --one-file-system
 2020-11-29 11:22:12 +01:00
Leo R. Lundgren
f70b10d0ee doc: Emphasize double transfer and duplication in copy command 2020-11-28 19:49:34 +01:00
greatroar
55bf76ba0c backup: Add --files-from-{verbatim,raw} options 2020-11-28 18:22:31 +01:00
Alexander Neumann
82ae942965 backup: Keep mountpoints for --one-file-system 
When a file system is mounted at a directory, lstat() returns attributes
of the root node of the mounted file system, including the device ID of
the other file system. The previous code used when --one-file-system is
specified excluded the directory itself because of that.

This commit changes the code so that mountpoints are kept as empty
directories, its attributes set to the root note of the mounted file
system. The behavior mimics `tar`, which does the same.
 2020-11-28 17:00:31 +01:00
Alexander Neumann
f576d3d826 Add tests 2020-11-28 17:00:31 +01:00
Alexander Neumann
037f0a4c91 Refactor device ID checking 2020-11-28 17:00:31 +01:00
Alexander Neumann
a3105799c9 Print warning when unlocking the repo fails 2020-11-28 11:31:25 +01:00
Kyle Brennan
666768cd17 cat: Respect --no-lock flag 2020-11-28 11:30:02 +01:00
Alexander Weiss
aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Weiss
67c938f232 Use PackSize in prune 2020-11-21 22:13:54 +01:00
Alexander Weiss
4960b841e6 Use PackSize in rebuild-index 2020-11-21 22:13:54 +01:00
Alexander Neumann
110a32a08b
Merge pull request #3113 from aawsome/fix-prune-stats-duplicates 
Fix statistics in prune for duplicates
 2020-11-19 20:34:11 +01:00
Alexander Weiss
cb5ec7ea6b Fix statistics in prune for duplicates 
Note that this fix only solves the statistics problem, if
all duplicates are marked for repacking.

If not all duplicates are marked for repacking, we lack the
information which

The situation that not all duplicates are marked for repacking can occur
when using the `max-repack-size` option
 2020-11-18 22:30:22 +01:00
Michael Eischer
625410f003 Limit progress bar updates to once per second on non-terminal outputs 
The code accidentally checked whether stdin is a terminal instead of
stdout, the former is not relevant here as the output is printed on
stdout.
 2020-11-18 22:12:07 +01:00
Alexander Neumann
d857fb6e59 Allow using --repository-file in init 2020-11-17 20:43:46 +01:00
Michael Eischer
1f43cac12d check: Only track data blobs when unused blobs should be reported 
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
 2020-11-15 18:43:07 +01:00
Michael Eischer
3500f9490c check: Simplify blob status tracking 
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
 2020-11-15 18:42:42 +01:00
Michael Eischer
caac38ed27 check: Tweak subset percentage over 100% error message 2020-11-15 18:13:50 +01:00
fgma
8f9cea8cc0 Check data subset: check random percentage subset 2020-11-15 18:13:50 +01:00
Alexander Neumann
3c0c0c132b
Merge pull request #3006 from aawsome/new-rebuild-index 
Reimplement rebuild-index and remove /internal/index
 2020-11-15 17:48:43 +01:00
kitone
3c03b35212 Fix #1681 should not try to create the mount point if it doesn't exist, rather return an error 2020-11-15 17:09:30 +01:00
Alexander Weiss
9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a Reimplement rebuild-index 2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5 Add extraObsolete to MasterIndex.Save 2020-11-15 07:05:09 +01:00
MichaelEischer
3601a9b6cd
Merge pull request #2690 from SkYNewZ/master 
Fix #2688: Handle comma-separated list tags when using backup command
 2020-11-14 23:19:42 +01:00
Michael Eischer
fdec8051ab tags: Tweak description of --add/set/remove options 2020-11-14 22:55:30 +01:00
MichaelEischer
333c5a19d4
Merge pull request #3082 from aawsome/check-sizes 
Check: check sizes of packs from index and packheader
 2020-11-14 22:37:42 +01:00
Quentin Lemaire
a8ad6b9a4b
fix(tags): Change tags list type according to restic.TagList 2020-11-14 16:24:58 +00:00
Quentin Lemaire
b0882b3f3c
fix(snapshots): Update help message to better match the 'forget' command one 2020-11-14 15:48:57 +00:00
Quentin Lemaire
ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list 2020-11-14 15:48:56 +00:00
Alexander Weiss
80dcfca191 check: Check sizes computed from index and pack header 2020-11-14 00:42:49 +01:00
Alexander Neumann
59ea5a4208 diff: Correctly count top-level blobs 2020-11-13 21:11:21 +01:00
greatroar
63e32c44c0 Improve error reporting from restic diff 
Instead of a stacktrace, restic diff 111 222 now reports:

	Fatal: no matching ID found for prefix "111"
 2020-11-11 16:40:40 +01:00
greatroar
21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter 
This fixes two race conditions while cleaning up the code.
 2020-11-09 12:12:35 +01:00
Alexander Neumann
a8d21b5dcf prune: Warn if no cache is present 2020-11-08 20:25:35 +01:00
Alexander Neumann
823d0afd6e backup: Always print parent snapshot info 2020-11-08 20:25:35 +01:00
Alexander Neumann
47277c4b4c Add comments, clarify computation 2020-11-06 20:23:30 +01:00
Alexander Weiss
d2e53730d6 Add test that repo.List is only called once 2020-11-06 20:23:30 +01:00
Alexander Weiss
fd33030556 Use in-memory index to rebuild index in prune 2020-11-06 20:23:30 +01:00
Alexander Neumann
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command 
Reimplementation of prune
 2020-11-05 10:12:19 +01:00
Alexander Neumann
d35d279455 Set development version for 0.11.0 2020-11-05 09:41:40 +01:00
Alexander Neumann
b7e1ece1e0 Add version for 0.11.0 2020-11-05 09:41:35 +01:00
fgma
916b2d303b
vss: fix potential crash (not reachable in restic) (#3045) 
HasSufficientPrivilegesForVSS() now returns an error
 2020-11-04 22:14:18 +01:00
Alexander Neumann
1ca60bccfb Refactor condition for MaxRepackBytes 
Don't depend on the string (opts.MaxRepackSize) for the condition,
instead check if there's a (positive) limit configured.
 2020-11-03 16:42:21 +01:00
Alexander Neumann
7f86eb4ec0 Move helper function 2020-11-03 16:42:21 +01:00
Alexander Neumann
c1a3de4a6e Refactor max-unused calculation, add unlimited option 
Add a callback to the PruneOptions struct which calculates the number of
bytes allowed to be unused after prune is done. This way, the logic is
closer to the option parsing code.

Also, add an explicit option `unlimited` for the use case when storage
does not matter but bandwidth and time do. Internally, this sets the
maximum number of unused bytes to MaxUint64.

Rework the documentation slightly so that no more "packs" are
mentioned and it talks about "files" instead.

Make it clear in the documentation that the percentage given to
`--max-unused` is relative to the whole repository size after pruning is
done. If specified, it must be below 100%, otherwise the repository
would contain 100% of unused data, which is pointless.

I had a hard time coming up with the correct formula to calculate the
maximum number of unused bytes based on the number of used bytes. For a
fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of
unused bytes `x` the following holds:

      x ≤ p * (u+x)
    ⇔ x ≤ p*u + p*x
    ⇔ x - p*x ≤ p*u
    ⇔ x * (1-p) ≤ p*u
    ⇔ x ≤ p/(1-p) * u
 2020-11-03 16:42:21 +01:00