octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-16 18:15:20 +00:00
Code Issues Releases Wiki Activity
5,990 Commits 7 Branches 33 Tags 66 MiB
a0fa9c6e9f
Commit Graph

1171 Commits

Author SHA1 Message Date
MichaelEischer
4d576c2f79
Merge pull request #3217 from M4a1x/read-data-subset-fix 
Fix missing rand seed for restic check --read-data-subset=x%
 2021-01-29 23:34:13 +01:00
Max Stabel
f9e1fa26ff Fix missing rand seed for restic check --read-data-subset=x% 2021-01-29 23:18:35 +01:00
Alexander Weiss
e08e65dc30 prune: Simplify logic selecting packs to repack 2021-01-29 22:27:22 +01:00
Alexander Weiss
daeb4cdf8f prune: Fix statistics for --repack-cacheable-only 2021-01-29 22:27:22 +01:00
Alexander Neumann
a16ce65295
Merge pull request #3244 from MichaelEischer/better-damage-reports 
Print more details about possible repository damages
 2021-01-29 11:45:45 +01:00
rawtaz
de7e3a0648
Merge pull request #2823 from greatroar/trust-mtime 
Add --ignore-ctime flag to backup and document change detection
 2021-01-29 00:02:19 +01:00
greatroar
6bd8a2faaa backup: Add --ignore-ctime option and document change detection 2021-01-28 23:42:10 +01:00
Michael Eischer
58b5679f14 prune: reword missing blobs error 
The previous wording could be understood such that the prune run did
damage the repository.
 2021-01-28 21:48:24 +01:00
Michael Eischer
7b8886c052 prune: report missing but unneeded pack files 
This indicates a damaged repository so add some output to help with
debugging.
 2021-01-28 21:46:01 +01:00
Michael Eischer
ff95999246 rebuild-index: report added/removed/reindexed files 
This should help with investigating missing pack files.
 2021-01-28 21:46:01 +01:00
Michael Eischer
b71c52797a find: correctly expand multiple blob ids 
For example `restic find --show-pack-id --blob f78dc991 5b9e4366 ddd8c7d4`
would previously only expand one blob if all of them belong to the same
file.
 2021-01-28 21:21:54 +01:00
MichaelEischer
82140967d3
Merge pull request #3228 from aawsome/prune-all-trees 
prune: Remove all unused trees
 2021-01-28 21:04:27 +01:00
MichaelEischer
35033d9b79
Merge pull request #3177 from MichaelEischer/fix-2759 
prune: don't print stacktrace on console
 2021-01-28 20:28:45 +01:00
Alexander Neumann
4d40c70214
Merge pull request #3211 from MichaelEischer/sftp-speedup 
Speed-up caching/pack download via SFTP
 2021-01-28 14:16:55 +01:00
Alexander Neumann
44169d0dc4
Merge pull request #3205 from MichaelEischer/fix-quiet-verbose 
Properly check that --quiet and --verbose are not combined
 2021-01-28 13:53:31 +01:00
Alexander Neumann
bdfedf1f5b
Merge pull request #3173 from MichaelEischer/unify-index-loading 
Unify index loading
 2021-01-28 13:50:42 +01:00
Michael Eischer
e2b0072441 check: add progress bar to the tree structure check 2021-01-28 11:10:50 +01:00
Michael Eischer
505f8a2229 progress/counter: Support updating the progress bar maximum 2021-01-28 11:10:47 +01:00
Michael Eischer
eda8c67616 restic: let FindUsedBlobs handle multiple snapshots at once 2021-01-28 11:08:43 +01:00
Michael Eischer
258ce0c1e5 parallel: report progress for StreamTrees 
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
 2021-01-28 11:08:43 +01:00
Michael Eischer
3d6a3e2555 copy: Remove treeCloner struct 2021-01-28 11:08:43 +01:00
Michael Eischer
0caad1e890 copy: parallelize tree walk 2021-01-28 11:08:43 +01:00
Alexander Neumann
a4689eb3b9
Merge pull request #3199 from MichaelEischer/non-interactive-counter 
Don't print progress on non-interactive terminals
 2021-01-28 10:53:38 +01:00
Alexander Weiss
d7dc19a496 prune: Always repack packs containing tree blobs 2021-01-15 16:42:04 +01:00
Michael Eischer
f3442ce8a5 Test that WriteTo of a backend's Load remains accessible 2021-01-03 22:23:53 +01:00
Michael Eischer
97388b3504 Properly check that --quiet and --verbose are not combined 
If --verbose is specified once, then globalOptions.Verbose == 1.
Previously --quiet --verbose would silently ignore the --verbose flag.
 2020-12-30 21:24:18 +01:00
MichaelEischer
e1efc193e1
Merge pull request #3139 from aawsome/prune-healing 
prune: Add healing of repository in some situations
 2020-12-29 22:17:27 +01:00
Alexander Weiss
f0113139ea prune: Correct error message 2020-12-29 20:20:05 +01:00
Alexander Weiss
f6df94a50e prune: Add self-healing 
Allow prune to heal situations where blobs in the index are missing or
the corresponding packfiles are damaged if those blobs are not needed.
 2020-12-29 20:20:05 +01:00
MichaelEischer
31e56f1ad5
Merge pull request #3197 from SkYNewZ/fix/3183 
Fix tag handling for multiple tag lists
 2020-12-29 18:44:38 +01:00
greatroar
dec5008369 Warn when unlock fails instead of returning an error 
Only one caller was checking the error.
 2020-12-29 17:48:20 +01:00
Quentin Lemaire
25ecf9eafb
fix(cmd_tag): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Quentin Lemaire
e88f3fb80c
fix(cmd_backup): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Michael Eischer
05a987b07c Support values less than 1 for RESTIC_PROGRESS_FPS 
For example set the variable to 0.016666 to print the progress once per
minute.
 2020-12-29 16:32:18 +01:00
Michael Eischer
85fe5feadb Unify progress report frequency calculation 2020-12-29 16:03:43 +01:00
Michael Eischer
969141b5e9 Honor RESTIC_PROGRESS_FPS env variable on non-interactive terminals 
This makes it possible to use the environment variable to also get
regular progress updates on non-interactive terminals.
 2020-12-29 16:03:43 +01:00
Michael Eischer
c2ef049f1b ui/progress: don't print progress on non-interactive terminals 
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058.
 2020-12-29 16:03:43 +01:00
greatroar
18531e3d6f Portability fixes to internal/restic 
syscall.Mknod is not available on AIX.
 2020-12-29 01:35:01 +01:00
DRON-666
332b1896d1 Some options fixes 
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
 2020-12-23 23:26:04 +03:00
Alexander Weiss
d8d2cc6dd9 Simplify rebuild-index code 2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Michael Eischer
ccc84af73d debug/list: parallelize index loading 2020-12-22 22:36:18 +01:00
Alexander Neumann
edf89e1c74 forget: Enable --dry-run together with --prune 2020-12-22 20:58:02 +01:00
Michael Eischer
cfea79d0c5 prune: don't print stacktrace on console 2020-12-19 14:28:48 +01:00
DRON-666
2ca76afc2b dump: Add new option --archive 2020-12-19 01:09:47 +03:00
DRON-666
0256f95994 dump: Split tar and walk logic 2020-12-19 01:09:47 +03:00
Alexander Neumann
0d5b764f90
Merge pull request #3130 from aawsome/snapshots-parallel 
Make loading snapshots parallel
 2020-12-06 21:08:18 +01:00
Alexander Weiss
e329623771 Remove LoadAllSnapshots 2020-12-06 05:22:27 +01:00
Alexander Weiss
5b9ee56335 Add ForAllSnapshots 2020-12-06 05:04:21 +01:00
Alexander Weiss
83c8a9b058 Bugfix: packSizeFromList should save size from List() 2020-12-05 20:58:36 +01:00
Michael Eischer
f72f6c9c80 Fix debug build 2020-11-29 18:44:36 +01:00
MichaelEischer
52b98f7f95
Merge pull request #3017 from greatroar/files-from0 
Add backup options --files-from-verbatim and --files-from-raw
 2020-11-29 18:15:21 +01:00
Alexander Neumann
04d856e601
Merge pull request #3136 from restic/rawtaz-copy-doc 
doc: Emphasize double transfer and duplication in copy command
 2020-11-29 13:57:10 +01:00
Alexander Neumann
a7b49c4889
Merge pull request #3119 from restic/keep-mountpoints 
Keep mountpoints as empty directories for --one-file-system
 2020-11-29 11:22:12 +01:00
Leo R. Lundgren
f70b10d0ee doc: Emphasize double transfer and duplication in copy command 2020-11-28 19:49:34 +01:00
greatroar
55bf76ba0c backup: Add --files-from-{verbatim,raw} options 2020-11-28 18:22:31 +01:00
Alexander Neumann
82ae942965 backup: Keep mountpoints for --one-file-system 
When a file system is mounted at a directory, lstat() returns attributes
of the root node of the mounted file system, including the device ID of
the other file system. The previous code used when --one-file-system is
specified excluded the directory itself because of that.

This commit changes the code so that mountpoints are kept as empty
directories, its attributes set to the root note of the mounted file
system. The behavior mimics `tar`, which does the same.
 2020-11-28 17:00:31 +01:00
Alexander Neumann
f576d3d826 Add tests 2020-11-28 17:00:31 +01:00
Alexander Neumann
037f0a4c91 Refactor device ID checking 2020-11-28 17:00:31 +01:00
Alexander Neumann
a3105799c9 Print warning when unlocking the repo fails 2020-11-28 11:31:25 +01:00
Kyle Brennan
666768cd17 cat: Respect --no-lock flag 2020-11-28 11:30:02 +01:00
Alexander Weiss
aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Weiss
67c938f232 Use PackSize in prune 2020-11-21 22:13:54 +01:00
Alexander Weiss
4960b841e6 Use PackSize in rebuild-index 2020-11-21 22:13:54 +01:00
Alexander Neumann
110a32a08b
Merge pull request #3113 from aawsome/fix-prune-stats-duplicates 
Fix statistics in prune for duplicates
 2020-11-19 20:34:11 +01:00
Alexander Weiss
cb5ec7ea6b Fix statistics in prune for duplicates 
Note that this fix only solves the statistics problem, if
all duplicates are marked for repacking.

If not all duplicates are marked for repacking, we lack the
information which

The situation that not all duplicates are marked for repacking can occur
when using the `max-repack-size` option
 2020-11-18 22:30:22 +01:00
Michael Eischer
625410f003 Limit progress bar updates to once per second on non-terminal outputs 
The code accidentally checked whether stdin is a terminal instead of
stdout, the former is not relevant here as the output is printed on
stdout.
 2020-11-18 22:12:07 +01:00
Alexander Neumann
d857fb6e59 Allow using --repository-file in init 2020-11-17 20:43:46 +01:00
Michael Eischer
1f43cac12d check: Only track data blobs when unused blobs should be reported 
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
 2020-11-15 18:43:07 +01:00
Michael Eischer
3500f9490c check: Simplify blob status tracking 
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
 2020-11-15 18:42:42 +01:00
Michael Eischer
caac38ed27 check: Tweak subset percentage over 100% error message 2020-11-15 18:13:50 +01:00
fgma
8f9cea8cc0 Check data subset: check random percentage subset 2020-11-15 18:13:50 +01:00
Alexander Neumann
3c0c0c132b
Merge pull request #3006 from aawsome/new-rebuild-index 
Reimplement rebuild-index and remove /internal/index
 2020-11-15 17:48:43 +01:00
kitone
3c03b35212 Fix #1681 should not try to create the mount point if it doesn't exist, rather return an error 2020-11-15 17:09:30 +01:00
Alexander Weiss
9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a Reimplement rebuild-index 2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5 Add extraObsolete to MasterIndex.Save 2020-11-15 07:05:09 +01:00
MichaelEischer
3601a9b6cd
Merge pull request #2690 from SkYNewZ/master 
Fix #2688: Handle comma-separated list tags when using backup command
 2020-11-14 23:19:42 +01:00
Michael Eischer
fdec8051ab tags: Tweak description of --add/set/remove options 2020-11-14 22:55:30 +01:00
MichaelEischer
333c5a19d4
Merge pull request #3082 from aawsome/check-sizes 
Check: check sizes of packs from index and packheader
 2020-11-14 22:37:42 +01:00
Quentin Lemaire
a8ad6b9a4b
fix(tags): Change tags list type according to restic.TagList 2020-11-14 16:24:58 +00:00
Quentin Lemaire
b0882b3f3c
fix(snapshots): Update help message to better match the 'forget' command one 2020-11-14 15:48:57 +00:00
Quentin Lemaire
ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list 2020-11-14 15:48:56 +00:00
Alexander Weiss
80dcfca191 check: Check sizes computed from index and pack header 2020-11-14 00:42:49 +01:00
Alexander Neumann
59ea5a4208 diff: Correctly count top-level blobs 2020-11-13 21:11:21 +01:00
greatroar
63e32c44c0 Improve error reporting from restic diff 
Instead of a stacktrace, restic diff 111 222 now reports:

	Fatal: no matching ID found for prefix "111"
 2020-11-11 16:40:40 +01:00
greatroar
21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter 
This fixes two race conditions while cleaning up the code.
 2020-11-09 12:12:35 +01:00
Alexander Neumann
a8d21b5dcf prune: Warn if no cache is present 2020-11-08 20:25:35 +01:00
Alexander Neumann
823d0afd6e backup: Always print parent snapshot info 2020-11-08 20:25:35 +01:00
Alexander Neumann
47277c4b4c Add comments, clarify computation 2020-11-06 20:23:30 +01:00
Alexander Weiss
d2e53730d6 Add test that repo.List is only called once 2020-11-06 20:23:30 +01:00
Alexander Weiss
fd33030556 Use in-memory index to rebuild index in prune 2020-11-06 20:23:30 +01:00
Alexander Neumann
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command 
Reimplementation of prune
 2020-11-05 10:12:19 +01:00
Alexander Neumann
d35d279455 Set development version for 0.11.0 2020-11-05 09:41:40 +01:00
Alexander Neumann
b7e1ece1e0 Add version for 0.11.0 2020-11-05 09:41:35 +01:00
fgma
916b2d303b
vss: fix potential crash (not reachable in restic) (#3045) 
HasSufficientPrivilegesForVSS() now returns an error
 2020-11-04 22:14:18 +01:00
Alexander Neumann
1ca60bccfb Refactor condition for MaxRepackBytes 
Don't depend on the string (opts.MaxRepackSize) for the condition,
instead check if there's a (positive) limit configured.
 2020-11-03 16:42:21 +01:00
Alexander Neumann
7f86eb4ec0 Move helper function 2020-11-03 16:42:21 +01:00
Alexander Neumann
c1a3de4a6e Refactor max-unused calculation, add unlimited option 
Add a callback to the PruneOptions struct which calculates the number of
bytes allowed to be unused after prune is done. This way, the logic is
closer to the option parsing code.

Also, add an explicit option `unlimited` for the use case when storage
does not matter but bandwidth and time do. Internally, this sets the
maximum number of unused bytes to MaxUint64.

Rework the documentation slightly so that no more "packs" are
mentioned and it talks about "files" instead.

Make it clear in the documentation that the percentage given to
`--max-unused` is relative to the whole repository size after pruning is
done. If specified, it must be below 100%, otherwise the repository
would contain 100% of unused data, which is pointless.

I had a hard time coming up with the correct formula to calculate the
maximum number of unused bytes based on the number of used bytes. For a
fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of
unused bytes `x` the following holds:

      x ≤ p * (u+x)
    ⇔ x ≤ p*u + p*x
    ⇔ x - p*x ≤ p*u
    ⇔ x * (1-p) ≤ p*u
    ⇔ x ≤ p/(1-p) * u
 2020-11-03 16:42:21 +01:00
Alexander Neumann
f8c4dd7b1a Split packe rewrite logic into two case branches 
The comma is too sublte, let's split this into two separate branches.
 2020-11-03 16:42:21 +01:00
Alexander Neumann
a5b80452fe Add comment that usedBlobs is modified 2020-11-03 16:42:21 +01:00
Alexander Neumann
aff1e220f5 Split struct members, add comments 2020-11-03 16:42:21 +01:00
Alexander Neumann
095155d9ce Remove RepackSmall 2020-11-03 16:42:21 +01:00
Alexander Weiss
b2f5381737 Make realistic forget --prune --dryrun 2020-11-03 16:42:21 +01:00
Alexander Weiss
7f9a0a5907 Reimplementation of prune 2020-11-03 16:42:21 +01:00
Alexander Weiss
3b591ed987 Add Verboseff 2020-11-03 16:42:21 +01:00
Alexander Neumann
21ba15577e Return an error for invalid sizes 2020-11-03 11:53:49 +01:00
Alexander Neumann
6509c207f4
Merge pull request #2997 from MichaelEischer/faster-excludes 
Speedup exclude/include checking
 2020-11-02 11:07:21 +01:00
Alexander Neumann
445b845267
Merge pull request #2978 from MichaelEischer/warn-tree-error 
Warn if backup failed to read tree blob
 2020-11-02 10:14:12 +01:00
Alexander Neumann
3ff37215df
Merge pull request #2935 from MichaelEischer/upgrade-minio 
Upgrade minio SDK to version 7
 2020-11-02 09:09:10 +01:00
MichaelEischer
31b8d7a639
Merge pull request #2274 from fgma/master 
Support for Volume Shadow Copy Service (VSS) on windows
 2020-10-24 15:43:34 +02:00
fgma
5695f9ebd2 vss: Implement VSS support for Windows 
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.

Snapshotting volumes also covers mountpoints but skips UNC paths.
 2020-10-24 11:35:57 +02:00
greatroar
3807d13bdc restic mount: don't umount unless we actually tried to mount 
Previously, failure to open a repository would result in an "unable to
umount" message.
 2020-10-21 12:15:05 +02:00
Alexander Neumann
56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings 
Fix some goreport warnings
 2020-10-12 20:44:56 +02:00
Michael Eischer
e638b46a13 Embed context into ReaderAt 
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
 2020-10-09 22:39:07 +02:00
Michael Eischer
2964d2ad15 Skip unlocking for nil locks 
Now that lockRepo receives a context, it is possible that it is canceled
before a lock was created. Thus `unlockRepo` must be able to handle this
case.
 2020-10-09 22:39:06 +02:00
Michael Eischer
0c9efa9c2a Pass context to lockRepo 2020-10-09 22:39:06 +02:00
Michael Eischer
37a5e2d681 rest: use global context on repository creation 2020-10-09 22:39:06 +02:00
Michael Eischer
dc2e664209 integration_fuse_test: use global context 
No need to use the TODO context.
 2020-10-09 22:37:56 +02:00
Michael Eischer
603bb0e309 restore: Use proper context while loading snapshot 2020-10-09 22:37:56 +02:00
Michael Eischer
27456f6545 debug: use proper context 
This allows the debug commands to be properly interrupted.
 2020-10-09 22:37:56 +02:00
Michael Eischer
c458e114d4 pass context to Find / FindSnapshot 
This allows proper interruption of restic while it searches for
snapshots or key files.
 2020-10-09 22:37:56 +02:00
Michael Eischer
45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
Michael Eischer
307a6ba3a3 Upgrade minio sdk to v7 
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
 2020-10-09 22:37:24 +02:00
Michael Eischer
50da20d93d Warn if backup failed to read tree blob 2020-10-09 22:36:27 +02:00
Alexander Neumann
30cb553c8d
Merge pull request #2932 from MichaelEischer/proper-rclone-create 
Call rclone.Create to create a new repository for the rclone backend
 2020-10-09 21:29:15 +02:00
Michael Eischer
bcc3bddcf4 filter: only check whether a child path could match when necessary 
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.3ms ± 9%    21.7ms ± 6%   -6.68%  (p=0.004 n=10+10)
FilterPatterns/Absolute-4       13.9ms ± 7%    10.0ms ± 5%  -27.61%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4       51.4ms ± 7%    47.0ms ± 7%   -8.51%  (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4     551ms ± 9%     190ms ± 1%  -65.41%  (p=0.000 n=10+8)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.665 n=10+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.480 n=9+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    14.3MB ± 0%     ~     (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
 2020-10-07 20:47:52 +02:00
Michael Eischer
375c2a56de filter: Parse filter patterns only once 
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       30.3ms ±10%    23.6ms ±20%  -22.12%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       49.0ms ± 3%    32.3ms ± 8%  -33.94%  (p=0.000 n=8+10)
FilterPatterns/Wildcard-4        345ms ± 9%     334ms ±17%     ~     (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4     3.93s ± 2%     0.71s ± 7%  -81.98%  (p=0.000 n=9+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       4.63MB ± 0%    3.57MB ± 0%  -22.98%  (p=0.000 n=9+9)
FilterPatterns/Absolute-4       8.54MB ± 0%    3.57MB ± 0%  -58.20%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        146MB ± 0%     141MB ± 0%   -2.93%  (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4     907MB ± 0%       4MB ± 0%  -99.61%  (p=0.000 n=9+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        66.6k ± 0%     22.2k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4        88.7k ± 0%     22.2k ± 0%  -75.00%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.70M ± 0%     1.63M ± 0%   -3.92%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     4.46M ± 0%     0.02M ± 0%  -99.50%  (p=0.000 n=10+10)
 2020-10-07 20:47:27 +02:00
MichaelEischer
0ae02f3030
Merge pull request #2844 from aawsome/prune-integration-tests 
Add prune integration tests for many edge cases
 2020-10-06 23:16:41 +02:00
Lorenz Bausch
af4100e07d
Replace repo/repos with repository/repositories in --no-lock help text 2020-10-06 22:49:48 +02:00
Alexander Weiss
6822a58413 Add prune integration tests for many edge cases 2020-10-06 20:20:05 +02:00
Michael Eischer
1579d2a8ec Remove some unused assignments 2020-10-06 14:55:13 +02:00
Michael Eischer
eba5dd831f Fix typos reported by misspell 2020-10-06 14:55:13 +02:00
Leo R. Lundgren
e105a3f391 cache/check: Clarify error message when given arguments 2020-10-06 00:08:59 +02:00
rawtaz
5d8cfff3f2
Merge pull request #2988 from greatroar/snapshots-json 
Regression test for #2979
 2020-10-05 21:02:52 +02:00
MichaelEischer
1ab4c710e1
Merge pull request #2987 from greatroar/no-password-for-help 
Don't read password for generate, help or self-update
 2020-10-05 20:16:19 +02:00
greatroar
feedf0ebce Don't read password for generate, help or self-update 
Fixes #2951.
 2020-10-05 19:54:07 +02:00
MichaelEischer
1a490acd67
Merge pull request #2982 from greatroar/archiver-error-handling 
Check error in archiver before calling Select
 2020-10-05 19:19:47 +02:00
greatroar
6bee0aafc2 Regression test for #2979 2020-10-05 15:08:09 +02:00
Leo R. Lundgren
137d20a06a snapshots: Make --json output [] instead of null when no snapshots 2020-10-05 13:14:10 +02:00
greatroar
c4e2203e45 Check error in archiver before calling Select 
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
 2020-10-05 11:11:04 +02:00
rawtaz
bbeb439f41
Merge pull request #2937 from andreaso/self-update-output-path-fix 
Don't require `self-update --output` placeholder file
 2020-10-03 13:35:20 +02:00
Andreas Olsson
ce14df303b
Don't require self-update --output placeholder file 
This removes the requirement on `restic self-update --output` to point
to a path of an existing file, to overwrite. In case the specified
path does exist we still want to verify that it's a regular file,
rather than a directory or a device, which gets overwritten.

We also want to verify that a path to a new file exists within an
existing directory. The alternative being running into that issue
after the actual download, etc has completed.

While at it I also replace `errors.Errorf` with the more appropriately
verbose `errors.Fatalf`.

Resolves #2491
 2020-10-03 07:02:23 +02:00
MichaelEischer
3c6671b18b
Merge pull request #2973 from restic/rawtaz-repo2 
Correct data type for --repo2 option
 2020-10-02 22:22:41 +02:00
rawtaz
0cce6dc31c
Correct data type for --repo2 option 2020-10-02 21:10:52 +02:00
rawtaz
6253ff0187
Clarify max verbose level in option description 
Clarifies that the max verbosity level is 3, in the description of the --verbose option.
 2020-10-02 20:25:34 +02:00
jtagcat
a687261804 --help: --compact: clarify description 2020-10-02 16:55:56 +03:00
Michael Eischer
61035d68bc Add test for --repository-file 2020-10-01 00:50:27 +02:00
Andreas Oberritter
97f7855de3 Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE) 
As an alternative to -r, this allows to read the repository URL
from a file in order to prevent certain types of information leaks,
especially for URLs containing credentials.

Fixes #1458, fixes #2900.
 2020-10-01 00:50:26 +02:00
rawtaz
34ea960559
Merge pull request #2966 from MichaelEischer/recover-help-typo 
recover: Fix typo in the command help
 2020-09-30 18:16:26 +02:00
Michael Eischer
2a79c1a44d recover: Fix typo in the command help 
Reported-by: Steve Divskinsy <stevesbrain@users.noreply.github.com>
 2020-09-30 17:44:34 +02:00
Michael Eischer
aea9f7d286 clarify verbose password prompt 2020-09-30 17:25:54 +02:00
plumbeo
009bd907f2
Don’t print too many messages when deleting files 
Print detailed debug messages on file deletions only when --verbose=2 is used
 2020-09-27 14:24:04 +02:00
MichaelEischer
14b312f00d
Merge pull request #2658 from creativeprojects/issue-2241 
Don't echo authentication passwords (rest backend)
 2020-09-22 22:17:53 +02:00
Fred
206cadfab4 Hide password from repository URLs 2020-09-22 22:00:51 +02:00
MichaelEischer
4875f7b659
Merge pull request #2614 from greatroar/simplify-fs 
Simplify internal/fs
 2020-09-21 22:05:10 +02:00
yoshiera
3cf29a777d Fix nil check in rejectBySize 2020-09-21 19:20:24 +08:00
Michael Eischer
9333f707fa init: use Create method for rclone backend 
This properly issues the initial repository creation command

Fixes #1896
 2020-09-19 19:11:43 +02:00
Alexander Neumann
429f97b887 Set development version for 0.10.0 2020-09-19 17:38:47 +02:00
Alexander Neumann
40832b2927 Add version for 0.10.0 2020-09-19 17:38:26 +02:00
Michael Eischer
f003410402 init: Add --copy-chunker-params option 
This allows creating multiple repositories with identical chunker
parameters which is required for working deduplication when copying
snapshots between different repositories.
 2020-09-19 16:53:05 +02:00
Michael Eischer
655430550b Extract parameters for second repository from copy command 2020-09-19 16:07:55 +02:00
Leo R. Lundgren
311ad2d2d0 Improve wording for --password-file and related options 2020-09-19 15:47:32 +02:00
Leo R. Lundgren
ffe6dce7e7 Improve wording for --password-command and --password-command2 options 2020-09-19 14:23:25 +02:00
yoshiera
ac4b8c98ac Support excluding files by size 2020-09-19 19:52:09 +08:00
greatroar
9abef3bf1a Move internal/fs.TestChdir to internal/test.Chdir 2020-09-17 10:43:33 +02:00
Michael Eischer
f5c448aa65 diff: Optimize diff calculation for shared subtrees 
When the diff calculation compares two trees with identical id then no
differences between them can ever show up. Optimize for that case by
simply traversing the tree only once to collect all referenced blobs for
a proper calculation of added and removed blobs.

Just skipping the common subtrees is not possible as this would skew the
results if the added or removed blobs are shared with one of the
subtrees.
 2020-09-13 16:28:46 +02:00
Michael Eischer
c0fc85d303 diff: Add integration test 2020-09-13 16:28:46 +02:00
Alexander Neumann
0c48e515f0
Merge pull request #2630 from MichaelEischer/fix-staticcheck 
Fix lots of small issues reported by staticcheck
 2020-09-13 16:19:22 +02:00
rawtaz
d3a286928a
Merge pull request #2905 from MichaelEischer/fix-ls-help 
ls: Explicitly enforce that the user specifies a snapshot ID
 2020-09-05 10:44:01 +02:00
Michael Eischer
1ede018ea6 error variable names should start with 'Err' 2020-09-05 10:07:17 +02:00
Michael Eischer
d0329cf3eb Adjust comments to match name of exported methods 2020-09-05 10:07:16 +02:00
Michael Eischer
b22655367c integration_test: Replace fprintf without format string 2020-09-05 10:07:16 +02:00
Michael Eischer
068a3ce23f Remove redundant return 2020-09-05 10:07:16 +02:00
Michael Eischer
d19f05c960 Use buf.String() instead of string(buf.Bytes()) 2020-09-05 10:07:16 +02:00
Michael Eischer
460e2ffbf6 Collapse a few boolean operations 2020-09-05 10:07:14 +02:00
Michael Eischer
49b6aac3fa Use !bytes.Equal instead of bytes.Compare !=0 to check for inequality 2020-09-05 10:06:23 +02:00
Michael Eischer
2f8335554c Remove a few unused variables 2020-09-05 10:06:23 +02:00
Michael Eischer
37113282ca Merge a few variable declaration and initializations 2020-09-05 10:05:34 +02:00
Michael Eischer
337725c354 Use sort.Strings 2020-09-05 10:05:34 +02:00
MichaelEischer
2ddb7ffb7e
Merge pull request #2255 from Kidswiss/tar 
Fix dumping issues with / and the first sub level
 2020-09-01 21:52:17 +02:00
Michael Eischer
81dcfea11a dump: Mention tar output in command help text 2020-08-31 22:43:10 +02:00
MichaelEischer
55071ee367
Merge pull request #2859 from buschjost/stats-filter-by-tag-and-path 
Add filter by tag and path to stats command
 2020-08-31 22:11:01 +02:00
Michael Eischer
bcf44a9c3f dump: Properly handle file paths without / prefix 
filepath.Rel failed if the requested path did not start with a `/` e.g.
`restic` instead of `/restic`.
 2020-08-30 18:25:42 +02:00
Oliver Buschjost
a7b4c19abf Fix display of warning about ignored filters 2020-08-30 17:14:56 +02:00
Oliver Buschjost
d3fcfeba3a Add filter by tag and path to stats command 2020-08-30 17:14:48 +02:00
Michael Eischer
e69449bf2c dump: Add test for splitPath 2020-08-30 16:00:48 +02:00
MichaelEischer
ea81a0e282
Merge pull request #2606 from middelink/fix-323 
Add copy functionality.
 2020-08-30 10:18:24 +02:00
Simon Beck
d6f739ec22 dump: Extract tar code to internal/dump 2020-08-29 21:14:34 +02:00
Simon Beck
b98598e55f dump: Fix dumping issues with / and the first sub level 
There was an issue that prevented the dump command from working
correctly when either:

* `/` contained multiple nodes (e.g. `restic backup /`)
* dumping a file in the first sublevel was attempted (e.g. `/foo`)
 2020-08-29 21:14:29 +02:00
Michael Eischer
d5f86effa1 ls: Explicitly enforce that the user specifies a snapshot ID 
The help messages suggested that the `ls` command work without
explicitly passing a snapshot ID. However, this was never the case:
without a snapshot ID the command just failed with the error
`Ignoring "", it is not a snapshot id`.

Fixes #2299
 2020-08-29 14:28:53 +02:00
Michael Eischer
412623b848 copy: Reuse buffer for downloaded blobs 2020-08-29 10:48:44 +02:00
Michael Eischer
9a4796594a integration tests: Fix checking of wrong snapshot 2020-08-29 10:48:44 +02:00
Michael Eischer
15374d22e9 integration tests: Add basic tests for copy command 2020-08-29 10:48:44 +02:00
Michael Eischer
88ad58d6cd integration tests: Redirect directory diff into intermediate buffer 2020-08-29 10:48:44 +02:00
Michael Eischer
591a8c4cdf integration tests: Deduplicate backup test-data setup code 2020-08-29 10:48:44 +02:00
Michael Eischer
ec9a53b7e8 copy: Mark and skip previously copied snapshots 
Use the `Original` field of the copied snapshot to store a persistent
snapshot ID. This can either be the ID of the source snapshot if
`Original` was not yet set or the previous value stored in the
`Original` field. In order to still copy snapshots modified using the
tags command the source snapshot is compared to all snapshots in the
destination repository which have the same persistent ID. Snapshots are
only considered equal if all fields except `Original` and `Parent`
match. That way modified snapshots are still copied while avoiding
duplicate copies at the same time.
 2020-08-29 10:48:44 +02:00
Alexander Weiss
9867c4bbb4 Add flag --iexclude-file to backup 2020-08-28 21:04:41 +02:00
Jason Cox
2447f3f110 Update usage strings to put flags before args 
The standard UNIX-style ordering of command-line arguments places
optional flags before other positional arguments. All of restic's
commands support this ordering, but some of the usage strings showed the
flags after the positional arguments (which restic also parses just
fine). This change updates the doc strings to reflect the standard
ordering.

Because the `restic help` command comes directly from Cobra, there does
not appear to be a way to update the argument ordering in its usage
string, so it maintains the non-standard ordering (positional arguments
before optional flags).
 2020-08-28 02:16:22 +00:00
Michael Eischer
b0a8c4ad6c copy: Only process each tree once 
This speeds up copying multiple overlapping snapshots from one
repository to another, as we only have to copy the changed parts of
later snapshots.
 2020-08-26 22:17:52 +02:00
Michael Eischer
908b23fda0 copy: Update for modernized repository interface 2020-08-26 22:17:52 +02:00
greatroar
4508d406ef copy: Remove separate SaveIndex in restic copy 
Flush does this now.
 2020-08-26 22:17:52 +02:00
Pauline Middelink
7048cc3e58 Add copy functionality 
Add a copy command to copy snapshots between repositories. It allows the user
to specify a destination repository, password, password-file, password-command
or key-hint to supply the necessary details to open the destination repository.

You need to supply a list of snapshots to copy, snapshots which already exist
in the destination repository will be skipped.

Note, when using the network this becomes rather slow, as it needs to read the
blocks, decrypt them using the source key, then encrypt them again using the
destination key before finally writing them out to the destination repository.
 2020-08-26 22:17:52 +02:00
Michael Eischer
17995dec7a Unify progress bar of check and prune commands 2020-08-25 22:47:38 +02:00
Alexander Neumann
b1b3f1ecb6
Merge pull request #2674 from MichaelEischer/prune-strict-checks 
prune: Stricter error checks
 2020-08-23 10:30:41 +02:00
Bruce Dillahunty
fe69b83074
Update key password prompt (#2847) 2020-08-19 21:42:08 +02:00
Michael Eischer
08d24ff99e prune: Include ID of all missing blobs in error message 2020-08-16 11:36:14 +02:00
Michael Eischer
1c84aceb39 prune: Test for abort on damaged repositories 2020-08-16 11:36:12 +02:00
Michael Eischer
575ed9a47e Test that rebuild-index errors when old index cannot be removed 2020-08-16 11:34:01 +02:00
Michael Eischer
8f811642c3 Add support for integration tests to wrap the backend 2020-08-16 11:34:01 +02:00
Michael Eischer
744a15247d prune/rebuild-index: Fail if an old index cannot be removed 
The old behavior was problematic in the context of rebuild-index as it
could leave old, possibly invalid index files behind without returning a
fatal error.

Prune calls rebuildIndex before removing any data from the repository.
For this use case failing to delete an old index MUST be treated as a
fatal error. Otherwise the index could still contain an old index file
that refers to blobs/packs that were later on deleted by prune. Later
backup runs will assume that the affected blobs already exist in the
repository which results in a backup which misses data.
 2020-08-16 11:34:01 +02:00
Michael Eischer
3ba19869be prune: Abort if any used blobs are missing 
The previous check only approximately verified whether all required
blobs were found. However, after forgetting a few snapshots the
repository contains lots of unused blobs whose number can be sufficient
to make up for missing packs.

When coupled with a malfunctioning backend that temporarily returns broken
data this could cause restic to regard the corresponding packs as
invalid and thereby delete data that's still in use. This change lets
restic play it safe and refuse to delete anything if data is missing.
 2020-08-16 11:34:01 +02:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" (#2885) 
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
 2020-08-16 11:16:38 +02:00
MichaelEischer
08dee8a52b
Merge pull request #2865 from greatroar/unused 
Dead code removal
 2020-08-08 16:15:28 +02:00
MichaelEischer
84b6f1ec53
Merge pull request #2874 from MichaelEischer/fix-verbose 
Adjust description of --verbose=n parameter
 2020-08-04 23:21:05 +02:00
MichaelEischer
06fb4ea3f0
Merge pull request #2805 from jwilk-forks/verbose 
Fix examples of --verbose with argument
 2020-08-04 23:11:35 +02:00
Michael Eischer
e38d415173 Adjust description of --verbose=n parameter 2020-08-04 23:07:53 +02:00
greatroar
d81a396944 Dead code removal 
Found by running golangci-lint on the entire code base.
 2020-08-04 08:38:57 +02:00
MichaelEischer
38a2f9c07b
Merge pull request #2864 from greatroar/dump-error 
Fix error handling in dump (err != err)
 2020-08-03 20:09:42 +02:00
rawtaz
5af2815627
Merge pull request #2821 from renard/mount-nolock 
cmd/mount: honur --no-lock flag
 2020-08-03 20:09:26 +02:00
Sébastien Gross
9be4fe3e84 cmd/mount: honour --no-lock flag 
Do not lock the repository if --no-lock global flag is set. This allows
to mount repositories which are archived on a read only system.

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
 2020-08-03 19:45:39 +02:00
Michael Eischer
05116e4787 prune: Cleanup progress bar handling while repacking 2020-08-03 19:32:46 +02:00
Michael Eischer
04f79b9642 prune: Stop progress bar after searching used blobs 2020-08-03 19:31:49 +02:00
greatroar
8b358935a0 Fix error handling in dump (err != err) 2020-08-03 09:43:44 +02:00
MichaelEischer
66d089e239
Merge pull request #2841 from aawsome/optimize-getUsedBlobs 
Extract get used blobs in prune as separate function
 2020-08-01 22:40:33 +02:00
Alexander Weiss
9762bec091 Use optimized getUsedBlobs in prune 2020-08-01 21:07:31 +02:00
Alexander Weiss
d3692f5b81 Delete files in parallel in rebuild-index 2020-08-01 20:43:18 +02:00
Alexander Weiss
1c0b61204b Delete files in parallel in forget 2020-08-01 20:43:18 +02:00
Alexander Weiss
2ee654763b Delete files in parallel 2020-08-01 20:39:24 +02:00
Alexander Neumann
b7b479b668
Merge pull request #2599 from MichaelEischer/tweak-mem-usage 
Reduce memory usage when searching for used blobs
 2020-08-01 13:49:02 +02:00
Alexander Neumann
2580eef2aa
Merge pull request #2318 from classmarkets/2175-named-keys 
Allow specifying user and host when adding keys
 2020-08-01 13:06:31 +02:00
Michael Eischer
184103647a FindUsedBlobs: merge seen into blobs BlobSet 
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.

Suggested-by: Alexander Weiss <alex@weissfam.de>
 2020-08-01 12:29:16 +02:00
Michael Eischer
3ce9893e0b Don't print a stacktrace if some files could not be read 2020-07-28 23:52:48 +02:00
Alexander Weiss
9d1fb94c6c make Lookup() return all blobs 
+ simplify syntax
 2020-07-25 21:18:34 +02:00
MichaelEischer
020cab8e08
Merge pull request #2787 from greatroar/no-blobsize-cache 
Remove blob size cache from restic mount
 2020-07-25 20:46:35 +02:00
greatroar
07da61baee Remove blob size cache from restic mount 2020-07-25 19:39:14 +02:00
Alexander Weiss
9175795fdb Check contents in archiver 
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
 2020-07-25 08:18:28 +02:00
Michael Eischer
ddf0b8cd0b checker: Properly distinguish between data and tree blobs 
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
 2020-07-20 22:58:39 +02:00
David Sommer
5729d967f5 backup: Return exit status code 3 when failing to read source data 
The backup command used to return a zero exit code as long as a snapshot
could be created successfully, even if some of the source files could not
be read (in which case the snapshot would contain the rest of the files).

This made it hard for automation/scripts to detect failures/incomplete
backups by looking at the exit code. Restic now returns the following exit
codes for the backup command:

 - 0 when the command was successful
 - 1 when there was a fatal error (no snapshot created)
 - 3 when some source data could not be read (incomplete snapshot created)
 2020-07-20 22:19:01 +02:00
greatroar
58719e1f47 Replace mount's per-file cache by a global LRU cache 2020-07-12 18:27:16 +02:00
Jakub Wilk
4e6a9767de Fix examples of --verbose with argument 2020-06-23 20:22:14 +02:00
Andreas Olsson
0fcef2ec23
Use https:// for linking CACHEDIR.TAG specs 
That site might not have supported https:// when those links were
originally added. It does now.

Also dropping the _spec.html_ ending of the url, there being a `<link
rel="canonical" ...>` tag suggesting that that no longer being the
preferred address.
 2020-06-21 19:36:26 +02:00
Alexander Weiss
b5543cff5d Loop over index files for 'list blobs' 
=> reduces memory consumption a lot!
 2020-06-14 17:25:34 +02:00
Michael Eischer
bb1e258bb7 cat/stats: Get stdout from globalOptions 2020-06-14 11:08:11 +02:00
Michael Eischer
182655bc88 Replace fmt.Printf/Println/Fprintf with wrapper functions 
cmd/restic/globals.go already provides Printf, Println and Warnf wrapper
which get their output streams from the globalOptions object. This
allows for stream replacements when testing.
 2020-06-14 11:08:11 +02:00
MichaelEischer
dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256 
Use Minio's optimized SHA-256
 2020-06-12 23:32:58 +02:00
MichaelEischer
735a8074d5
Merge pull request #2773 from aawsome/index-uploads+knownblobs 
Fix non-intuitive repo behavior
 2020-06-12 22:41:04 +02:00
Alexander Weiss
70347e95d5 disable index uploads for prune command 
+ modifications of changelog
 2020-06-12 09:24:38 +02:00