fe04d024c7
Use LRU cache in restic dump
2021-09-24 15:45:08 +02:00
897d8e662c
Add --insecure-tls flag to disable SSL cert verification
...
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2021-09-21 10:52:40 -04:00
58efe21eca
Merge pull request #3264 from amozoss/upstream-master
...
Refactor backup progress
2021-09-19 14:54:42 +02:00
92ae951ffa
Report timing from restic restore --verify
2021-09-19 12:53:09 +02:00
fbb0e6499a
ui: consolidate backup ui in ui/backup package
2021-09-12 16:20:15 +02:00
a144c986f2
backup: Reenable JSON status updates with redirected output
...
After the refactoring status updates were no longer printed in quiet
mode or when the output is not an interactive terminal. However, the
JSON output is often piped to e.g. another program. Thus, don't set the
update frequency to 0 in that case. The status updates are still
disabled for backup --quiet.
This also reduces the status update frequency to 60fps compared to a
potentially much higher value before the refactoring.
2021-09-12 15:26:40 +02:00
d62bfed65d
ui: move SetDryRun to ProgressReporter
2021-09-12 15:25:58 +02:00
448419990c
Refactor backup progress
...
Move the shared logic into the progress
Allows logic to be shared with forth coming restore status
2021-09-12 15:25:58 +02:00
8fe122d675
Merge pull request #3488 from MichaelEischer/rebuild-broken-index
...
Fix `rebuild-index` for damaged index
2021-09-07 17:00:23 +02:00
fa5ca8af81
Merge pull request #2856 from aawsome/remove-readahead
...
Simplify cache logic
2021-09-04 20:25:49 +02:00
b45d88e124
Merge pull request #3496 from juergenhoetzel/imporove-snapshot-filter-warning-message
...
Improve snapshot filter warning message
2021-09-03 23:42:27 +02:00
bc4cbd775b
Merge pull request #2880 from aawsome/enhance-recover
...
Improve recover command
2021-09-03 23:40:43 +02:00
bce87922c0
Improve recover
...
- only save directories not referenced by any snapshot
- dont't write empty snapshot
- use progress bar
2021-09-03 21:36:57 +02:00
81876d5c1b
Simplify cache logic
2021-09-03 21:01:00 +02:00
7f0aa49f45
cmd/restic: Streamline progress printing
...
* PrintProgress no longer does unnecessary Sprintf calls, and performs
fewer allocations in general
* newProgressMax's callback checks whether the terminal supports
line updates once instead of once per call
* the callback looks up the terminal width once per call instead of
twice (on Windows)
* the status shortening now uses the Unicode-aware version from
internal/ui/termstatus (future-proofing)
2021-09-03 11:48:22 +02:00
409e4936af
Improve snapshot filter warning message
...
Include the root-cause why the snapshot prefix is ignored.
2021-08-25 15:46:21 +02:00
877fc9f352
rebuild-index: test that invalid indexes are skipped and removed
2021-08-22 18:24:19 +02:00
64258a2c2a
rebuild-index: Ignore broken index files
...
Previously, an e.g. truncated index could prevent rebuild-index from
working.
2021-08-22 18:23:47 +02:00
d90efd7704
Fix test
2021-08-09 10:30:10 +02:00
9fe5a87785
Merge pull request #3429 from MichaelEischer/safe-key-change
...
key: Check that new key works before deleting the old one
2021-08-09 10:07:15 +02:00
4126435663
resolve rawtaz's review comments
...
make majority of suggestions from review by @rawtaz verbatim, with one clarification on my part in changelog
2021-08-04 21:19:29 +02:00
d107a2cfdf
Separate dry run tests
2021-08-04 21:19:29 +02:00
77bf148460
backup: add --dry-run/-n flag to show what would happen.
...
This can be used to check how large a backup is or validate exclusions.
It does not actually write any data to the underlying backend. This is
implemented as a simple overlay backend that accepts writes without
forwarding them, passes through reads, and generally does the minimal
necessary to pretend that progress is actually happening.
Fixes #1542
Example usage:
$ restic -vv --dry-run . | grep add
new /changelog/unreleased/issue-1542, saved in 0.000s (350 B added)
modified /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added)
modified /cmd/restic/global.go, saved in 0.000s (0 B added)
new /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added)
new /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added)
modified /internal/backend/test/tests.go, saved in 0.000s (0 B added)
modified /internal/repository/repository.go, saved in 0.000s (20.707 KiB added)
modified /internal/ui/backup.go, saved in 0.000s (9.110 KiB added)
modified /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added)
modified /restic, saved in 0.131s (25.542 MiB added)
Would add to the repo: 25.892 MiB
2021-08-04 21:19:29 +02:00
7049f1cbfc
Set development version for 0.12.1
2021-08-03 11:45:39 +02:00
dc7a8aab24
Add version for 0.12.1
2021-08-03 11:45:36 +02:00
77551597b2
Merge pull request #3416 from torfason/keep-hourly-within
...
Keep hourly within
2021-07-30 10:36:41 +02:00
92f293cd0b
Merge pull request #3427 from MichaelEischer/find-packs-from-index
...
find: List missing pack files based on the index
2021-07-30 10:33:02 +02:00
74ebc650ab
forget: Add --keep-within-hourly (and friends)
...
Allow keeping hourly/daily/weekly/monthly/yearly snapshots for a given time period.
This adds the following flags/parameters to restic forget:
--keep-within-hourly duration
--keep-within-daily duration
--keep-within-weekly duration
--keep-within-monthly duration
--keep-within-yearly duration
Includes following changes:
- Add tests for --keep-within-hourly (and friends)
- Add documentation for --keep-within-hourly (and friends)
- Add changelog for --keep-within-hourly (and friends)
2021-07-24 16:14:43 +00:00
c707d71b72
Merge pull request #3401 from MichaelEischer/goroutine-shutdown-cleanups
...
Goroutine shutdown cleanups
2021-07-11 16:32:28 +02:00
0880afe67b
Use our generate command instead of cobra's complete command
2021-07-10 19:44:18 +02:00
3442dc87fb
find: Address review comments
2021-07-06 21:04:34 +02:00
95b44490a0
find: search blob ids for pack in index if pack is missing
...
If a pack file is missing try to determine the contained pack ids based
on the repository index. This helps with assessing the damage to a
repository before running `rebuild-index`.
2021-07-06 21:04:34 +02:00
3caab3c7ac
find: Print not found pack files
2021-07-06 21:04:34 +02:00
40745b4f82
find: stop file listing after resolving all pack files
2021-07-06 21:04:34 +02:00
6c01078f3d
find: support resolving multiple pack ids to blobs
...
Just passing the list of blobs to packsToBlobs would also work in most
cases, however, it could cause unexpected results when multiple pack
files have the same prefix. Forget found prefixes to prevent this.
2021-07-06 21:04:34 +02:00
aad8864835
prune: Add missing newlines in error descriptions
2021-06-20 14:25:40 +02:00
454b6d608e
key: Add test that failed key saves don't break the repository
2021-06-13 13:46:48 +02:00
6add186867
key: Check that a new key file actually works
2021-06-12 23:09:19 +02:00
fe43f53528
mount: enable fuse readahead
...
Apparently readahead was disabled by default. Enable readahead with the
Linux default size of 128kB. Larger values seem to have no effect.
This can speed up reading from the fuse mount by at least factor 5.
Speedup for a 1G random file stored in a local repository:
(Only one result shown, but times were quite stable, restarted restic
after each command)
$ dd if=/dev/urandom bs=1M count=1024 of=rand
$ shasum -a 256 tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d tmp/rand
before: $ time shasum -a 256 fuse/snapshots/latest/tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d fuse/snapshots/latest/tmp/rand
real 0m18.294s
user 0m4.522s
sys 0m3.305s
before: $ time cat fuse/snapshots/latest/tmp/rand > /dev/null
real 0m14.924s
user 0m0.000s
sys 0m4.625s
after: $ time shasum -a 256 fuse/snapshots/latest/tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d fuse/snapshots/latest/tmp/rand
real 0m6.106s
user 0m3.115s
sys 0m0.182s
after: $ time cat fuse/snapshots/latest/tmp/rand > /dev/null
real 0m3.096s
user 0m0.017s
sys 0m0.241s
2021-06-12 17:07:30 +02:00
fdbd65485e
Merge pull request #3402 from MichaelEischer/misc-fixes
...
Various small code cleanups
2021-05-24 11:30:31 +02:00
2daf033156
Merge pull request #3403 from MichaelEischer/fast-cat
...
cat: only load index if really necessary
2021-05-24 11:02:10 +02:00
5dad45f005
Merge pull request #3312 from soraxas/master
...
Bump cobra and add completions for fish
2021-05-23 13:58:27 +02:00
61b368ddea
cat: only load index if really necessary
2021-05-23 13:11:55 +02:00
fd8bce8184
backup: cleanly shutdown goroutines on error
2021-05-23 13:02:44 +02:00
d7322a5f36
restic ls --json: print "size":0 for empty files
2021-05-21 21:06:00 +02:00
9cc1ecdd45
bump cobra and add completions for fish
...
Signed-off-by: Tin Lai <oscar@tinyiu.com>
2021-05-21 13:47:52 +10:00
af3de702c7
Merge pull request #3332 from restic/debug-1999
...
Merge `debug examine` to salvage damaged pack files
2021-05-18 09:38:40 +02:00
226cd8d4d1
Merge pull request #3310 from MichaelEischer/copy-unstable-tree
...
`copy` raw tree blobs
2021-05-18 09:36:51 +02:00
4cabad8c34
Merge pull request #3325 from MichaelEischer/fix-mintty-output
...
Fix windows terminal output for mintty
2021-05-18 09:29:24 +02:00
5767c65c62
delete: properly close fileChan if context is canceled
2021-05-17 21:05:54 +02:00
5a87a0ba0a
find: use Str() to format short ids
2021-05-17 20:56:49 +02:00
90d75651e6
backup: Improve wording for --one-file-system description
2021-05-15 00:06:27 +02:00
2a92b68e65
cmd/snapshots: Add option to limit snapshots list
...
This patch adds a `--latest` option to limit snapshots list to the n
last snapshots. It is very similar to the `--last` one but does not
limit to one entry. It also deprecates the `--last` flag usage in
favor of `--latest 1`
Output example:
$ restic snapshots --latest 2
repository 0d3eb989 opened successfully, password is correct
ID Time Host Tags Paths
------------------------------------------------------------
5a33bdcc 2020-12-14 12:30:00 local /home
73887d8e 2020-12-15 12:30:00 local /home
------------------------------------------------------------
2 snapshots
Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2021-05-13 20:18:23 +02:00
5e6af77b7a
Unify interactive terminal detection code
...
Previously the progress bar / status update interval used
stdoutIsTerminal to determine whether it is possible to update the
progress bar or not. However, its implementation differed from the
detection within the backup command which included additional checks to
detect the presence of mintty on Windows. mintty behaves like a terminal
but uses pipes for communication.
This adds stdoutCanUpdateStatus() which calls the same terminal detection
code used by backup. This ensures that all commands consistently switch
between interactive and non-interactive terminal mode.
stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus()
does so. This is required to properly handle the special case of mintty.
2021-04-11 20:02:09 +02:00
efb10b3c40
Merge pull request #3347 from jazcap53/fix_ambiguous_warning
...
Change ambiguous Warning message
2021-03-27 12:54:07 +01:00
9f44129c2f
Change ambiguous Warning message
...
Fixes #3346
On branch fix_ambiguous_warning
Changes to be committed:
modified: cmd_backup.go
2021-03-24 10:44:47 -04:00
5592c17e4a
Display 'created new cache in ' message only if output is terminal
2021-03-22 11:31:08 +02:00
54d58edacc
debug: fix usage for examine command
2021-03-10 22:22:33 +01:00
5975ed61f3
debug: fix linter warning
2021-03-10 21:22:53 +01:00
dc62ec5933
debug: use Printf/Warnf for output
2021-03-10 21:20:21 +01:00
547d9b384d
debug: cleanup repair code
2021-03-10 21:15:38 +01:00
fa7b9d5dfe
debug: Cleanup pack size checks
2021-03-10 20:57:14 +01:00
6774fc6454
debug: check arguments and cleanup error handling
2021-03-10 20:43:22 +01:00
096f15db5c
debug: extract examinePack function
2021-03-10 20:21:05 +01:00
84491ff40b
debug: store repaired and correct blobs
2021-03-10 20:03:44 +01:00
b3c3121622
debug: Save raw decrypt (disregarding signature)
2021-03-10 20:03:44 +01:00
ce4b6d0874
examine: add byte repair mode
2021-03-10 20:03:44 +01:00
52061e817c
debug: fix percentage
2021-03-10 20:03:44 +01:00
133ac42a0b
debug: check file content hash
2021-03-10 20:03:44 +01:00
90f975fa1c
debug: make output less verbose
2021-03-10 20:03:44 +01:00
086993bae1
debug: check packs not in index, implement repair
2021-03-10 20:03:44 +01:00
d6f78163d4
Add 'debug examine' command to debug #1999
2021-03-10 20:03:44 +01:00
c9b4fadd91
Merge pull request #3294 from Achilleshiel/fix-copy-repofile
...
Add RepositoryFile2 Option for secondary repository
2021-03-08 22:50:31 +01:00
da458a55db
Cleanup comments in secondary repo test
2021-03-08 22:41:13 +01:00
9ccdba9df6
Add repositoryFile2 option
...
The `init` and `copy` commands can now use `--repository-file2` flag and
the `$RESTIC_REPOSITORY_FILE2` environment variable.
This also fixes the conflict with the `--repository-file` and `--repo2`
flag.
Tests are added for the initSecondaryGlobalOpts function.
This adds a NOK function to the test helper functions. This NOK tests if
err is not nil, and otherwise fail the test.
With the NOK function a couple of sad paths are tested in the
initSecondaryGlobalOpts function.
In total the tests checks wether the following are passed correct:
- Password
- PasswordFile
- Repo
- RepositoryFile
The following situation must return an error to pass the test:
- no Repo or RepositoryFile defined
- Repo and RepositoryFile defined both
2021-03-08 22:41:13 +01:00
427781928f
copy: test that trees with unstable json encoding are properly copied
2021-03-08 22:16:48 +01:00
2fc7abac35
copy: copy raw bytes of tree blobs
...
This avoids problems when for some reason the JSON encoding changes.
This also ensures forward compatibility with future restic versions
which might e.g. add new fields to the tree metadata.
2021-03-08 22:16:48 +01:00
18fccb5995
backup: In case of error also Wait() for terminal goroutine
...
This prevents a race condition where the final summary message can get lost.
2021-03-08 22:11:34 +01:00
814a399e4c
Merge pull request #3283 from dennypage/master
...
Treat an empty password as a fatal error for repository init.
2021-02-28 00:49:21 +01:00
0f41e99ea7
Treat an empty password as a fatal error for repository init.
2021-02-28 00:40:00 +01:00
9e852af5be
check: Fix crash of --read-data-subset=x% on empty repository
...
Rounding up to at least one pack file should only be done if at least a
pack file exists.
2021-02-27 16:05:36 +01:00
65bd2a9a49
backup: Correctly handle --quiet flag
...
The quiet flag changes the backup output to assume a non-interactive
terminal. However, the output progress interval was not set to 0 by
default.
2021-02-15 22:14:58 +01:00
bb53fcfc0d
Set development version for 0.12.0
2021-02-14 11:44:24 +01:00
27f241334e
Add version for 0.12.0
2021-02-14 11:44:20 +01:00
c0f2c1d871
ls: Check for non-nil error before calling panic()
2021-02-07 21:12:54 +01:00
39a26066f7
rebuild-index: add missing bar.Done()
2021-01-31 18:28:02 +01:00
b3dc127af5
Merge pull request #3207 from aawsome/filerestorer-fix-errorhandling
...
Fix error handling in filerestorer and fix #3166
2021-01-31 17:47:46 +01:00
6e942693ba
Fix #3166
2021-01-31 14:22:57 +01:00
027a51529d
prune: Improve error message for missing files
...
This commit changes the error message so that a list of file names is
printed. Before, just the raw map was printed, which is not a great user
interface.
2021-01-31 11:31:27 +01:00
f867e65bcd
Fix issues reported by staticcheck
2021-01-30 20:43:53 +01:00
aef3658a5f
Address review comments
2021-01-30 20:02:37 +01:00
200f09522d
Add more error checks
2021-01-30 20:02:37 +01:00
cbd88c457a
backup: Improve error handling
2021-01-30 20:02:37 +01:00
1a0eb05bfa
errcheck: Add more error checks
2021-01-30 20:02:37 +01:00
4d576c2f79
Merge pull request #3217 from M4a1x/read-data-subset-fix
...
Fix missing rand seed for restic check --read-data-subset=x%
2021-01-29 23:34:13 +01:00
f9e1fa26ff
Fix missing rand seed for restic check --read-data-subset=x%
2021-01-29 23:18:35 +01:00
e08e65dc30
prune: Simplify logic selecting packs to repack
2021-01-29 22:27:22 +01:00
daeb4cdf8f
prune: Fix statistics for --repack-cacheable-only
2021-01-29 22:27:22 +01:00
a16ce65295
Merge pull request #3244 from MichaelEischer/better-damage-reports
...
Print more details about possible repository damages
2021-01-29 11:45:45 +01:00
de7e3a0648
Merge pull request #2823 from greatroar/trust-mtime
...
Add --ignore-ctime flag to backup and document change detection
2021-01-29 00:02:19 +01:00
6bd8a2faaa
backup: Add --ignore-ctime option and document change detection
2021-01-28 23:42:10 +01:00
58b5679f14
prune: reword missing blobs error
...
The previous wording could be understood such that the prune run did
damage the repository.
2021-01-28 21:48:24 +01:00
7b8886c052
prune: report missing but unneeded pack files
...
This indicates a damaged repository so add some output to help with
debugging.
2021-01-28 21:46:01 +01:00
ff95999246
rebuild-index: report added/removed/reindexed files
...
This should help with investigating missing pack files.
2021-01-28 21:46:01 +01:00
b71c52797a
find: correctly expand multiple blob ids
...
For example `restic find --show-pack-id --blob f78dc991 5b9e4366 ddd8c7d4`
would previously only expand one blob if all of them belong to the same
file.
2021-01-28 21:21:54 +01:00
82140967d3
Merge pull request #3228 from aawsome/prune-all-trees
...
prune: Remove all unused trees
2021-01-28 21:04:27 +01:00
35033d9b79
Merge pull request #3177 from MichaelEischer/fix-2759
...
prune: don't print stacktrace on console
2021-01-28 20:28:45 +01:00
4d40c70214
Merge pull request #3211 from MichaelEischer/sftp-speedup
...
Speed-up caching/pack download via SFTP
2021-01-28 14:16:55 +01:00
44169d0dc4
Merge pull request #3205 from MichaelEischer/fix-quiet-verbose
...
Properly check that --quiet and --verbose are not combined
2021-01-28 13:53:31 +01:00
bdfedf1f5b
Merge pull request #3173 from MichaelEischer/unify-index-loading
...
Unify index loading
2021-01-28 13:50:42 +01:00
e2b0072441
check: add progress bar to the tree structure check
2021-01-28 11:10:50 +01:00
505f8a2229
progress/counter: Support updating the progress bar maximum
2021-01-28 11:10:47 +01:00
eda8c67616
restic: let FindUsedBlobs handle multiple snapshots at once
2021-01-28 11:08:43 +01:00
258ce0c1e5
parallel: report progress for StreamTrees
...
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
2021-01-28 11:08:43 +01:00
3d6a3e2555
copy: Remove treeCloner struct
2021-01-28 11:08:43 +01:00
0caad1e890
copy: parallelize tree walk
2021-01-28 11:08:43 +01:00
a4689eb3b9
Merge pull request #3199 from MichaelEischer/non-interactive-counter
...
Don't print progress on non-interactive terminals
2021-01-28 10:53:38 +01:00
d7dc19a496
prune: Always repack packs containing tree blobs
2021-01-15 16:42:04 +01:00
f3442ce8a5
Test that WriteTo of a backend's Load remains accessible
2021-01-03 22:23:53 +01:00
97388b3504
Properly check that --quiet and --verbose are not combined
...
If --verbose is specified once, then globalOptions.Verbose == 1.
Previously --quiet --verbose would silently ignore the --verbose flag.
2020-12-30 21:24:18 +01:00
e1efc193e1
Merge pull request #3139 from aawsome/prune-healing
...
prune: Add healing of repository in some situations
2020-12-29 22:17:27 +01:00
f0113139ea
prune: Correct error message
2020-12-29 20:20:05 +01:00
f6df94a50e
prune: Add self-healing
...
Allow prune to heal situations where blobs in the index are missing or
the corresponding packfiles are damaged if those blobs are not needed.
2020-12-29 20:20:05 +01:00
31e56f1ad5
Merge pull request #3197 from SkYNewZ/fix/3183
...
Fix tag handling for multiple tag lists
2020-12-29 18:44:38 +01:00
dec5008369
Warn when unlock fails instead of returning an error
...
Only one caller was checking the error.
2020-12-29 17:48:20 +01:00
25ecf9eafb
fix(cmd_tag): Use restic.TagLists
2020-12-29 17:12:46 +01:00
e88f3fb80c
fix(cmd_backup): Use restic.TagLists
2020-12-29 17:12:46 +01:00
05a987b07c
Support values less than 1 for RESTIC_PROGRESS_FPS
...
For example set the variable to 0.016666 to print the progress once per
minute.
2020-12-29 16:32:18 +01:00
85fe5feadb
Unify progress report frequency calculation
2020-12-29 16:03:43 +01:00
969141b5e9
Honor RESTIC_PROGRESS_FPS env variable on non-interactive terminals
...
This makes it possible to use the environment variable to also get
regular progress updates on non-interactive terminals.
2020-12-29 16:03:43 +01:00
c2ef049f1b
ui/progress: don't print progress on non-interactive terminals
...
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058 .
2020-12-29 16:03:43 +01:00
18531e3d6f
Portability fixes to internal/restic
...
syscall.Mknod is not available on AIX.
2020-12-29 01:35:01 +01:00
332b1896d1
Some `options` fixes
...
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
2020-12-23 23:26:04 +03:00
d8d2cc6dd9
Simplify rebuild-index code
2020-12-22 23:01:58 +01:00
68b74e359e
Count packs directly in RebuildIndexFiles
2020-12-22 23:01:58 +01:00
ccc84af73d
debug/list: parallelize index loading
2020-12-22 22:36:18 +01:00
edf89e1c74
forget: Enable --dry-run together with --prune
2020-12-22 20:58:02 +01:00
cfea79d0c5
prune: don't print stacktrace on console
2020-12-19 14:28:48 +01:00
2ca76afc2b
dump: Add new option --archive
2020-12-19 01:09:47 +03:00
0256f95994
dump: Split tar and walk logic
2020-12-19 01:09:47 +03:00
0d5b764f90
Merge pull request #3130 from aawsome/snapshots-parallel
...
Make loading snapshots parallel
2020-12-06 21:08:18 +01:00
e329623771
Remove LoadAllSnapshots
2020-12-06 05:22:27 +01:00
5b9ee56335
Add ForAllSnapshots
2020-12-06 05:04:21 +01:00
83c8a9b058
Bugfix: packSizeFromList should save size from List()
2020-12-05 20:58:36 +01:00
f72f6c9c80
Fix debug build
2020-11-29 18:44:36 +01:00
52b98f7f95
Merge pull request #3017 from greatroar/files-from0
...
Add backup options --files-from-verbatim and --files-from-raw
2020-11-29 18:15:21 +01:00
04d856e601
Merge pull request #3136 from restic/rawtaz-copy-doc
...
doc: Emphasize double transfer and duplication in copy command
2020-11-29 13:57:10 +01:00
a7b49c4889
Merge pull request #3119 from restic/keep-mountpoints
...
Keep mountpoints as empty directories for --one-file-system
2020-11-29 11:22:12 +01:00
f70b10d0ee
doc: Emphasize double transfer and duplication in copy command
2020-11-28 19:49:34 +01:00
55bf76ba0c
backup: Add --files-from-{verbatim,raw} options
2020-11-28 18:22:31 +01:00
82ae942965
backup: Keep mountpoints for --one-file-system
...
When a file system is mounted at a directory, lstat() returns attributes
of the root node of the mounted file system, including the device ID of
the other file system. The previous code used when --one-file-system is
specified excluded the directory itself because of that.
This commit changes the code so that mountpoints are kept as empty
directories, its attributes set to the root note of the mounted file
system. The behavior mimics `tar`, which does the same.
2020-11-28 17:00:31 +01:00
f576d3d826
Add tests
2020-11-28 17:00:31 +01:00
037f0a4c91
Refactor device ID checking
2020-11-28 17:00:31 +01:00
a3105799c9
Print warning when unlocking the repo fails
2020-11-28 11:31:25 +01:00
666768cd17
cat: Respect --no-lock flag
2020-11-28 11:30:02 +01:00
aa7a5f19c2
Use BlobHandle in index methods
2020-11-22 20:41:12 +01:00
92bd448691
Make BlobHandle substruct of Blob
2020-11-22 20:41:10 +01:00
67c938f232
Use PackSize in prune
2020-11-21 22:13:54 +01:00
4960b841e6
Use PackSize in rebuild-index
2020-11-21 22:13:54 +01:00
110a32a08b
Merge pull request #3113 from aawsome/fix-prune-stats-duplicates
...
Fix statistics in prune for duplicates
2020-11-19 20:34:11 +01:00
cb5ec7ea6b
Fix statistics in prune for duplicates
...
Note that this fix only solves the statistics problem, if
all duplicates are marked for repacking.
If not all duplicates are marked for repacking, we lack the
information which
The situation that not all duplicates are marked for repacking can occur
when using the `max-repack-size` option
2020-11-18 22:30:22 +01:00
625410f003
Limit progress bar updates to once per second on non-terminal outputs
...
The code accidentally checked whether stdin is a terminal instead of
stdout, the former is not relevant here as the output is printed on
stdout.
2020-11-18 22:12:07 +01:00
d857fb6e59
Allow using --repository-file in init
2020-11-17 20:43:46 +01:00
1f43cac12d
check: Only track data blobs when unused blobs should be reported
...
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00
3500f9490c
check: Simplify blob status tracking
...
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
2020-11-15 18:42:42 +01:00
caac38ed27
check: Tweak subset percentage over 100% error message
2020-11-15 18:13:50 +01:00
8f9cea8cc0
Check data subset: check random percentage subset
2020-11-15 18:13:50 +01:00
3c0c0c132b
Merge pull request #3006 from aawsome/new-rebuild-index
...
Reimplement rebuild-index and remove /internal/index
2020-11-15 17:48:43 +01:00
3c03b35212
Fix #1681 should not try to create the mount point if it doesn't exist, rather return an error
2020-11-15 17:09:30 +01:00
9607cad267
Remove internal/index
2020-11-15 07:05:09 +01:00
30b6a0878a
Reimplement rebuild-index
2020-11-15 07:05:09 +01:00
1ec628ddf5
Add extraObsolete to MasterIndex.Save
2020-11-15 07:05:09 +01:00
3601a9b6cd
Merge pull request #2690 from SkYNewZ/master
...
Fix #2688 : Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
fdec8051ab
tags: Tweak description of --add/set/remove options
2020-11-14 22:55:30 +01:00
333c5a19d4
Merge pull request #3082 from aawsome/check-sizes
...
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
a8ad6b9a4b
fix(tags): Change tags list type according to restic.TagList
2020-11-14 16:24:58 +00:00
b0882b3f3c
fix(snapshots): Update help message to better match the 'forget' command one
2020-11-14 15:48:57 +00:00
ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list
2020-11-14 15:48:56 +00:00
80dcfca191
check: Check sizes computed from index and pack header
2020-11-14 00:42:49 +01:00
59ea5a4208
diff: Correctly count top-level blobs
2020-11-13 21:11:21 +01:00
63e32c44c0
Improve error reporting from restic diff
...
Instead of a stacktrace, restic diff 111 222 now reports:
Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
21b787a4d1
Stop Counters where they're constructed and started
2020-11-09 13:03:31 +01:00
ddca699cd2
Replace restic.Progress with new progress.Counter
...
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
a8d21b5dcf
prune: Warn if no cache is present
2020-11-08 20:25:35 +01:00
823d0afd6e
backup: Always print parent snapshot info
2020-11-08 20:25:35 +01:00
47277c4b4c
Add comments, clarify computation
2020-11-06 20:23:30 +01:00
d2e53730d6
Add test that repo.List is only called once
2020-11-06 20:23:30 +01:00
fd33030556
Use in-memory index to rebuild index in prune
2020-11-06 20:23:30 +01:00
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command
...
Reimplementation of prune
2020-11-05 10:12:19 +01:00
d35d279455
Set development version for 0.11.0
2020-11-05 09:41:40 +01:00
b7e1ece1e0
Add version for 0.11.0
2020-11-05 09:41:35 +01:00
916b2d303b
vss: fix potential crash (not reachable in restic) ( #3045 )
...
HasSufficientPrivilegesForVSS() now returns an error
2020-11-04 22:14:18 +01:00
1ca60bccfb
Refactor condition for MaxRepackBytes
...
Don't depend on the string (opts.MaxRepackSize) for the condition,
instead check if there's a (positive) limit configured.
2020-11-03 16:42:21 +01:00
7f86eb4ec0
Move helper function
2020-11-03 16:42:21 +01:00
c1a3de4a6e
Refactor max-unused calculation, add `unlimited` option
...
Add a callback to the PruneOptions struct which calculates the number of
bytes allowed to be unused after prune is done. This way, the logic is
closer to the option parsing code.
Also, add an explicit option `unlimited` for the use case when storage
does not matter but bandwidth and time do. Internally, this sets the
maximum number of unused bytes to MaxUint64.
Rework the documentation slightly so that no more "packs" are
mentioned and it talks about "files" instead.
Make it clear in the documentation that the percentage given to
`--max-unused` is relative to the whole repository size after pruning is
done. If specified, it must be below 100%, otherwise the repository
would contain 100% of unused data, which is pointless.
I had a hard time coming up with the correct formula to calculate the
maximum number of unused bytes based on the number of used bytes. For a
fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of
unused bytes `x` the following holds:
x ≤ p * (u+x)
⇔ x ≤ p*u + p*x
⇔ x - p*x ≤ p*u
⇔ x * (1-p) ≤ p*u
⇔ x ≤ p/(1-p) * u
2020-11-03 16:42:21 +01:00
f8c4dd7b1a
Split packe rewrite logic into two case branches
...
The comma is too sublte, let's split this into two separate branches.
2020-11-03 16:42:21 +01:00
a5b80452fe
Add comment that usedBlobs is modified
2020-11-03 16:42:21 +01:00
aff1e220f5
Split struct members, add comments
2020-11-03 16:42:21 +01:00
095155d9ce
Remove RepackSmall
2020-11-03 16:42:21 +01:00
b2f5381737
Make realistic forget --prune --dryrun
2020-11-03 16:42:21 +01:00
7f9a0a5907
Reimplementation of prune
2020-11-03 16:42:21 +01:00
3b591ed987
Add Verboseff
2020-11-03 16:42:21 +01:00
21ba15577e
Return an error for invalid sizes
2020-11-03 11:53:49 +01:00
6509c207f4
Merge pull request #2997 from MichaelEischer/faster-excludes
...
Speedup exclude/include checking
2020-11-02 11:07:21 +01:00
445b845267
Merge pull request #2978 from MichaelEischer/warn-tree-error
...
Warn if backup failed to read tree blob
2020-11-02 10:14:12 +01:00
3ff37215df
Merge pull request #2935 from MichaelEischer/upgrade-minio
...
Upgrade minio SDK to version 7
2020-11-02 09:09:10 +01:00
31b8d7a639
Merge pull request #2274 from fgma/master
...
Support for Volume Shadow Copy Service (VSS) on windows
2020-10-24 15:43:34 +02:00
5695f9ebd2
vss: Implement VSS support for Windows
...
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.
Snapshotting volumes also covers mountpoints but skips UNC paths.
2020-10-24 11:35:57 +02:00
3807d13bdc
restic mount: don't umount unless we actually tried to mount
...
Previously, failure to open a repository would result in an "unable to
umount" message.
2020-10-21 12:15:05 +02:00
56883817d8
Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
...
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
e638b46a13
Embed context into ReaderAt
...
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
2020-10-09 22:39:07 +02:00
2964d2ad15
Skip unlocking for nil locks
...
Now that lockRepo receives a context, it is possible that it is canceled
before a lock was created. Thus `unlockRepo` must be able to handle this
case.
2020-10-09 22:39:06 +02:00
0c9efa9c2a
Pass context to lockRepo
2020-10-09 22:39:06 +02:00
37a5e2d681
rest: use global context on repository creation
2020-10-09 22:39:06 +02:00
dc2e664209
integration_fuse_test: use global context
...
No need to use the TODO context.
2020-10-09 22:37:56 +02:00
603bb0e309
restore: Use proper context while loading snapshot
2020-10-09 22:37:56 +02:00
27456f6545
debug: use proper context
...
This allows the debug commands to be properly interrupted.
2020-10-09 22:37:56 +02:00
c458e114d4
pass context to Find / FindSnapshot
...
This allows proper interruption of restic while it searches for
snapshots or key files.
2020-10-09 22:37:56 +02:00
45e9a55c62
Wire context into backend layout detection
2020-10-09 22:37:24 +02:00
307a6ba3a3
Upgrade minio sdk to v7
...
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
2020-10-09 22:37:24 +02:00
50da20d93d
Warn if backup failed to read tree blob
2020-10-09 22:36:27 +02:00
30cb553c8d
Merge pull request #2932 from MichaelEischer/proper-rclone-create
...
Call rclone.Create to create a new repository for the rclone backend
2020-10-09 21:29:15 +02:00
bcc3bddcf4
filter: only check whether a child path could match when necessary
...
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.
name old time/op new time/op delta
FilterPatterns/Relative-4 23.3ms ± 9% 21.7ms ± 6% -6.68% (p=0.004 n=10+10)
FilterPatterns/Absolute-4 13.9ms ± 7% 10.0ms ± 5% -27.61% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 51.4ms ± 7% 47.0ms ± 7% -8.51% (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4 551ms ± 9% 190ms ± 1% -65.41% (p=0.000 n=10+8)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.665 n=10+9)
FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.480 n=9+10)
FilterPatterns/Wildcard-4 14.3MB ± 0% 14.3MB ± 0% ~ (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal)
FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)
2020-10-07 20:47:52 +02:00
375c2a56de
filter: Parse filter patterns only once
...
name old time/op new time/op delta
FilterPatterns/Relative-4 30.3ms ±10% 23.6ms ±20% -22.12% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 49.0ms ± 3% 32.3ms ± 8% -33.94% (p=0.000 n=8+10)
FilterPatterns/Wildcard-4 345ms ± 9% 334ms ±17% ~ (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4 3.93s ± 2% 0.71s ± 7% -81.98% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
FilterPatterns/Relative-4 4.63MB ± 0% 3.57MB ± 0% -22.98% (p=0.000 n=9+9)
FilterPatterns/Absolute-4 8.54MB ± 0% 3.57MB ± 0% -58.20% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 146MB ± 0% 141MB ± 0% -2.93% (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4 907MB ± 0% 4MB ± 0% -99.61% (p=0.000 n=9+9)
name old allocs/op new allocs/op delta
FilterPatterns/Relative-4 66.6k ± 0% 22.2k ± 0% -66.67% (p=0.000 n=10+10)
FilterPatterns/Absolute-4 88.7k ± 0% 22.2k ± 0% -75.00% (p=0.000 n=10+10)
FilterPatterns/Wildcard-4 1.70M ± 0% 1.63M ± 0% -3.92% (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4 4.46M ± 0% 0.02M ± 0% -99.50% (p=0.000 n=10+10)
2020-10-07 20:47:27 +02:00
0ae02f3030
Merge pull request #2844 from aawsome/prune-integration-tests
...
Add prune integration tests for many edge cases
2020-10-06 23:16:41 +02:00
af4100e07d
Replace repo/repos with repository/repositories in --no-lock help text
2020-10-06 22:49:48 +02:00
6822a58413
Add prune integration tests for many edge cases
2020-10-06 20:20:05 +02:00
1579d2a8ec
Remove some unused assignments
2020-10-06 14:55:13 +02:00
eba5dd831f
Fix typos reported by misspell
2020-10-06 14:55:13 +02:00
e105a3f391
cache/check: Clarify error message when given arguments
2020-10-06 00:08:59 +02:00
5d8cfff3f2
Merge pull request #2988 from greatroar/snapshots-json
...
Regression test for #2979
2020-10-05 21:02:52 +02:00
1ab4c710e1
Merge pull request #2987 from greatroar/no-password-for-help
...
Don't read password for generate, help or self-update
2020-10-05 20:16:19 +02:00
feedf0ebce
Don't read password for generate, help or self-update
...
Fixes #2951 .
2020-10-05 19:54:07 +02:00
1a490acd67
Merge pull request #2982 from greatroar/archiver-error-handling
...
Check error in archiver before calling Select
2020-10-05 19:19:47 +02:00
6bee0aafc2
Regression test for #2979
2020-10-05 15:08:09 +02:00
137d20a06a
snapshots: Make --json output [] instead of null when no snapshots
2020-10-05 13:14:10 +02:00
c4e2203e45
Check error in archiver before calling Select
...
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
2020-10-05 11:11:04 +02:00
bbeb439f41
Merge pull request #2937 from andreaso/self-update-output-path-fix
...
Don't require `self-update --output` placeholder file
2020-10-03 13:35:20 +02:00
ce14df303b
Don't require `self-update --output` placeholder file
...
This removes the requirement on `restic self-update --output` to point
to a path of an existing file, to overwrite. In case the specified
path does exist we still want to verify that it's a regular file,
rather than a directory or a device, which gets overwritten.
We also want to verify that a path to a new file exists within an
existing directory. The alternative being running into that issue
after the actual download, etc has completed.
While at it I also replace `errors.Errorf` with the more appropriately
verbose `errors.Fatalf`.
Resolves #2491
2020-10-03 07:02:23 +02:00
3c6671b18b
Merge pull request #2973 from restic/rawtaz-repo2
...
Correct data type for --repo2 option
2020-10-02 22:22:41 +02:00
0cce6dc31c
Correct data type for --repo2 option
2020-10-02 21:10:52 +02:00
6253ff0187
Clarify max verbose level in option description
...
Clarifies that the max verbosity level is 3, in the description of the --verbose option.
2020-10-02 20:25:34 +02:00
a687261804
--help: --compact: clarify description
2020-10-02 16:55:56 +03:00
61035d68bc
Add test for --repository-file
2020-10-01 00:50:27 +02:00
97f7855de3
Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE)
...
As an alternative to -r, this allows to read the repository URL
from a file in order to prevent certain types of information leaks,
especially for URLs containing credentials.
Fixes #1458 , fixes #2900 .
2020-10-01 00:50:26 +02:00
34ea960559
Merge pull request #2966 from MichaelEischer/recover-help-typo
...
recover: Fix typo in the command help
2020-09-30 18:16:26 +02:00
2a79c1a44d
recover: Fix typo in the command help
...
Reported-by: Steve Divskinsy <stevesbrain@users.noreply.github.com>
2020-09-30 17:44:34 +02:00
aea9f7d286
clarify verbose password prompt
2020-09-30 17:25:54 +02:00
009bd907f2
Don’t print too many messages when deleting files
...
Print detailed debug messages on file deletions only when --verbose=2 is used
2020-09-27 14:24:04 +02:00
14b312f00d
Merge pull request #2658 from creativeprojects/issue-2241
...
Don't echo authentication passwords (rest backend)
2020-09-22 22:17:53 +02:00
greatroar
Sam Lucidi
MichaelEischer
greatroar
Dan Willoughby
Alexander Neumann
Alexander Weiss
Juergen Hoetzel
Alexander Neumann
erin
Ryan Hitchman
Magnus Thor Torfason
rawtaz
Tin Lai
Leo R. Lundgren
Sébastien Gross
Andrew Jarcho
Hendrik Luup
Wouter Horlings
Juergen Hoetzel
Denny Page
Max Stabel
Quentin Lemaire
DRON-666
Kyle Brennan
fgma
kitone
fgma
Lorenz Bausch
Andreas Olsson
jtagcat
Andreas Oberritter
plumbeo