1192 Commits

Author	SHA1	Message	Date
Michael Eischer	da458a55db	Cleanup comments in secondary repo test	2021-03-08 22:41:13 +01:00
Wouter Horlings	9ccdba9df6	Add repositoryFile2 option ... The `init` and `copy` commands can now use `--repository-file2` flag and the `$RESTIC_REPOSITORY_FILE2` environment variable. This also fixes the conflict with the `--repository-file` and `--repo2` flag. Tests are added for the initSecondaryGlobalOpts function. This adds a NOK function to the test helper functions. This NOK tests if err is not nil, and otherwise fail the test. With the NOK function a couple of sad paths are tested in the initSecondaryGlobalOpts function. In total the tests checks wether the following are passed correct: - Password - PasswordFile - Repo - RepositoryFile The following situation must return an error to pass the test: - no Repo or RepositoryFile defined - Repo and RepositoryFile defined both	2021-03-08 22:41:13 +01:00
Michael Eischer	427781928f	copy: test that trees with unstable json encoding are properly copied	2021-03-08 22:16:48 +01:00
Michael Eischer	2fc7abac35	copy: copy raw bytes of tree blobs ... This avoids problems when for some reason the JSON encoding changes. This also ensures forward compatibility with future restic versions which might e.g. add new fields to the tree metadata.	2021-03-08 22:16:48 +01:00
Juergen Hoetzel	18fccb5995	backup: In case of error also Wait() for terminal goroutine ... This prevents a race condition where the final summary message can get lost.	2021-03-08 22:11:34 +01:00
MichaelEischer	814a399e4c	Merge pull request #3283 from dennypage/master ... Treat an empty password as a fatal error for repository init.	2021-02-28 00:49:21 +01:00
Denny Page	0f41e99ea7	Treat an empty password as a fatal error for repository init.	2021-02-28 00:40:00 +01:00
Michael Eischer	9e852af5be	check: Fix crash of --read-data-subset=x% on empty repository ... Rounding up to at least one pack file should only be done if at least a pack file exists.	2021-02-27 16:05:36 +01:00
Michael Eischer	65bd2a9a49	backup: Correctly handle --quiet flag ... The quiet flag changes the backup output to assume a non-interactive terminal. However, the output progress interval was not set to 0 by default.	2021-02-15 22:14:58 +01:00
Alexander Neumann	bb53fcfc0d	Set development version for 0.12.0	2021-02-14 11:44:24 +01:00
Alexander Neumann	27f241334e	Add version for 0.12.0	2021-02-14 11:44:20 +01:00
Alexander Neumann	c0f2c1d871	ls: Check for non-nil error before calling panic()	2021-02-07 21:12:54 +01:00
Alexander Weiss	39a26066f7	rebuild-index: add missing bar.Done()	2021-01-31 18:28:02 +01:00
MichaelEischer	b3dc127af5	Merge pull request #3207 from aawsome/filerestorer-fix-errorhandling ... Fix error handling in filerestorer and fix #3166	2021-01-31 17:47:46 +01:00
Alexander Weiss	6e942693ba	Fix #3166	2021-01-31 14:22:57 +01:00
Alexander Neumann	027a51529d	prune: Improve error message for missing files ... This commit changes the error message so that a list of file names is printed. Before, just the raw map was printed, which is not a great user interface.	2021-01-31 11:31:27 +01:00
Alexander Neumann	f867e65bcd	Fix issues reported by staticcheck	2021-01-30 20:43:53 +01:00
Alexander Neumann	aef3658a5f	Address review comments	2021-01-30 20:02:37 +01:00
Alexander Neumann	200f09522d	Add more error checks	2021-01-30 20:02:37 +01:00
Alexander Neumann	cbd88c457a	backup: Improve error handling	2021-01-30 20:02:37 +01:00
Alexander Neumann	1a0eb05bfa	errcheck: Add more error checks	2021-01-30 20:02:37 +01:00
MichaelEischer	4d576c2f79	Merge pull request #3217 from M4a1x/read-data-subset-fix ... Fix missing rand seed for restic check --read-data-subset=x%	2021-01-29 23:34:13 +01:00
Max Stabel	f9e1fa26ff	Fix missing rand seed for restic check --read-data-subset=x%	2021-01-29 23:18:35 +01:00
Alexander Weiss	e08e65dc30	prune: Simplify logic selecting packs to repack	2021-01-29 22:27:22 +01:00
Alexander Weiss	daeb4cdf8f	prune: Fix statistics for --repack-cacheable-only	2021-01-29 22:27:22 +01:00
Alexander Neumann	a16ce65295	Merge pull request #3244 from MichaelEischer/better-damage-reports ... Print more details about possible repository damages	2021-01-29 11:45:45 +01:00
rawtaz	de7e3a0648	Merge pull request #2823 from greatroar/trust-mtime ... Add --ignore-ctime flag to backup and document change detection	2021-01-29 00:02:19 +01:00
greatroar	6bd8a2faaa	backup: Add --ignore-ctime option and document change detection	2021-01-28 23:42:10 +01:00
Michael Eischer	58b5679f14	prune: reword missing blobs error ... The previous wording could be understood such that the prune run did damage the repository.	2021-01-28 21:48:24 +01:00
Michael Eischer	7b8886c052	prune: report missing but unneeded pack files ... This indicates a damaged repository so add some output to help with debugging.	2021-01-28 21:46:01 +01:00
Michael Eischer	ff95999246	rebuild-index: report added/removed/reindexed files ... This should help with investigating missing pack files.	2021-01-28 21:46:01 +01:00
Michael Eischer	b71c52797a	find: correctly expand multiple blob ids ... For example `restic find --show-pack-id --blob f78dc991 5b9e4366 ddd8c7d4` would previously only expand one blob if all of them belong to the same file.	2021-01-28 21:21:54 +01:00
MichaelEischer	82140967d3	Merge pull request #3228 from aawsome/prune-all-trees ... prune: Remove all unused trees	2021-01-28 21:04:27 +01:00
MichaelEischer	35033d9b79	Merge pull request #3177 from MichaelEischer/fix-2759 ... prune: don't print stacktrace on console	2021-01-28 20:28:45 +01:00
Alexander Neumann	4d40c70214	Merge pull request #3211 from MichaelEischer/sftp-speedup ... Speed-up caching/pack download via SFTP	2021-01-28 14:16:55 +01:00
Alexander Neumann	44169d0dc4	Merge pull request #3205 from MichaelEischer/fix-quiet-verbose ... Properly check that --quiet and --verbose are not combined	2021-01-28 13:53:31 +01:00
Alexander Neumann	bdfedf1f5b	Merge pull request #3173 from MichaelEischer/unify-index-loading ... Unify index loading	2021-01-28 13:50:42 +01:00
Michael Eischer	e2b0072441	check: add progress bar to the tree structure check	2021-01-28 11:10:50 +01:00
Michael Eischer	505f8a2229	progress/counter: Support updating the progress bar maximum	2021-01-28 11:10:47 +01:00
Michael Eischer	eda8c67616	restic: let FindUsedBlobs handle multiple snapshots at once	2021-01-28 11:08:43 +01:00
Michael Eischer	258ce0c1e5	parallel: report progress for StreamTrees ... This assigns an id to each tree root and then keeps track of how many tree loads (i.e. trees referenced for the first time) are pending per tree root. Once a tree root and its subtrees were fully processed there are no more pending tree loads and the tree root is reported as processed.	2021-01-28 11:08:43 +01:00
Michael Eischer	3d6a3e2555	copy: Remove treeCloner struct	2021-01-28 11:08:43 +01:00
Michael Eischer	0caad1e890	copy: parallelize tree walk	2021-01-28 11:08:43 +01:00
Alexander Neumann	a4689eb3b9	Merge pull request #3199 from MichaelEischer/non-interactive-counter ... Don't print progress on non-interactive terminals	2021-01-28 10:53:38 +01:00
Alexander Weiss	d7dc19a496	prune: Always repack packs containing tree blobs	2021-01-15 16:42:04 +01:00
Michael Eischer	f3442ce8a5	Test that WriteTo of a backend's Load remains accessible	2021-01-03 22:23:53 +01:00
Michael Eischer	97388b3504	Properly check that --quiet and --verbose are not combined ... If --verbose is specified once, then globalOptions.Verbose == 1. Previously --quiet --verbose would silently ignore the --verbose flag.	2020-12-30 21:24:18 +01:00
MichaelEischer	e1efc193e1	Merge pull request #3139 from aawsome/prune-healing ... prune: Add healing of repository in some situations	2020-12-29 22:17:27 +01:00
Alexander Weiss	f0113139ea	prune: Correct error message	2020-12-29 20:20:05 +01:00
Alexander Weiss	f6df94a50e	prune: Add self-healing ... Allow prune to heal situations where blobs in the index are missing or the corresponding packfiles are damaged if those blobs are not needed.	2020-12-29 20:20:05 +01:00
MichaelEischer	31e56f1ad5	Merge pull request #3197 from SkYNewZ/fix/3183 ... Fix tag handling for multiple tag lists	2020-12-29 18:44:38 +01:00
greatroar	dec5008369	Warn when unlock fails instead of returning an error ... Only one caller was checking the error.	2020-12-29 17:48:20 +01:00
Quentin Lemaire	25ecf9eafb	fix(cmd_tag): Use restic.TagLists	2020-12-29 17:12:46 +01:00
Quentin Lemaire	e88f3fb80c	fix(cmd_backup): Use restic.TagLists	2020-12-29 17:12:46 +01:00
Michael Eischer	05a987b07c	Support values less than 1 for RESTIC_PROGRESS_FPS ... For example set the variable to 0.016666 to print the progress once per minute.	2020-12-29 16:32:18 +01:00
Michael Eischer	85fe5feadb	Unify progress report frequency calculation	2020-12-29 16:03:43 +01:00
Michael Eischer	969141b5e9	Honor RESTIC_PROGRESS_FPS env variable on non-interactive terminals ... This makes it possible to use the environment variable to also get regular progress updates on non-interactive terminals.	2020-12-29 16:03:43 +01:00
Michael Eischer	c2ef049f1b	ui/progress: don't print progress on non-interactive terminals ... This reverts to the old behavior of not printing progress updates on non-interactive terminals. It was accidentally changed in #3058.	2020-12-29 16:03:43 +01:00
greatroar	18531e3d6f	Portability fixes to internal/restic ... syscall.Mknod is not available on AIX.	2020-12-29 01:35:01 +01:00
DRON-666	332b1896d1	Some options fixes ... Add tests for bool type. Fix subtle bug in TestOptionsApplyInvalid. Fix options list formatting.	2020-12-23 23:26:04 +03:00
Alexander Weiss	d8d2cc6dd9	Simplify rebuild-index code	2020-12-22 23:01:58 +01:00
Alexander Weiss	68b74e359e	Count packs directly in RebuildIndexFiles	2020-12-22 23:01:58 +01:00
Michael Eischer	ccc84af73d	debug/list: parallelize index loading	2020-12-22 22:36:18 +01:00
Alexander Neumann	edf89e1c74	forget: Enable --dry-run together with --prune	2020-12-22 20:58:02 +01:00
Michael Eischer	cfea79d0c5	prune: don't print stacktrace on console	2020-12-19 14:28:48 +01:00
DRON-666	2ca76afc2b	dump: Add new option --archive	2020-12-19 01:09:47 +03:00
DRON-666	0256f95994	dump: Split tar and walk logic	2020-12-19 01:09:47 +03:00
Alexander Neumann	0d5b764f90	Merge pull request #3130 from aawsome/snapshots-parallel ... Make loading snapshots parallel	2020-12-06 21:08:18 +01:00
Alexander Weiss	e329623771	Remove LoadAllSnapshots	2020-12-06 05:22:27 +01:00
Alexander Weiss	5b9ee56335	Add ForAllSnapshots	2020-12-06 05:04:21 +01:00
Alexander Weiss	83c8a9b058	Bugfix: packSizeFromList should save size from List()	2020-12-05 20:58:36 +01:00
Michael Eischer	f72f6c9c80	Fix debug build	2020-11-29 18:44:36 +01:00
MichaelEischer	52b98f7f95	Merge pull request #3017 from greatroar/files-from0 ... Add backup options --files-from-verbatim and --files-from-raw	2020-11-29 18:15:21 +01:00
Alexander Neumann	04d856e601	Merge pull request #3136 from restic/rawtaz-copy-doc ... doc: Emphasize double transfer and duplication in copy command	2020-11-29 13:57:10 +01:00
Alexander Neumann	a7b49c4889	Merge pull request #3119 from restic/keep-mountpoints ... Keep mountpoints as empty directories for --one-file-system	2020-11-29 11:22:12 +01:00
Leo R. Lundgren	f70b10d0ee	doc: Emphasize double transfer and duplication in copy command	2020-11-28 19:49:34 +01:00
greatroar	55bf76ba0c	backup: Add --files-from-{verbatim,raw} options	2020-11-28 18:22:31 +01:00
Alexander Neumann	82ae942965	backup: Keep mountpoints for --one-file-system ... When a file system is mounted at a directory, lstat() returns attributes of the root node of the mounted file system, including the device ID of the other file system. The previous code used when --one-file-system is specified excluded the directory itself because of that. This commit changes the code so that mountpoints are kept as empty directories, its attributes set to the root note of the mounted file system. The behavior mimics `tar`, which does the same.	2020-11-28 17:00:31 +01:00
Alexander Neumann	f576d3d826	Add tests	2020-11-28 17:00:31 +01:00
Alexander Neumann	037f0a4c91	Refactor device ID checking	2020-11-28 17:00:31 +01:00
Alexander Neumann	a3105799c9	Print warning when unlocking the repo fails	2020-11-28 11:31:25 +01:00
Kyle Brennan	666768cd17	cat: Respect --no-lock flag	2020-11-28 11:30:02 +01:00
Alexander Weiss	aa7a5f19c2	Use BlobHandle in index methods	2020-11-22 20:41:12 +01:00
Alexander Weiss	92bd448691	Make BlobHandle substruct of Blob	2020-11-22 20:41:10 +01:00
Alexander Weiss	67c938f232	Use PackSize in prune	2020-11-21 22:13:54 +01:00
Alexander Weiss	4960b841e6	Use PackSize in rebuild-index	2020-11-21 22:13:54 +01:00
Alexander Neumann	110a32a08b	Merge pull request #3113 from aawsome/fix-prune-stats-duplicates ... Fix statistics in prune for duplicates	2020-11-19 20:34:11 +01:00
Alexander Weiss	cb5ec7ea6b	Fix statistics in prune for duplicates ... Note that this fix only solves the statistics problem, if all duplicates are marked for repacking. If not all duplicates are marked for repacking, we lack the information which The situation that not all duplicates are marked for repacking can occur when using the `max-repack-size` option	2020-11-18 22:30:22 +01:00
Michael Eischer	625410f003	Limit progress bar updates to once per second on non-terminal outputs ... The code accidentally checked whether stdin is a terminal instead of stdout, the former is not relevant here as the output is printed on stdout.	2020-11-18 22:12:07 +01:00
Alexander Neumann	d857fb6e59	Allow using --repository-file in init	2020-11-17 20:43:46 +01:00
Michael Eischer	1f43cac12d	check: Only track data blobs when unused blobs should be reported ... This improves the memory usage of check a lot as it now only has to track tree blobs when run using the default parameters.	2020-11-15 18:43:07 +01:00
Michael Eischer	3500f9490c	check: Simplify blob status tracking ... UnusedBlobs now directly reads the list of existing blobs from the repository index. This removes the need for the blobStatusExists flag, which in turn allows converting the blobRefs map into a BlobSet.	2020-11-15 18:42:42 +01:00
Michael Eischer	caac38ed27	check: Tweak subset percentage over 100% error message	2020-11-15 18:13:50 +01:00
fgma	8f9cea8cc0	Check data subset: check random percentage subset	2020-11-15 18:13:50 +01:00
Alexander Neumann	3c0c0c132b	Merge pull request #3006 from aawsome/new-rebuild-index ... Reimplement rebuild-index and remove /internal/index	2020-11-15 17:48:43 +01:00
kitone	3c03b35212	Fix #1681 should not try to create the mount point if it doesn't exist, rather return an error	2020-11-15 17:09:30 +01:00
Alexander Weiss	9607cad267	Remove internal/index	2020-11-15 07:05:09 +01:00
Alexander Weiss	30b6a0878a	Reimplement rebuild-index	2020-11-15 07:05:09 +01:00
Alexander Weiss	1ec628ddf5	Add extraObsolete to MasterIndex.Save	2020-11-15 07:05:09 +01:00
MichaelEischer	3601a9b6cd	Merge pull request #2690 from SkYNewZ/master ... Fix #2688: Handle comma-separated list tags when using backup command	2020-11-14 23:19:42 +01:00
Michael Eischer	fdec8051ab	tags: Tweak description of --add/set/remove options	2020-11-14 22:55:30 +01:00
MichaelEischer	333c5a19d4	Merge pull request #3082 from aawsome/check-sizes ... Check: check sizes of packs from index and packheader	2020-11-14 22:37:42 +01:00
Quentin Lemaire	a8ad6b9a4b	fix(tags): Change tags list type according to restic.TagList	2020-11-14 16:24:58 +00:00
Quentin Lemaire	b0882b3f3c	fix(snapshots): Update help message to better match the 'forget' command one	2020-11-14 15:48:57 +00:00
Quentin Lemaire	ae441d3134	fix(backup): Switch tags cobra type to handle comma-separated list	2020-11-14 15:48:56 +00:00
Alexander Weiss	80dcfca191	check: Check sizes computed from index and pack header	2020-11-14 00:42:49 +01:00
Alexander Neumann	59ea5a4208	diff: Correctly count top-level blobs	2020-11-13 21:11:21 +01:00
greatroar	63e32c44c0	Improve error reporting from restic diff ... Instead of a stacktrace, restic diff 111 222 now reports: Fatal: no matching ID found for prefix "111"	2020-11-11 16:40:40 +01:00
greatroar	21b787a4d1	Stop Counters where they're constructed and started	2020-11-09 13:03:31 +01:00
greatroar	ddca699cd2	Replace restic.Progress with new progress.Counter ... This fixes two race conditions while cleaning up the code.	2020-11-09 12:12:35 +01:00
Alexander Neumann	a8d21b5dcf	prune: Warn if no cache is present	2020-11-08 20:25:35 +01:00
Alexander Neumann	823d0afd6e	backup: Always print parent snapshot info	2020-11-08 20:25:35 +01:00
Alexander Neumann	47277c4b4c	Add comments, clarify computation	2020-11-06 20:23:30 +01:00
Alexander Weiss	d2e53730d6	Add test that repo.List is only called once	2020-11-06 20:23:30 +01:00
Alexander Weiss	fd33030556	Use in-memory index to rebuild index in prune	2020-11-06 20:23:30 +01:00
Alexander Neumann	5144141321	Merge pull request #2718 from aawsome/new-cleanup-command ... Reimplementation of prune	2020-11-05 10:12:19 +01:00
Alexander Neumann	d35d279455	Set development version for 0.11.0	2020-11-05 09:41:40 +01:00
Alexander Neumann	b7e1ece1e0	Add version for 0.11.0	2020-11-05 09:41:35 +01:00
fgma	916b2d303b	vss: fix potential crash (not reachable in restic) (#3045) ... HasSufficientPrivilegesForVSS() now returns an error	2020-11-04 22:14:18 +01:00
Alexander Neumann	1ca60bccfb	Refactor condition for MaxRepackBytes ... Don't depend on the string (opts.MaxRepackSize) for the condition, instead check if there's a (positive) limit configured.	2020-11-03 16:42:21 +01:00
Alexander Neumann	7f86eb4ec0	Move helper function	2020-11-03 16:42:21 +01:00
Alexander Neumann	c1a3de4a6e	Refactor max-unused calculation, add unlimited option ... Add a callback to the PruneOptions struct which calculates the number of bytes allowed to be unused after prune is done. This way, the logic is closer to the option parsing code. Also, add an explicit option `unlimited` for the use case when storage does not matter but bandwidth and time do. Internally, this sets the maximum number of unused bytes to MaxUint64. Rework the documentation slightly so that no more "packs" are mentioned and it talks about "files" instead. Make it clear in the documentation that the percentage given to `--max-unused` is relative to the whole repository size after pruning is done. If specified, it must be below 100%, otherwise the repository would contain 100% of unused data, which is pointless. I had a hard time coming up with the correct formula to calculate the maximum number of unused bytes based on the number of used bytes. For a fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of unused bytes `x` the following holds: x ≤ p * (u+x) ⇔ x ≤ p*u + p*x ⇔ x - p*x ≤ p*u ⇔ x * (1-p) ≤ p*u ⇔ x ≤ p/(1-p) * u	2020-11-03 16:42:21 +01:00
Alexander Neumann	f8c4dd7b1a	Split packe rewrite logic into two case branches ... The comma is too sublte, let's split this into two separate branches.	2020-11-03 16:42:21 +01:00
Alexander Neumann	a5b80452fe	Add comment that usedBlobs is modified	2020-11-03 16:42:21 +01:00
Alexander Neumann	aff1e220f5	Split struct members, add comments	2020-11-03 16:42:21 +01:00
Alexander Neumann	095155d9ce	Remove RepackSmall	2020-11-03 16:42:21 +01:00
Alexander Weiss	b2f5381737	Make realistic forget --prune --dryrun	2020-11-03 16:42:21 +01:00
Alexander Weiss	7f9a0a5907	Reimplementation of prune	2020-11-03 16:42:21 +01:00
Alexander Weiss	3b591ed987	Add Verboseff	2020-11-03 16:42:21 +01:00
Alexander Neumann	21ba15577e	Return an error for invalid sizes	2020-11-03 11:53:49 +01:00
Alexander Neumann	6509c207f4	Merge pull request #2997 from MichaelEischer/faster-excludes ... Speedup exclude/include checking	2020-11-02 11:07:21 +01:00
Alexander Neumann	445b845267	Merge pull request #2978 from MichaelEischer/warn-tree-error ... Warn if backup failed to read tree blob	2020-11-02 10:14:12 +01:00
Alexander Neumann	3ff37215df	Merge pull request #2935 from MichaelEischer/upgrade-minio ... Upgrade minio SDK to version 7	2020-11-02 09:09:10 +01:00
MichaelEischer	31b8d7a639	Merge pull request #2274 from fgma/master ... Support for Volume Shadow Copy Service (VSS) on windows	2020-10-24 15:43:34 +02:00
fgma	5695f9ebd2	vss: Implement VSS support for Windows ... The VSS support works for 32 and 64-bit windows, this includes a check that the restic version matches the OS architecture as required by VSS. The backup operation will fail the user has not sufficient permissions to use VSS. Snapshotting volumes also covers mountpoints but skips UNC paths.	2020-10-24 11:35:57 +02:00
greatroar	3807d13bdc	restic mount: don't umount unless we actually tried to mount ... Previously, failure to open a repository would result in an "unable to umount" message.	2020-10-21 12:15:05 +02:00
Alexander Neumann	56883817d8	Merge pull request #2990 from MichaelEischer/fix-goreport-warnings ... Fix some goreport warnings	2020-10-12 20:44:56 +02:00
Michael Eischer	e638b46a13	Embed context into ReaderAt ... The io.Reader interface does not support contexts, such that it is necessary to embed the context into the backendReaderAt struct. This has the problem that a reader might suddenly stop working when it's contained context is canceled. However, this is now problem here as the reader instances never escape the calling function.	2020-10-09 22:39:07 +02:00
Michael Eischer	2964d2ad15	Skip unlocking for nil locks ... Now that lockRepo receives a context, it is possible that it is canceled before a lock was created. Thus `unlockRepo` must be able to handle this case.	2020-10-09 22:39:06 +02:00
Michael Eischer	0c9efa9c2a	Pass context to lockRepo	2020-10-09 22:39:06 +02:00
Michael Eischer	37a5e2d681	rest: use global context on repository creation	2020-10-09 22:39:06 +02:00
Michael Eischer	dc2e664209	integration_fuse_test: use global context ... No need to use the TODO context.	2020-10-09 22:37:56 +02:00
Michael Eischer	603bb0e309	restore: Use proper context while loading snapshot	2020-10-09 22:37:56 +02:00
Michael Eischer	27456f6545	debug: use proper context ... This allows the debug commands to be properly interrupted.	2020-10-09 22:37:56 +02:00
Michael Eischer	c458e114d4	pass context to Find / FindSnapshot ... This allows proper interruption of restic while it searches for snapshots or key files.	2020-10-09 22:37:56 +02:00
Michael Eischer	45e9a55c62	Wire context into backend layout detection	2020-10-09 22:37:24 +02:00
Michael Eischer	307a6ba3a3	Upgrade minio sdk to v7 ... This changes are primarily straightforward modifications to pass the parameters in the now expected way.	2020-10-09 22:37:24 +02:00
Michael Eischer	50da20d93d	Warn if backup failed to read tree blob	2020-10-09 22:36:27 +02:00
Alexander Neumann	30cb553c8d	Merge pull request #2932 from MichaelEischer/proper-rclone-create ... Call rclone.Create to create a new repository for the rclone backend	2020-10-09 21:29:15 +02:00
Michael Eischer	bcc3bddcf4	filter: only check whether a child path could match when necessary ... When checking excludes there is no need to test whether a child path could also match the pattern, as it is by definition excluded. Previously childMayMatch was calculated but then discarded. For simple absolute paths this can account for half the time spent for checking pattern matches. name old time/op new time/op delta FilterPatterns/Relative-4 23.3ms ± 9% 21.7ms ± 6% -6.68% (p=0.004 n=10+10) FilterPatterns/Absolute-4 13.9ms ± 7% 10.0ms ± 5% -27.61% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 51.4ms ± 7% 47.0ms ± 7% -8.51% (p=0.001 n=9+9) FilterPatterns/ManyNoMatch-4 551ms ± 9% 190ms ± 1% -65.41% (p=0.000 n=10+8) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.665 n=10+9) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.480 n=9+10) FilterPatterns/Wildcard-4 14.3MB ± 0% 14.3MB ± 0% ~ (p=0.431 n=9+10) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:47:52 +02:00
Michael Eischer	375c2a56de	filter: Parse filter patterns only once ... name old time/op new time/op delta FilterPatterns/Relative-4 30.3ms ±10% 23.6ms ±20% -22.12% (p=0.000 n=10+10) FilterPatterns/Absolute-4 49.0ms ± 3% 32.3ms ± 8% -33.94% (p=0.000 n=8+10) FilterPatterns/Wildcard-4 345ms ± 9% 334ms ±17% ~ (p=0.315 n=10+10) FilterPatterns/ManyNoMatch-4 3.93s ± 2% 0.71s ± 7% -81.98% (p=0.000 n=9+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 4.63MB ± 0% 3.57MB ± 0% -22.98% (p=0.000 n=9+9) FilterPatterns/Absolute-4 8.54MB ± 0% 3.57MB ± 0% -58.20% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 146MB ± 0% 141MB ± 0% -2.93% (p=0.000 n=9+9) FilterPatterns/ManyNoMatch-4 907MB ± 0% 4MB ± 0% -99.61% (p=0.000 n=9+9) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 66.6k ± 0% 22.2k ± 0% -66.67% (p=0.000 n=10+10) FilterPatterns/Absolute-4 88.7k ± 0% 22.2k ± 0% -75.00% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 1.70M ± 0% 1.63M ± 0% -3.92% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 4.46M ± 0% 0.02M ± 0% -99.50% (p=0.000 n=10+10)	2020-10-07 20:47:27 +02:00
MichaelEischer	0ae02f3030	Merge pull request #2844 from aawsome/prune-integration-tests ... Add prune integration tests for many edge cases	2020-10-06 23:16:41 +02:00
Lorenz Bausch	af4100e07d	Replace repo/repos with repository/repositories in --no-lock help text	2020-10-06 22:49:48 +02:00
Alexander Weiss	6822a58413	Add prune integration tests for many edge cases	2020-10-06 20:20:05 +02:00
Michael Eischer	1579d2a8ec	Remove some unused assignments	2020-10-06 14:55:13 +02:00
Michael Eischer	eba5dd831f	Fix typos reported by misspell	2020-10-06 14:55:13 +02:00
Leo R. Lundgren	e105a3f391	cache/check: Clarify error message when given arguments	2020-10-06 00:08:59 +02:00
rawtaz	5d8cfff3f2	Merge pull request #2988 from greatroar/snapshots-json ... Regression test for #2979	2020-10-05 21:02:52 +02:00
MichaelEischer	1ab4c710e1	Merge pull request #2987 from greatroar/no-password-for-help ... Don't read password for generate, help or self-update	2020-10-05 20:16:19 +02:00
greatroar	feedf0ebce	Don't read password for generate, help or self-update ... Fixes #2951.	2020-10-05 19:54:07 +02:00
MichaelEischer	1a490acd67	Merge pull request #2982 from greatroar/archiver-error-handling ... Check error in archiver before calling Select	2020-10-05 19:19:47 +02:00
greatroar	6bee0aafc2	Regression test for #2979	2020-10-05 15:08:09 +02:00
Leo R. Lundgren	137d20a06a	snapshots: Make --json output [] instead of null when no snapshots	2020-10-05 13:14:10 +02:00
greatroar	c4e2203e45	Check error in archiver before calling Select ... The archiver first called the Select function for a path before checking whether the Lstat on that path actually worked. The RejectFuncs in exclude.go worked around this by checking whether they received a nil os.FileInfo. Checking first is more obvious and requires less code.	2020-10-05 11:11:04 +02:00
rawtaz	bbeb439f41	Merge pull request #2937 from andreaso/self-update-output-path-fix ... Don't require `self-update --output` placeholder file	2020-10-03 13:35:20 +02:00
Andreas Olsson	ce14df303b	Don't require self-update --output placeholder file ... This removes the requirement on `restic self-update --output` to point to a path of an existing file, to overwrite. In case the specified path does exist we still want to verify that it's a regular file, rather than a directory or a device, which gets overwritten. We also want to verify that a path to a new file exists within an existing directory. The alternative being running into that issue after the actual download, etc has completed. While at it I also replace `errors.Errorf` with the more appropriately verbose `errors.Fatalf`. Resolves #2491	2020-10-03 07:02:23 +02:00
MichaelEischer	3c6671b18b	Merge pull request #2973 from restic/rawtaz-repo2 ... Correct data type for --repo2 option	2020-10-02 22:22:41 +02:00
rawtaz	0cce6dc31c	Correct data type for --repo2 option	2020-10-02 21:10:52 +02:00
rawtaz	6253ff0187	Clarify max verbose level in option description ... Clarifies that the max verbosity level is 3, in the description of the --verbose option.	2020-10-02 20:25:34 +02:00
jtagcat	a687261804	--help: --compact: clarify description	2020-10-02 16:55:56 +03:00
Michael Eischer	61035d68bc	Add test for --repository-file	2020-10-01 00:50:27 +02:00
Andreas Oberritter	97f7855de3	Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE) ... As an alternative to -r, this allows to read the repository URL from a file in order to prevent certain types of information leaks, especially for URLs containing credentials. Fixes #1458, fixes #2900.	2020-10-01 00:50:26 +02:00
rawtaz	34ea960559	Merge pull request #2966 from MichaelEischer/recover-help-typo ... recover: Fix typo in the command help	2020-09-30 18:16:26 +02:00
Michael Eischer	2a79c1a44d	recover: Fix typo in the command help ... Reported-by: Steve Divskinsy <stevesbrain@users.noreply.github.com>	2020-09-30 17:44:34 +02:00
Michael Eischer	aea9f7d286	clarify verbose password prompt	2020-09-30 17:25:54 +02:00
plumbeo	009bd907f2	Don’t print too many messages when deleting files ... Print detailed debug messages on file deletions only when --verbose=2 is used	2020-09-27 14:24:04 +02:00
MichaelEischer	14b312f00d	Merge pull request #2658 from creativeprojects/issue-2241 ... Don't echo authentication passwords (rest backend)	2020-09-22 22:17:53 +02:00
Fred	206cadfab4	Hide password from repository URLs	2020-09-22 22:00:51 +02:00
MichaelEischer	4875f7b659	Merge pull request #2614 from greatroar/simplify-fs ... Simplify internal/fs	2020-09-21 22:05:10 +02:00
yoshiera	3cf29a777d	Fix nil check in rejectBySize	2020-09-21 19:20:24 +08:00
Michael Eischer	9333f707fa	init: use Create method for rclone backend ... This properly issues the initial repository creation command Fixes #1896	2020-09-19 19:11:43 +02:00
Alexander Neumann	429f97b887	Set development version for 0.10.0	2020-09-19 17:38:47 +02:00
Alexander Neumann	40832b2927	Add version for 0.10.0	2020-09-19 17:38:26 +02:00
Michael Eischer	f003410402	init: Add --copy-chunker-params option ... This allows creating multiple repositories with identical chunker parameters which is required for working deduplication when copying snapshots between different repositories.	2020-09-19 16:53:05 +02:00
Michael Eischer	655430550b	Extract parameters for second repository from copy command	2020-09-19 16:07:55 +02:00
Leo R. Lundgren	311ad2d2d0	Improve wording for --password-file and related options	2020-09-19 15:47:32 +02:00
Leo R. Lundgren	ffe6dce7e7	Improve wording for --password-command and --password-command2 options	2020-09-19 14:23:25 +02:00
yoshiera	ac4b8c98ac	Support excluding files by size	2020-09-19 19:52:09 +08:00
greatroar	9abef3bf1a	Move internal/fs.TestChdir to internal/test.Chdir	2020-09-17 10:43:33 +02:00
Michael Eischer	f5c448aa65	diff: Optimize diff calculation for shared subtrees ... When the diff calculation compares two trees with identical id then no differences between them can ever show up. Optimize for that case by simply traversing the tree only once to collect all referenced blobs for a proper calculation of added and removed blobs. Just skipping the common subtrees is not possible as this would skew the results if the added or removed blobs are shared with one of the subtrees.	2020-09-13 16:28:46 +02:00
Michael Eischer	c0fc85d303	diff: Add integration test	2020-09-13 16:28:46 +02:00
Alexander Neumann	0c48e515f0	Merge pull request #2630 from MichaelEischer/fix-staticcheck ... Fix lots of small issues reported by staticcheck	2020-09-13 16:19:22 +02:00
rawtaz	d3a286928a	Merge pull request #2905 from MichaelEischer/fix-ls-help ... ls: Explicitly enforce that the user specifies a snapshot ID	2020-09-05 10:44:01 +02:00
Michael Eischer	1ede018ea6	error variable names should start with 'Err'	2020-09-05 10:07:17 +02:00
Michael Eischer	d0329cf3eb	Adjust comments to match name of exported methods	2020-09-05 10:07:16 +02:00
Michael Eischer	b22655367c	integration_test: Replace fprintf without format string	2020-09-05 10:07:16 +02:00
Michael Eischer	068a3ce23f	Remove redundant return	2020-09-05 10:07:16 +02:00
Michael Eischer	d19f05c960	Use buf.String() instead of string(buf.Bytes())	2020-09-05 10:07:16 +02:00
Michael Eischer	460e2ffbf6	Collapse a few boolean operations	2020-09-05 10:07:14 +02:00
Michael Eischer	49b6aac3fa	Use !bytes.Equal instead of bytes.Compare !=0 to check for inequality	2020-09-05 10:06:23 +02:00
Michael Eischer	2f8335554c	Remove a few unused variables	2020-09-05 10:06:23 +02:00
Michael Eischer	37113282ca	Merge a few variable declaration and initializations	2020-09-05 10:05:34 +02:00
Michael Eischer	337725c354	Use sort.Strings	2020-09-05 10:05:34 +02:00
MichaelEischer	2ddb7ffb7e	Merge pull request #2255 from Kidswiss/tar ... Fix dumping issues with / and the first sub level	2020-09-01 21:52:17 +02:00
Michael Eischer	81dcfea11a	dump: Mention tar output in command help text	2020-08-31 22:43:10 +02:00
MichaelEischer	55071ee367	Merge pull request #2859 from buschjost/stats-filter-by-tag-and-path ... Add filter by tag and path to stats command	2020-08-31 22:11:01 +02:00
Michael Eischer	bcf44a9c3f	dump: Properly handle file paths without / prefix ... filepath.Rel failed if the requested path did not start with a `/` e.g. `restic` instead of `/restic`.	2020-08-30 18:25:42 +02:00
Oliver Buschjost	a7b4c19abf	Fix display of warning about ignored filters	2020-08-30 17:14:56 +02:00
Oliver Buschjost	d3fcfeba3a	Add filter by tag and path to stats command	2020-08-30 17:14:48 +02:00
Michael Eischer	e69449bf2c	dump: Add test for splitPath	2020-08-30 16:00:48 +02:00
MichaelEischer	ea81a0e282	Merge pull request #2606 from middelink/fix-323 ... Add copy functionality.	2020-08-30 10:18:24 +02:00
Simon Beck	d6f739ec22	dump: Extract tar code to internal/dump	2020-08-29 21:14:34 +02:00
Simon Beck	b98598e55f	dump: Fix dumping issues with / and the first sub level ... There was an issue that prevented the dump command from working correctly when either: * `/` contained multiple nodes (e.g. `restic backup /`) * dumping a file in the first sublevel was attempted (e.g. `/foo`)	2020-08-29 21:14:29 +02:00
Michael Eischer	d5f86effa1	ls: Explicitly enforce that the user specifies a snapshot ID ... The help messages suggested that the `ls` command work without explicitly passing a snapshot ID. However, this was never the case: without a snapshot ID the command just failed with the error `Ignoring "", it is not a snapshot id`. Fixes #2299	2020-08-29 14:28:53 +02:00
Michael Eischer	412623b848	copy: Reuse buffer for downloaded blobs	2020-08-29 10:48:44 +02:00
Michael Eischer	9a4796594a	integration tests: Fix checking of wrong snapshot	2020-08-29 10:48:44 +02:00
Michael Eischer	15374d22e9	integration tests: Add basic tests for copy command	2020-08-29 10:48:44 +02:00
Michael Eischer	88ad58d6cd	integration tests: Redirect directory diff into intermediate buffer	2020-08-29 10:48:44 +02:00
Michael Eischer	591a8c4cdf	integration tests: Deduplicate backup test-data setup code	2020-08-29 10:48:44 +02:00
Michael Eischer	ec9a53b7e8	copy: Mark and skip previously copied snapshots ... Use the `Original` field of the copied snapshot to store a persistent snapshot ID. This can either be the ID of the source snapshot if `Original` was not yet set or the previous value stored in the `Original` field. In order to still copy snapshots modified using the tags command the source snapshot is compared to all snapshots in the destination repository which have the same persistent ID. Snapshots are only considered equal if all fields except `Original` and `Parent` match. That way modified snapshots are still copied while avoiding duplicate copies at the same time.	2020-08-29 10:48:44 +02:00
Alexander Weiss	9867c4bbb4	Add flag --iexclude-file to backup	2020-08-28 21:04:41 +02:00
Jason Cox	2447f3f110	Update usage strings to put flags before args ... The standard UNIX-style ordering of command-line arguments places optional flags before other positional arguments. All of restic's commands support this ordering, but some of the usage strings showed the flags after the positional arguments (which restic also parses just fine). This change updates the doc strings to reflect the standard ordering. Because the `restic help` command comes directly from Cobra, there does not appear to be a way to update the argument ordering in its usage string, so it maintains the non-standard ordering (positional arguments before optional flags).	2020-08-28 02:16:22 +00:00
Michael Eischer	b0a8c4ad6c	copy: Only process each tree once ... This speeds up copying multiple overlapping snapshots from one repository to another, as we only have to copy the changed parts of later snapshots.	2020-08-26 22:17:52 +02:00
Michael Eischer	908b23fda0	copy: Update for modernized repository interface	2020-08-26 22:17:52 +02:00
greatroar	4508d406ef	copy: Remove separate SaveIndex in restic copy ... Flush does this now.	2020-08-26 22:17:52 +02:00
Pauline Middelink	7048cc3e58	Add copy functionality ... Add a copy command to copy snapshots between repositories. It allows the user to specify a destination repository, password, password-file, password-command or key-hint to supply the necessary details to open the destination repository. You need to supply a list of snapshots to copy, snapshots which already exist in the destination repository will be skipped. Note, when using the network this becomes rather slow, as it needs to read the blocks, decrypt them using the source key, then encrypt them again using the destination key before finally writing them out to the destination repository.	2020-08-26 22:17:52 +02:00
Michael Eischer	17995dec7a	Unify progress bar of check and prune commands	2020-08-25 22:47:38 +02:00
Alexander Neumann	b1b3f1ecb6	Merge pull request #2674 from MichaelEischer/prune-strict-checks ... prune: Stricter error checks	2020-08-23 10:30:41 +02:00
Bruce Dillahunty	fe69b83074	Update key password prompt (#2847)	2020-08-19 21:42:08 +02:00
Michael Eischer	08d24ff99e	prune: Include ID of all missing blobs in error message	2020-08-16 11:36:14 +02:00
Michael Eischer	1c84aceb39	prune: Test for abort on damaged repositories	2020-08-16 11:36:12 +02:00
Michael Eischer	575ed9a47e	Test that rebuild-index errors when old index cannot be removed	2020-08-16 11:34:01 +02:00
Michael Eischer	8f811642c3	Add support for integration tests to wrap the backend	2020-08-16 11:34:01 +02:00
Michael Eischer	744a15247d	prune/rebuild-index: Fail if an old index cannot be removed ... The old behavior was problematic in the context of rebuild-index as it could leave old, possibly invalid index files behind without returning a fatal error. Prune calls rebuildIndex before removing any data from the repository. For this use case failing to delete an old index MUST be treated as a fatal error. Otherwise the index could still contain an old index file that refers to blobs/packs that were later on deleted by prune. Later backup runs will assume that the affected blobs already exist in the repository which results in a backup which misses data.	2020-08-16 11:34:01 +02:00
Michael Eischer	3ba19869be	prune: Abort if any used blobs are missing ... The previous check only approximately verified whether all required blobs were found. However, after forgetting a few snapshots the repository contains lots of unused blobs whose number can be sufficient to make up for missing packs. When coupled with a malfunctioning backend that temporarily returns broken data this could cause restic to regard the corresponding packs as invalid and thereby delete data that's still in use. This change lets restic play it safe and refuse to delete anything if data is missing.	2020-08-16 11:34:01 +02:00
aawsome	0fed6a8dfc	Use "pack file" instead of "data file" (#2885) ... - changed variable names, especially changed DataFile into PackFile - changed in some comments - always use "pack file" in docu	2020-08-16 11:16:38 +02:00
MichaelEischer	08dee8a52b	Merge pull request #2865 from greatroar/unused ... Dead code removal	2020-08-08 16:15:28 +02:00
MichaelEischer	84b6f1ec53	Merge pull request #2874 from MichaelEischer/fix-verbose ... Adjust description of --verbose=n parameter	2020-08-04 23:21:05 +02:00
MichaelEischer	06fb4ea3f0	Merge pull request #2805 from jwilk-forks/verbose ... Fix examples of --verbose with argument	2020-08-04 23:11:35 +02:00
Michael Eischer	e38d415173	Adjust description of --verbose=n parameter	2020-08-04 23:07:53 +02:00
greatroar	d81a396944	Dead code removal ... Found by running golangci-lint on the entire code base.	2020-08-04 08:38:57 +02:00
MichaelEischer	38a2f9c07b	Merge pull request #2864 from greatroar/dump-error ... Fix error handling in dump (err != err)	2020-08-03 20:09:42 +02:00
rawtaz	5af2815627	Merge pull request #2821 from renard/mount-nolock ... cmd/mount: honur --no-lock flag	2020-08-03 20:09:26 +02:00
Sébastien Gross	9be4fe3e84	cmd/mount: honour --no-lock flag ... Do not lock the repository if --no-lock global flag is set. This allows to mount repositories which are archived on a read only system. Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>	2020-08-03 19:45:39 +02:00
Michael Eischer	05116e4787	prune: Cleanup progress bar handling while repacking	2020-08-03 19:32:46 +02:00
Michael Eischer	04f79b9642	prune: Stop progress bar after searching used blobs	2020-08-03 19:31:49 +02:00
greatroar	8b358935a0	Fix error handling in dump (err != err)	2020-08-03 09:43:44 +02:00
MichaelEischer	66d089e239	Merge pull request #2841 from aawsome/optimize-getUsedBlobs ... Extract get used blobs in prune as separate function	2020-08-01 22:40:33 +02:00
Alexander Weiss	9762bec091	Use optimized getUsedBlobs in prune	2020-08-01 21:07:31 +02:00
Alexander Weiss	d3692f5b81	Delete files in parallel in rebuild-index	2020-08-01 20:43:18 +02:00