restic

mirror of https://github.com/octoleo/restic.git synced 2024-11-23 13:17:42 +00:00

Author	SHA1	Message	Date
DRON-666	33adb58817	Minor fixes and linter suggestions	2020-12-19 02:04:17 +03:00
DRON-666	89ab6d557e	dump: Add zip dumper	2020-12-19 01:09:47 +03:00
DRON-666	0256f95994	dump: Split tar and walk logic	2020-12-19 01:09:47 +03:00
Michael Eischer	34b6130a0e	restic: parallelize lock file loading	2020-12-18 20:46:16 +01:00
greatroar	66d904c905	Make invalid handles permanent errors	2020-12-17 12:47:53 +01:00
greatroar	746dbda413	Mark "ssh exited" errors in SFTP as permanent	2020-12-17 12:43:09 +01:00
greatroar	f7784bddb3	Don't retry when "no space left on device" in local backend Also adds relevant documentation to the restic.Backend interface.	2020-12-17 12:43:09 +01:00
Alexander Neumann	e96677cafb	Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables swift: Add support for id based keystone v3 auth parameters	2020-12-12 16:27:38 +01:00
Michael Eischer	1d69341e88	swift: Add support for id based keystone v3 auth parameters This adds support for the following environment variables, which were previously missing: OS_USER_ID User ID for keystone v3 authentication OS_USER_DOMAIN_ID User domain ID for keystone v3 authentication OS_PROJECT_DOMAIN_ID Project domain ID for keystone v3 authentication OS_TRUST_ID Trust ID for keystone v3 authentication	2020-12-11 19:22:34 +01:00
Alexander Neumann	36c5d39c2c	Fix issues reported by semgrep	2020-12-11 09:41:59 +01:00
Alexander Neumann	7facc8ccc1	Merge pull request #2505 from aawsome/fix-repo-configfile Fix repo configfile	2020-12-07 07:52:37 +01:00
Alexander Neumann	ba31c6fdaa	Merge pull request #3150 from MichaelEischer/fix-windows-redir-output termstatus: Fix canUpdateStatus detection for redirected output on windows	2020-12-06 21:17:27 +01:00
Alexander Neumann	b58799d83a	Merge pull request #3152 from MichaelEischer/fix-backup-background-hang backup: Fix shutdown hang when running in the background on linux	2020-12-06 21:16:00 +01:00
Alexander Neumann	0d5b764f90	Merge pull request #3130 from aawsome/snapshots-parallel Make loading snapshots parallel	2020-12-06 21:08:18 +01:00
Michael Eischer	b48f579530	termstatus: Fix canUpdateStatus detection for redirected output The canUpdateStatus check was simplified in #2608, but it accidentally flipped the condition. The correct check is as follows: If the output is a pipe then restic probably runs in mintty/cygwin. In that case it's possible to update the output status. In all other cases it isn't. This commit inverts to condition again to offer the previous and correct behavior.	2020-12-06 19:02:42 +01:00
Michael Eischer	401ef92c5f	backup: Fix shutdown hang when running in the background On shutdown the backup commands waits for the terminal output goroutine to stop. However while running in the background the goroutine ignored the canceled context.	2020-12-06 18:53:41 +01:00
Alexander Weiss	e329623771	Remove LoadAllSnapshots	2020-12-06 05:22:27 +01:00
Alexander Weiss	26f85779be	Parallelize ForAllSnapshots	2020-12-06 05:09:58 +01:00
Alexander Weiss	5b9ee56335	Add ForAllSnapshots	2020-12-06 05:04:21 +01:00
Michael Eischer	d05c88a5d6	fuse: Properly set uid/gid for directories In #2584 this was changed to use the uid/gid of the root node. This would be okay for the top-level directory of a snapshot, however, this change also applied to normal directories within a snapshot. This change reverts the problematic part and adds a test that directory attributes are represented correctly.	2020-11-30 23:41:49 +01:00
MichaelEischer	f2959127b6	Merge pull request #3065 from greatroar/local-subdirs Don't recurse in local backend's List if not required	2020-11-29 19:03:59 +01:00
Alexander Weiss	aa7a5f19c2	Use BlobHandle in index methods	2020-11-22 20:41:12 +01:00
Alexander Weiss	e3013271a6	Harmonize naming	2020-11-22 20:41:12 +01:00
Alexander Weiss	92bd448691	Make BlobHandle substruct of Blob	2020-11-22 20:41:10 +01:00
Alexander Weiss	a851c53cbe	Use PackSize in checker	2020-11-21 22:13:54 +01:00
Alexander Weiss	ce5d630681	Add MasterIndex.PackSize()	2020-11-21 22:13:54 +01:00
Alexander Weiss	c3ddde9e7d	Return hdrSize in ListPack	2020-11-21 22:13:54 +01:00
greatroar	8e213e82fc	backend/local: replace fs.Walk with custom walker This code is more strict in what it expects to find in the backend: depending on the layout, either a directory full of files or a directory full of such directories.	2020-11-19 16:46:42 +01:00
Alexander Neumann	75eff92b56	Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init do not require gs bucket permissions to init repository	2020-11-18 16:53:45 +01:00
eleith	a24e986b2b	do not require gs bucket permissions to init repository a gs service account may only have object permissions on an existing bucket but no bucket create/get permissions. these service accounts currently are blocked from initialization a restic repository because restic can not determine if the bucket exists. this PR updates the logic to assume the bucket exists when the bucket attribute request results in a permissions denied error. this way, restic can still initialize a repository if the service account does have object permissions fixes: https://github.com/restic/restic/issues/3100	2020-11-18 06:14:11 -08:00
rawtaz	1b6e8c888f	Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param Remove `--drive-use-trash=false` from rclone param	2020-11-17 11:21:20 +01:00
Michael Eischer	1f43cac12d	check: Only track data blobs when unused blobs should be reported This improves the memory usage of check a lot as it now only has to track tree blobs when run using the default parameters.	2020-11-15 18:43:07 +01:00
Michael Eischer	6da66c15d8	check: Simplify referenced blob tracking The result is identical as long as the context in not canceled. However, in that case the result is incomplete anyways.	2020-11-15 18:42:55 +01:00
Michael Eischer	3500f9490c	check: Simplify blob status tracking UnusedBlobs now directly reads the list of existing blobs from the repository index. This removes the need for the blobStatusExists flag, which in turn allows converting the blobRefs map into a BlobSet.	2020-11-15 18:42:42 +01:00
Michael Eischer	b8c7543a55	check: Merge 'size could not be found' and 'not found in index' errors By construction these two errors always show up in pairs: 'size could not be found' is printed when the blob is not found in the repository index. That blob is also part of the `blobs` array. Later on, check iterates over that array and checks whether the blob is marked as existing. Which cannot be the case as that mark is generated by iterating over the repository index. The merged warning no longer reports the blob index within a file. That information could also be derived by printing the affected tree using `cat` and searching for the blob.	2020-11-15 18:41:50 +01:00
Alexander Neumann	3c0c0c132b	Merge pull request #3006 from aawsome/new-rebuild-index Reimplement rebuild-index and remove /internal/index	2020-11-15 17:48:43 +01:00
Alexander Neumann	9968220652	Merge pull request #2850 from greatroar/packer-malloc Decrease allocation rate in internal/pack	2020-11-15 17:19:49 +01:00
greatroar	ab2b7d7f9a	Decrease allocation rate in internal/pack internal/repository benchmark results: name old time/op new time/op delta PackerManager-8 179ms ± 1% 181ms ± 1% +0.78% (p=0.009 n=10+10) name old speed new speed delta PackerManager-8 294MB/s ± 1% 292MB/s ± 1% -0.77% (p=0.009 n=10+10) name old alloc/op new alloc/op delta PackerManager-8 91.3kB ± 0% 72.2kB ± 0% -20.92% (p=0.000 n=9+7) name old allocs/op new allocs/op delta PackerManager-8 1.38k ± 0% 0.76k ± 0% -45.20% (p=0.000 n=10+7)	2020-11-15 16:51:47 +01:00
greatroar	9a8a2cae4c	Move pack testing logic to test file	2020-11-15 16:45:49 +01:00
Alexander Weiss	9607cad267	Remove internal/index	2020-11-15 07:05:09 +01:00
Alexander Weiss	30b6a0878a	Reimplement rebuild-index	2020-11-15 07:05:09 +01:00
Alexander Weiss	187c8fb259	Parallelize MasterIndex.Save()	2020-11-15 07:05:09 +01:00
Alexander Weiss	1ec628ddf5	Add extraObsolete to MasterIndex.Save	2020-11-15 07:05:09 +01:00
Alexander Weiss	5898cb341f	Use CreateIndexFromPacks() in test	2020-11-15 07:05:05 +01:00
Alexander Weiss	43732bb885	Add CreateIndexFromPacks()	2020-11-15 07:04:51 +01:00
greatroar	c45f8ee075	Replace restic.statT interface by concrete types name old time/op new time/op delta NodeFillUser-8 1.81µs ± 9% 1.50µs ± 5% -17.07% (p=0.000 n=19+20) NodeFromFileInfo-8 1.76µs ± 4% 1.49µs ± 6% -15.63% (p=0.000 n=20+19) name old alloc/op new alloc/op delta NodeFillUser-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20) NodeFromFileInfo-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20) name old allocs/op new allocs/op delta NodeFillUser-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20) NodeFromFileInfo-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)	2020-11-14 23:23:26 +01:00
MichaelEischer	3601a9b6cd	Merge pull request #2690 from SkYNewZ/master Fix #2688: Handle comma-separated list tags when using backup command	2020-11-14 23:19:42 +01:00
MichaelEischer	333c5a19d4	Merge pull request #3082 from aawsome/check-sizes Check: check sizes of packs from index and packheader	2020-11-14 22:37:42 +01:00
Quentin Lemaire	ae441d3134	fix(backup): Switch tags cobra type to handle comma-separated list	2020-11-14 15:48:56 +00:00
Alexander Weiss	17bb77b1f9	check: Also check blob length and offset	2020-11-14 00:42:49 +01:00
Alexander Weiss	80dcfca191	check: Check sizes computed from index and pack header	2020-11-14 00:42:49 +01:00
tofran	94a154c7ca	Remove `--drive-use-trash=false` from rclone param Google drive trash retention policy changed making this no longer a good default a go Issue #3095	2020-11-13 22:58:48 +00:00
Alexander Neumann	04d1983800	Merge pull request #3090 from fgma/vss-fix-386 vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386	2020-11-13 21:30:34 +01:00
Alexander Neumann	8ad9f88993	helpers: Improve error message	2020-11-12 20:38:31 +01:00
fgma	f9223cd827	vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386	2020-11-12 19:31:00 +01:00
rawtaz	dfb9326b1b	Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag Extended option to select V1 API for ListObjects on S3 backend	2020-11-11 20:52:44 +01:00
Alexander Neumann	4a0b7328ec	s3: Remove dots for config description	2020-11-11 20:20:35 +01:00
greatroar	63e32c44c0	Improve error reporting from restic diff Instead of a stacktrace, restic diff 111 222 now reports: Fatal: no matching ID found for prefix "111"	2020-11-11 16:40:40 +01:00
Nick Douma	829959390a	Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1	2020-11-11 11:54:38 +01:00
Nick Douma	ccd55d529d	Add s3.list-objects-v1 extended option and default to false	2020-11-11 11:54:36 +01:00
Nick Douma	4ddcc17135	Add support for boolean extended options	2020-11-11 11:54:27 +01:00
MichaelEischer	407843c5f9	Merge pull request #3034 from hoyho/gh_master bugfix: omit ENOTDATA for extended attributes	2020-11-09 22:38:49 +01:00
MichaelEischer	46d31ab86d	Merge pull request #3058 from greatroar/counter Replace restic.Progress with new progress.Counter (fixes two race conditions)	2020-11-09 22:19:09 +01:00
Alexander Neumann	c986823d3f	Merge pull request #3048 from aawsome/check-pack-index check: check index for packs that are read	2020-11-09 20:12:32 +01:00
Alexander Weiss	239931578c	check: check index for packs that are read	2020-11-09 17:28:14 +01:00
hoyho	9df52327cc	bugfix: omit ENOTDATA for extended attributes Signed-off-by: hoyho <luohaihao@gmail.com>	2020-11-10 00:20:34 +08:00
greatroar	21b787a4d1	Stop Counters where they're constructed and started	2020-11-09 13:03:31 +01:00
greatroar	ddca699cd2	Replace restic.Progress with new progress.Counter This fixes two race conditions while cleaning up the code.	2020-11-09 12:12:35 +01:00
Alexander Neumann	3bf43d7951	Merge pull request #2475 from restic/use-github-actions Use GitHub Actions for CI	2020-11-08 17:30:12 +01:00
Alexander Neumann	5cf42884c8	Run tests on GitHub Actions	2020-11-08 16:04:57 +01:00
Alexander Weiss	826cfa0533	fix context in archiver tests	2020-11-08 08:24:24 +01:00
Alexander Weiss	fef408a8bd	Return context error in mem backend	2020-11-08 00:05:53 +01:00
greatroar	a2d4209322	Don't recurse in local backend's List if not required Due to the return if !isFile, the IsDir branch in List was never taken and subdirectories were traversed recursively. Also replaced isFile by an IsRegular check, which has been equivalent since Go 1.12 (golang/go@a2a3dd00c9).	2020-11-07 08:54:13 +01:00
Alexander Weiss	fd33030556	Use in-memory index to rebuild index in prune	2020-11-06 20:23:30 +01:00
Alexander Weiss	38cc4393f6	Add Masterindex.Save(); Add Index.Packs()	2020-11-06 20:23:30 +01:00
Alexander Weiss	aaf1c44362	Fix #3062	2020-11-05 17:05:42 +01:00
Ivan Andreev	ab2790d9de	Fix http2 stream reset between restic and rest backends #3014	2020-11-05 15:57:40 +03:00
Nick Craig-Wood	86b5d8ffaa	s3: add bucket-lookup parameter to select path or dns style bucket lookup This is to enable restic working with Alibaba cloud Fixes #2528	2020-11-05 12:20:10 +01:00
Alexander Neumann	ae5302c7a8	Add comment that keepBlobs is modified	2020-11-05 10:33:38 +01:00
Alexander Neumann	866a52ad4e	Remove unneeded seek The file returned from DownloadAndHash() is already seeked to the start of the file.	2020-11-05 10:31:49 +01:00
Alexander Neumann	a4507610a0	Fix typo	2020-11-05 10:31:49 +01:00
Alexander Neumann	7def2d8ea7	Use a non-constant seed	2020-11-05 10:31:49 +01:00
Alexander Neumann	ee0112ab3b	Clarify message about expected error	2020-11-05 10:31:49 +01:00
Michael Eischer	b373f164fe	prune: Parallelize repack command	2020-11-05 10:31:49 +01:00
Alexander Neumann	5144141321	Merge pull request #2718 from aawsome/new-cleanup-command Reimplementation of prune	2020-11-05 10:12:19 +01:00
fgma	916b2d303b	vss: fix potential crash (not reachable in restic) (#3045 ) HasSufficientPrivilegesForVSS() now returns an error	2020-11-04 22:14:18 +01:00
Alexander Neumann	a06f5c28c0	Merge pull request #3055 from greatroar/cleanup Don't separately allocate sync.{Mutex,Once} if not necessary	2020-11-04 11:11:21 +01:00
greatroar	c2f3eee5af	Don't separately allocate sync.{Mutex,Once} if not necessary Separate allocation of synchronization devices suggests they're shared between objects, but they're not.	2020-11-04 09:55:43 +01:00
Alexander Weiss	b2f5381737	Make realistic forget --prune --dryrun	2020-11-03 16:42:21 +01:00
Alexander Weiss	ce7d613749	Add Blob.Handle()	2020-11-03 16:42:21 +01:00
Alexander Weiss	581d90cf91	Make some pack parameters public	2020-11-03 16:42:21 +01:00
Alexander Neumann	9a88fb253b	Merge pull request #3051 from greatroar/sanitize-env Sanitize environment before starting backend processes (rclone, ssh)	2020-11-02 21:18:57 +01:00
Alexander Neumann	f14436953a	Merge pull request #3050 from greatroar/widechars Fix string truncation in ui/termstatus	2020-11-02 16:58:51 +01:00
greatroar	11fbaaae9a	Sanitize environment before starting backend processes (rclone, ssh) The restic security model includes full trust of the local machine, so this should not fix any actual security problems, but it's better to be safe than sorry. Fixes #2192.	2020-11-02 16:41:23 +01:00
greatroar	3ed84ff0c6	Fix string truncation in ui/termstatus Fixes #3046.	2020-11-02 12:50:49 +01:00
Alexander Neumann	8e965ed4eb	Merge pull request #3039 from MichaelEischer/tar-proper-dirs Properly dump directories to tar	2020-11-02 12:30:26 +01:00
Alexander Neumann	5f0fa2129e	Improve readability It's time to use a switch statement.	2020-11-02 11:24:26 +01:00
Alexander Neumann	6509c207f4	Merge pull request #2997 from MichaelEischer/faster-excludes Speedup exclude/include checking	2020-11-02 11:07:21 +01:00
Alexander Neumann	445b845267	Merge pull request #2978 from MichaelEischer/warn-tree-error Warn if backup failed to read tree blob	2020-11-02 10:14:12 +01:00
Alexander Neumann	3ff37215df	Merge pull request #2935 from MichaelEischer/upgrade-minio Upgrade minio SDK to version 7	2020-11-02 09:09:10 +01:00
Sergio Rubio	e708628cfd	Remove unused function Not currently used, and it'd need to be added to the MasterIndex interface first.	2020-10-28 13:24:49 +01:00
MichaelEischer	bb4b3481a6	Merge pull request #3031 from greatroar/debug-no-seek Allow debug logging to pipes and terminals	2020-10-25 17:59:06 +01:00
MichaelEischer	ad3a52e6f0	Merge pull request #3026 from greatroar/refactor-ui internal/ui refactoring	2020-10-25 17:51:39 +01:00
Michael Eischer	8d7d6ad2d5	dump: include username in tar	2020-10-24 22:42:01 +02:00
Michael Eischer	fe09e6f865	dump: test proper permissions and directory name	2020-10-24 22:42:01 +02:00
Michael Eischer	1e3c9a2c11	dump: Fix file permission to tar mapping The file permissions included a go specific directory bit which accidentially forced the usage of the GNU header format. This leads to problems with 7zip on Windows or when extended attributes are used.	2020-10-24 22:42:01 +02:00
Michael Eischer	e21dcb0eea	dump: Additional ACL tests	2020-10-24 22:41:01 +02:00
MichaelEischer	31b8d7a639	Merge pull request #2274 from fgma/master Support for Volume Shadow Copy Service (VSS) on windows	2020-10-24 15:43:34 +02:00
fgma	5695f9ebd2	vss: Implement VSS support for Windows The VSS support works for 32 and 64-bit windows, this includes a check that the restic version matches the OS architecture as required by VSS. The backup operation will fail the user has not sufficient permissions to use VSS. Snapshotting volumes also covers mountpoints but skips UNC paths.	2020-10-24 11:35:57 +02:00
greatroar	63be3704d9	Allow debug logging to pipes and terminals	2020-10-19 22:30:30 +02:00
greatroar	35419de232	Simplify ui.StdioWrapper.Write Instead of looping to find line breaks, make it look for the last one.	2020-10-17 20:30:46 +02:00
greatroar	863a590a81	Refactor termstatus.Term.{Print,Error} methods	2020-10-17 20:30:46 +02:00
greatroar	7c0b6a82db	Remove unused public method ui.linesWriter.Flush	2020-10-17 20:30:24 +02:00
MichaelEischer	96a912b65a	Merge pull request #3025 from aawsome/fix-index-ids Fix setting of ID in DecodeIndex	2020-10-17 09:54:45 +02:00
Alexander Weiss	b44ecde8b0	Fix setting of ID in DecodeIndex	2020-10-17 09:12:58 +02:00
MichaelEischer	39fe1e96fe	Merge pull request #3018 from greatroar/background-check Fix IsProcessBackground on Linux with stdin redirection	2020-10-16 22:50:19 +02:00
MichaelEischer	4ba237bb93	Merge pull request #3019 from greatroar/refactor-decodeindex Refactor index decoding	2020-10-15 23:22:33 +02:00
greatroar	b27375f5ce	defer close(ch) outside repository.RunWorkers	2020-10-14 15:50:16 +02:00
greatroar	720e0ee0c7	if cond { return true }; return false => return cond	2020-10-13 20:56:43 +02:00
greatroar	27db3ec262	Refactor index decoding Decoding old-format indices no longer requires loading and decrypting twice.	2020-10-13 20:47:50 +02:00
greatroar	f80b07b2c8	Fix IsProcessBackground on Linux with stdin redirection The previous implementation assumed that stdin was a terminal. It now checks the terminal's fd.	2020-10-13 13:12:02 +02:00
MichaelEischer	41a45ae908	Merge pull request #3016 from greatroar/uid-gid-lookup Simplify/optimize cached UID/GID lookups	2020-10-12 23:24:55 +02:00
Alexander Neumann	56883817d8	Merge pull request #2990 from MichaelEischer/fix-goreport-warnings Fix some goreport warnings	2020-10-12 20:44:56 +02:00
greatroar	f0cd16e5ea	Cache uint32-typed ids in lookup{Username,Group} NodeFillUser-8 1.92µs ± 5% 1.75µs ± 3% -8.89% (p=0.000 n=10+10) NodeFromFileInfo-8 1.89µs ± 7% 1.76µs ± 4% -6.69% (p=0.001 n=10+10) name old alloc/op new alloc/op delta NodeFillUser-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10) NodeFromFileInfo-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10) name old allocs/op new allocs/op delta NodeFillUser-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10) NodeFromFileInfo-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)	2020-10-12 15:08:15 +02:00
greatroar	a03fe4562e	Remove unused error return in lookup{Username,Group}	2020-10-12 14:48:23 +02:00
Alexander Neumann	c193cea119	Pass Context to NewRestorer()	2020-10-10 15:24:26 +02:00
Alexander Neumann	f0d49ca600	Merge pull request #2933 from MichaelEischer/less-context-todo Replace most usages of context.TODO()	2020-10-10 15:03:44 +02:00
kitone	6099f81692	Fix #1212 restore code produces inconsistent timestamps/permissions. Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.	2020-10-10 13:46:44 +02:00
kitone	9d7f616190	Improve restorer debug log information	2020-10-10 13:46:43 +02:00
kitone	295ddb9e57	Add test case for inconsistent timestamps and permissions restoration Reproduce from https://github.com/restic/restic/issues/1212	2020-10-10 13:46:43 +02:00
Michael Eischer	e638b46a13	Embed context into ReaderAt The io.Reader interface does not support contexts, such that it is necessary to embed the context into the backendReaderAt struct. This has the problem that a reader might suddenly stop working when it's contained context is canceled. However, this is now problem here as the reader instances never escape the calling function.	2020-10-09 22:39:07 +02:00
Michael Eischer	d6cfe857b7	pass proper context into MasterIndex.RebuildIndex	2020-10-09 22:39:07 +02:00
Michael Eischer	37a5e2d681	rest: use global context on repository creation	2020-10-09 22:39:06 +02:00
Michael Eischer	4b0fcaed45	unlock: use proper context for locks cleanup The list operation used by RemoveStaleLocks or RemoveAllLocks will already be canceled by the passed in context. Therefore we can also just cancel the remove operation as the unlock command won't process all lock files anyways.	2020-10-09 22:37:56 +02:00
Michael Eischer	645a6efaf2	restorer: remove redundant type specification	2020-10-09 22:37:56 +02:00
Michael Eischer	a449450021	init: pass proper context to master key generation This is no change in behavior as a canceled context did later on cause the config file creation to fail. Therefore this change just lets the repository initialization fail a bit earlier.	2020-10-09 22:37:56 +02:00
Michael Eischer	603bb0e309	restore: Use proper context while loading snapshot	2020-10-09 22:37:56 +02:00
Michael Eischer	c458e114d4	pass context to Find / FindSnapshot This allows proper interruption of restic while it searches for snapshots or key files.	2020-10-09 22:37:56 +02:00
Michael Eischer	45e9a55c62	Wire context into backend layout detection	2020-10-09 22:37:24 +02:00
Michael Eischer	307a6ba3a3	Upgrade minio sdk to v7 This changes are primarily straightforward modifications to pass the parameters in the now expected way.	2020-10-09 22:37:24 +02:00
Michael Eischer	50da20d93d	Warn if backup failed to read tree blob	2020-10-09 22:36:27 +02:00
Alexander Neumann	5fd3dbccb7	Merge pull request #2963 from MichaelEischer/fix-status-deadlock backup: Fix possible deadlock of scanner goroutine	2020-10-09 21:35:50 +02:00
Alexander Neumann	35655a481b	Merge pull request #2931 from MichaelEischer/reduce-debug-overhead Reduce debug log overhead	2020-10-09 21:35:11 +02:00
Alexander Neumann	30cb553c8d	Merge pull request #2932 from MichaelEischer/proper-rclone-create Call rclone.Create to create a new repository for the rclone backend	2020-10-09 21:29:15 +02:00
MichaelEischer	356ac404cd	Merge pull request #2996 from greatroar/setconsolecursorposition Use windows.SetConsoleCursorPosition in ui/termstatus	2020-10-09 18:19:02 +02:00
Michael Eischer	88c8e903d2	filter: Fix glob matching on absolute path marker on windows A pattern part containing "/" is used to mark a path or a pattern as absolute. However, on Windows the path separator is "\" such that glob patterns like "?" could match the marker. The code now explicitly skips the marker when the pattern does not represent an absolute path.	2020-10-09 16:11:05 +02:00
greatroar	740758a5fa	Optimize filter pattern matching By replacing "**" with "", checking for this special path component can be reduced to a length-zero check. name old time/op new time/op delta FilterLines-8 44.7ms ± 5% 44.9ms ± 5% ~ (p=0.631 n=10+10) FilterPatterns/Relative-8 13.6ms ± 4% 13.4ms ± 5% ~ (p=0.165 n=10+10) FilterPatterns/Absolute-8 10.9ms ± 5% 10.7ms ± 4% ~ (p=0.052 n=10+10) FilterPatterns/Wildcard-8 53.7ms ± 5% 50.4ms ± 5% -6.00% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-8 128ms ± 2% 95ms ± 1% -25.54% (p=0.000 n=10+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=1.000 n=9+8) FilterPatterns/Absolute-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.903 n=9+8) FilterPatterns/Wildcard-8 19.7MB ± 0% 19.7MB ± 0% -0.00% (p=0.022 n=10+9) FilterPatterns/ManyNoMatch-8 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-8 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-8 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-8 88.7k ± 0% 88.7k ± 0% ~ (all equal) FilterPatterns/ManyNoMatch-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-09 15:51:14 +02:00
MichaelEischer	862ee4b2c9	Merge pull request #2970 from labkode/fopenskip Skip fopen for file change check to avoid network penalty	2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador	958dc6aafc	Skip fopen for file change check to avoid network penalty	2020-10-09 10:53:28 +02:00
MichaelEischer	88cc444779	Merge pull request #2934 from MichaelEischer/upgrade-backoff Upgrade github.com/cenkalti/backoff module	2020-10-08 20:33:30 +02:00
Michael Eischer	b79f18209f	Upgrade github.com/cenkalti/backoff module We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats an operation endlessly when a retry count of 0 is specified. This required a few fixes for the tests.	2020-10-07 22:04:59 +02:00
Michael Eischer	8388e67c4b	filter: cleanup path separator conversion	2020-10-07 21:14:07 +02:00
Michael Eischer	0acc3c5923	filter: special case patterns without globbing characters In case a part of a path is a simple string, we can just check for equality without complex parsing in filepath.Match. name old time/op new time/op delta FilterLines-4 34.8ms ±17% 41.2ms ±23% +18.36% (p=0.000 n=10+10) FilterPatterns/Relative-4 21.7ms ± 6% 12.1ms ±23% -44.46% (p=0.000 n=10+10) FilterPatterns/Absolute-4 10.0ms ± 5% 9.1ms ±11% -9.80% (p=0.006 n=10+9) FilterPatterns/Wildcard-4 47.0ms ± 7% 42.2ms ± 5% -10.19% (p=0.000 n=9+10) FilterPatterns/ManyNoMatch-4 190ms ± 1% 131ms ±20% -31.47% (p=0.000 n=8+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.870 n=9+9) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.145 n=10+10) FilterPatterns/Wildcard-4 14.3MB ± 0% 19.7MB ± 0% +37.91% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.421 n=10+9) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:55:43 +02:00
Michael Eischer	54a124de3b	filter: explicitly test separate ListWithChild function	2020-10-07 20:47:52 +02:00
Michael Eischer	bcc3bddcf4	filter: only check whether a child path could match when necessary When checking excludes there is no need to test whether a child path could also match the pattern, as it is by definition excluded. Previously childMayMatch was calculated but then discarded. For simple absolute paths this can account for half the time spent for checking pattern matches. name old time/op new time/op delta FilterPatterns/Relative-4 23.3ms ± 9% 21.7ms ± 6% -6.68% (p=0.004 n=10+10) FilterPatterns/Absolute-4 13.9ms ± 7% 10.0ms ± 5% -27.61% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 51.4ms ± 7% 47.0ms ± 7% -8.51% (p=0.001 n=9+9) FilterPatterns/ManyNoMatch-4 551ms ± 9% 190ms ± 1% -65.41% (p=0.000 n=10+8) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.665 n=10+9) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.480 n=9+10) FilterPatterns/Wildcard-4 14.3MB ± 0% 14.3MB ± 0% ~ (p=0.431 n=9+10) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:47:52 +02:00
Michael Eischer	17c53efb0d	filter: Optimize double wildcard expansion This only allocates a single slice to expand the double wildcard and only copies the pattern prefix once. name old time/op new time/op delta FilterPatterns/Relative-4 22.7ms ± 5% 23.3ms ± 9% ~ (p=0.353 n=10+10) FilterPatterns/Absolute-4 14.2ms ±13% 13.9ms ± 7% ~ (p=0.853 n=10+10) FilterPatterns/Wildcard-4 266ms ±16% 51ms ± 7% -80.67% (p=0.000 n=10+9) FilterPatterns/ManyNoMatch-4 554ms ± 6% 551ms ± 9% ~ (p=0.436 n=10+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.349 n=10+10) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.073 n=10+9) FilterPatterns/Wildcard-4 141MB ± 0% 14MB ± 0% -89.89% (p=0.000 n=10+9) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 1.63M ± 0% 0.09M ± 0% -94.56% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:47:48 +02:00
Michael Eischer	7959796269	filter: Special case for absolute paths name old time/op new time/op delta FilterPatterns/Relative-4 23.6ms ±20% 22.7ms ± 5% ~ (p=0.684 n=10+10) FilterPatterns/Absolute-4 32.3ms ± 8% 14.2ms ±13% -56.01% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 334ms ±17% 266ms ±16% -20.56% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 709ms ± 7% 554ms ± 6% -21.89% (p=0.000 n=10+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% +0.00% (p=0.046 n=9+10) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.464 n=10+10) FilterPatterns/Wildcard-4 141MB ± 0% 141MB ± 0% ~ (p=0.163 n=9+10) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 1.63M ± 0% 1.63M ± 0% ~ (p=0.072 n=10+10) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:47:29 +02:00
Michael Eischer	375c2a56de	filter: Parse filter patterns only once name old time/op new time/op delta FilterPatterns/Relative-4 30.3ms ±10% 23.6ms ±20% -22.12% (p=0.000 n=10+10) FilterPatterns/Absolute-4 49.0ms ± 3% 32.3ms ± 8% -33.94% (p=0.000 n=8+10) FilterPatterns/Wildcard-4 345ms ± 9% 334ms ±17% ~ (p=0.315 n=10+10) FilterPatterns/ManyNoMatch-4 3.93s ± 2% 0.71s ± 7% -81.98% (p=0.000 n=9+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 4.63MB ± 0% 3.57MB ± 0% -22.98% (p=0.000 n=9+9) FilterPatterns/Absolute-4 8.54MB ± 0% 3.57MB ± 0% -58.20% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 146MB ± 0% 141MB ± 0% -2.93% (p=0.000 n=9+9) FilterPatterns/ManyNoMatch-4 907MB ± 0% 4MB ± 0% -99.61% (p=0.000 n=9+9) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 66.6k ± 0% 22.2k ± 0% -66.67% (p=0.000 n=10+10) FilterPatterns/Absolute-4 88.7k ± 0% 22.2k ± 0% -75.00% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 1.70M ± 0% 1.63M ± 0% -3.92% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 4.46M ± 0% 0.02M ± 0% -99.50% (p=0.000 n=10+10)	2020-10-07 20:47:27 +02:00
Michael Eischer	b8eacd1364	filter: Reduce redundant path and pattern splitting A single call to filter.List will split the path only once and also split each search pattern only once and use it for both match and childMatch. name old time/op new time/op delta FilterPatterns/Relative-4 62.1ms ±15% 30.3ms ±10% -51.22% (p=0.000 n=9+10) FilterPatterns/Absolute-4 111ms ±10% 49ms ± 3% -56.08% (p=0.000 n=10+8) FilterPatterns/Wildcard-4 393ms ±15% 345ms ± 9% -12.30% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 10.0s ± 3% 3.9s ± 2% -60.53% (p=0.000 n=10+9) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 16.4MB ± 0% 4.6MB ± 0% -71.76% (p=0.000 n=10+9) FilterPatterns/Absolute-4 31.4MB ± 0% 8.5MB ± 0% -72.77% (p=0.000 n=9+10) FilterPatterns/Wildcard-4 168MB ± 0% 146MB ± 0% -13.19% (p=0.000 n=10+9) FilterPatterns/ManyNoMatch-4 3.23GB ± 0% 0.91GB ± 0% -71.96% (p=0.000 n=10+9) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 178k ± 0% 67k ± 0% -62.50% (p=0.000 n=10+10) FilterPatterns/Absolute-4 266k ± 0% 89k ± 0% -66.67% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 1.87M ± 0% 1.70M ± 0% -9.47% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 17.7M ± 0% 4.5M ± 0% -74.87% (p=0.000 n=9+10)	2020-10-07 18:13:19 +02:00
Michael Eischer	e73c281142	filter: Benchmark absolute paths, wildcards and long filter lists	2020-10-07 17:54:36 +02:00
Michael Eischer	fdd3b14db3	filter: test some corner cases	2020-10-07 17:09:44 +02:00
Michael Eischer	f4282aa6fd	local: mark repository files as read-only This is intended to prevent accidental modifications of data files. Marking the files as read-only was accidentally removed in #1258.	2020-10-07 12:29:37 +02:00
Michael Eischer	40ee17167e	local: Ignore permission errors on chmod call in Save/Remove operation The file is already created with the proper permissions, thus the chmod call is not critical. However, some file systems have don't allow modifications of the file permissions. Similarly the chmod call in the Remove operation should not prevent it from working.	2020-10-07 12:29:37 +02:00
greatroar	f3e933f0c1	Use windows.SetConsoleCursorPosition in ui/termstatus	2020-10-07 11:18:54 +02:00
MichaelEischer	09d39e260d	Merge pull request #2980 from greatroar/error-wrapping Remove repetitive error wrapping from internal/cache	2020-10-06 15:19:00 +02:00
Michael Eischer	1579d2a8ec	Remove some unused assignments	2020-10-06 14:55:13 +02:00
Michael Eischer	eba5dd831f	Fix typos reported by misspell	2020-10-06 14:55:13 +02:00
Michael Eischer	9ffb698c8d	Modernize context import in internal/fuse Migrated using `go fix`.	2020-10-06 14:55:13 +02:00
Michael Eischer	efbb850d92	Remove a few redundant type specifiers This is the result of running `gofmt -s -w */.go`	2020-10-06 14:55:13 +02:00
greatroar	673dda77c0	Less repetitive error messages in internal/cache Many instances of errors.Wrap in this package would produce messages like "Open: open <filename>: no such file or directory"; those now omit the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion now appends its own name to the error message, rather than the function that failed, to make it easier to spot. Other function names (e.g., Load) are already added further up in the call chain.	2020-10-05 20:28:54 +02:00
MichaelEischer	1a490acd67	Merge pull request #2982 from greatroar/archiver-error-handling Check error in archiver before calling Select	2020-10-05 19:19:47 +02:00
MichaelEischer	c84643c6a9	Merge pull request #2981 from greatroar/mkdircachedir Simplify cache directory creation	2020-10-05 18:15:23 +02:00
greatroar	1e1a1f3078	Upgrade cespare/xxhash to 2.1.1	2020-10-05 17:25:39 +02:00
greatroar	c4e2203e45	Check error in archiver before calling Select The archiver first called the Select function for a path before checking whether the Lstat on that path actually worked. The RejectFuncs in exclude.go worked around this by checking whether they received a nil os.FileInfo. Checking first is more obvious and requires less code.	2020-10-05 11:11:04 +02:00
greatroar	7d0fa1a686	Simplify cache directory creation	2020-10-05 10:46:45 +02:00
Ingo Gottwald	8b8e230771	Swap deprecated GCS lib with replacement	2020-10-03 18:55:56 +02:00
Ingo Gottwald	00cedd22aa	Replace deprecated method in gs backend	2020-10-01 10:02:42 +02:00
Michael Eischer	d44df9d00d	backup: Always remove the status lines once a backup ends	2020-09-30 23:13:10 +02:00
Michael Eischer	8d0ba55ecd	backup: Fix possible deadlock of scanner goroutine When the backup is interrupted for some reason while the scanner is still active this could lead to a deadlock. Interruptions are triggered by canceling the context object used by both the backup progress UI and the scanner. It is possible that a context is canceled between the respective check in the scanner and it calling the `ReportTotal` method of the UI. The latter method sends a message to the UI goroutine. However, a canceled context will also stop that goroutine, which can cause the channel send operation to block indefinitely. This is resolved by adding a `closed` channel which is closed once the UI goroutine is stopped and serves as an escape hatch for reported UI updates. This change covers not just the ReportTotal method but all potentially affected methods of the progress UI implementation.	2020-09-30 23:13:10 +02:00
MichaelEischer	48d0ab5276	Merge pull request #2959 from restic/keep-text backup: Correct keep policy text	2020-09-30 17:57:00 +02:00
MichaelEischer	fd02407863	Merge pull request #2849 from classmarkets/gcs-access-token gs: support authentication with access token	2020-09-30 17:42:56 +02:00
Leo R. Lundgren	028c9a5343	backup: Correct keep policy text Makes the following corrections to the "Applying Policy:" output: - keep the last 1 snapshots snapshots => keep 1 latest snapshots - keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots	2020-09-28 14:26:53 +02:00
MichaelEischer	14b312f00d	Merge pull request #2658 from creativeprojects/issue-2241 Don't echo authentication passwords (rest backend)	2020-09-22 22:17:53 +02:00
Fred	206cadfab4	Hide password from repository URLs	2020-09-22 22:00:51 +02:00
MichaelEischer	4875f7b659	Merge pull request #2614 from greatroar/simplify-fs Simplify internal/fs	2020-09-21 22:05:10 +02:00
greatroar	0d65b78168	Simplify os.ModeType\|os.ModeCharDevice => os.ModeType Since Go 1.12, ModeCharDevice is included in ModeType: golang/go@a2a3dd00c9	2020-09-21 14:21:32 +02:00
greatroar	95ebba85ff	Remove stray Printf from internal/fs	2020-09-21 14:21:32 +02:00
greatroar	59b343a9bf	Remove OS-specific versions of fs.MkdirAll Go has supported Windows paths correctly since 1.11, see https://github.com/golang/go/issues/10900 and the commit referenced there.	2020-09-21 14:21:32 +02:00
greatroar	1557c58eef	Fix and simplify fs.Reader fakeDir.{Readdir,Readdirnames} weren't handling the case n == 0 correctly. fakeFileInfo.sys is always nil, so omit the field.	2020-09-21 14:21:32 +02:00
greatroar	c504aa505c	Remove unused fs.Rename	2020-09-21 14:21:32 +02:00
greatroar	1b20f6beec	Remove io.Writer from fs.File It was only used in a single test, which now uses plain *os.File instead.	2020-09-21 14:21:32 +02:00
Michael Eischer	8c36317b71	rclone: use configured number of connections during create	2020-09-19 19:11:43 +02:00
Michael Eischer	60a5c35de9	rclone: close connection to rclone if open fails	2020-09-19 19:11:43 +02:00
Michael Eischer	07f4e7d10b	Only log HTTP requests when a debug log is configured The logged HTTP requests are only visible when a debug log is configured.	2020-09-19 17:07:53 +02:00
Michael Eischer	f003410402	init: Add `--copy-chunker-params` option This allows creating multiple repositories with identical chunker parameters which is required for working deduplication when copying snapshots between different repositories.	2020-09-19 16:53:05 +02:00
Michael Eischer	9ad8250a78	Reduce overhead of debug calls if no log is enabled In case no debug log is configured, then calls to debug.Log only incur the costs to check a single boolean flag making the call really cheap.	2020-09-19 16:45:01 +02:00
greatroar	cb3f531050	Make Cache.Path private It's only used inside the package.	2020-09-18 11:17:29 +02:00
greatroar	23fcbb275a	Remove restic.Cache interface It was used in one code path, which then asserted its concrete type as *cache.Cache. Privatised some of the interface methods.	2020-09-18 10:48:13 +02:00
greatroar	6e3215a80d	Remove unused error type from internal/cache	2020-09-18 10:38:14 +02:00
greatroar	9abef3bf1a	Move internal/fs.TestChdir to internal/test.Chdir	2020-09-17 10:43:33 +02:00
Alexander Neumann	0c48e515f0	Merge pull request #2630 from MichaelEischer/fix-staticcheck Fix lots of small issues reported by staticcheck	2020-09-13 16:19:22 +02:00
Alexander Neumann	97950ab81a	options: Fix test for Go >= 1.15	2020-09-12 17:36:44 +02:00
MichaelEischer	88664ba222	Merge pull request #2893 from MichaelEischer/restore-preallocate restorer: Preallocate files	2020-09-08 22:43:05 +02:00
Michael Eischer	121233e1b3	Add preallocate tests	2020-09-07 21:41:47 +02:00
Michael Eischer	8cc9514879	restorer: pre-allocate files before loading chunks	2020-09-07 21:41:47 +02:00
Michael Eischer	2e7d475029	Process packs in order of first appearance	2020-09-06 21:25:35 +02:00
Michael Eischer	c46edcd9d6	error strings should not end with punctuation	2020-09-05 10:07:17 +02:00
Michael Eischer	1ede018ea6	error variable names should start with 'Err'	2020-09-05 10:07:17 +02:00
Michael Eischer	b77e933d80	Convert underscore variable names to camelCase	2020-09-05 10:07:16 +02:00
Michael Eischer	d0329cf3eb	Adjust comments to match name of exported methods	2020-09-05 10:07:16 +02:00
Michael Eischer	dc31529fc3	Unindent else block after if block ending with a return statement	2020-09-05 10:07:16 +02:00
Michael Eischer	4784540f04	repository: Simplify worker group code	2020-09-05 10:07:16 +02:00
Michael Eischer	84ea2389ae	archiver_test: Only skip symlinks on windows	2020-09-05 10:07:16 +02:00
Michael Eischer	b4a7ce86cf	uint cannot be less than zero	2020-09-05 10:07:16 +02:00
Michael Eischer	7ee0964880	crypto: Remove unused error	2020-09-05 10:07:16 +02:00
Michael Eischer	f6f11400c2	restic: Remove unused variable value	2020-09-05 10:07:16 +02:00
Michael Eischer	e2dc5034d3	restic/file: Use FileType for all File constants The data type of a variable is only carried on when using iota.	2020-09-05 10:07:16 +02:00
Michael Eischer	9a1b3cb5d9	restorer: Remove unused treeID struct member from test	2020-09-05 10:07:16 +02:00
Michael Eischer	ee05501ce7	archiver: Remove unused chmod function	2020-09-05 10:07:16 +02:00
Michael Eischer	014600bee6	archiver: Remove unused variable from test	2020-09-05 10:07:16 +02:00
Michael Eischer	d9a80e07b9	repository: Simplify index age calculation	2020-09-05 10:07:16 +02:00
Michael Eischer	460e2ffbf6	Collapse a few boolean operations	2020-09-05 10:07:14 +02:00
Michael Eischer	2f8335554c	Remove a few unused variables	2020-09-05 10:06:23 +02:00
Michael Eischer	37113282ca	Merge a few variable declaration and initializations	2020-09-05 10:05:34 +02:00
Michael Eischer	337725c354	Use sort.Strings	2020-09-05 10:05:34 +02:00
Michael Eischer	dcf9ded977	dump: Use slashes as path separators in tar on windows This conversion is necessary as paths returned by filepath.Rel use the operating-system native path separator	2020-08-30 18:26:50 +02:00
Simon Beck	fe6445e0f4	dump: Added unit tests for tar output	2020-08-30 16:00:48 +02:00
Simon Beck	80a11960dd	dump: Always dump relative paths into tarballs Tarballs should only contain relative paths.	2020-08-29 21:14:34 +02:00
Simon Beck	d6f739ec22	dump: Extract tar code to internal/dump	2020-08-29 21:14:34 +02:00
Michael Eischer	b25978a53c	backup: Fix reporting of directory count in summary Previously the directory stats were reported immediately after calling `SaveDir`. However, as the latter method saves the tree asynchronously the stats were still initialized to their nil value. The stats are now reported via a callback similar to the one used for the fileSaver.	2020-08-27 22:43:51 +02:00
Alexander Neumann	b1b3f1ecb6	Merge pull request #2674 from MichaelEischer/prune-strict-checks prune: Stricter error checks	2020-08-23 10:30:41 +02:00
Hristo Trendev	51c22f4223	local backend: ignore not supported error on sync() Closes #2395	2020-08-22 01:27:07 +02:00
Michael Eischer	f4b9544ab2	prune: Add test that repack aborts on wrong blob	2020-08-16 11:34:01 +02:00
Michael Eischer	367449dede	prune: Reduce memory allocations while repacking The slicing operator `slice[low:high]` default to 0 for the lower bound and len(slice) for the upper bound when either or both are not specified. Fix the code to use `cap(slice)` to check for the slice capacity.	2020-08-16 11:34:01 +02:00
Michael Eischer	7042bafea5	prune: Abort repacking when a pack contains a wrong blob If a blob in a pack file can be decrypted successfully but contains data that results in a different hash than stated in the header pack, then abort repacking. As both the pack header and the blob are cryptographically verified this either means than a malicious entity tampered with the backup or indicates hardware problems on the client. prune should fail with an error in both cases.	2020-08-16 11:34:01 +02:00
aawsome	0fed6a8dfc	Use "pack file" instead of "data file" (#2885 ) - changed variable names, especially changed DataFile into PackFile - changed in some comments - always use "pack file" in docu	2020-08-16 11:16:38 +02:00
MichaelEischer	eca0f0ad24	Merge pull request #2863 from aawsome/index-no-duplicates Don't save exact duplicates in merged index	2020-08-08 18:24:14 +02:00
Alexander Weiss	b112533812	Don't save exact duplicates when merging indexes	2020-08-05 06:32:02 +02:00
Alexander Weiss	5e63294355	Add benchmark MasterIndexAlloc	2020-08-05 06:32:02 +02:00
Michael Eischer	05116e4787	prune: Cleanup progress bar handling while repacking	2020-08-03 19:32:46 +02:00
Alexander Neumann	b7b479b668	Merge pull request #2599 from MichaelEischer/tweak-mem-usage Reduce memory usage when searching for used blobs	2020-08-01 13:49:02 +02:00
Alexander Neumann	4cf9656f12	Merge pull request #2861 from MichaelEischer/fix-rclone-crash rclone: Don't panic after unexpected subprocess exit	2020-08-01 13:08:50 +02:00
Alexander Neumann	2580eef2aa	Merge pull request #2318 from classmarkets/2175-named-keys Allow specifying user and host when adding keys	2020-08-01 13:06:31 +02:00
Michael Eischer	a178e5628e	Remove duplicate TreeLoader interface	2020-08-01 12:29:16 +02:00
Michael Eischer	af66a62c04	FindUsedBlobs: Test that seen blobs are skipped This also copies the TreeLoader interface from internal/walker to allow stubbing the repository in the call to `FindUsedBlobs`.	2020-08-01 12:29:16 +02:00
Michael Eischer	9ea1a78bd4	FindUsedBlobs: Check for seen blobs before loading trees The only effective change in behavior is that that toplevel nodes can also be skipped.	2020-08-01 12:29:16 +02:00
Michael Eischer	184103647a	FindUsedBlobs: merge seen into blobs BlobSet The seen BlobSet always contained a subset of the entries in blobs. Thus use blobs instead and avoid the memory overhead of the second set. Suggested-by: Alexander Weiss <alex@weissfam.de>	2020-08-01 12:29:16 +02:00
Michael Eischer	c81b122374	rclone: Don't panic after unexpected subprocess exit As the connection to the rclone child process is now closed after an unexpected subprocess exit, later requests will cause the http2 transport to try to reestablish a new connection. As previously this never should have happened, the connection called panic in that case. This panic is now replaced with a simple error message, as it no longer indicates an internal problem.	2020-08-01 12:17:40 +02:00
Alexander Weiss	f8316948d1	Optimize FUSE - make command `restic mount` faster and consume less memory - Add Open() functionality to dir - only access index for blobs when file is read - Implement NodeOpener and put one-time file stuff there - Add comment about locking as suggested by bazil.org/fuse => Thanks at Michael Eischer for suggesting the last two improvements	2020-07-28 23:01:18 +02:00
Michael Eischer	ea97ff1ba4	rclone: Skip crash test when rclone is not found	2020-07-26 12:06:18 +02:00
Michael Eischer	01b9581453	rclone: Better field names for stdio conn	2020-07-26 00:29:25 +02:00
Michael Eischer	3cd927d180	rclone: Give rclone time to finish before closing stdin pipe Calling `Close()` on the rclone backend sometimes failed during test execution with 'signal: Broken pipe'. The stdio connection closed both the stdin and stdout file descriptors at the same moment, therefore giving rclone no chance to properly send any final http2 data frames. Now the stdin connection to rclone is closed first and will only be forcefully closed after a timeout. In case rclone exits before the timeout then the stdio connection will be closed normally.	2020-07-26 00:29:25 +02:00
greatroar	bf7b1f12ea	rclone: Add test for pipe handling when rclone exits	2020-07-26 00:29:25 +02:00
Michael Eischer	8554332894	rclone: Close rclone side of stdio_conn pipes restic did not notice when the rclone subprocess exited unexpectedly. restic manually created pipes for stdin and stdout and used these for the connection to the rclone subprocess. The process creating a pipe gets file descriptors for the sender and receiver side of a pipe and passes them on to the subprocess. The expected behavior would be that reads or writes in the parent process fail / return once the child process dies as a pipe would now just have a reader or writer but not both. However, this never happened as restic kept the reader and writer file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe` close the subprocess side of pipes once the child process was started and close the parent process side of pipes once wait has finished. We can't use these functions as we need access to the raw `os.File` so just replicate that behavior.	2020-07-26 00:29:25 +02:00
greatroar	3e93b36ca4	Make rclone.New private	2020-07-26 00:28:45 +02:00
Michael Eischer	c847aace35	Rename Index interface to MasterIndex The interface is now only implemented by repository.MasterIndex.	2020-07-25 21:19:46 +02:00
Alexander Weiss	9d1fb94c6c	make Lookup() return all blobs + simplify syntax	2020-07-25 21:18:34 +02:00
MichaelEischer	020cab8e08	Merge pull request #2787 from greatroar/no-blobsize-cache Remove blob size cache from restic mount	2020-07-25 20:46:35 +02:00
greatroar	07da61baee	Remove blob size cache from restic mount	2020-07-25 19:39:14 +02:00
MichaelEischer	3d530dfc91	Merge pull request #2827 from aawsome/archiver-test-contents Make self-healing work when backing up with parent snapshot	2020-07-25 13:13:18 +02:00
MichaelEischer	bbc960f957	Merge pull request #2635 from greatroar/optimize-sortbycached Optimize sorting blobs by cache status	2020-07-25 12:35:42 +02:00
greatroar	309598c237	Simplify sortCachedPacksFirst test in internal/repository The test now uses the fact that the sort is stable. It's not guaranteed to be, but the test is cleaner and more exhaustive. sortCachedPacksFirst no longer needs a return value.	2020-07-25 12:12:59 +02:00
greatroar	03d23e6faa	Speed up blob sorting in internal/repository name old time/op new time/op delta SortCachedPacksFirst-8 208µs ± 3% 186µs ± 3% -10.74% (p=0.000 n=10+8) name old alloc/op new alloc/op delta SortCachedPacksFirst-8 213kB ± 0% 139kB ± 0% -34.62% (p=0.000 n=10+10) name old allocs/op new allocs/op delta SortCachedPacksFirst-8 1.03k ± 0% 1.03k ± 0% -0.19% (p=0.000 n=10+10)	2020-07-25 12:12:59 +02:00
greatroar	b10acd2af7	Test and benchmark blob sorting in internal/repository	2020-07-25 12:12:58 +02:00
Alexander Weiss	9175795fdb	Check contents in archiver When backing up with a parent snapshot and the file is not changed, also check if contents are still available in index.	2020-07-25 08:18:28 +02:00
rawtaz	5d8d70542f	Merge pull request #2852 from MichaelEischer/drop-go-1.11 Drop support for Go version 1.11	2020-07-25 01:17:36 +02:00
Michael Eischer	7c23381a2b	Drop support for Go version 1.11	2020-07-24 18:52:39 +02:00
MichaelEischer	34181b13a2	Merge pull request #2328 from MichaelEischer/no-repeated-checks Fix duplicate tree checks within `restic check`	2020-07-22 22:08:02 +02:00
Alexander Weiss	a666a6d576	Add tests and merge indexes in index benchmarks	2020-07-22 21:54:02 +02:00
Alexander Weiss	e388d962a5	Merge final indexes together for faster index access	2020-07-22 21:54:02 +02:00
Peter Schultz	758b44b9c0	gs: support authentication with access token In the Google Cloud Storage backend, support specifying access tokens directly, as an alternative to a credentials file. This is useful when restic is used non-interactively by some other program that is already authenticated and eliminates the need to store long lived credentials. The access token is specified in the GOOGLE_ACCESS_TOKEN environment variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.	2020-07-22 16:23:03 +02:00
Alexander Weiss	3b7a3711e6	Add more realistic index benchmarks + reduce test size of BenchmarkMasterIndexLookupParallel	2020-07-21 07:18:20 +02:00
Michael Eischer	9b0e718852	checker: Test that blob types are not confused	2020-07-20 23:43:47 +02:00
MichaelEischer	82c908871d	Merge pull request #2812 from greatroar/chaining Chaining hash table for repository.Index	2020-07-20 23:29:36 +02:00
Michael Eischer	ddf0b8cd0b	checker: Properly distinguish between data and tree blobs If a data blob and a tree blob with the same ID (= same content) exist, then the checker did not report a data or tree blob as unused when the blob of the other type was still in use.	2020-07-20 22:58:39 +02:00
Michael Eischer	2d0c138c9b	checker: Test that check only decodes trees once The `DuplicateTree` flag is necessary to ensure that failures cannot be swallowed. The old checker implementation ignores errors from LoadTree if the corresponding tree was already checked.	2020-07-20 22:51:53 +02:00
Michael Eischer	ef325ffc02	checker: Cleanup error handling code This change only moves code around but does not result in any change in behavior.	2020-07-20 22:51:53 +02:00
Michael Eischer	7a165f32a9	checker: Traverse trees in depth-first order Backups traverse the file tree in depth-first order and saves trees on the way back up. This results in tree packs filled in a way comparable to the reverse Polish notation. In order to check tree blobs in that order, the treeFilter would have to delay the forwarding of tree nodes until all children of it are processed which would complicate the implementation. Therefore do the next similar thing and traverse the tree in depth-first order, but process trees already on the way down. The tree blob ids are added in reverse order to the backlog, which is once again reverted when removing the ids from the back of the backlog.	2020-07-20 22:51:53 +02:00
Michael Eischer	36c69e3ca7	checker: Unify blobs, processed trees and referenced blobs map The blobRefs map and the processedTrees IDSet are merged to reduce the memory usage. The blobRefs map now uses separate flags to track blob usage as data or tree blob. This prevents skipping of trees whose content is identical to an already processed data blob. A third flag tracks whether a blob exists or not, which removes the need for the blobs IDSet.	2020-07-20 22:51:47 +02:00
Michael Eischer	35d8413639	checker: Remove dead index map	2020-07-20 22:37:31 +02:00
Michael Eischer	c66a0e408c	checker: Reduce cost of debug log Avoid duplicate allocation of the Subtree list.	2020-07-20 22:37:31 +02:00
Michael Eischer	70f4c014ef	checker: Decode identical tree nodes only once Even though the checkTreeWorker skips already processed chunks, filterTrees did queue the same tree blob on every occurence. This becomes a serious performance bottleneck for larger number of snapshots that cover mostly the same directories. Therefore decode a tree blob exactly once.	2020-07-20 22:37:31 +02:00
Michael Eischer	f0d8710611	Add benchmark for checker scaling with snapshot count	2020-07-20 22:37:31 +02:00
Alexander Neumann	8074879c5f	Remove 'go generate'	2020-07-19 17:28:42 +02:00
greatroar	7bda28f31f	Chaining hash table for repository.Index These are faster to construct but slower to access. The allocation rate is halved, the peak memory usage almost halved compared to standard map. Benchmark results on linux/amd64, -benchtime=3s -count=20: name old time/op new time/op delta PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.231 n=20+20) DecodeIndex-8 4.54s ± 0% 4.30s ± 0% -5.20% (p=0.000 n=18+17) DecodeIndexParallel-8 4.54s ± 0% 4.30s ± 0% -5.22% (p=0.000 n=19+18) IndexHasUnknown-8 44.4ns ± 5% 50.5ns ±11% +13.82% (p=0.000 n=19+17) IndexHasKnown-8 48.3ns ± 0% 51.5ns ±12% +6.68% (p=0.001 n=16+20) IndexAlloc-8 758ms ± 1% 616ms ± 1% -18.69% (p=0.000 n=19+19) IndexAllocParallel-8 234ms ± 3% 204ms ± 2% -12.60% (p=0.000 n=20+18) MasterIndexLookupSingleIndex-8 122ns ± 0% 145ns ± 9% +18.44% (p=0.000 n=14+20) MasterIndexLookupMultipleIndex-8 369ns ± 2% 429ns ± 8% +16.27% (p=0.000 n=20+20) MasterIndexLookupSingleIndexUnknown-8 68.4ns ± 5% 74.9ns ±13% +9.47% (p=0.000 n=20+20) MasterIndexLookupMultipleIndexUnknown-8 315ns ± 3% 369ns ±11% +17.14% (p=0.000 n=20+20) MasterIndexLookupParallel/known,indices=5-8 743ns ± 1% 816ns ± 2% +9.87% (p=0.000 n=17+17) MasterIndexLookupParallel/unknown,indices=5-8 238ns ± 1% 260ns ± 2% +9.14% (p=0.000 n=19+20) MasterIndexLookupParallel/known,indices=10-8 1.01µs ± 3% 1.11µs ± 2% +9.79% (p=0.000 n=19+20) MasterIndexLookupParallel/unknown,indices=10-8 222ns ± 0% 269ns ± 2% +20.83% (p=0.000 n=16+20) MasterIndexLookupParallel/known,indices=20-8 1.06µs ± 2% 1.19µs ± 2% +12.95% (p=0.000 n=19+18) MasterIndexLookupParallel/unknown,indices=20-8 413ns ± 1% 530ns ± 1% +28.19% (p=0.000 n=18+20) SaveAndEncrypt-8 30.2ms ± 1% 30.4ms ± 0% +0.71% (p=0.000 n=19+19) LoadTree-8 540µs ± 1% 576µs ± 1% +6.73% (p=0.000 n=20+20) LoadBlob-8 5.64ms ± 0% 5.64ms ± 0% ~ (p=0.883 n=18+17) LoadAndDecrypt-8 5.93ms ± 0% 5.95ms ± 1% ~ (p=0.247 n=20+19) LoadIndex-8 25.1ms ± 0% 24.5ms ± 1% -2.54% (p=0.000 n=18+17) name old speed new speed delta PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.229 n=20+20) SaveAndEncrypt-8 139MB/s ± 1% 138MB/s ± 0% -0.71% (p=0.000 n=19+19) LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% ~ (p=0.890 n=18+17) LoadAndDecrypt-8 169MB/s ± 0% 168MB/s ± 1% ~ (p=0.227 n=20+19) name old alloc/op new alloc/op delta PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.772 n=12+19) IndexAlloc-8 786MB ± 0% 400MB ± 0% -49.04% (p=0.000 n=20+18) IndexAllocParallel-8 786MB ± 0% 401MB ± 0% -49.04% (p=0.000 n=19+15) SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% +0.00% (p=0.000 n=19+19) name old allocs/op new allocs/op delta PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal) IndexAlloc-8 977k ± 0% 907k ± 0% -7.18% (p=0.000 n=20+20) IndexAllocParallel-8 977k ± 0% 907k ± 0% -7.17% (p=0.000 n=19+15) SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)	2020-07-19 13:58:22 +02:00
greatroar	255ba83c4b	Parallel index benchmarks + benchmark optimizations createRandomIndex was using the global RNG, which locks on every call It was also using twice as many random numbers as necessary and doing a float division in every iteration of the inner loop. BenchmarkDecodeIndex was using too short an input, especially for a parallel version. (It may now be using one that is a bit large.) Results on linux/amd64, -benchtime=3s -count=20: name old time/op new time/op delta PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.165 n=20+20) DecodeIndex-8 13.6µs ± 2% 4539886.8µs ± 0% +33293901.38% (p=0.000 n=20+18) IndexHasUnknown-8 44.4ns ± 7% 44.4ns ± 5% ~ (p=0.873 n=20+19) IndexHasKnown-8 49.2ns ± 3% 48.3ns ± 0% -1.86% (p=0.000 n=20+16) IndexAlloc-8 802ms ± 1% 758ms ± 1% -5.51% (p=0.000 n=20+19) MasterIndexLookupSingleIndex-8 124ns ± 1% 122ns ± 0% -1.41% (p=0.000 n=20+14) MasterIndexLookupMultipleIndex-8 373ns ± 2% 369ns ± 2% -1.13% (p=0.001 n=20+20) MasterIndexLookupSingleIndexUnknown-8 67.8ns ± 3% 68.4ns ± 5% ~ (p=0.753 n=20+20) MasterIndexLookupMultipleIndexUnknown-8 316ns ± 3% 315ns ± 3% ~ (p=0.846 n=20+20) SaveAndEncrypt-8 30.5ms ± 1% 30.2ms ± 1% -1.09% (p=0.000 n=19+19) LoadTree-8 527µs ± 1% 540µs ± 1% +2.37% (p=0.000 n=19+20) LoadBlob-8 5.65ms ± 0% 5.64ms ± 0% -0.21% (p=0.000 n=19+18) LoadAndDecrypt-8 7.07ms ± 2% 5.93ms ± 0% -16.15% (p=0.000 n=19+20) LoadIndex-8 32.1ms ± 2% 25.1ms ± 0% -21.64% (p=0.000 n=20+18) name old speed new speed delta PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.159 n=20+20) SaveAndEncrypt-8 138MB/s ± 1% 139MB/s ± 1% +1.10% (p=0.000 n=19+19) LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% +0.21% (p=0.000 n=19+18) LoadAndDecrypt-8 141MB/s ± 2% 169MB/s ± 0% +19.24% (p=0.000 n=19+20) name old alloc/op new alloc/op delta PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.826 n=19+12) IndexAlloc-8 786MB ± 0% 786MB ± 0% +0.01% (p=0.000 n=20+20) SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% -0.00% (p=0.012 n=20+19) name old allocs/op new allocs/op delta PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal) IndexAlloc-8 977k ± 0% 977k ± 0% +0.01% (p=0.022 n=20+20) SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)	2020-07-19 13:58:05 +02:00
Lars Lehtonen	9ac90cf5cd	internal/fuse: fix dropped test error	2020-07-12 21:42:31 -07:00
greatroar	58719e1f47	Replace mount's per-file cache by a global LRU cache	2020-07-12 18:27:16 +02:00
greatroar	d42c169458	Fix quadratic file reading in restic mount	2020-07-12 18:27:16 +02:00
greatroar	02bec13ef2	Fix repository_test.BenchmarkSaveAndEncrypt The benchmark was actually testing the speed of index lookups. name old time/op new time/op delta SaveAndEncrypt-8 101ns ± 2% 31505824ns ± 1% +31311591.31% (p=0.000 n=10+10) name old speed new speed delta SaveAndEncrypt-8 41.7TB/s ± 2% 0.0TB/s ± 1% -100.00% (p=0.000 n=10+10) name old alloc/op new alloc/op delta SaveAndEncrypt-8 1.00B ± 0% 20989508.40B ± 0% +2098950740.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta SaveAndEncrypt-8 0.00 123.00 ± 0% +Inf% (p=0.000 n=10+9) (The actual speed is ca. 131MiB/s.)	2020-07-05 17:41:42 +02:00
MichaelEischer	212607dc8a	Merge pull request #2760 from greatroar/backend-benchmark Fix backend benchmarks + a micro-optimization	2020-06-17 23:17:05 +02:00
greatroar	190d8e2f51	Flatten backend.LimitedReadCloser structure This inlines the io.LimitedReader into the LimitedReadCloser body to achieve fewer allocations. Results on linux/amd64: name old time/op new time/op delta Backend/BenchmarkLoadPartialFile-8 412µs ± 4% 413µs ± 4% ~ (p=0.634 n=17+17) Backend/BenchmarkLoadPartialFileOffset-8 455µs ±13% 441µs ±10% ~ (p=0.072 n=20+18) name old speed new speed delta Backend/BenchmarkLoadPartialFile-8 10.2GB/s ± 3% 10.2GB/s ± 4% ~ (p=0.817 n=16+17) Backend/BenchmarkLoadPartialFileOffset-8 9.25GB/s ±12% 9.54GB/s ± 9% ~ (p=0.072 n=20+18) name old alloc/op new alloc/op delta Backend/BenchmarkLoadPartialFile-8 888B ± 0% 872B ± 0% -1.80% (p=0.000 n=15+15) Backend/BenchmarkLoadPartialFileOffset-8 888B ± 0% 872B ± 0% -1.80% (p=0.000 n=15+15) name old allocs/op new allocs/op delta Backend/BenchmarkLoadPartialFile-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=15+15) Backend/BenchmarkLoadPartialFileOffset-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=15+15)	2020-06-17 13:11:45 +02:00
greatroar	f4cd2a7120	Make backend benchmarks fairer by removing checks Checking whether the right data is returned takes up half the time in some benchmarks. Results for local backend benchmarks on linux/amd64: name old time/op new time/op delta Backend/BenchmarkLoadFile-8 4.89ms ± 0% 2.72ms ± 1% -44.26% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFile-8 936µs ± 6% 439µs ±15% -53.07% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFileOffset-8 940µs ± 1% 456µs ±10% -51.50% (p=0.008 n=5+5) Backend/BenchmarkSave-8 23.9ms ±14% 24.8ms ±41% ~ (p=0.690 n=5+5) name old speed new speed delta Backend/BenchmarkLoadFile-8 3.43GB/s ± 0% 6.16GB/s ± 1% +79.40% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFile-8 4.48GB/s ± 6% 9.63GB/s ±14% +114.78% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFileOffset-8 4.46GB/s ± 1% 9.22GB/s ±10% +106.74% (p=0.008 n=5+5) Backend/BenchmarkSave-8 706MB/s ±13% 698MB/s ±31% ~ (p=0.690 n=5+5)	2020-06-17 13:11:45 +02:00
Alexander Weiss	1361341c58	don't save duplicate packIDs when using internal/repository/Index.Store	2020-06-14 07:56:24 +02:00
Alexander Weiss	ce4a2f4ca6	save packIDs and duplicates separately A side remark to the definition of Index.blob: Another possibility would have been to use: blob map[restic.BlobHandle]indexEntry This would have led to the following sizes: key: 32 + 1 = 33 bytes value: 8 bytes indexEntry: 8 + 4 + 4 = 16 bytes each packID: 32 bytes To save N index entries, we would therefore have needed: N OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes More precicely, using a pointer instead of a direct entry is the better memory choice if: OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1) Under the assumption of OF=1.5, this means using pointers would have been the better choice if sizeof(indexEntry) > 24 bytes.	2020-06-14 07:56:21 +02:00
Alexander Weiss	cf979e2b81	make offset and length uint32	2020-06-14 07:50:19 +02:00
Michael Eischer	d92e2c5769	simplify index code	2020-06-14 07:50:19 +02:00
Alexander Weiss	7419844885	add changelog, benchmark, memory calculation	2020-06-14 07:50:15 +02:00
Alexander Weiss	d3c59d18e5	Fix inconsistency of saving/loading config file Fix saving/loading config file: Always set ID to a zero ID.	2020-06-13 16:30:23 +02:00
MichaelEischer	dd7b4f54f5	Merge pull request #2709 from greatroar/minio-sha256 Use Minio's optimized SHA-256	2020-06-12 23:32:58 +02:00
MichaelEischer	6896c6449b	Merge pull request #2779 from greatroar/archiver-comment Fix up comment on archiver.BlobSaver.Save	2020-06-12 23:04:21 +02:00
MichaelEischer	735a8074d5	Merge pull request #2773 from aawsome/index-uploads+knownblobs Fix non-intuitive repo behavior	2020-06-12 22:41:04 +02:00
Alexander Weiss	70347e95d5	disable index uploads for prune command + modifications of changelog	2020-06-12 09:24:38 +02:00
greatroar	0fa3091c78	Fix up comment on archiver.BlobSaver.Save	2020-06-11 13:40:30 +02:00
Alexander Weiss	91906911b0	Fix non-intuitive repository behavior - The SaveBlob method now checks for duplicates. - Moves handling of pending blobs to MasterIndex. -> also cleans up pending index entries when they are saved in the index -> when using SaveBlob no need to care about index any longer - Always check for full index and save it when storing packs. -> removes the need of an index uploader -> also removes the verbose "uploaded intermediate index" messages - The Flush method now also saves the index - Fix race condition when checking and saving full/non-finalized indexes	2020-06-11 13:05:23 +02:00
MichaelEischer	6856d1e422	Merge pull request #2749 from aawsome/fix-fullindex Change condition for full index	2020-06-10 20:40:19 +02:00
Alexander Weiss	8c1261ff02	changed condition for full index	2020-06-07 22:00:49 +02:00
MichaelEischer	84475aa3a8	Merge pull request #2730 from greatroar/mount-build-tags Simplify build tags for restic mount	2020-05-23 20:33:43 +02:00
Martin Michlmayr	5cc1760fdf	Fix typos	2020-05-16 14:05:26 +08:00
greatroar	649cbec6c5	Simplify build tags for restic mount This command can only be built on Darwin, FreeBSD and Linux (and if we upgrade bazil.org/fuse, only FreeBSD and Linux: https://github.com/bazil/fuse/issues/224). Listing the few supported operating systems explicitly here makes porting restic to new platforms easier.	2020-05-12 11:30:41 +02:00
greatroar	9f7cd69f13	Move Index.FindBlob to tests	2020-04-29 10:57:01 +02:00
greatroar	f97a680887	Fix repository benchmarks BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of their time measuring SHA-256 performance in their checks.	2020-04-28 07:57:29 +02:00
greatroar	42a3db05b0	Use Minio's optimized SHA-256 internal/repository benchmarks on an Intel i7-3770k: name old speed new speed delta PackerManager-8 209MB/s ± 1% 291MB/s ± 1% +38.94% (p=0.008 n=5+5) SaveAndEncrypt-8 112MB/s ± 1% 135MB/s ± 1% +20.25% (p=0.008 n=5+5)	2020-04-28 07:57:18 +02:00
greatroar	e7d7b85d59	Merge Repository.{LoadBlob,loadBlob} Pushing the allocation logic down into the former loadBlob body means that fewer allocations have to be performed: name old time/op new time/op delta LoadTree-8 478µs ± 1% 481µs ± 2% ~ (p=0.315 n=9+10) LoadBlob-8 11.6ms ± 1% 11.6ms ± 2% ~ (p=0.393 n=10+10) LoadAndDecrypt-8 13.3ms ± 3% 13.3ms ± 3% ~ (p=0.905 n=10+9) LoadIndex-8 33.6ms ± 2% 33.2ms ± 1% -1.15% (p=0.028 n=10+9) name old alloc/op new alloc/op delta LoadTree-8 41.2kB ± 0% 41.1kB ± 0% -0.23% (p=0.000 n=10+10) LoadBlob-8 2.28kB ± 0% 2.18kB ± 0% -4.21% (p=0.000 n=10+10) LoadAndDecrypt-8 2.10MB ± 0% 2.10MB ± 0% ~ (all equal) LoadIndex-8 5.22MB ± 0% 5.22MB ± 0% ~ (p=0.631 n=10+10) name old allocs/op new allocs/op delta LoadTree-8 652 ± 0% 651 ± 0% -0.15% (p=0.000 n=10+10) LoadBlob-8 24.0 ± 0% 23.0 ± 0% -4.17% (p=0.000 n=10+10) LoadAndDecrypt-8 30.0 ± 0% 30.0 ± 0% ~ (all equal) LoadIndex-8 30.2k ± 0% 30.2k ± 0% ~ (p=0.610 n=10+10) name old speed new speed delta LoadBlob-8 86.4MB/s ± 1% 85.9MB/s ± 2% ~ (p=0.393 n=10+10) LoadAndDecrypt-8 75.4MB/s ± 3% 75.4MB/s ± 3% ~ (p=0.858 n=10+9)	2020-04-23 10:04:20 +02:00
greatroar	be5a0ff59f	Centralize buffer allocation and size checking in Repository.LoadBlob Benchmark results for internal/repository: name old time/op new time/op delta LoadTree-8 479µs ± 2% 478µs ± 1% ~ (p=0.780 n=10+9) LoadBlob-8 11.6ms ± 2% 11.6ms ± 1% ~ (p=0.631 n=10+10) LoadAndDecrypt-8 13.2ms ± 2% 13.3ms ± 3% ~ (p=0.631 n=10+10) name old alloc/op new alloc/op delta LoadTree-8 41.2kB ± 0% 41.2kB ± 0% ~ (all equal) LoadBlob-8 2.28kB ± 0% 2.28kB ± 0% ~ (all equal) LoadAndDecrypt-8 2.10MB ± 0% 2.10MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta LoadTree-8 652 ± 0% 652 ± 0% ~ (all equal) LoadBlob-8 24.0 ± 0% 24.0 ± 0% ~ (all equal) LoadAndDecrypt-8 30.0 ± 0% 30.0 ± 0% ~ (all equal) name old speed new speed delta LoadBlob-8 86.2MB/s ± 2% 86.4MB/s ± 1% ~ (p=0.594 n=10+10) LoadAndDecrypt-8 75.7MB/s ± 2% 75.4MB/s ± 3% ~ (p=0.617 n=10+10)	2020-04-23 10:04:20 +02:00
MichaelEischer	4f00564574	Merge pull request #2621 from greatroar/fixes Some small fixes	2020-04-18 18:07:08 +02:00
MichaelEischer	f77477129f	Merge pull request #2584 from greatroar/mount-cache-uid-gid Cache uid and gid for top directories in internal/fuse	2020-04-18 17:45:14 +02:00
greatroar	2e31120f89	Remove unused argument to restic.fakeFile	2020-04-18 17:40:13 +02:00
greatroar	8fb2c0d3c1	Typo in crypto test name	2020-04-18 17:39:06 +02:00
greatroar	072cf7b02d	Fix debug messages in internal/fuse	2020-04-18 17:39:06 +02:00
greatroar	df66daa5c9	Fix context usage in backend tests Found by go vet. This is also the only complaint is has.	2020-04-18 17:39:06 +02:00
MichaelEischer	16710454f4	Merge pull request #2628 from MichaelEischer/one-element-pack-lists cache: Don't sort one element pack lists	2020-04-18 17:09:06 +02:00
MichaelEischer	7910ff4c0e	Merge pull request #2648 from nairb774/iowritestring termstatus: Use io.WriteString to output messages.	2020-04-18 13:48:42 +02:00
MichaelEischer	c4da9d1e90	Merge pull request #2638 from greatroar/no-close-in-packer Don't Close in Packer.Finalize	2020-04-18 13:07:21 +02:00
MichaelEischer	a1352906e2	Merge pull request #2622 from greatroar/optimize-packer-manager Fix PackerManager benchmark and optimize hashing.Writer	2020-04-18 12:46:34 +02:00
MichaelEischer	b7c0d4d8bf	Merge pull request #2644 from greatroar/signal-notify-buffered Make all signal.Notify channels buffered	2020-04-18 11:31:02 +02:00
Michael Eischer	bdf7ba20cb	archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError The Save methods of the BlobSaver, FileSaver and TreeSaver return early on when the archiver is stopped due to an error. For that they select on both the tomb.Dying() and context.Done() channels, which can lead to a race condition when the tomb is killed due to an error: The tomb first closes its Dying channel before canceling all child contexts. Archiver.SaveDir only aborts its execution once the context was canceled. When the tomb killing is paused between closing its Dying channel and canceling the child contexts, this lets the FileSaver/TreeSaver.Save methods return immediately, however, ScanDir still reads further files causing the test case to fail. As a killed tomb always cancels all child contexts and as the Savers always use a context bound to the tomb, it is sufficient to just use context.Done() as escape hatch in the Save functions. This fixes the mismatch between SaveDir and Save. Adjust the tests to use contexts bound to the tomb for all interactions with the Savers.	2020-04-13 18:23:17 +02:00
Brian Atkinson	b8da7b1f4d	termstatus: Use io.WriteString to output messages. The previous implementation was repeating the implementation that is found inside of io.WriteString. Simplify by making use of the stdlib's implementation.	2020-03-26 14:55:00 -07:00
Peter Schultz	90fc639a67	Allow specifying user and host when adding keys The username and hostname for new keys can be specified with the new --user and --host flags, respectively. The flags are used only by the `key add` command and are otherwise ignored. This allows adding keys with for a desired user and host without having to run restic as that particular user on that particular host, making automated key management easier. Co-authored-by: James TD Smith <ahktenzero@mohorovi.cc>	2020-03-23 13:11:10 +01:00
rawtaz	9efbe98879	Merge pull request #2623 from alrs/internal-restic-err-before-close internal/restic: close os.File after checking for error	2020-03-18 22:23:20 +01:00
rawtaz	7d9300efca	Merge pull request #2637 from greatroar/unused Remove Go 1.5 compatibility code	2020-03-18 22:22:15 +01:00
greatroar	47d4d5bf1b	Make all signal.Notify channels buffered	2020-03-12 20:59:39 +01:00
greatroar	74a64c47e4	Move testing logic to test file in internal/pack	2020-03-09 14:32:28 +01:00
greatroar	a23e9c86ba	Remove closing logic from Packer.Finalize The method only ever receives *hashing.Writers, which don't implement io.Closer. These come from packerManager.findPacker and have their actual writers closed in Repository.savePacker. Moving the closing logic to hashing.Writer results in "file already closed" errors.	2020-03-09 14:31:45 +01:00
greatroar	4de12bf593	Remove restic.RandReader math/rand.Rand has implemented Reader since Go 1.6. The repacking tests are not deterministic, but they weren't before, either.	2020-03-09 10:00:28 +01:00
greatroar	8cf3bb8737	Revert "Put host last in SSH command line" This reverts commit `e1969d1e33`.	2020-03-08 16:45:33 +01:00
Michael Eischer	b46cc6d57e	repository: Don't sort one element pack lists When loading a blob, restic first looks up pack files containing the blob. To avoid unnecessary work an already cached pack file is preferred. However, if there is only a single pack file to choose from (which is the normal case) sorting the one-element list won't change anything. Therefore avoid the unnecessary cache check in that case.	2020-03-07 10:26:06 +01:00
Lars Lehtonen	4a2156d3f0	internal/restic: close os.File after checking for error	2020-03-05 16:22:46 -08:00
greatroar	41fee11f66	Micro-optimization for hashing.Writer/PackerManager name old time/op new time/op delta PackerManager-8 247ms ± 1% 246ms ± 1% -0.43% (p=0.001 n=18+18) name old speed new speed delta PackerManager-8 213MB/s ± 1% 214MB/s ± 1% +0.43% (p=0.001 n=18+18) name old alloc/op new alloc/op delta PackerManager-8 92.2kB ± 0% 91.5kB ± 0% -0.82% (p=0.000 n=19+20) name old allocs/op new allocs/op delta PackerManager-8 1.43k ± 0% 1.41k ± 0% -1.67% (p=0.000 n=20+20)	2020-03-05 22:30:04 +01:00
greatroar	b592614061	Improve PackerManager benchmark The previous benchmark spent much of its time allocating RNGs and generating too many random numbers. It now spends 90% of its time hashing and half of the rest writing to files. name old time/op new time/op delta PackerManager-8 319ms ± 1% 247ms ± 1% -22.48% (p=0.000 n=20+18) name old speed new speed delta PackerManager-8 143MB/s ± 1% 213MB/s ± 1% +48.63% (p=0.000 n=10+18) name old alloc/op new alloc/op delta PackerManager-8 635kB ± 0% 92kB ± 0% -85.48% (p=0.000 n=10+19) name old allocs/op new allocs/op delta PackerManager-8 1.64k ± 0% 1.43k ± 0% -12.76% (p=0.000 n=10+20)	2020-03-05 22:30:03 +01:00
greatroar	b7c3039eb2	Remove Go 1.5 compatibility code from PackerManager benchmark This alone is enough to speed up the benchmark by ~10%.	2020-03-05 22:29:06 +01:00
Alexander Neumann	52abec967f	Merge pull request #2605 from middelink/fix-2604 Fix running tests on a SELinux enabled system	2020-03-01 20:25:28 +01:00
Pauline Middelink	2828a9c2b0	Fix running tests on a SELinux enabled system Archivers TestMetadataChanged incorrectly clears the Extended Attributes from the expected metadata of the temporary file. This is incorrect as on SELinux enabled filesystem, as the kernel will automaticly add a SElinux label. However, since ExtendedAttributes{} != ExtendedAttributes{nil} we still need to clear them if there are no attributes found.	2020-03-01 20:23:22 +01:00
Alexander Neumann	95da6c1c1d	Merge pull request #2589 from greatroar/no-stable-sort Replace sort.Stable by sort.Strings	2020-03-01 19:40:28 +01:00
Alexander Neumann	0c03a80fc4	Merge pull request #2592 from greatroar/sftp-ipv6 Support IPv6 in SFTP backend	2020-03-01 19:38:44 +01:00
Alexander Neumann	c50f91b7f9	Merge pull request #2602 from greatroar/no-bufpool Remove sync.Pool from internal/repository	2020-03-01 10:50:57 +01:00
Alexander Neumann	e851d29565	Merge pull request #2608 from greatroar/simplify-termstatus Simplify termstatus	2020-03-01 10:48:29 +01:00
Alexander Neumann	b67b7ebfe6	Merge pull request #2583 from greatroar/unused Remove some unused or duplicated code	2020-03-01 10:46:17 +01:00
greatroar	751eba0e68	Remove unnecessary pipe checks in termstatus canUpdateStatus has already determined that the file descriptor is not a pipe.	2020-02-29 18:03:49 +01:00
greatroar	7447c44484	Use golang.org/x/sys/windows in termstatus Some functionality is missing, but at least the types are all defined. Replaced short, word, dword by their Go names to match the x/sys convention.	2020-02-29 18:03:49 +01:00
greatroar	c8a672fa29	Remove code copy-pasted from x/crypto/ssh/terminal	2020-02-29 18:03:49 +01:00
greatroar	863ba76494	Drop mattn/go-isatty in favor of crypto/ssh/terminal	2020-02-29 18:03:47 +01:00
greatroar	8526cc6647	Remove sync.Pool from internal/repository The pool was used improperly, causing more allocations to be performed than without it. name old time/op new time/op delta SaveAndEncrypt-8 36.8ms ± 2% 36.9ms ± 2% ~ (p=0.218 n=10+10) name old speed new speed delta SaveAndEncrypt-8 114MB/s ± 2% 114MB/s ± 2% ~ (p=0.218 n=10+10) name old alloc/op new alloc/op delta SaveAndEncrypt-8 21.1MB ± 0% 21.0MB ± 0% -0.44% (p=0.000 n=10+10) name old allocs/op new allocs/op delta SaveAndEncrypt-8 79.0 ± 0% 77.0 ± 0% -2.53% (p=0.000 n=10+10)	2020-02-29 17:54:46 +01:00
greatroar	5cd0bce452	Honor RESTIC_CACHE_DIR on Mac and Windows	2020-02-28 15:44:32 +01:00
rawtaz	58bd165253	Merge pull request #2581 from aawsome/multiple-hostnames Allow multiple hostnames tags	2020-02-27 08:35:23 +01:00
rawtaz	65d3fb6b33	Merge pull request #2603 from greatroar/restorer-waitgroup Fix unsafe sync.WaitGroup usage in restorer.fileRestorer	2020-02-27 00:30:59 +01:00
greatroar	de5516a90e	Fix sync.WaitGroup usage in restorer.fileRestorer	2020-02-27 00:07:49 +01:00
greatroar	4f6fd9fb98	Remove remnant of Go 1.9 compatibility code from tests	2020-02-26 22:23:38 +01:00
Alexander Weiss	9a9101d144	Support specifying multiple host flags for various commands The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`, `stats` and `tag` commands will now take into account multiple `--host` and `-H` flags.	2020-02-26 22:17:59 +01:00
Alexander Neumann	99fd80a585	Remove all workarounds for Go < 1.11	2020-02-26 20:35:13 +01:00
Igor Fedorenko	f17ffa0283	restorer: Allow writing target file blobs out of order Much simpler implementation that guarantees each required pack is downloaded only once (and hence does not need to manage pack cache). Also improves large file restore performance. Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2020-02-26 16:14:45 +01:00
greatroar	5e2afd91e7	Assert that archiver.Tree implements fmt.Stringer	2020-02-26 11:05:38 +01:00
greatroar	79b882e901	Merge duplicated readdir functionality internal/archiver.readdir and internal/fs.ReadDir were unused. internal/fs.ReadDirNames and internal/archiver.readdirnames were doing nearly the same thing, except one sorted its output and opened with fs.O_NOFOLLOW. Both were only used in internal/archiver.	2020-02-26 11:05:38 +01:00
greatroar	1b502fa9ef	Cache uid and gid for top directories in internal/fuse	2020-02-24 10:46:09 +01:00
greatroar	e1969d1e33	Put host last in SSH command line This is how the SSH manpage says the command line should look, and the "--" prevents mistakes in hostnames from being interpreted as options.	2020-02-19 15:53:20 +01:00
greatroar	6ac6bca7a1	Support IPv6 in SFTP backend The previous code was doing its own hostname:port splitting, which caused IPv6 addresses to be misinterpreted.	2020-02-19 15:42:12 +01:00
greatroar	3a6feb0596	Replace sort.Stable by sort.Strings Calling the slow, O(n lg² n) sort.Stable is equivalent to sort.Strings for a slice of unique strings.	2020-02-18 19:41:06 +01:00
greatroar	2f8aa2ce30	Remove unused fs.FS from archiver.FileSaver	2020-02-18 10:39:14 +01:00
Michael Eischer	e1f722d266	archiver: Fix flaky TestArchiverAbortEarlyOnError Each of the random test files was split into the same five blobs. The test fails once the fifth blob is passed on to `SaveBlob`. That is for certain interleavings of goroutine execution it would be possible for the test to trigger the testErr just after storing the first file. The fixed test uses a different file content for each of the nine files and fails after writing the fourth blob. The file content is also small enough to ensure that for each file only a single blob is saved. This guarantees that the test cannot fail before reading the first four files. FileReadConcurrency = 2 allows up to two files queued for processing. Therefore the test can at most open the sixth file before it has to save the fourth file / blob which triggers the testErr.	2020-02-14 23:16:13 +01:00
Michael Eischer	42a3292bcf	Better name for jsonstatus package internal/ui/jsonstatus and termstatus sound similar but are not related in any way. Instead `internal/ui/backup` and `internal/ui/jsonstatus/status` are the counterparts. Rename the latter to `internal/ui/json/backup` to make this clear.	2020-02-13 21:14:20 +01:00
Michael Eischer	ef70a2fcb3	Fix mangled JSON output by backup command jsonstatus wrote the JSON output without synchronization to the stdio_wrapper which caused mangling between different status lines. Use the Print and Error methods of termstatus instead which use a central goroutine to synchronize output.	2020-02-13 21:14:20 +01:00
rawtaz	680a14afa1	Merge pull request #2530 from restic/fix-sftp-mkdirall sftp: Use MkdirAll provided by the client	2020-02-13 01:12:59 +01:00
rawtaz	d8da9c4401	Merge pull request #2577 from alrs/fix-internal-errs internal: Fix code and test dropped errors	2020-02-12 23:41:54 +01:00
rawtaz	299f5971f2	Merge pull request #2560 from brualan/master Two small improvements to code quality	2020-02-12 22:40:20 +01:00
Lars Lehtonen	72734d59b5	internal/archiver: fix dropped error	2020-02-12 13:37:37 -08:00
Lars Lehtonen	3ed54e762e	internal/backend/sftp: fix dropped test error	2020-02-12 13:36:21 -08:00
Lars Lehtonen	fea835b4e2	internal/repository: fix dropped test error	2020-02-12 13:33:54 -08:00
Lars Lehtonen	16b321b140	internal/restic: fix dropped test error	2020-02-12 13:32:45 -08:00
Michael Eischer	760863e7f9	archiver: Fix TestRacyFileSwap on windows	2020-02-11 21:09:47 +01:00
Michael Eischer	a135699397	Close file if file type has changed after initial stat	2020-02-11 21:09:47 +01:00
Lars Lehtonen	57815d9cd6	restic/internal/ui: fix gofmt nit that popped up in Go 1.13	2020-02-10 11:06:47 -08:00
Daniel Hoffend	e7cdf2acbb	fix backup --json total_bytes_processed output Closes #2429	2020-02-09 02:02:45 +01:00
Alexander Bruyako	38ddfbc4d3	simpler error return	2020-01-27 18:41:46 +03:00
Alexander Bruyako	da48b925ff	remove unnecessary error return I was running "golangci-lint" and found this two warnings internal/checker/checker.go:135:18: (Checker).LoadIndex$3 - result 0 (error) is always nil (unparam) final := func() error { ^ internal/repository/repository.go:457:18: (Repository).LoadIndex$3 - result 0 (error) is always nil (unparam) final := func() error { ^ It turns out that these functions are used only in "RunWorkers(...)", which is used only two times in whole project right after this "final" functions. And because these "final" functions always return "nil", I've descided, that it would be better to remove requriments for "final" func to return error to avoid magick "return nil" at their end.	2020-01-27 18:28:21 +03:00
rawtaz	d70a4a9350	Merge pull request #2373 from vrusinov/issue-2372 Ignore username difference in TestMetadataChanged	2020-01-05 21:07:34 +01:00
Alexander Neumann	2cd9c7ef16	sftp: Use MkdirAll provided by the client Closes #2518	2020-01-01 17:26:38 +01:00
Ward Vandewege	69a0d0ee90	fix typos	2019-12-21 21:00:28 -05:00
streambinder	97e5ce4344	internal: backend: sftp: support user@domain parsing as user	2019-12-19 13:15:37 +01:00
rawtaz	da4473aba6	Merge pull request #2391 from kastenhq/refresh-lock-time Update Lock.Time in lock.Refresh()	2019-11-26 21:44:29 +01:00
rawtaz	6e85a58045	Merge pull request #2465 from whs-dot-hk/fix-test-metadata-changed Fix test metadata changed	2019-11-26 21:42:42 +01:00
rawtaz	e14c4b1737	Merge pull request #2484 from restic/add-s3-region s3: Allow specifying region	2019-11-22 15:51:17 +01:00
Alexander Neumann	409909a7f5	Add option description for Region	2019-11-22 15:09:09 +01:00
mdauphin	df500a372d	Add AWS_REGION env var to specify s3 region	2019-11-22 15:04:48 +01:00
Alexander Neumann	a6e8af7e0f	Update minio-go	2019-11-22 14:50:46 +01:00
rawtaz	41fe9318b1	Merge pull request #2425 from thiell/restic_cache_dir_env Add support for $RESTIC_CACHE_DIR	2019-11-20 21:18:57 +01:00
Leo R. Lundgren	4d2aa18273	Accept both singular and plural for --group-by (#2330 )	2019-11-20 19:25:48 +01:00
rawtaz	26d1f9f4ba	Merge pull request #2307 from gary-kim/fix-2306-password-retries Allow multiple retries for interactive password input	2019-11-20 18:30:20 +01:00
whs	7cacba0394	Assume WithAtime default to false	2019-11-06 16:38:46 +08:00
Stephane Thiell	0e897ef7b8	Add support for $RESTIC_CACHE_DIR Add support for restic-specific $RESTIC_CACHE_DIR environment variable to override the cache directory like --cache-dir would have.	2019-09-26 15:59:56 -07:00
Eric Hamilton	1596d06f8e	Update Lock.Time in lock.Refresh()	2019-09-04 11:38:35 -07:00
Vladimir Rusinov	db20c0b8d0	Ignore username difference in TestMetadataChanged. In some (rare) cases "fake" UID 51234 may exist in a system running a test. When this is the case, `cmp.Equal(want, node3)` will fail based on difference between empty string and an actual username present in a system. Fixes github issue #2372	2019-08-13 22:25:00 +01:00
Alexander Neumann	01c51b3449	Merge pull request #2321 from restic/fix-index-error-handling Check errors returned by LoadIndex()	2019-07-27 09:37:18 +02:00
Alexandr Bruyako	e19622e4b1	start using fileMode const	2019-07-01 00:26:00 +03:00
Alexandr Bruyako	38ea7ed4f6	remove unused code	2019-07-01 00:24:45 +03:00
Alexandr Bruyako	76d1866444	avoiding unnecessary type conversions	2019-06-30 23:58:00 +03:00
Alexandr Bruyako	8b22fe29cf	improved slice copying	2019-06-30 23:56:36 +03:00
Alexandr Bruyako	02014be76c	simplified prefix removal, removed unnecessary if-else statements	2019-06-30 23:34:47 +03:00
Alexandr Bruyako	16eeed2ad5	simplified string sorting by using a more suitable function	2019-06-30 23:20:32 +03:00
Alexander Neumann	88716794e3	Check errors returned by LoadIndex() Bug was reported in the forum here: https://forum.restic.net/t/check-rebuild-index-prune/1848/13	2019-06-30 21:34:53 +02:00
Gary Kim	fea2464d4d	Allow multiple retries for interactive password input Restic used to quit if the repository password was typed incorrectly once. Restic will now ask the user again for the repository password if typed incorrectly. The user will now get three tries to input the correct password before restic quits.	2019-06-13 20:11:02 +08:00
Garry McNulty	8066195e6e	fs: Handle absolute pathname for --stdin-filename Return valid directory info from Lstat() for parent directories of the specified filename. Previously only "/" and "." were valid directories. Also set directory mode as this is checked by archiver. Closes #2063	2019-05-08 15:21:27 +02:00
Garry McNulty	f7f14cf8c9	fs: Add file info base name check in reader tests (#2063 )	2019-05-08 15:21:27 +02:00
Garry McNulty	5096f3b491	fs: Update directory check in reader tests (#2063 )	2019-05-08 15:21:27 +02:00
Alexander Neumann	920d458a4a	archiver: Use untyped constants for testing FileInfo	2019-05-05 14:57:38 +02:00
Alexander Neumann	b016dc2ff0	archiver/Windows: Skip test new-content-same-filestamp	2019-05-05 14:02:11 +02:00
Alexander Neumann	355db0bc29	windows: Use LastWriteTime for ctime and mtime Windows does not have a concept of a `change time` in the sense as Unix has it: the field `CreationTime` of the `Win32FileAttributeData` struct is not updated when attributes or content is changed. So from now on we're using the `LastWriteTime` as the `change time` on Windows.	2019-05-05 14:02:11 +02:00
Alexander Neumann	6e2fe73189	archiver: Move tests back into the same file Move all Archiver tests back into `archiver_test.go` and add some tiny helpers to mock what `lstat` returns (for Windows and Unix separately).	2019-05-05 14:02:11 +02:00
Alexander Neumann	303a5dab6a	archiver: Clarify value in test struct Since I could not remember what the value for `Check` means this commit renames it to `SameFile`: when set to true, the test should make sure that `FileChanged` should return false (=file is unmodified).	2019-05-05 12:57:00 +02:00
Gábor Lipták	bce6438d22	Don't run TestMetadataChanged test on Windows Signed-off-by: Gábor Lipták <gliptak@gmail.com>	2019-04-27 21:23:47 -04:00
Alexander Neumann	919dd2ac84	Merge pull request #2252 from restic/fix-2249 Read fresh metadata for unmodified files	2019-04-25 09:15:50 +02:00
Alexander Neumann	870bc5108e	Merge pull request #2253 from restic/fix-2174 Make sure timestamps are valid	2019-04-25 09:14:43 +02:00
Alexander Neumann	a6481b3707	Merge pull request #2212 from cbane/check-ctime Examine file ctime when checking if files have changed.	2019-04-25 08:58:27 +02:00
Courtney Bane	0ebfc55ee3	Use existing setTimestamp function for ctime test and improve error checking.	2019-04-24 20:53:08 -05:00
Courtney Bane	35b7607802	Don't check ctime when ignoring inode.	2019-04-24 20:53:08 -05:00
Alexander Neumann	939f3e972c	node: Make sure year of all timestamps is valid Sometimes restic gets bogus timestamps which cannot be converted to JSON, because the stdlib JSON encoder returns an error if the year is not within [0, 9999]. We now make sure that we at least record _some_ timestamp and cap the year either to 0000 or 9999. Before, restic would refuse to save the file at all, so this improves the status quo. This fixes #2174 and #1173	2019-04-24 16:39:36 +02:00
Alexander Neumann	389067fb8b	Only use list of blobs for old node Closes #2249	2019-04-24 15:07:26 +02:00
Alexander Neumann	4b0ca9ddab	Add test for #2249	2019-04-24 15:07:23 +02:00
Courtney Bane	b8c2544dcb	Examine file ctime when checking if files have changed.	2019-04-23 21:54:35 -05:00
Alexander Neumann	18ec27a0da	Merge pull request #2243 from treymo/ls-length-2104 backend find: Check prefix length < snapshot IDs when searching	2019-04-23 09:41:19 +02:00
Jan Niklas Richter	733519d895	Refactor duplicate code for grouping snapshots This commit is a followup to the addition of the --group-by flag for the snapshots command. Adding the grouping code there introduced duplicated code (the forget command also does grouping). This commit refactors boths sides to only use shared code.	2019-04-22 14:57:53 +02:00
Trey Moore	5ac9c1157a	Prevent "slice bounds out of range" error if prefix is longer than snapshot IDs. This includes tests as well as other tests for the backend_find.Find function. Issue #2104.	2019-04-16 20:26:20 -07:00
Alexander Neumann	6b700d02f5	Merge pull request #2217 from restic/improve-memory-usage WIP: improve memory usage	2019-04-13 15:07:07 +02:00
Alexander Neumann	7304738872	check: Reduce default parallelism from 40 to 5	2019-04-13 13:38:39 +02:00
Alexander Neumann	66efa425bf	Reuse buffer in worker functions	2019-04-13 13:38:39 +02:00
Alexander Neumann	d51e9d1b98	Add []byte to repo.LoadAndDecrypt and utils.LoadAll This commit changes the signatures for repository.LoadAndDecrypt and utils.LoadAll to allow passing in a []byte as the buffer to use. This buffer is enlarged as needed, and returned back to the caller for further use. In later commits, this allows reducing allocations by reusing a buffer for multiple calls, e.g. in a worker function.	2019-04-13 13:38:39 +02:00
Alexander Neumann	e046428c94	Replace FilesInParallel with an errgroup.Group	2019-04-13 13:38:39 +02:00
Alexander Neumann	75906edef5	mock: Remove unused repository type	2019-04-13 13:38:39 +02:00
Alexander Neumann	203d775190	restic: Make JSON unmarshal for ID more efficient This commit reduces several allocations in UnmarshalJSON() by decoding the hex string directly in a single step.	2019-04-13 13:38:39 +02:00
Benoît Knecht	3c112d9cae	s3: Add config option to set storage class The `s3.storage-class` option can be passed to restic (using `-o`) to specify the storage class to be used for S3 objects created by restic. The storage class is passed as-is to S3, so it needs to be understood by the API. On AWS, it can be one of `STANDARD`, `STANDARD_IA`, `ONEZONE_IA`, `INTELLIGENT_TIERING` and `REDUCED_REDUNDANCY`. If unspecified, the default storage class is used (`STANDARD` on AWS). You can mix storage classes in the same bucket, and the setting isn't stored in the restic repository, so be sure to specify it with each command that writes to S3. Closes #706	2019-03-26 16:37:07 +01:00
Alexander Neumann	c1058005c3	Fix build on FreeBSD for Go < 1.12	2019-03-16 20:52:37 +01:00
Alexander Neumann	ca73808649	Merge pull request #2197 from dmgk/master Fix build on FreeBSD with Go 1.12	2019-03-16 20:23:19 +01:00
Igor Fedorenko	bf9a507148	Fix rare 'file already closed' during restore Fixes #2183 Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>	2019-03-16 17:01:20 +01:00
Alexander Neumann	65b476ead9	Fix gofmt	2019-03-16 13:29:05 +01:00
Alexander Neumann	aaa1cc2c26	Merge pull request 2193 from restic/allow-empty-rclone-args rclone: Rework backend option parsing	2019-03-16 12:17:38 +01:00
Alexander Neumann	95434cff16	Merge pull request 2156 from kayrus/swift-app-cred Swift: introduce application credential auth support	2019-03-16 12:17:17 +01:00
Heiko Bornholdt	db8f5864fc	Add --ignore-inode option to backup cmd revised version of https://github.com/restic/restic/pull/2047	2019-03-10 21:24:29 +01:00
Dmitri Goutnik	1d8b21cdad	Fix build on FreeBSD with Go 1.12	2019-03-06 09:41:49 -05:00
Alexander Neumann	3865b59716	rclone: Rework backend option parsing This change allows passing no arguments to rclone, using `-o rclone.args=""`. It is helpful when running rclone remotely via SSH using a key with a forced command (via `command=` in `authorized_keys`).	2019-03-02 10:36:42 +01:00
Alexander Neumann	e277a92a2f	self-update: Don't cancel download after 30 seconds	2019-02-23 11:15:18 +01:00
kayrus	6ebcfe7c18	Swift: introduce application credential auth support	2019-02-14 14:19:05 +01:00

... 7 8 9 10 11 ...

1323 Commits