1559 Commits

Author	SHA1	Message	Date
Alexander Neumann	fb5d9345a7	Merge pull request #3510 from MichaelEischer/fix-archiver-early-on-abort ... archiver: Fix TestArchiverAbortEarlyOnError test	2021-10-16 15:37:41 +02:00
kitone	95eb859b54	Honor RESTIC_CACHE_DIR environment variable ... Fix #3382: restic check doesn't obey the RESTIC_CACHE_DIR environment variable	2021-10-10 16:00:02 +02:00
Michael Eischer	257740b0cc	b2: Successful delete if file does not exist ... When deleting a file, B2 sometimes returns a "500 Service Unavailable" error but nevertheless correctly deletes the file. Due to retries in the B2 library blazer, we sometimes also see a "400 File not present" error. The retries of restic for the delete request then fail with "404 File with such name does not exist.". As we have to rely on request retries in a distributed system to handle temporary errors, also consider a delete request to be successful if the file is reported as not existing. This should be safe as B2 claims to provide a strongly consistent bucket listing and thus a missing file shouldn't mysteriously show up again later on.	2021-10-09 23:51:12 +02:00
MichaelEischer	f2a3b3b4a1	Merge pull request #3537 from greatroar/dump-lru ... Check cap instead of len in bloblru	2021-10-09 20:00:27 +02:00
Garry McNulty	708d7a2574	s3: Add warning if key ID or secret is empty ... Also add debug message if no credential types are available. Closes #2388	2021-10-06 23:13:40 +01:00
greatroar	634a9c162d	Check cap instead of len in bloblru ... restic dump uses bloblru.Cache to keep buffers alive, but also reuses evicted buffers. That means large buffers may be used to store small blobs, causing the cache to think it's using less memory than it actually does.	2021-10-03 09:34:17 +02:00
greatroar	c892c0bab9	internal/restic: Don't allocate in Tree.Insert ... name old time/op new time/op delta BuildTree-8 34.6µs ± 4% 7.0µs ± 3% -79.68% (p=0.000 n=18+19) name old alloc/op new alloc/op delta BuildTree-8 34.0kB ± 0% 0.9kB ± 0% -97.37% (p=0.000 n=20+20) name old allocs/op new allocs/op delta BuildTree-8 108 ± 0% 1 ± 0% -99.07% (p=0.000 n=20+20)	2021-09-26 18:08:48 +02:00
MichaelEischer	78dac2fd48	Merge pull request #3523 from greatroar/swift-v2 ... Upgrade ncw/swift to v2	2021-09-24 21:41:36 +02:00
Michael Eischer	5ea8bba1a1	swift: restore context err check for list()	2021-09-24 21:19:46 +02:00
MichaelEischer	a5e103a212	Merge pull request #3522 from greatroar/dump-lru ... Use LRU cache in restic dump	2021-09-24 20:33:58 +02:00
greatroar	e7ec0453b1	Upgrade ncw/swift to v2	2021-09-24 19:08:37 +02:00
greatroar	fe04d024c7	Use LRU cache in restic dump	2021-09-24 15:45:08 +02:00
Uli Martens	718966a81a	Move Blobcache into dedicated internal package	2021-09-24 15:45:00 +02:00
greatroar	4f33eca634	Remove unused Writer arg to internal/dump.writeDump	2021-09-24 15:40:42 +02:00
Sam Lucidi	897d8e662c	Add --insecure-tls flag to disable SSL cert verification ... Signed-off-by: Sam Lucidi <slucidi@redhat.com>	2021-09-21 10:52:40 -04:00
MichaelEischer	1827b16ade	Merge pull request #3519 from greatroar/maphash ... Replace siphash by hash/maphash	2021-09-19 19:46:03 +02:00
greatroar	8d2996eaaa	Replace siphash by hash/maphash ... In Go 1.17.1, maphash has become quite a bit faster than siphash, so we can drop one third-party dependency. maphash is just an interface to the standard Go map's hash function, which we already trust for other use cases. Benchmark results on linux/amd64, -benchtime=3s: name old time/op new time/op delta IndexHasUnknown-8 50.6ns ±10% 41.0ns ±19% -18.92% (p=0.000 n=9+10) IndexHasKnown-8 52.6ns ±12% 41.5ns ±12% -21.13% (p=0.000 n=9+10) IndexMapHash-8 3.64µs ± 1% 2.00µs ± 0% -45.09% (p=0.000 n=10+9) IndexAlloc-8 700ms ± 1% 601ms ± 6% -14.18% (p=0.000 n=8+10) IndexAllocParallel-8 205ms ± 5% 192ms ± 8% -6.18% (p=0.043 n=10+10) MasterIndexAlloc-8 319ms ± 1% 279ms ± 5% -12.58% (p=0.000 n=10+10) MasterIndexLookupSingleIndex-8 156ns ± 8% 147ns ± 6% -5.46% (p=0.023 n=10+10) MasterIndexLookupMultipleIndex-8 150ns ± 7% 142ns ± 8% -5.69% (p=0.007 n=10+10) MasterIndexLookupSingleIndexUnknown-8 74.4ns ± 6% 72.0ns ± 9% ~ (p=0.175 n=10+9) MasterIndexLookupMultipleIndexUnknown-8 67.4ns ± 9% 65.5ns ± 7% ~ (p=0.340 n=9+9) MasterIndexLookupParallel/known,indices=25-8 461ns ± 2% 445ns ± 2% -3.49% (p=0.000 n=10+10) MasterIndexLookupParallel/unknown,indices=25-8 408ns ±11% 378ns ± 5% -7.22% (p=0.035 n=10+9) MasterIndexLookupParallel/known,indices=50-8 479ns ± 1% 437ns ± 4% -8.82% (p=0.000 n=10+10) MasterIndexLookupParallel/unknown,indices=50-8 406ns ± 8% 343ns ±15% -15.44% (p=0.001 n=10+10) MasterIndexLookupParallel/known,indices=100-8 480ns ± 1% 455ns ± 5% -5.15% (p=0.000 n=8+10) MasterIndexLookupParallel/unknown,indices=100-8 391ns ±18% 382ns ± 8% ~ (p=0.315 n=10+10) MasterIndexLookupBlobSize-8 71.0ns ± 8% 57.2ns ±11% -19.36% (p=0.000 n=9+10) PackerManager-8 254ms ± 1% 254ms ± 1% ~ (p=0.285 n=15+15) name old speed new speed delta IndexMapHash-8 1.12GB/s ± 1% 2.05GB/s ± 0% +82.13% (p=0.000 n=10+9) PackerManager-8 208MB/s ± 1% 207MB/s ± 1% ~ (p=0.281 n=15+15) name old alloc/op new alloc/op delta IndexMapHash-8 0.00B 0.00B ~ (all equal) IndexAlloc-8 400MB ± 0% 400MB ± 0% ~ (p=1.000 n=9+10) IndexAllocParallel-8 401MB ± 0% 401MB ± 0% +0.00% (p=0.000 n=10+10) MasterIndexAlloc-8 258MB ± 0% 262MB ± 0% +1.42% (p=0.000 n=9+10) PackerManager-8 73.1kB ± 0% 73.1kB ± 0% ~ (p=0.382 n=13+13) name old allocs/op new allocs/op delta IndexMapHash-8 0.00 0.00 ~ (all equal) IndexAlloc-8 907k ± 0% 907k ± 0% -0.00% (p=0.000 n=10+10) IndexAllocParallel-8 907k ± 0% 907k ± 0% +0.00% (p=0.009 n=10+10) MasterIndexAlloc-8 327k ± 0% 317k ± 0% -3.06% (p=0.000 n=10+10) PackerManager-8 744 ± 0% 744 ± 0% ~ (all equal)	2021-09-19 16:05:18 +02:00
MichaelEischer	58efe21eca	Merge pull request #3264 from amozoss/upstream-master ... Refactor backup progress	2021-09-19 14:54:42 +02:00
Michael Eischer	921e328b56	restore: Fix linting error	2021-09-19 14:41:07 +02:00
Michael Eischer	2cdc0719af	restorer: Sanitize verify errors	2021-09-19 14:01:26 +02:00
greatroar	bdcdfaf6b4	restore --verify: buffer reuse consistency and comment	2021-09-19 13:11:27 +02:00
greatroar	2b94742ca5	Replace no-op closures in restorer by nil check	2021-09-19 13:11:16 +02:00
greatroar	d357744104	Handle canceled context in restore --verify properly	2021-09-19 13:11:05 +02:00
greatroar	d4225ec803	Simplify buffer growing in Restorer.verifyFile ... Suggested-by: Igor Fedorenko <igor@ifedorenko.com>	2021-09-19 13:10:55 +02:00
greatroar	de8521ae56	Report number of successes from Restorer.VerifyFiles ... Previously, the number of attempts was reported.	2021-09-19 13:10:44 +02:00
greatroar	bb066cf7d3	Concurrent Restorer.VerifyFiles ... Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).	2021-09-19 13:10:41 +02:00
greatroar	556424d61b	Reuse buffer in Restorer.VerifyFiles ... Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).	2021-09-19 12:55:31 +02:00
greatroar	973fa921cb	Test and document Restorer.VerifyFiles	2021-09-19 12:52:11 +02:00
Michael Eischer	e0d615c264	archiver: Fix TestArchiverAbortEarlyOnError test ... This can be caused when the test has uploaded four blobs, then queues two blobs for upload which are delayed. Then a seventh file can be opened which lead to a test failure.	2021-09-12 22:17:17 +02:00
Michael Eischer	fbb0e6499a	ui: consolidate backup ui in ui/backup package	2021-09-12 16:20:15 +02:00
Michael Eischer	cccb0d4064	ui: assert that ProgressPrinter interface is implemented	2021-09-12 15:26:40 +02:00
Michael Eischer	d62bfed65d	ui: move SetDryRun to ProgressReporter	2021-09-12 15:25:58 +02:00
Michael Eischer	77b129ec74	ui: remove unused Summary field	2021-09-12 15:25:58 +02:00
Michael Eischer	3024239e40	ui/json: restore summary output	2021-09-12 15:25:58 +02:00
Michael Eischer	5ccf583b8a	ui: restore a few comments	2021-09-12 15:25:58 +02:00
Michael Eischer	80cbaf6d38	ui: Simplify ReportTotal methods	2021-09-12 15:25:58 +02:00
Dan Willoughby	448419990c	Refactor backup progress ... Move the shared logic into the progress Allows logic to be shared with forth coming restore status	2021-09-12 15:25:58 +02:00
MichaelEischer	fa5ca8af81	Merge pull request #2856 from aawsome/remove-readahead ... Simplify cache logic	2021-09-04 20:25:49 +02:00
MichaelEischer	b45d88e124	Merge pull request #3496 from juergenhoetzel/imporove-snapshot-filter-warning-message ... Improve snapshot filter warning message	2021-09-03 23:42:27 +02:00
MichaelEischer	a29777f467	Merge pull request #3501 from greatroar/printprogress ... Streamline progress printing in cmd/restic	2021-09-03 23:34:36 +02:00
Alexander Weiss	81876d5c1b	Simplify cache logic	2021-09-03 21:01:00 +02:00
greatroar	5aaa3e93c1	internal/ui/termstatus: Optimize and publish Truncate ... name old time/op new time/op delta TruncateASCII-8 347ns ± 1% 69ns ± 1% -80.02% (p=0.000 n=9+10) TruncateUnicode-8 447ns ± 3% 348ns ± 1% -22.04% (p=0.000 n=10+10)	2021-09-03 11:48:22 +02:00
MichaelEischer	ec2e3b260e	Merge pull request #3499 from greatroar/wrappedconn-pointer ... rclone: Return one fewer value from run	2021-08-31 21:48:36 +02:00
greatroar	950b818274	rclone: Return one fewer value from run	2021-08-26 18:12:08 +02:00
Juergen Hoetzel	defe19fdf6	Quote snapshot prefix in error string	2021-08-25 16:11:28 +02:00
Michael Eischer	68370feeee	backends: Remove TestSaveFilenames test ... Filenames are expected to match the sha256 sum of the file content. This rule is now enforced by the rest server thus making this test useless.	2021-08-15 18:24:16 +02:00
Michael Eischer	574c83e47f	rest: Fix test to use paths which are the sha256 sum of the data	2021-08-15 18:19:43 +02:00
Michael Eischer	e6a5801155	rest: Fix test backend url ... The rest config normally uses prepareURL to sanitize URLs and ensures that the URL ends with a slash. However, the test used an URL without a trailing slash, which after the rest server changes causes test failures.	2021-08-15 18:16:17 +02:00
Michael Eischer	f4c5dec05d	backend: test that a wrong hash fails an upload	2021-08-04 22:17:46 +02:00
Michael Eischer	7c1903e1ee	panic if hash returns an error ... Add a sanity check that the interface contract is honoured.	2021-08-04 22:17:46 +02:00
Michael Eischer	51b7e3119b	mem: calculate md5 content hash for uploads ... The mem backend is primarily used for testing. This ensures that the upload hash calculation gets appropriate test coverage.	2021-08-04 22:17:46 +02:00
Michael Eischer	a009b39e4c	gs/swift: calculate md5 content hash for upload	2021-08-04 22:17:46 +02:00
Michael Eischer	1d3e99f475	azure: check upload using md5 content hash ... For files below 256MB this uses the md5 hash calculated while assembling the pack file. For larger files the hash for each 100MB part is calculated on the fly. That hash is also reused as temporary filename. As restic only uploads encrypted data which includes among others a random initialization vector, the file hash shouldn't be susceptible to md5 collision attacks (even though the algorithm is broken).	2021-08-04 22:17:46 +02:00
Michael Eischer	9aa2eff384	Add plumbing to calculate backend specific file hash for upload ... This enables the backends to request the calculation of a backend-specific hash. For the currently supported backends this will always be MD5. The hash calculation happens as early as possible, for pack files this is during assembly of the pack file. That way the hash would even capture corruptions of the temporary pack file on disk.	2021-08-04 22:17:46 +02:00
Michael Eischer	ee2f14eaf0	s3: enable content hash calculation for uploads	2021-08-04 22:12:12 +02:00
greatroar	6586e90acf	Modernize internal/cache error handling	2021-08-04 22:02:42 +02:00
greatroar	ea04f40eb3	Save cached files to a temporary location first	2021-08-04 22:02:42 +02:00
greatroar	f9b6f8fd45	Replace duplicate type checking in cache with a function	2021-08-04 22:02:42 +02:00
greatroar	81e2499d19	Sync directory to get durable writes in local backend	2021-08-04 21:51:53 +02:00
greatroar	195a5cf996	Save files under temporary name in local backend ... Fixes #3435.	2021-08-04 21:51:53 +02:00
Alexander Weiss	38a8a48a25	Simplify dry run backend	2021-08-04 21:19:29 +02:00
Ryan Hitchman	77bf148460	backup: add --dry-run/-n flag to show what would happen. ... This can be used to check how large a backup is or validate exclusions. It does not actually write any data to the underlying backend. This is implemented as a simple overlay backend that accepts writes without forwarding them, passes through reads, and generally does the minimal necessary to pretend that progress is actually happening. Fixes #1542 Example usage: $ restic -vv --dry-run . | grep add new /changelog/unreleased/issue-1542, saved in 0.000s (350 B added) modified /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added) modified /cmd/restic/global.go, saved in 0.000s (0 B added) new /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added) new /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added) modified /internal/backend/test/tests.go, saved in 0.000s (0 B added) modified /internal/repository/repository.go, saved in 0.000s (20.707 KiB added) modified /internal/ui/backup.go, saved in 0.000s (9.110 KiB added) modified /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added) modified /restic, saved in 0.131s (25.542 MiB added) Would add to the repo: 25.892 MiB	2021-08-04 21:19:29 +02:00
MichaelEischer	533ac4fd95	Merge pull request #3467 from greatroar/wrappedconn-pointer ... Use rclone.wrappedConn by pointer	2021-08-04 21:18:53 +02:00
David le Blanc	326fefcd80	Allow --tag and --keep-tag to match untagged snapshots	2021-08-02 23:06:20 +02:00
greatroar	fa3eed1998	Use rclone.wrappedConn by pointer ... This shaves a kilobyte off the Linux binary by not generating a non-pointer interface implementation.	2021-08-01 09:11:50 +02:00
MichaelEischer	5571c3f7fd	Merge pull request #3453 from MichaelEischer/http2-zero-length-workaround ... rest: Workaround Http2 zero-length reply bug	2021-07-31 20:30:06 +02:00
Alexander Neumann	d8ea10db8c	rest: Rework handling HTTP2 zero-length replies bug ... Add comment that the check is based on the stdlib HTTP2 client. Refactor the checks into a function. Return an error if the value in the Content-Length header cannot be parsed.	2021-07-31 17:12:24 +02:00
Magnus Thor Torfason	2081bd12fb	forget: Ensure future snapshots do not affect --keep-within-* ... Ensure that only snapshots made in the past are taken into account when running restic forget with the within switches (--keep-within, --keep-within- hourly, and friends)	2021-07-24 16:14:43 +00:00
Magnus Thor Torfason	74ebc650ab	forget: Add --keep-within-hourly (and friends) ... Allow keeping hourly/daily/weekly/monthly/yearly snapshots for a given time period. This adds the following flags/parameters to restic forget: --keep-within-hourly duration --keep-within-daily duration --keep-within-weekly duration --keep-within-monthly duration --keep-within-yearly duration Includes following changes: - Add tests for --keep-within-hourly (and friends) - Add documentation for --keep-within-hourly (and friends) - Add changelog for --keep-within-hourly (and friends)	2021-07-24 16:14:43 +00:00
Michael Eischer	097ed659b2	rest: test that zero-length replies over HTTP2 work correctly ... The first test function ensures that the workaround works as expected. And the second test function is intended to fail as soon as the issue has been fixed in golang to allow us to eventually remove the workaround.	2021-07-10 17:22:42 +02:00
Michael Eischer	185a55026b	rest: workaround for HTTP2 zero-length replies bug ... The golang http client does not return an error when a HTTP2 reply includes a non-zero content length but does not return any data at all. This scenario can occur e.g. when using rclone when a file stored in a backend seems to be accessible but then fails to download.	2021-07-10 16:59:01 +02:00
Michael Eischer	bd316d3893	restore: Test partial pack downloads in filerestorer	2021-06-29 21:11:30 +02:00
Michael Eischer	e8bbb05328	restore: Correctly handle partial pack download errors ... Failed pack/blob downloads should be retried. For blobs that fail decryption assume that the pack file is really damaged and try to restore the remaining blobs.	2021-06-29 20:54:16 +02:00
MichaelEischer	c1eb7ac1a1	Merge pull request #3420 from greatroar/local-errs ... Modernize error handling in local backend	2021-06-20 14:20:40 +02:00
greatroar	e5f0f67ba0	Modernize error handling in local backend ... * Stop prepending the operation name: it's already part of os.PathError, leading to repetitive errors like "Chmod: chmod /foo/bar: operation not permitted". * Use errors.Is to check for specific errors.	2021-06-18 11:13:27 +02:00
MichaelEischer	a476752962	Merge pull request #3421 from greatroar/s3-fileinfo ... Return s3.fileInfos by pointer	2021-06-12 18:55:18 +02:00
MichaelEischer	e8d20ea32c	Merge pull request #3409 from greatroar/lchown-mknod ... Make restic.{lchown,mknod} regular functions	2021-06-12 18:22:38 +02:00
greatroar	0d4f16b6ba	Return s3.fileInfos by pointer ... Since the fileInfos are returned in a []interface, they're already allocated on the heap. Making them pointers explicitly means the compiler doesn't need to generate fileInfo and *fileInfo versions of the methods on this type. The binary becomes about 7KiB smaller on Linux/amd64.	2021-06-07 19:48:43 +02:00
greatroar	0666c4d244	Make restic.{lchown,mknod} regular functions ... This way, they can be inlined and dead code can be removed on Windows. Also fixed some comments.	2021-05-27 22:51:40 +02:00
rawtaz	fdbd65485e	Merge pull request #3402 from MichaelEischer/misc-fixes ... Various small code cleanups	2021-05-24 11:30:31 +02:00
Alexander Neumann	4cabad8c34	Merge pull request #3325 from MichaelEischer/fix-mintty-output ... Fix windows terminal output for mintty	2021-05-18 09:29:24 +02:00
Michael Eischer	cf92c58460	Properly wrap errors in readerat helper	2021-05-17 21:08:23 +02:00
Michael Eischer	75c990504d	azure/gs: Fix default value in connections help text	2021-05-17 20:56:51 +02:00
Michael Eischer	55bea6e7a6	filter: Fix crash for '**' pattern	2021-05-14 23:50:31 +02:00
MichaelEischer	6d8ceefd67	Merge pull request #3373 from greatroar/simplify-limiter ... Simplify internal/limiter	2021-05-14 20:27:31 +02:00
MichaelEischer	64b00d28b1	Merge pull request #3345 from greatroar/sftp-enospc ... Check for ENOSPC and remove broken files in SFTP	2021-05-13 20:09:38 +02:00
greatroar	ae170e2b38	Simplify internal/limiter	2021-04-24 11:54:43 +02:00
Michael Eischer	7cb8ea69ba	Add test to mintty pipe detection	2021-04-11 20:02:09 +02:00
Michael Eischer	80564a9bc9	Properly detect mintty output redirection ... mintty on windows always uses pipes to connect stdout between processes and for the terminal output. The previous implementation always assumed that stdout connected to a pipe means that stdout is displayed on a mintty terminal. However, this detection breaks when using pipes to connect processes and for powershell which uses pipes when redirecting to a file. Now the pipe filename is queried and matched against the pattern used by msys / cygwin when connected to the terminal. In all other cases assume that a pipe is just a regular pipe.	2021-04-11 20:02:09 +02:00
Michael Eischer	5e6af77b7a	Unify interactive terminal detection code ... Previously the progress bar / status update interval used stdoutIsTerminal to determine whether it is possible to update the progress bar or not. However, its implementation differed from the detection within the backup command which included additional checks to detect the presence of mintty on Windows. mintty behaves like a terminal but uses pipes for communication. This adds stdoutCanUpdateStatus() which calls the same terminal detection code used by backup. This ensures that all commands consistently switch between interactive and non-interactive terminal mode. stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus() does so. This is required to properly handle the special case of mintty.	2021-04-11 20:02:09 +02:00
MichaelEischer	cc254dfefe	Merge pull request #3362 from greatroar/darwin-preallocate ... Use FcntlFstore to preallocate on Mac	2021-04-10 22:47:56 +02:00
greatroar	23531be272	Use FcntlFstore to preallocate on Mac	2021-04-10 16:54:07 +02:00
rawtaz	74c0607c92	Merge pull request #3319 from MichaelEischer/skip-prealloc-test ... restorer: Skip preallocate test if not supported by the filesystem	2021-04-07 18:59:06 +02:00
greatroar	dc88ca79b6	Handle lack of space and remove broken files in SFTP backend	2021-03-27 15:02:14 +01:00
Alexander Neumann	88a23521dd	Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check ... s3: Fix sanity check	2021-03-11 13:13:46 +01:00
greatroar	187a77fb27	Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist	2021-03-10 21:05:24 +01:00
Michael Eischer	2a9f0f19b6	s3: Fix sanity check ... The sanity check shouldn't replace the error message if there is already one.	2021-03-08 20:23:57 +01:00
Michael Eischer	f70aca6f6f	restorer: Skip preallocate test if not supported by the filesystem	2021-03-04 20:33:46 +01:00
Michael Eischer	a293fd9aef	local: Ignore files in intermediate folders ... For example the data folder of a repository might contain hidden files which caused list operations to fail.	2021-02-27 13:47:55 +01:00
Alexander Neumann	9985368d46	Merge pull request #3255 from MichaelEischer/restorer-check-error ... restorer: Check dropped error	2021-02-03 16:10:46 +01:00
Alexander Neumann	2dd592a06c	Merge pull request #3254 from lorenz/allow-http2 ... Allow HTTP/2	2021-02-02 21:08:20 +01:00
Michael Eischer	47faf69230	restorer: Check dropped error	2021-01-31 18:06:28 +01:00
Alexander Weiss	5e22ae10f1	Add error handling for fileRestorer	2021-01-31 14:22:57 +01:00
Alexander Weiss	573221aa40	Add failing test for fileRestorer	2021-01-31 13:40:42 +01:00
Lorenz Brun	5427119205	Allow HTTP/2	2021-01-31 02:44:30 +01:00
Michael Eischer	e0867c9682	backend: try to cleanup test leftovers	2021-01-30 21:23:20 +01:00
Michael Eischer	f740b2fb23	mem: check upload length before storing upload	2021-01-30 21:23:20 +01:00
Alexander Neumann	0e5f2fff71	Merge pull request #3243 from restic/fix-scanner-overlap ... backup: Fix total size for overlapping targets	2021-01-30 21:17:21 +01:00
Alexander Neumann	04ca69cc78	Address issues reported by golint	2021-01-30 20:45:57 +01:00
Alexander Neumann	f867e65bcd	Fix issues reported by staticcheck	2021-01-30 20:43:53 +01:00
Alexander Neumann	0858fbf6aa	Add more error handling	2021-01-30 20:19:47 +01:00
Alexander Neumann	aef3658a5f	Address review comments	2021-01-30 20:02:37 +01:00
Alexander Neumann	200f09522d	Add more error checks	2021-01-30 20:02:37 +01:00
Alexander Neumann	3c753c071c	errcheck: More error handling	2021-01-30 20:02:37 +01:00
Alexander Neumann	16313bfcc9	errcheck: Add error check for MergeFinalIndexes()	2021-01-30 20:02:37 +01:00
Alexander Neumann	75f53955ee	errcheck: Add error checks ... Most added checks are straight forward.	2021-01-30 20:02:37 +01:00
Michael Eischer	a53778cd83	rest: handle dropped error in save operation	2021-01-30 19:25:04 +01:00
Michael Eischer	8a486eafed	gs: Don't drop error when finishing upload ... The error returned when finishing the upload of an object was dropped. This could cause silent upload failures and thus data loss in certain cases. When a MD5 hash for the uploaded blob is specified, a wrong hash/damaged upload would return its error via the Close() whose error was dropped.	2021-01-30 13:31:32 +01:00
Alexander Neumann	cdd704920d	azure: Pass data length to Azure libray ... The azureAdapter was used directly without a pointer, but the Len() method was only defined with a pointer receiver (which means Len() is not present on a azureAdapter{}, only on a pointer to it).	2021-01-29 21:08:41 +01:00
Alexander Neumann	bbdf18c4a2	Merge pull request #3176 from MichaelEischer/backend-content-length ... Pass upload size to backends and sanity check it	2021-01-29 20:33:44 +01:00
Michael Eischer	1f583b3d8e	backend: test that incomplete uploads fail	2021-01-29 13:51:53 +01:00
Michael Eischer	c73316a111	backends: add sanity check for the uploaded file size ... Bugs in the error handling while uploading a file to the backend could cause incomplete files, e.g. https://github.com/golang/go/issues/42400 which could affect the local backend. Proactively add sanity checks which will treat an upload as failed if the reported upload size does not match the actual file size.	2021-01-29 13:51:51 +01:00
Michael Eischer	4526d5d197	swift: explicitly pass upload size to library ... This allows properly setting the content-length which could help the server-side to detect incomplete uploads.	2021-01-29 13:50:46 +01:00
Michael Eischer	dca9b6f5db	azure: explicitly pass upload size ... Previously the fallback from the azure library was to read the whole blob into memory and use that to determine the upload size.	2021-01-29 13:50:46 +01:00
Alexander Neumann	5c617859ab	backup/scanner: Fix total size for overlapping targets ... Before, the scanner would could files twice if they were included in the list of backup targets twice, e.g. `restic backup foo foo/bar` would could the file `foo/bar` twice. This commit uses the tree structure from the archiver to run the scanner, so both parts see the same files.	2021-01-29 11:31:36 +01:00
Alexander Neumann	81211750ba	archiver/tree: Introduce functions Leaf() and NodeNames()	2021-01-29 11:11:28 +01:00
greatroar	6bd8a2faaa	backup: Add --ignore-ctime option and document change detection	2021-01-28 23:42:10 +01:00
MichaelEischer	43cb26010a	Merge pull request #3242 from greatroar/fprintln ... internal/ui/termstatus: Use Fprintln to get a newline	2021-01-28 20:34:06 +01:00
Alexander Neumann	4d40c70214	Merge pull request #3211 from MichaelEischer/sftp-speedup ... Speed-up caching/pack download via SFTP	2021-01-28 14:16:55 +01:00
Alexander Neumann	6aa7e9f9c6	Merge pull request #3174 from MichaelEischer/parallelize-lock-loading ... Parallelize lock file loading	2021-01-28 13:52:12 +01:00
Alexander Neumann	bdfedf1f5b	Merge pull request #3173 from MichaelEischer/unify-index-loading ... Unify index loading	2021-01-28 13:50:42 +01:00
greatroar	b9cfe6f68a	internal/ui/termstatus: Use Fprintln to get a newline	2021-01-28 13:30:10 +01:00
Michael Eischer	68608a89ad	restic: add comment about StreamTrees shutdown	2021-01-28 11:10:50 +01:00
Michael Eischer	ddb7697d29	restic: Test progress reporting of StreamTrees	2021-01-28 11:10:50 +01:00
Michael Eischer	313ad0e32f	progress/counter: Fix test for final report call	2021-01-28 11:10:50 +01:00
Michael Eischer	e2b0072441	check: add progress bar to the tree structure check	2021-01-28 11:10:50 +01:00
Michael Eischer	505f8a2229	progress/counter: Support updating the progress bar maximum	2021-01-28 11:10:47 +01:00
Michael Eischer	eda8c67616	restic: let FindUsedBlobs handle multiple snapshots at once	2021-01-28 11:08:43 +01:00
Michael Eischer	258ce0c1e5	parallel: report progress for StreamTrees ... This assigns an id to each tree root and then keeps track of how many tree loads (i.e. trees referenced for the first time) are pending per tree root. Once a tree root and its subtrees were fully processed there are no more pending tree loads and the tree root is reported as processed.	2021-01-28 11:08:43 +01:00
Michael Eischer	f2a1b125cb	restic: Actually parallelize FindUsedBlobs	2021-01-28 11:08:43 +01:00
Michael Eischer	6e03f80ca2	check: Split the parallelized tree loader into a reusable component ... The actual code change is minimal	2021-01-28 11:08:43 +01:00
Michael Eischer	1d7bb01a6b	check: Cleanup tree loading and switch to use errgroup ... The helper methods are now wired up in the Structure method.	2021-01-28 11:08:43 +01:00
Alexander Neumann	a4689eb3b9	Merge pull request #3199 from MichaelEischer/non-interactive-counter ... Don't print progress on non-interactive terminals	2021-01-28 10:53:38 +01:00
Alexander Neumann	c5a66e9181	ui: Simlify channel receive	2021-01-28 10:42:02 +01:00
Michael Eischer	678e75e1c2	sftp: enforce use of optimized upload method ... ReadFrom was already used by Save before, this just ensures that this won't accidentally change in the future.	2021-01-03 22:23:53 +01:00
Michael Eischer	6b5b29dbee	limiter: add unit tests	2021-01-03 22:23:53 +01:00
Michael Eischer	f35f2c48cd	limiter: support WriteTo in LimitBackend for read rate limiting	2021-01-03 22:23:53 +01:00
Michael Eischer	bcb852a8d0	hashing: support WriteTo in the reader	2021-01-03 22:23:53 +01:00
Michael Eischer	69d5b4c36b	restorer: lower-case variable name	2021-01-03 13:55:59 +01:00
Alexander Weiss	34a33565c8	Fix loadBlob in filerestorer	2021-01-01 08:06:04 +01:00
Alexander Weiss	7409225fa8	Add filerestorer test where only parts of pack are used	2021-01-01 07:24:46 +01:00
Alexander Weiss	07b3f65a6f	filesrestorer: Re-use buffer	2021-01-01 07:24:46 +01:00
Alexander Weiss	3e0acf1395	restore: Don't save (part of) pack in memory	2021-01-01 07:24:46 +01:00
Alexander Neumann	8b84c96d9d	Merge pull request #3204 from MichaelEischer/archiver-tomb-race ... archiver: fix race condition during worker startup	2020-12-30 20:04:20 +01:00
Michael Eischer	debc4a3a99	archiver: fix race condition during worker startup ... When the tomb is created with a canceled context, then the workers started via `t.Go` exist nearly immediately. Once for the first time all started goroutines have been stopped, it is not allowed to issue further calls to `t.Go`. This is a problem when the started goroutines exit immediately, as for example the first goroutine might already have stopped before starting the second one, which is not allowed as once the first goroutines has stopped no goroutines were running. To fix this race condition the startup and main task of the archiver now also run within a `t.Go` function. This also allows unifying the error handling as it is no longer necessary to distinguish between errors returned by the workers or the saveTree processing. The tomb now just returns the first error encountered, which should also be the most descriptive one.	2020-12-30 17:31:22 +01:00
MichaelEischer	31e56f1ad5	Merge pull request #3197 from SkYNewZ/fix/3183 ... Fix tag handling for multiple tag lists	2020-12-29 18:44:38 +01:00
Alexander Neumann	b2efa0af39	Merge pull request #3164 from MichaelEischer/improve-context-cancel ... Improve context cancel handling in archiver and backends	2020-12-29 17:03:42 +01:00
Michael Eischer	cff4955a48	ui: remove dead struct member	2020-12-29 16:32:18 +01:00
Michael Eischer	92da5168e1	ui: force backup progress update on signal	2020-12-29 16:32:18 +01:00
Michael Eischer	34afc93ddc	ui/progress: extract signal handling into own package	2020-12-29 16:32:18 +01:00
Michael Eischer	023eea6463	ui: don't shorten non-interactive progress output	2020-12-29 16:32:18 +01:00
Michael Eischer	684600cf42	ui: update status for the backup command on non-interactive terminals ... Allow the backup command to print status on non-interactive terminals. The output is disabled by setting a MinUpdatePause == 0.	2020-12-29 16:03:43 +01:00
Michael Eischer	13ce981794	ui: cleanup backup status shutdown	2020-12-29 16:03:43 +01:00
Michael Eischer	c2ef049f1b	ui/progress: don't print progress on non-interactive terminals ... This reverts to the old behavior of not printing progress updates on non-interactive terminals. It was accidentally changed in #3058.	2020-12-29 16:03:43 +01:00
Quentin Lemaire	334d8ce724	feat(tags): Create Flatten() method	2020-12-29 10:59:46 +01:00
greatroar	3b09ae9074	AIX port	2020-12-29 01:35:01 +01:00
greatroar	18531e3d6f	Portability fixes to internal/restic ... syscall.Mknod is not available on AIX.	2020-12-29 01:35:01 +01:00
Michael Eischer	d0ca8fb0b8	backend: test that a canceled context prevents RetryBackend operations	2020-12-28 21:06:47 +01:00
Michael Eischer	08b7f2b58d	archiver: test that context canceled error is not dropped	2020-12-28 21:06:47 +01:00
Michael Eischer	e483b63c40	retrybackend: Fail operations when context is already canceled ... Depending on the used backend, operations started with a canceled context may fail or not. For example the local backend still works in large parts when called with a canceled context. Backends transfering data via http don't work. It is also not possible to retry failed operations in that state as the RetryBackend will abort with a 'context canceled' error. Ensure uniform behavior of all backends by checking for a canceled context by checking for a canceled context as a first step in the RetryBackend. This ensures uniform behavior across all backends, as backends are always wrapped in a RetryBackend.	2020-12-28 21:06:47 +01:00
Michael Eischer	fc60b560ba	archiver: Let saveTree report a canceled context as an error ... If the context was canceled then saveTree might receive a treeID or not depending on the timing. This could cause saveTree to incorrectly return a nil treeID as valid. Fix this always returning an error when the context was canceled in the meantime.	2020-12-28 21:06:47 +01:00
Michael Eischer	736e964317	archiver: Don't loose error if background context is canceled ... A canceled background context lets the blob/tree/fileSavers exit without reporting an error. The error handling previously replaced a 'context canceled' error received by the main backup method with the error reported by the savers. However, in case of a canceled background context that error is nil, causing restic to loose the error and save a snapshot with a nil tree.	2020-12-28 21:06:47 +01:00
DRON-666	332b1896d1	Some options fixes ... Add tests for bool type. Fix subtle bug in TestOptionsApplyInvalid. Fix options list formatting.	2020-12-23 23:26:04 +03:00
MichaelEischer	cb6b0f6255	Merge pull request #3181 from tWido/feature-no-retry-permission ... Don't retry when "Permission denied" occurs in local backend	2020-12-23 20:24:29 +01:00
MichaelEischer	1e73aac610	Merge pull request #3179 from aawsome/check-filesize ... check: Remove filesize counter	2020-12-23 20:22:02 +01:00
Alexander Weiss	2a1add7538	check: remove file size counter	2020-12-23 02:34:31 +01:00
tWido	7dab113035	Don't retry when "Permission denied" occurs in local backend	2020-12-22 23:41:12 +01:00
Michael Eischer	de99207046	repository: tweak comment for packs method	2020-12-22 23:01:58 +01:00
Alexander Weiss	68b74e359e	Count packs directly in RebuildIndexFiles	2020-12-22 23:01:58 +01:00
Michael Eischer	b9f5d3fe13	repository: Add test for ForAllIndexes	2020-12-22 22:36:18 +01:00
Michael Eischer	a12c5f1d37	repository: move otherwise unused LoadIndex to tests	2020-12-22 22:36:18 +01:00
Michael Eischer	24474a36f4	repository: deduplicate index loading implementation	2020-12-22 22:36:18 +01:00
Michael Eischer	96904f8972	check: extract parallel index loading	2020-12-22 22:36:18 +01:00
Alexander Neumann	6c514adb8a	ui/progress: Use mutex instead of atomic ... The counter value needs to be aligned to 64 bit in memory for the atomic functions to work on some platform (such as 32 bit ARM). The atomic package says in its documentation: > These functions require great care to be used correctly. Except for > special, low-level applications, synchronization is better done with > channels or the facilities of the sync package. This commit replaces the atomic functions with a simple sync.Mutex, so we don't have to care about alignment.	2020-12-22 21:03:27 +01:00
DRON-666	83b10dbb12	Deduplicate dumper closing logic	2020-12-19 02:42:46 +03:00
DRON-666	e136dd8696	Deduplicate test code	2020-12-19 02:06:54 +03:00
DRON-666	33adb58817	Minor fixes and linter suggestions	2020-12-19 02:04:17 +03:00
DRON-666	89ab6d557e	dump: Add zip dumper	2020-12-19 01:09:47 +03:00
DRON-666	0256f95994	dump: Split tar and walk logic	2020-12-19 01:09:47 +03:00
Michael Eischer	34b6130a0e	restic: parallelize lock file loading	2020-12-18 20:46:16 +01:00
greatroar	66d904c905	Make invalid handles permanent errors	2020-12-17 12:47:53 +01:00
greatroar	746dbda413	Mark "ssh exited" errors in SFTP as permanent	2020-12-17 12:43:09 +01:00
greatroar	f7784bddb3	Don't retry when "no space left on device" in local backend ... Also adds relevant documentation to the restic.Backend interface.	2020-12-17 12:43:09 +01:00
Alexander Neumann	e96677cafb	Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables ... swift: Add support for id based keystone v3 auth parameters	2020-12-12 16:27:38 +01:00
Michael Eischer	1d69341e88	swift: Add support for id based keystone v3 auth parameters ... This adds support for the following environment variables, which were previously missing: OS_USER_ID User ID for keystone v3 authentication OS_USER_DOMAIN_ID User domain ID for keystone v3 authentication OS_PROJECT_DOMAIN_ID Project domain ID for keystone v3 authentication OS_TRUST_ID Trust ID for keystone v3 authentication	2020-12-11 19:22:34 +01:00
Alexander Neumann	36c5d39c2c	Fix issues reported by semgrep	2020-12-11 09:41:59 +01:00
Alexander Neumann	7facc8ccc1	Merge pull request #2505 from aawsome/fix-repo-configfile ... Fix repo configfile	2020-12-07 07:52:37 +01:00
Alexander Neumann	ba31c6fdaa	Merge pull request #3150 from MichaelEischer/fix-windows-redir-output ... termstatus: Fix canUpdateStatus detection for redirected output on windows	2020-12-06 21:17:27 +01:00
Alexander Neumann	b58799d83a	Merge pull request #3152 from MichaelEischer/fix-backup-background-hang ... backup: Fix shutdown hang when running in the background on linux	2020-12-06 21:16:00 +01:00
Alexander Neumann	0d5b764f90	Merge pull request #3130 from aawsome/snapshots-parallel ... Make loading snapshots parallel	2020-12-06 21:08:18 +01:00
Michael Eischer	b48f579530	termstatus: Fix canUpdateStatus detection for redirected output ... The canUpdateStatus check was simplified in #2608, but it accidentally flipped the condition. The correct check is as follows: If the output is a pipe then restic probably runs in mintty/cygwin. In that case it's possible to update the output status. In all other cases it isn't. This commit inverts to condition again to offer the previous and correct behavior.	2020-12-06 19:02:42 +01:00
Michael Eischer	401ef92c5f	backup: Fix shutdown hang when running in the background ... On shutdown the backup commands waits for the terminal output goroutine to stop. However while running in the background the goroutine ignored the canceled context.	2020-12-06 18:53:41 +01:00
Alexander Weiss	e329623771	Remove LoadAllSnapshots	2020-12-06 05:22:27 +01:00
Alexander Weiss	26f85779be	Parallelize ForAllSnapshots	2020-12-06 05:09:58 +01:00
Alexander Weiss	5b9ee56335	Add ForAllSnapshots	2020-12-06 05:04:21 +01:00
Michael Eischer	d05c88a5d6	fuse: Properly set uid/gid for directories ... In #2584 this was changed to use the uid/gid of the root node. This would be okay for the top-level directory of a snapshot, however, this change also applied to normal directories within a snapshot. This change reverts the problematic part and adds a test that directory attributes are represented correctly.	2020-11-30 23:41:49 +01:00
MichaelEischer	f2959127b6	Merge pull request #3065 from greatroar/local-subdirs ... Don't recurse in local backend's List if not required	2020-11-29 19:03:59 +01:00
Alexander Weiss	aa7a5f19c2	Use BlobHandle in index methods	2020-11-22 20:41:12 +01:00
Alexander Weiss	e3013271a6	Harmonize naming	2020-11-22 20:41:12 +01:00
Alexander Weiss	92bd448691	Make BlobHandle substruct of Blob	2020-11-22 20:41:10 +01:00
Alexander Weiss	a851c53cbe	Use PackSize in checker	2020-11-21 22:13:54 +01:00
Alexander Weiss	ce5d630681	Add MasterIndex.PackSize()	2020-11-21 22:13:54 +01:00
Alexander Weiss	c3ddde9e7d	Return hdrSize in ListPack	2020-11-21 22:13:54 +01:00
greatroar	8e213e82fc	backend/local: replace fs.Walk with custom walker ... This code is more strict in what it expects to find in the backend: depending on the layout, either a directory full of files or a directory full of such directories.	2020-11-19 16:46:42 +01:00
Alexander Neumann	75eff92b56	Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init ... do not require gs bucket permissions to init repository	2020-11-18 16:53:45 +01:00
eleith	a24e986b2b	do not require gs bucket permissions to init repository ... a gs service account may only have object permissions on an existing bucket but no bucket create/get permissions. these service accounts currently are blocked from initialization a restic repository because restic can not determine if the bucket exists. this PR updates the logic to assume the bucket exists when the bucket attribute request results in a permissions denied error. this way, restic can still initialize a repository if the service account does have object permissions fixes: https://github.com/restic/restic/issues/3100	2020-11-18 06:14:11 -08:00
rawtaz	1b6e8c888f	Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param ... Remove `--drive-use-trash=false` from rclone param	2020-11-17 11:21:20 +01:00
Michael Eischer	1f43cac12d	check: Only track data blobs when unused blobs should be reported ... This improves the memory usage of check a lot as it now only has to track tree blobs when run using the default parameters.	2020-11-15 18:43:07 +01:00
Michael Eischer	6da66c15d8	check: Simplify referenced blob tracking ... The result is identical as long as the context in not canceled. However, in that case the result is incomplete anyways.	2020-11-15 18:42:55 +01:00
Michael Eischer	3500f9490c	check: Simplify blob status tracking ... UnusedBlobs now directly reads the list of existing blobs from the repository index. This removes the need for the blobStatusExists flag, which in turn allows converting the blobRefs map into a BlobSet.	2020-11-15 18:42:42 +01:00
Michael Eischer	b8c7543a55	check: Merge 'size could not be found' and 'not found in index' errors ... By construction these two errors always show up in pairs: 'size could not be found' is printed when the blob is not found in the repository index. That blob is also part of the `blobs` array. Later on, check iterates over that array and checks whether the blob is marked as existing. Which cannot be the case as that mark is generated by iterating over the repository index. The merged warning no longer reports the blob index within a file. That information could also be derived by printing the affected tree using `cat` and searching for the blob.	2020-11-15 18:41:50 +01:00
Alexander Neumann	3c0c0c132b	Merge pull request #3006 from aawsome/new-rebuild-index ... Reimplement rebuild-index and remove /internal/index	2020-11-15 17:48:43 +01:00
Alexander Neumann	9968220652	Merge pull request #2850 from greatroar/packer-malloc ... Decrease allocation rate in internal/pack	2020-11-15 17:19:49 +01:00
greatroar	ab2b7d7f9a	Decrease allocation rate in internal/pack ... internal/repository benchmark results: name old time/op new time/op delta PackerManager-8 179ms ± 1% 181ms ± 1% +0.78% (p=0.009 n=10+10) name old speed new speed delta PackerManager-8 294MB/s ± 1% 292MB/s ± 1% -0.77% (p=0.009 n=10+10) name old alloc/op new alloc/op delta PackerManager-8 91.3kB ± 0% 72.2kB ± 0% -20.92% (p=0.000 n=9+7) name old allocs/op new allocs/op delta PackerManager-8 1.38k ± 0% 0.76k ± 0% -45.20% (p=0.000 n=10+7)	2020-11-15 16:51:47 +01:00
greatroar	9a8a2cae4c	Move pack testing logic to test file	2020-11-15 16:45:49 +01:00
Alexander Weiss	9607cad267	Remove internal/index	2020-11-15 07:05:09 +01:00
Alexander Weiss	30b6a0878a	Reimplement rebuild-index	2020-11-15 07:05:09 +01:00
Alexander Weiss	187c8fb259	Parallelize MasterIndex.Save()	2020-11-15 07:05:09 +01:00
Alexander Weiss	1ec628ddf5	Add extraObsolete to MasterIndex.Save	2020-11-15 07:05:09 +01:00
Alexander Weiss	5898cb341f	Use CreateIndexFromPacks() in test	2020-11-15 07:05:05 +01:00
Alexander Weiss	43732bb885	Add CreateIndexFromPacks()	2020-11-15 07:04:51 +01:00
greatroar	c45f8ee075	Replace restic.statT interface by concrete types ... name old time/op new time/op delta NodeFillUser-8 1.81µs ± 9% 1.50µs ± 5% -17.07% (p=0.000 n=19+20) NodeFromFileInfo-8 1.76µs ± 4% 1.49µs ± 6% -15.63% (p=0.000 n=20+19) name old alloc/op new alloc/op delta NodeFillUser-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20) NodeFromFileInfo-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20) name old allocs/op new allocs/op delta NodeFillUser-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20) NodeFromFileInfo-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)	2020-11-14 23:23:26 +01:00
MichaelEischer	3601a9b6cd	Merge pull request #2690 from SkYNewZ/master ... Fix #2688: Handle comma-separated list tags when using backup command	2020-11-14 23:19:42 +01:00
MichaelEischer	333c5a19d4	Merge pull request #3082 from aawsome/check-sizes ... Check: check sizes of packs from index and packheader	2020-11-14 22:37:42 +01:00
Quentin Lemaire	ae441d3134	fix(backup): Switch tags cobra type to handle comma-separated list	2020-11-14 15:48:56 +00:00
Alexander Weiss	17bb77b1f9	check: Also check blob length and offset	2020-11-14 00:42:49 +01:00
Alexander Weiss	80dcfca191	check: Check sizes computed from index and pack header	2020-11-14 00:42:49 +01:00
tofran	94a154c7ca	Remove --drive-use-trash=false from rclone param ... Google drive trash retention policy changed making this no longer a good default a go Issue #3095	2020-11-13 22:58:48 +00:00
Alexander Neumann	04d1983800	Merge pull request #3090 from fgma/vss-fix-386 ... vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386	2020-11-13 21:30:34 +01:00
Alexander Neumann	8ad9f88993	helpers: Improve error message	2020-11-12 20:38:31 +01:00
fgma	f9223cd827	vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386	2020-11-12 19:31:00 +01:00
rawtaz	dfb9326b1b	Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag ... Extended option to select V1 API for ListObjects on S3 backend	2020-11-11 20:52:44 +01:00
Alexander Neumann	4a0b7328ec	s3: Remove dots for config description	2020-11-11 20:20:35 +01:00
greatroar	63e32c44c0	Improve error reporting from restic diff ... Instead of a stacktrace, restic diff 111 222 now reports: Fatal: no matching ID found for prefix "111"	2020-11-11 16:40:40 +01:00
Nick Douma	829959390a	Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1	2020-11-11 11:54:38 +01:00
Nick Douma	ccd55d529d	Add s3.list-objects-v1 extended option and default to false	2020-11-11 11:54:36 +01:00
Nick Douma	4ddcc17135	Add support for boolean extended options	2020-11-11 11:54:27 +01:00
MichaelEischer	407843c5f9	Merge pull request #3034 from hoyho/gh_master ... bugfix: omit ENOTDATA for extended attributes	2020-11-09 22:38:49 +01:00
MichaelEischer	46d31ab86d	Merge pull request #3058 from greatroar/counter ... Replace restic.Progress with new progress.Counter (fixes two race conditions)	2020-11-09 22:19:09 +01:00
Alexander Neumann	c986823d3f	Merge pull request #3048 from aawsome/check-pack-index ... check: check index for packs that are read	2020-11-09 20:12:32 +01:00
Alexander Weiss	239931578c	check: check index for packs that are read	2020-11-09 17:28:14 +01:00
hoyho	9df52327cc	bugfix: omit ENOTDATA for extended attributes ... Signed-off-by: hoyho <luohaihao@gmail.com>	2020-11-10 00:20:34 +08:00
greatroar	21b787a4d1	Stop Counters where they're constructed and started	2020-11-09 13:03:31 +01:00
greatroar	ddca699cd2	Replace restic.Progress with new progress.Counter ... This fixes two race conditions while cleaning up the code.	2020-11-09 12:12:35 +01:00
Alexander Neumann	3bf43d7951	Merge pull request #2475 from restic/use-github-actions ... Use GitHub Actions for CI	2020-11-08 17:30:12 +01:00
Alexander Neumann	5cf42884c8	Run tests on GitHub Actions	2020-11-08 16:04:57 +01:00
Alexander Weiss	826cfa0533	fix context in archiver tests	2020-11-08 08:24:24 +01:00
Alexander Weiss	fef408a8bd	Return context error in mem backend	2020-11-08 00:05:53 +01:00
greatroar	a2d4209322	Don't recurse in local backend's List if not required ... Due to the return if !isFile, the IsDir branch in List was never taken and subdirectories were traversed recursively. Also replaced isFile by an IsRegular check, which has been equivalent since Go 1.12 (golang/go@a2a3dd00c9).	2020-11-07 08:54:13 +01:00
Alexander Weiss	fd33030556	Use in-memory index to rebuild index in prune	2020-11-06 20:23:30 +01:00
Alexander Weiss	38cc4393f6	Add Masterindex.Save(); Add Index.Packs()	2020-11-06 20:23:30 +01:00
Alexander Weiss	aaf1c44362	Fix #3062	2020-11-05 17:05:42 +01:00
Ivan Andreev	ab2790d9de	Fix http2 stream reset between restic and rest backends #3014	2020-11-05 15:57:40 +03:00
Nick Craig-Wood	86b5d8ffaa	s3: add bucket-lookup parameter to select path or dns style bucket lookup ... This is to enable restic working with Alibaba cloud Fixes #2528	2020-11-05 12:20:10 +01:00
Alexander Neumann	ae5302c7a8	Add comment that keepBlobs is modified	2020-11-05 10:33:38 +01:00
Alexander Neumann	866a52ad4e	Remove unneeded seek ... The file returned from DownloadAndHash() is already seeked to the start of the file.	2020-11-05 10:31:49 +01:00
Alexander Neumann	a4507610a0	Fix typo	2020-11-05 10:31:49 +01:00
Alexander Neumann	7def2d8ea7	Use a non-constant seed	2020-11-05 10:31:49 +01:00
Alexander Neumann	ee0112ab3b	Clarify message about expected error	2020-11-05 10:31:49 +01:00
Michael Eischer	b373f164fe	prune: Parallelize repack command	2020-11-05 10:31:49 +01:00
Alexander Neumann	5144141321	Merge pull request #2718 from aawsome/new-cleanup-command ... Reimplementation of prune	2020-11-05 10:12:19 +01:00
fgma	916b2d303b	vss: fix potential crash (not reachable in restic) (#3045) ... HasSufficientPrivilegesForVSS() now returns an error	2020-11-04 22:14:18 +01:00
Alexander Neumann	a06f5c28c0	Merge pull request #3055 from greatroar/cleanup ... Don't separately allocate sync.{Mutex,Once} if not necessary	2020-11-04 11:11:21 +01:00
greatroar	c2f3eee5af	Don't separately allocate sync.{Mutex,Once} if not necessary ... Separate allocation of synchronization devices suggests they're shared between objects, but they're not.	2020-11-04 09:55:43 +01:00
Alexander Weiss	b2f5381737	Make realistic forget --prune --dryrun	2020-11-03 16:42:21 +01:00
Alexander Weiss	ce7d613749	Add Blob.Handle()	2020-11-03 16:42:21 +01:00
Alexander Weiss	581d90cf91	Make some pack parameters public	2020-11-03 16:42:21 +01:00
Alexander Neumann	9a88fb253b	Merge pull request #3051 from greatroar/sanitize-env ... Sanitize environment before starting backend processes (rclone, ssh)	2020-11-02 21:18:57 +01:00
Alexander Neumann	f14436953a	Merge pull request #3050 from greatroar/widechars ... Fix string truncation in ui/termstatus	2020-11-02 16:58:51 +01:00
greatroar	11fbaaae9a	Sanitize environment before starting backend processes (rclone, ssh) ... The restic security model includes full trust of the local machine, so this should not fix any actual security problems, but it's better to be safe than sorry. Fixes #2192.	2020-11-02 16:41:23 +01:00
greatroar	3ed84ff0c6	Fix string truncation in ui/termstatus ... Fixes #3046.	2020-11-02 12:50:49 +01:00
Alexander Neumann	8e965ed4eb	Merge pull request #3039 from MichaelEischer/tar-proper-dirs ... Properly dump directories to tar	2020-11-02 12:30:26 +01:00
Alexander Neumann	5f0fa2129e	Improve readability ... It's time to use a switch statement.	2020-11-02 11:24:26 +01:00
Alexander Neumann	6509c207f4	Merge pull request #2997 from MichaelEischer/faster-excludes ... Speedup exclude/include checking	2020-11-02 11:07:21 +01:00
Alexander Neumann	445b845267	Merge pull request #2978 from MichaelEischer/warn-tree-error ... Warn if backup failed to read tree blob	2020-11-02 10:14:12 +01:00
Alexander Neumann	3ff37215df	Merge pull request #2935 from MichaelEischer/upgrade-minio ... Upgrade minio SDK to version 7	2020-11-02 09:09:10 +01:00
Sergio Rubio	e708628cfd	Remove unused function ... Not currently used, and it'd need to be added to the MasterIndex interface first.	2020-10-28 13:24:49 +01:00
MichaelEischer	bb4b3481a6	Merge pull request #3031 from greatroar/debug-no-seek ... Allow debug logging to pipes and terminals	2020-10-25 17:59:06 +01:00
MichaelEischer	ad3a52e6f0	Merge pull request #3026 from greatroar/refactor-ui ... internal/ui refactoring	2020-10-25 17:51:39 +01:00
Michael Eischer	8d7d6ad2d5	dump: include username in tar	2020-10-24 22:42:01 +02:00
Michael Eischer	fe09e6f865	dump: test proper permissions and directory name	2020-10-24 22:42:01 +02:00
Michael Eischer	1e3c9a2c11	dump: Fix file permission to tar mapping ... The file permissions included a go specific directory bit which accidentially forced the usage of the GNU header format. This leads to problems with 7zip on Windows or when extended attributes are used.	2020-10-24 22:42:01 +02:00
Michael Eischer	e21dcb0eea	dump: Additional ACL tests	2020-10-24 22:41:01 +02:00
MichaelEischer	31b8d7a639	Merge pull request #2274 from fgma/master ... Support for Volume Shadow Copy Service (VSS) on windows	2020-10-24 15:43:34 +02:00
fgma	5695f9ebd2	vss: Implement VSS support for Windows ... The VSS support works for 32 and 64-bit windows, this includes a check that the restic version matches the OS architecture as required by VSS. The backup operation will fail the user has not sufficient permissions to use VSS. Snapshotting volumes also covers mountpoints but skips UNC paths.	2020-10-24 11:35:57 +02:00
greatroar	63be3704d9	Allow debug logging to pipes and terminals	2020-10-19 22:30:30 +02:00
greatroar	35419de232	Simplify ui.StdioWrapper.Write ... Instead of looping to find line breaks, make it look for the last one.	2020-10-17 20:30:46 +02:00
greatroar	863a590a81	Refactor termstatus.Term.{Print,Error} methods	2020-10-17 20:30:46 +02:00
greatroar	7c0b6a82db	Remove unused public method ui.linesWriter.Flush	2020-10-17 20:30:24 +02:00
MichaelEischer	96a912b65a	Merge pull request #3025 from aawsome/fix-index-ids ... Fix setting of ID in DecodeIndex	2020-10-17 09:54:45 +02:00
Alexander Weiss	b44ecde8b0	Fix setting of ID in DecodeIndex	2020-10-17 09:12:58 +02:00
MichaelEischer	39fe1e96fe	Merge pull request #3018 from greatroar/background-check ... Fix IsProcessBackground on Linux with stdin redirection	2020-10-16 22:50:19 +02:00
MichaelEischer	4ba237bb93	Merge pull request #3019 from greatroar/refactor-decodeindex ... Refactor index decoding	2020-10-15 23:22:33 +02:00
greatroar	b27375f5ce	defer close(ch) outside repository.RunWorkers	2020-10-14 15:50:16 +02:00
greatroar	720e0ee0c7	if cond { return true }; return false => return cond	2020-10-13 20:56:43 +02:00
greatroar	27db3ec262	Refactor index decoding ... Decoding old-format indices no longer requires loading and decrypting twice.	2020-10-13 20:47:50 +02:00
greatroar	f80b07b2c8	Fix IsProcessBackground on Linux with stdin redirection ... The previous implementation assumed that stdin was a terminal. It now checks the terminal's fd.	2020-10-13 13:12:02 +02:00
MichaelEischer	41a45ae908	Merge pull request #3016 from greatroar/uid-gid-lookup ... Simplify/optimize cached UID/GID lookups	2020-10-12 23:24:55 +02:00
Alexander Neumann	56883817d8	Merge pull request #2990 from MichaelEischer/fix-goreport-warnings ... Fix some goreport warnings	2020-10-12 20:44:56 +02:00
greatroar	f0cd16e5ea	Cache uint32-typed ids in lookup{Username,Group} ... NodeFillUser-8 1.92µs ± 5% 1.75µs ± 3% -8.89% (p=0.000 n=10+10) NodeFromFileInfo-8 1.89µs ± 7% 1.76µs ± 4% -6.69% (p=0.001 n=10+10) name old alloc/op new alloc/op delta NodeFillUser-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10) NodeFromFileInfo-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10) name old allocs/op new allocs/op delta NodeFillUser-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10) NodeFromFileInfo-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)	2020-10-12 15:08:15 +02:00
greatroar	a03fe4562e	Remove unused error return in lookup{Username,Group}	2020-10-12 14:48:23 +02:00
Alexander Neumann	c193cea119	Pass Context to NewRestorer()	2020-10-10 15:24:26 +02:00
Alexander Neumann	f0d49ca600	Merge pull request #2933 from MichaelEischer/less-context-todo ... Replace most usages of context.TODO()	2020-10-10 15:03:44 +02:00
kitone	6099f81692	Fix #1212 restore code produces inconsistent timestamps/permissions. ... Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.	2020-10-10 13:46:44 +02:00
kitone	9d7f616190	Improve restorer debug log information	2020-10-10 13:46:43 +02:00
kitone	295ddb9e57	Add test case for inconsistent timestamps and permissions restoration ... Reproduce from https://github.com/restic/restic/issues/1212	2020-10-10 13:46:43 +02:00
Michael Eischer	e638b46a13	Embed context into ReaderAt ... The io.Reader interface does not support contexts, such that it is necessary to embed the context into the backendReaderAt struct. This has the problem that a reader might suddenly stop working when it's contained context is canceled. However, this is now problem here as the reader instances never escape the calling function.	2020-10-09 22:39:07 +02:00
Michael Eischer	d6cfe857b7	pass proper context into MasterIndex.RebuildIndex	2020-10-09 22:39:07 +02:00
Michael Eischer	37a5e2d681	rest: use global context on repository creation	2020-10-09 22:39:06 +02:00
Michael Eischer	4b0fcaed45	unlock: use proper context for locks cleanup ... The list operation used by RemoveStaleLocks or RemoveAllLocks will already be canceled by the passed in context. Therefore we can also just cancel the remove operation as the unlock command won't process all lock files anyways.	2020-10-09 22:37:56 +02:00
Michael Eischer	645a6efaf2	restorer: remove redundant type specification	2020-10-09 22:37:56 +02:00
Michael Eischer	a449450021	init: pass proper context to master key generation ... This is no change in behavior as a canceled context did later on cause the config file creation to fail. Therefore this change just lets the repository initialization fail a bit earlier.	2020-10-09 22:37:56 +02:00
Michael Eischer	603bb0e309	restore: Use proper context while loading snapshot	2020-10-09 22:37:56 +02:00
Michael Eischer	c458e114d4	pass context to Find / FindSnapshot ... This allows proper interruption of restic while it searches for snapshots or key files.	2020-10-09 22:37:56 +02:00
Michael Eischer	45e9a55c62	Wire context into backend layout detection	2020-10-09 22:37:24 +02:00
Michael Eischer	307a6ba3a3	Upgrade minio sdk to v7 ... This changes are primarily straightforward modifications to pass the parameters in the now expected way.	2020-10-09 22:37:24 +02:00
Michael Eischer	50da20d93d	Warn if backup failed to read tree blob	2020-10-09 22:36:27 +02:00
Alexander Neumann	5fd3dbccb7	Merge pull request #2963 from MichaelEischer/fix-status-deadlock ... backup: Fix possible deadlock of scanner goroutine	2020-10-09 21:35:50 +02:00
Alexander Neumann	35655a481b	Merge pull request #2931 from MichaelEischer/reduce-debug-overhead ... Reduce debug log overhead	2020-10-09 21:35:11 +02:00
Alexander Neumann	30cb553c8d	Merge pull request #2932 from MichaelEischer/proper-rclone-create ... Call rclone.Create to create a new repository for the rclone backend	2020-10-09 21:29:15 +02:00
MichaelEischer	356ac404cd	Merge pull request #2996 from greatroar/setconsolecursorposition ... Use windows.SetConsoleCursorPosition in ui/termstatus	2020-10-09 18:19:02 +02:00
Michael Eischer	88c8e903d2	filter: Fix glob matching on absolute path marker on windows ... A pattern part containing "/" is used to mark a path or a pattern as absolute. However, on Windows the path separator is "\" such that glob patterns like "?" could match the marker. The code now explicitly skips the marker when the pattern does not represent an absolute path.	2020-10-09 16:11:05 +02:00
greatroar	740758a5fa	Optimize filter pattern matching ... By replacing "**" with "", checking for this special path component can be reduced to a length-zero check. name old time/op new time/op delta FilterLines-8 44.7ms ± 5% 44.9ms ± 5% ~ (p=0.631 n=10+10) FilterPatterns/Relative-8 13.6ms ± 4% 13.4ms ± 5% ~ (p=0.165 n=10+10) FilterPatterns/Absolute-8 10.9ms ± 5% 10.7ms ± 4% ~ (p=0.052 n=10+10) FilterPatterns/Wildcard-8 53.7ms ± 5% 50.4ms ± 5% -6.00% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-8 128ms ± 2% 95ms ± 1% -25.54% (p=0.000 n=10+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=1.000 n=9+8) FilterPatterns/Absolute-8 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.903 n=9+8) FilterPatterns/Wildcard-8 19.7MB ± 0% 19.7MB ± 0% -0.00% (p=0.022 n=10+9) FilterPatterns/ManyNoMatch-8 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-8 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-8 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-8 88.7k ± 0% 88.7k ± 0% ~ (all equal) FilterPatterns/ManyNoMatch-8 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-09 15:51:14 +02:00
MichaelEischer	862ee4b2c9	Merge pull request #2970 from labkode/fopenskip ... Skip fopen for file change check to avoid network penalty	2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador	958dc6aafc	Skip fopen for file change check to avoid network penalty	2020-10-09 10:53:28 +02:00
MichaelEischer	88cc444779	Merge pull request #2934 from MichaelEischer/upgrade-backoff ... Upgrade github.com/cenkalti/backoff module	2020-10-08 20:33:30 +02:00
Michael Eischer	b79f18209f	Upgrade github.com/cenkalti/backoff module ... We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats an operation endlessly when a retry count of 0 is specified. This required a few fixes for the tests.	2020-10-07 22:04:59 +02:00
Michael Eischer	8388e67c4b	filter: cleanup path separator conversion	2020-10-07 21:14:07 +02:00
Michael Eischer	0acc3c5923	filter: special case patterns without globbing characters ... In case a part of a path is a simple string, we can just check for equality without complex parsing in filepath.Match. name old time/op new time/op delta FilterLines-4 34.8ms ±17% 41.2ms ±23% +18.36% (p=0.000 n=10+10) FilterPatterns/Relative-4 21.7ms ± 6% 12.1ms ±23% -44.46% (p=0.000 n=10+10) FilterPatterns/Absolute-4 10.0ms ± 5% 9.1ms ±11% -9.80% (p=0.006 n=10+9) FilterPatterns/Wildcard-4 47.0ms ± 7% 42.2ms ± 5% -10.19% (p=0.000 n=9+10) FilterPatterns/ManyNoMatch-4 190ms ± 1% 131ms ±20% -31.47% (p=0.000 n=8+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.870 n=9+9) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.145 n=10+10) FilterPatterns/Wildcard-4 14.3MB ± 0% 19.7MB ± 0% +37.91% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.421 n=10+9) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:55:43 +02:00
Michael Eischer	54a124de3b	filter: explicitly test separate ListWithChild function	2020-10-07 20:47:52 +02:00
Michael Eischer	bcc3bddcf4	filter: only check whether a child path could match when necessary ... When checking excludes there is no need to test whether a child path could also match the pattern, as it is by definition excluded. Previously childMayMatch was calculated but then discarded. For simple absolute paths this can account for half the time spent for checking pattern matches. name old time/op new time/op delta FilterPatterns/Relative-4 23.3ms ± 9% 21.7ms ± 6% -6.68% (p=0.004 n=10+10) FilterPatterns/Absolute-4 13.9ms ± 7% 10.0ms ± 5% -27.61% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 51.4ms ± 7% 47.0ms ± 7% -8.51% (p=0.001 n=9+9) FilterPatterns/ManyNoMatch-4 551ms ± 9% 190ms ± 1% -65.41% (p=0.000 n=10+8) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.665 n=10+9) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.480 n=9+10) FilterPatterns/Wildcard-4 14.3MB ± 0% 14.3MB ± 0% ~ (p=0.431 n=9+10) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 88.7k ± 0% 88.7k ± 0% ~ (all equal) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:47:52 +02:00
Michael Eischer	17c53efb0d	filter: Optimize double wildcard expansion ... This only allocates a single slice to expand the double wildcard and only copies the pattern prefix once. name old time/op new time/op delta FilterPatterns/Relative-4 22.7ms ± 5% 23.3ms ± 9% ~ (p=0.353 n=10+10) FilterPatterns/Absolute-4 14.2ms ±13% 13.9ms ± 7% ~ (p=0.853 n=10+10) FilterPatterns/Wildcard-4 266ms ±16% 51ms ± 7% -80.67% (p=0.000 n=10+9) FilterPatterns/ManyNoMatch-4 554ms ± 6% 551ms ± 9% ~ (p=0.436 n=10+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.349 n=10+10) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.073 n=10+9) FilterPatterns/Wildcard-4 141MB ± 0% 14MB ± 0% -89.89% (p=0.000 n=10+9) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 1.63M ± 0% 0.09M ± 0% -94.56% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:47:48 +02:00
Michael Eischer	7959796269	filter: Special case for absolute paths ... name old time/op new time/op delta FilterPatterns/Relative-4 23.6ms ±20% 22.7ms ± 5% ~ (p=0.684 n=10+10) FilterPatterns/Absolute-4 32.3ms ± 8% 14.2ms ±13% -56.01% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 334ms ±17% 266ms ±16% -20.56% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 709ms ± 7% 554ms ± 6% -21.89% (p=0.000 n=10+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 3.57MB ± 0% 3.57MB ± 0% +0.00% (p=0.046 n=9+10) FilterPatterns/Absolute-4 3.57MB ± 0% 3.57MB ± 0% ~ (p=0.464 n=10+10) FilterPatterns/Wildcard-4 141MB ± 0% 141MB ± 0% ~ (p=0.163 n=9+10) FilterPatterns/ManyNoMatch-4 3.57MB ± 0% 3.57MB ± 0% ~ (all equal) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Absolute-4 22.2k ± 0% 22.2k ± 0% ~ (all equal) FilterPatterns/Wildcard-4 1.63M ± 0% 1.63M ± 0% ~ (p=0.072 n=10+10) FilterPatterns/ManyNoMatch-4 22.2k ± 0% 22.2k ± 0% ~ (all equal)	2020-10-07 20:47:29 +02:00
Michael Eischer	375c2a56de	filter: Parse filter patterns only once ... name old time/op new time/op delta FilterPatterns/Relative-4 30.3ms ±10% 23.6ms ±20% -22.12% (p=0.000 n=10+10) FilterPatterns/Absolute-4 49.0ms ± 3% 32.3ms ± 8% -33.94% (p=0.000 n=8+10) FilterPatterns/Wildcard-4 345ms ± 9% 334ms ±17% ~ (p=0.315 n=10+10) FilterPatterns/ManyNoMatch-4 3.93s ± 2% 0.71s ± 7% -81.98% (p=0.000 n=9+10) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 4.63MB ± 0% 3.57MB ± 0% -22.98% (p=0.000 n=9+9) FilterPatterns/Absolute-4 8.54MB ± 0% 3.57MB ± 0% -58.20% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 146MB ± 0% 141MB ± 0% -2.93% (p=0.000 n=9+9) FilterPatterns/ManyNoMatch-4 907MB ± 0% 4MB ± 0% -99.61% (p=0.000 n=9+9) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 66.6k ± 0% 22.2k ± 0% -66.67% (p=0.000 n=10+10) FilterPatterns/Absolute-4 88.7k ± 0% 22.2k ± 0% -75.00% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 1.70M ± 0% 1.63M ± 0% -3.92% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 4.46M ± 0% 0.02M ± 0% -99.50% (p=0.000 n=10+10)	2020-10-07 20:47:27 +02:00
Michael Eischer	b8eacd1364	filter: Reduce redundant path and pattern splitting ... A single call to filter.List will split the path only once and also split each search pattern only once and use it for both match and childMatch. name old time/op new time/op delta FilterPatterns/Relative-4 62.1ms ±15% 30.3ms ±10% -51.22% (p=0.000 n=9+10) FilterPatterns/Absolute-4 111ms ±10% 49ms ± 3% -56.08% (p=0.000 n=10+8) FilterPatterns/Wildcard-4 393ms ±15% 345ms ± 9% -12.30% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 10.0s ± 3% 3.9s ± 2% -60.53% (p=0.000 n=10+9) name old alloc/op new alloc/op delta FilterPatterns/Relative-4 16.4MB ± 0% 4.6MB ± 0% -71.76% (p=0.000 n=10+9) FilterPatterns/Absolute-4 31.4MB ± 0% 8.5MB ± 0% -72.77% (p=0.000 n=9+10) FilterPatterns/Wildcard-4 168MB ± 0% 146MB ± 0% -13.19% (p=0.000 n=10+9) FilterPatterns/ManyNoMatch-4 3.23GB ± 0% 0.91GB ± 0% -71.96% (p=0.000 n=10+9) name old allocs/op new allocs/op delta FilterPatterns/Relative-4 178k ± 0% 67k ± 0% -62.50% (p=0.000 n=10+10) FilterPatterns/Absolute-4 266k ± 0% 89k ± 0% -66.67% (p=0.000 n=10+10) FilterPatterns/Wildcard-4 1.87M ± 0% 1.70M ± 0% -9.47% (p=0.000 n=10+10) FilterPatterns/ManyNoMatch-4 17.7M ± 0% 4.5M ± 0% -74.87% (p=0.000 n=9+10)	2020-10-07 18:13:19 +02:00
Michael Eischer	e73c281142	filter: Benchmark absolute paths, wildcards and long filter lists	2020-10-07 17:54:36 +02:00
Michael Eischer	fdd3b14db3	filter: test some corner cases	2020-10-07 17:09:44 +02:00
Michael Eischer	f4282aa6fd	local: mark repository files as read-only ... This is intended to prevent accidental modifications of data files. Marking the files as read-only was accidentally removed in #1258.	2020-10-07 12:29:37 +02:00
Michael Eischer	40ee17167e	local: Ignore permission errors on chmod call in Save/Remove operation ... The file is already created with the proper permissions, thus the chmod call is not critical. However, some file systems have don't allow modifications of the file permissions. Similarly the chmod call in the Remove operation should not prevent it from working.	2020-10-07 12:29:37 +02:00
greatroar	f3e933f0c1	Use windows.SetConsoleCursorPosition in ui/termstatus	2020-10-07 11:18:54 +02:00
MichaelEischer	09d39e260d	Merge pull request #2980 from greatroar/error-wrapping ... Remove repetitive error wrapping from internal/cache	2020-10-06 15:19:00 +02:00
Michael Eischer	1579d2a8ec	Remove some unused assignments	2020-10-06 14:55:13 +02:00
Michael Eischer	eba5dd831f	Fix typos reported by misspell	2020-10-06 14:55:13 +02:00
Michael Eischer	9ffb698c8d	Modernize context import in internal/fuse ... Migrated using `go fix`.	2020-10-06 14:55:13 +02:00
Michael Eischer	efbb850d92	Remove a few redundant type specifiers ... This is the result of running `gofmt -s -w **/*.go`	2020-10-06 14:55:13 +02:00
greatroar	673dda77c0	Less repetitive error messages in internal/cache ... Many instances of errors.Wrap in this package would produce messages like "Open: open <filename>: no such file or directory"; those now omit the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion now appends its own name to the error message, rather than the function that failed, to make it easier to spot. Other function names (e.g., Load) are already added further up in the call chain.	2020-10-05 20:28:54 +02:00
MichaelEischer	1a490acd67	Merge pull request #2982 from greatroar/archiver-error-handling ... Check error in archiver before calling Select	2020-10-05 19:19:47 +02:00
MichaelEischer	c84643c6a9	Merge pull request #2981 from greatroar/mkdircachedir ... Simplify cache directory creation	2020-10-05 18:15:23 +02:00
greatroar	1e1a1f3078	Upgrade cespare/xxhash to 2.1.1	2020-10-05 17:25:39 +02:00
greatroar	c4e2203e45	Check error in archiver before calling Select ... The archiver first called the Select function for a path before checking whether the Lstat on that path actually worked. The RejectFuncs in exclude.go worked around this by checking whether they received a nil os.FileInfo. Checking first is more obvious and requires less code.	2020-10-05 11:11:04 +02:00
greatroar	7d0fa1a686	Simplify cache directory creation	2020-10-05 10:46:45 +02:00
Ingo Gottwald	8b8e230771	Swap deprecated GCS lib with replacement	2020-10-03 18:55:56 +02:00
Ingo Gottwald	00cedd22aa	Replace deprecated method in gs backend	2020-10-01 10:02:42 +02:00
Michael Eischer	d44df9d00d	backup: Always remove the status lines once a backup ends	2020-09-30 23:13:10 +02:00
Michael Eischer	8d0ba55ecd	backup: Fix possible deadlock of scanner goroutine ... When the backup is interrupted for some reason while the scanner is still active this could lead to a deadlock. Interruptions are triggered by canceling the context object used by both the backup progress UI and the scanner. It is possible that a context is canceled between the respective check in the scanner and it calling the `ReportTotal` method of the UI. The latter method sends a message to the UI goroutine. However, a canceled context will also stop that goroutine, which can cause the channel send operation to block indefinitely. This is resolved by adding a `closed` channel which is closed once the UI goroutine is stopped and serves as an escape hatch for reported UI updates. This change covers not just the ReportTotal method but all potentially affected methods of the progress UI implementation.	2020-09-30 23:13:10 +02:00
MichaelEischer	48d0ab5276	Merge pull request #2959 from restic/keep-text ... backup: Correct keep policy text	2020-09-30 17:57:00 +02:00
MichaelEischer	fd02407863	Merge pull request #2849 from classmarkets/gcs-access-token ... gs: support authentication with access token	2020-09-30 17:42:56 +02:00
Leo R. Lundgren	028c9a5343	backup: Correct keep policy text ... Makes the following corrections to the "Applying Policy:" output: - keep the last 1 snapshots snapshots => keep 1 latest snapshots - keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots	2020-09-28 14:26:53 +02:00
MichaelEischer	14b312f00d	Merge pull request #2658 from creativeprojects/issue-2241 ... Don't echo authentication passwords (rest backend)	2020-09-22 22:17:53 +02:00
Fred	206cadfab4	Hide password from repository URLs	2020-09-22 22:00:51 +02:00
MichaelEischer	4875f7b659	Merge pull request #2614 from greatroar/simplify-fs ... Simplify internal/fs	2020-09-21 22:05:10 +02:00
greatroar	0d65b78168	Simplify os.ModeType|os.ModeCharDevice => os.ModeType ... Since Go 1.12, ModeCharDevice is included in ModeType: golang/go@a2a3dd00c9	2020-09-21 14:21:32 +02:00
greatroar	95ebba85ff	Remove stray Printf from internal/fs	2020-09-21 14:21:32 +02:00
greatroar	59b343a9bf	Remove OS-specific versions of fs.MkdirAll ... Go has supported Windows paths correctly since 1.11, see https://github.com/golang/go/issues/10900 and the commit referenced there.	2020-09-21 14:21:32 +02:00
greatroar	1557c58eef	Fix and simplify fs.Reader ... fakeDir.{Readdir,Readdirnames} weren't handling the case n == 0 correctly. fakeFileInfo.sys is always nil, so omit the field.	2020-09-21 14:21:32 +02:00
greatroar	c504aa505c	Remove unused fs.Rename	2020-09-21 14:21:32 +02:00
greatroar	1b20f6beec	Remove io.Writer from fs.File ... It was only used in a single test, which now uses plain *os.File instead.	2020-09-21 14:21:32 +02:00
Michael Eischer	8c36317b71	rclone: use configured number of connections during create	2020-09-19 19:11:43 +02:00
Michael Eischer	60a5c35de9	rclone: close connection to rclone if open fails	2020-09-19 19:11:43 +02:00
Michael Eischer	07f4e7d10b	Only log HTTP requests when a debug log is configured ... The logged HTTP requests are only visible when a debug log is configured.	2020-09-19 17:07:53 +02:00
Michael Eischer	f003410402	init: Add --copy-chunker-params option ... This allows creating multiple repositories with identical chunker parameters which is required for working deduplication when copying snapshots between different repositories.	2020-09-19 16:53:05 +02:00
Michael Eischer	9ad8250a78	Reduce overhead of debug calls if no log is enabled ... In case no debug log is configured, then calls to debug.Log only incur the costs to check a single boolean flag making the call really cheap.	2020-09-19 16:45:01 +02:00
greatroar	cb3f531050	Make Cache.Path private ... It's only used inside the package.	2020-09-18 11:17:29 +02:00
greatroar	23fcbb275a	Remove restic.Cache interface ... It was used in one code path, which then asserted its concrete type as *cache.Cache. Privatised some of the interface methods.	2020-09-18 10:48:13 +02:00
greatroar	6e3215a80d	Remove unused error type from internal/cache	2020-09-18 10:38:14 +02:00
greatroar	9abef3bf1a	Move internal/fs.TestChdir to internal/test.Chdir	2020-09-17 10:43:33 +02:00
Alexander Neumann	0c48e515f0	Merge pull request #2630 from MichaelEischer/fix-staticcheck ... Fix lots of small issues reported by staticcheck	2020-09-13 16:19:22 +02:00
Alexander Neumann	97950ab81a	options: Fix test for Go >= 1.15	2020-09-12 17:36:44 +02:00
MichaelEischer	88664ba222	Merge pull request #2893 from MichaelEischer/restore-preallocate ... restorer: Preallocate files	2020-09-08 22:43:05 +02:00
Michael Eischer	121233e1b3	Add preallocate tests	2020-09-07 21:41:47 +02:00
Michael Eischer	8cc9514879	restorer: pre-allocate files before loading chunks	2020-09-07 21:41:47 +02:00
Michael Eischer	2e7d475029	Process packs in order of first appearance	2020-09-06 21:25:35 +02:00
Michael Eischer	c46edcd9d6	error strings should not end with punctuation	2020-09-05 10:07:17 +02:00
Michael Eischer	1ede018ea6	error variable names should start with 'Err'	2020-09-05 10:07:17 +02:00
Michael Eischer	b77e933d80	Convert underscore variable names to camelCase	2020-09-05 10:07:16 +02:00
Michael Eischer	d0329cf3eb	Adjust comments to match name of exported methods	2020-09-05 10:07:16 +02:00
Michael Eischer	dc31529fc3	Unindent else block after if block ending with a return statement	2020-09-05 10:07:16 +02:00
Michael Eischer	4784540f04	repository: Simplify worker group code	2020-09-05 10:07:16 +02:00
Michael Eischer	84ea2389ae	archiver_test: Only skip symlinks on windows	2020-09-05 10:07:16 +02:00
Michael Eischer	b4a7ce86cf	uint cannot be less than zero	2020-09-05 10:07:16 +02:00
Michael Eischer	7ee0964880	crypto: Remove unused error	2020-09-05 10:07:16 +02:00
Michael Eischer	f6f11400c2	restic: Remove unused variable value	2020-09-05 10:07:16 +02:00
Michael Eischer	e2dc5034d3	restic/file: Use FileType for all File constants ... The data type of a variable is only carried on when using iota.	2020-09-05 10:07:16 +02:00
Michael Eischer	9a1b3cb5d9	restorer: Remove unused treeID struct member from test	2020-09-05 10:07:16 +02:00
Michael Eischer	ee05501ce7	archiver: Remove unused chmod function	2020-09-05 10:07:16 +02:00
Michael Eischer	014600bee6	archiver: Remove unused variable from test	2020-09-05 10:07:16 +02:00
Michael Eischer	d9a80e07b9	repository: Simplify index age calculation	2020-09-05 10:07:16 +02:00
Michael Eischer	460e2ffbf6	Collapse a few boolean operations	2020-09-05 10:07:14 +02:00
Michael Eischer	2f8335554c	Remove a few unused variables	2020-09-05 10:06:23 +02:00
Michael Eischer	37113282ca	Merge a few variable declaration and initializations	2020-09-05 10:05:34 +02:00
Michael Eischer	337725c354	Use sort.Strings	2020-09-05 10:05:34 +02:00
Michael Eischer	dcf9ded977	dump: Use slashes as path separators in tar on windows ... This conversion is necessary as paths returned by filepath.Rel use the operating-system native path separator	2020-08-30 18:26:50 +02:00
Simon Beck	fe6445e0f4	dump: Added unit tests for tar output	2020-08-30 16:00:48 +02:00
Simon Beck	80a11960dd	dump: Always dump relative paths into tarballs ... Tarballs should only contain relative paths.	2020-08-29 21:14:34 +02:00
Simon Beck	d6f739ec22	dump: Extract tar code to internal/dump	2020-08-29 21:14:34 +02:00
Michael Eischer	b25978a53c	backup: Fix reporting of directory count in summary ... Previously the directory stats were reported immediately after calling `SaveDir`. However, as the latter method saves the tree asynchronously the stats were still initialized to their nil value. The stats are now reported via a callback similar to the one used for the fileSaver.	2020-08-27 22:43:51 +02:00
Alexander Neumann	b1b3f1ecb6	Merge pull request #2674 from MichaelEischer/prune-strict-checks ... prune: Stricter error checks	2020-08-23 10:30:41 +02:00
Hristo Trendev	51c22f4223	local backend: ignore not supported error on sync() ... Closes #2395	2020-08-22 01:27:07 +02:00
Michael Eischer	f4b9544ab2	prune: Add test that repack aborts on wrong blob	2020-08-16 11:34:01 +02:00
Michael Eischer	367449dede	prune: Reduce memory allocations while repacking ... The slicing operator `slice[low:high]` default to 0 for the lower bound and len(slice) for the upper bound when either or both are not specified. Fix the code to use `cap(slice)` to check for the slice capacity.	2020-08-16 11:34:01 +02:00
Michael Eischer	7042bafea5	prune: Abort repacking when a pack contains a wrong blob ... If a blob in a pack file can be decrypted successfully but contains data that results in a different hash than stated in the header pack, then abort repacking. As both the pack header and the blob are cryptographically verified this either means than a malicious entity tampered with the backup or indicates hardware problems on the client. prune should fail with an error in both cases.	2020-08-16 11:34:01 +02:00
aawsome	0fed6a8dfc	Use "pack file" instead of "data file" (#2885) ... - changed variable names, especially changed DataFile into PackFile - changed in some comments - always use "pack file" in docu	2020-08-16 11:16:38 +02:00
MichaelEischer	eca0f0ad24	Merge pull request #2863 from aawsome/index-no-duplicates ... Don't save exact duplicates in merged index	2020-08-08 18:24:14 +02:00
Alexander Weiss	b112533812	Don't save exact duplicates when merging indexes	2020-08-05 06:32:02 +02:00
Alexander Weiss	5e63294355	Add benchmark MasterIndexAlloc	2020-08-05 06:32:02 +02:00
Michael Eischer	05116e4787	prune: Cleanup progress bar handling while repacking	2020-08-03 19:32:46 +02:00
Alexander Neumann	b7b479b668	Merge pull request #2599 from MichaelEischer/tweak-mem-usage ... Reduce memory usage when searching for used blobs	2020-08-01 13:49:02 +02:00
Alexander Neumann	4cf9656f12	Merge pull request #2861 from MichaelEischer/fix-rclone-crash ... rclone: Don't panic after unexpected subprocess exit	2020-08-01 13:08:50 +02:00
Alexander Neumann	2580eef2aa	Merge pull request #2318 from classmarkets/2175-named-keys ... Allow specifying user and host when adding keys	2020-08-01 13:06:31 +02:00
Michael Eischer	a178e5628e	Remove duplicate TreeLoader interface	2020-08-01 12:29:16 +02:00
Michael Eischer	af66a62c04	FindUsedBlobs: Test that seen blobs are skipped ... This also copies the TreeLoader interface from internal/walker to allow stubbing the repository in the call to `FindUsedBlobs`.	2020-08-01 12:29:16 +02:00
Michael Eischer	9ea1a78bd4	FindUsedBlobs: Check for seen blobs before loading trees ... The only effective change in behavior is that that toplevel nodes can also be skipped.	2020-08-01 12:29:16 +02:00
Michael Eischer	184103647a	FindUsedBlobs: merge seen into blobs BlobSet ... The seen BlobSet always contained a subset of the entries in blobs. Thus use blobs instead and avoid the memory overhead of the second set. Suggested-by: Alexander Weiss <alex@weissfam.de>	2020-08-01 12:29:16 +02:00
Michael Eischer	c81b122374	rclone: Don't panic after unexpected subprocess exit ... As the connection to the rclone child process is now closed after an unexpected subprocess exit, later requests will cause the http2 transport to try to reestablish a new connection. As previously this never should have happened, the connection called panic in that case. This panic is now replaced with a simple error message, as it no longer indicates an internal problem.	2020-08-01 12:17:40 +02:00
Alexander Weiss	f8316948d1	Optimize FUSE - make command restic mount faster and consume less memory ... - Add Open() functionality to dir - only access index for blobs when file is read - Implement NodeOpener and put one-time file stuff there - Add comment about locking as suggested by bazil.org/fuse => Thanks at Michael Eischer for suggesting the last two improvements	2020-07-28 23:01:18 +02:00
Michael Eischer	ea97ff1ba4	rclone: Skip crash test when rclone is not found	2020-07-26 12:06:18 +02:00
Michael Eischer	01b9581453	rclone: Better field names for stdio conn	2020-07-26 00:29:25 +02:00
Michael Eischer	3cd927d180	rclone: Give rclone time to finish before closing stdin pipe ... Calling `Close()` on the rclone backend sometimes failed during test execution with 'signal: Broken pipe'. The stdio connection closed both the stdin and stdout file descriptors at the same moment, therefore giving rclone no chance to properly send any final http2 data frames. Now the stdin connection to rclone is closed first and will only be forcefully closed after a timeout. In case rclone exits before the timeout then the stdio connection will be closed normally.	2020-07-26 00:29:25 +02:00
greatroar	bf7b1f12ea	rclone: Add test for pipe handling when rclone exits	2020-07-26 00:29:25 +02:00
Michael Eischer	8554332894	rclone: Close rclone side of stdio_conn pipes ... restic did not notice when the rclone subprocess exited unexpectedly. restic manually created pipes for stdin and stdout and used these for the connection to the rclone subprocess. The process creating a pipe gets file descriptors for the sender and receiver side of a pipe and passes them on to the subprocess. The expected behavior would be that reads or writes in the parent process fail / return once the child process dies as a pipe would now just have a reader or writer but not both. However, this never happened as restic kept the reader and writer file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe` close the subprocess side of pipes once the child process was started and close the parent process side of pipes once wait has finished. We can't use these functions as we need access to the raw `os.File` so just replicate that behavior.	2020-07-26 00:29:25 +02:00
greatroar	3e93b36ca4	Make rclone.New private	2020-07-26 00:28:45 +02:00
Michael Eischer	c847aace35	Rename Index interface to MasterIndex ... The interface is now only implemented by repository.MasterIndex.	2020-07-25 21:19:46 +02:00
Alexander Weiss	9d1fb94c6c	make Lookup() return all blobs ... + simplify syntax	2020-07-25 21:18:34 +02:00
MichaelEischer	020cab8e08	Merge pull request #2787 from greatroar/no-blobsize-cache ... Remove blob size cache from restic mount	2020-07-25 20:46:35 +02:00
greatroar	07da61baee	Remove blob size cache from restic mount	2020-07-25 19:39:14 +02:00
MichaelEischer	3d530dfc91	Merge pull request #2827 from aawsome/archiver-test-contents ... Make self-healing work when backing up with parent snapshot	2020-07-25 13:13:18 +02:00
MichaelEischer	bbc960f957	Merge pull request #2635 from greatroar/optimize-sortbycached ... Optimize sorting blobs by cache status	2020-07-25 12:35:42 +02:00
greatroar	309598c237	Simplify sortCachedPacksFirst test in internal/repository ... The test now uses the fact that the sort is stable. It's not guaranteed to be, but the test is cleaner and more exhaustive. sortCachedPacksFirst no longer needs a return value.	2020-07-25 12:12:59 +02:00
greatroar	03d23e6faa	Speed up blob sorting in internal/repository ... name old time/op new time/op delta SortCachedPacksFirst-8 208µs ± 3% 186µs ± 3% -10.74% (p=0.000 n=10+8) name old alloc/op new alloc/op delta SortCachedPacksFirst-8 213kB ± 0% 139kB ± 0% -34.62% (p=0.000 n=10+10) name old allocs/op new allocs/op delta SortCachedPacksFirst-8 1.03k ± 0% 1.03k ± 0% -0.19% (p=0.000 n=10+10)	2020-07-25 12:12:59 +02:00
greatroar	b10acd2af7	Test and benchmark blob sorting in internal/repository	2020-07-25 12:12:58 +02:00
Alexander Weiss	9175795fdb	Check contents in archiver ... When backing up with a parent snapshot and the file is not changed, also check if contents are still available in index.	2020-07-25 08:18:28 +02:00
rawtaz	5d8d70542f	Merge pull request #2852 from MichaelEischer/drop-go-1.11 ... Drop support for Go version 1.11	2020-07-25 01:17:36 +02:00
Michael Eischer	7c23381a2b	Drop support for Go version 1.11	2020-07-24 18:52:39 +02:00
MichaelEischer	34181b13a2	Merge pull request #2328 from MichaelEischer/no-repeated-checks ... Fix duplicate tree checks within `restic check`	2020-07-22 22:08:02 +02:00
Alexander Weiss	a666a6d576	Add tests and merge indexes in index benchmarks	2020-07-22 21:54:02 +02:00
Alexander Weiss	e388d962a5	Merge final indexes together for faster index access	2020-07-22 21:54:02 +02:00
Peter Schultz	758b44b9c0	gs: support authentication with access token ... In the Google Cloud Storage backend, support specifying access tokens directly, as an alternative to a credentials file. This is useful when restic is used non-interactively by some other program that is already authenticated and eliminates the need to store long lived credentials. The access token is specified in the GOOGLE_ACCESS_TOKEN environment variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.	2020-07-22 16:23:03 +02:00
Alexander Weiss	3b7a3711e6	Add more realistic index benchmarks ... + reduce test size of BenchmarkMasterIndexLookupParallel	2020-07-21 07:18:20 +02:00
Michael Eischer	9b0e718852	checker: Test that blob types are not confused	2020-07-20 23:43:47 +02:00
MichaelEischer	82c908871d	Merge pull request #2812 from greatroar/chaining ... Chaining hash table for repository.Index	2020-07-20 23:29:36 +02:00
Michael Eischer	ddf0b8cd0b	checker: Properly distinguish between data and tree blobs ... If a data blob and a tree blob with the same ID (= same content) exist, then the checker did not report a data or tree blob as unused when the blob of the other type was still in use.	2020-07-20 22:58:39 +02:00
Michael Eischer	2d0c138c9b	checker: Test that check only decodes trees once ... The `DuplicateTree` flag is necessary to ensure that failures cannot be swallowed. The old checker implementation ignores errors from LoadTree if the corresponding tree was already checked.	2020-07-20 22:51:53 +02:00
Michael Eischer	ef325ffc02	checker: Cleanup error handling code ... This change only moves code around but does not result in any change in behavior.	2020-07-20 22:51:53 +02:00
Michael Eischer	7a165f32a9	checker: Traverse trees in depth-first order ... Backups traverse the file tree in depth-first order and saves trees on the way back up. This results in tree packs filled in a way comparable to the reverse Polish notation. In order to check tree blobs in that order, the treeFilter would have to delay the forwarding of tree nodes until all children of it are processed which would complicate the implementation. Therefore do the next similar thing and traverse the tree in depth-first order, but process trees already on the way down. The tree blob ids are added in reverse order to the backlog, which is once again reverted when removing the ids from the back of the backlog.	2020-07-20 22:51:53 +02:00
Michael Eischer	36c69e3ca7	checker: Unify blobs, processed trees and referenced blobs map ... The blobRefs map and the processedTrees IDSet are merged to reduce the memory usage. The blobRefs map now uses separate flags to track blob usage as data or tree blob. This prevents skipping of trees whose content is identical to an already processed data blob. A third flag tracks whether a blob exists or not, which removes the need for the blobs IDSet.	2020-07-20 22:51:47 +02:00
Michael Eischer	35d8413639	checker: Remove dead index map	2020-07-20 22:37:31 +02:00
Michael Eischer	c66a0e408c	checker: Reduce cost of debug log ... Avoid duplicate allocation of the Subtree list.	2020-07-20 22:37:31 +02:00
Michael Eischer	70f4c014ef	checker: Decode identical tree nodes only once ... Even though the checkTreeWorker skips already processed chunks, filterTrees did queue the same tree blob on every occurence. This becomes a serious performance bottleneck for larger number of snapshots that cover mostly the same directories. Therefore decode a tree blob exactly once.	2020-07-20 22:37:31 +02:00
Michael Eischer	f0d8710611	Add benchmark for checker scaling with snapshot count	2020-07-20 22:37:31 +02:00
Alexander Neumann	8074879c5f	Remove 'go generate'	2020-07-19 17:28:42 +02:00
greatroar	7bda28f31f	Chaining hash table for repository.Index ... These are faster to construct but slower to access. The allocation rate is halved, the peak memory usage almost halved compared to standard map. Benchmark results on linux/amd64, -benchtime=3s -count=20: name old time/op new time/op delta PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.231 n=20+20) DecodeIndex-8 4.54s ± 0% 4.30s ± 0% -5.20% (p=0.000 n=18+17) DecodeIndexParallel-8 4.54s ± 0% 4.30s ± 0% -5.22% (p=0.000 n=19+18) IndexHasUnknown-8 44.4ns ± 5% 50.5ns ±11% +13.82% (p=0.000 n=19+17) IndexHasKnown-8 48.3ns ± 0% 51.5ns ±12% +6.68% (p=0.001 n=16+20) IndexAlloc-8 758ms ± 1% 616ms ± 1% -18.69% (p=0.000 n=19+19) IndexAllocParallel-8 234ms ± 3% 204ms ± 2% -12.60% (p=0.000 n=20+18) MasterIndexLookupSingleIndex-8 122ns ± 0% 145ns ± 9% +18.44% (p=0.000 n=14+20) MasterIndexLookupMultipleIndex-8 369ns ± 2% 429ns ± 8% +16.27% (p=0.000 n=20+20) MasterIndexLookupSingleIndexUnknown-8 68.4ns ± 5% 74.9ns ±13% +9.47% (p=0.000 n=20+20) MasterIndexLookupMultipleIndexUnknown-8 315ns ± 3% 369ns ±11% +17.14% (p=0.000 n=20+20) MasterIndexLookupParallel/known,indices=5-8 743ns ± 1% 816ns ± 2% +9.87% (p=0.000 n=17+17) MasterIndexLookupParallel/unknown,indices=5-8 238ns ± 1% 260ns ± 2% +9.14% (p=0.000 n=19+20) MasterIndexLookupParallel/known,indices=10-8 1.01µs ± 3% 1.11µs ± 2% +9.79% (p=0.000 n=19+20) MasterIndexLookupParallel/unknown,indices=10-8 222ns ± 0% 269ns ± 2% +20.83% (p=0.000 n=16+20) MasterIndexLookupParallel/known,indices=20-8 1.06µs ± 2% 1.19µs ± 2% +12.95% (p=0.000 n=19+18) MasterIndexLookupParallel/unknown,indices=20-8 413ns ± 1% 530ns ± 1% +28.19% (p=0.000 n=18+20) SaveAndEncrypt-8 30.2ms ± 1% 30.4ms ± 0% +0.71% (p=0.000 n=19+19) LoadTree-8 540µs ± 1% 576µs ± 1% +6.73% (p=0.000 n=20+20) LoadBlob-8 5.64ms ± 0% 5.64ms ± 0% ~ (p=0.883 n=18+17) LoadAndDecrypt-8 5.93ms ± 0% 5.95ms ± 1% ~ (p=0.247 n=20+19) LoadIndex-8 25.1ms ± 0% 24.5ms ± 1% -2.54% (p=0.000 n=18+17) name old speed new speed delta PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.229 n=20+20) SaveAndEncrypt-8 139MB/s ± 1% 138MB/s ± 0% -0.71% (p=0.000 n=19+19) LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% ~ (p=0.890 n=18+17) LoadAndDecrypt-8 169MB/s ± 0% 168MB/s ± 1% ~ (p=0.227 n=20+19) name old alloc/op new alloc/op delta PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.772 n=12+19) IndexAlloc-8 786MB ± 0% 400MB ± 0% -49.04% (p=0.000 n=20+18) IndexAllocParallel-8 786MB ± 0% 401MB ± 0% -49.04% (p=0.000 n=19+15) SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% +0.00% (p=0.000 n=19+19) name old allocs/op new allocs/op delta PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal) IndexAlloc-8 977k ± 0% 907k ± 0% -7.18% (p=0.000 n=20+20) IndexAllocParallel-8 977k ± 0% 907k ± 0% -7.17% (p=0.000 n=19+15) SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)	2020-07-19 13:58:22 +02:00
greatroar	255ba83c4b	Parallel index benchmarks + benchmark optimizations ... createRandomIndex was using the global RNG, which locks on every call It was also using twice as many random numbers as necessary and doing a float division in every iteration of the inner loop. BenchmarkDecodeIndex was using too short an input, especially for a parallel version. (It may now be using one that is a bit large.) Results on linux/amd64, -benchtime=3s -count=20: name old time/op new time/op delta PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.165 n=20+20) DecodeIndex-8 13.6µs ± 2% 4539886.8µs ± 0% +33293901.38% (p=0.000 n=20+18) IndexHasUnknown-8 44.4ns ± 7% 44.4ns ± 5% ~ (p=0.873 n=20+19) IndexHasKnown-8 49.2ns ± 3% 48.3ns ± 0% -1.86% (p=0.000 n=20+16) IndexAlloc-8 802ms ± 1% 758ms ± 1% -5.51% (p=0.000 n=20+19) MasterIndexLookupSingleIndex-8 124ns ± 1% 122ns ± 0% -1.41% (p=0.000 n=20+14) MasterIndexLookupMultipleIndex-8 373ns ± 2% 369ns ± 2% -1.13% (p=0.001 n=20+20) MasterIndexLookupSingleIndexUnknown-8 67.8ns ± 3% 68.4ns ± 5% ~ (p=0.753 n=20+20) MasterIndexLookupMultipleIndexUnknown-8 316ns ± 3% 315ns ± 3% ~ (p=0.846 n=20+20) SaveAndEncrypt-8 30.5ms ± 1% 30.2ms ± 1% -1.09% (p=0.000 n=19+19) LoadTree-8 527µs ± 1% 540µs ± 1% +2.37% (p=0.000 n=19+20) LoadBlob-8 5.65ms ± 0% 5.64ms ± 0% -0.21% (p=0.000 n=19+18) LoadAndDecrypt-8 7.07ms ± 2% 5.93ms ± 0% -16.15% (p=0.000 n=19+20) LoadIndex-8 32.1ms ± 2% 25.1ms ± 0% -21.64% (p=0.000 n=20+18) name old speed new speed delta PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.159 n=20+20) SaveAndEncrypt-8 138MB/s ± 1% 139MB/s ± 1% +1.10% (p=0.000 n=19+19) LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% +0.21% (p=0.000 n=19+18) LoadAndDecrypt-8 141MB/s ± 2% 169MB/s ± 0% +19.24% (p=0.000 n=19+20) name old alloc/op new alloc/op delta PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.826 n=19+12) IndexAlloc-8 786MB ± 0% 786MB ± 0% +0.01% (p=0.000 n=20+20) SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% -0.00% (p=0.012 n=20+19) name old allocs/op new allocs/op delta PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal) IndexAlloc-8 977k ± 0% 977k ± 0% +0.01% (p=0.022 n=20+20) SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)	2020-07-19 13:58:05 +02:00
Lars Lehtonen	9ac90cf5cd	internal/fuse: fix dropped test error	2020-07-12 21:42:31 -07:00
greatroar	58719e1f47	Replace mount's per-file cache by a global LRU cache	2020-07-12 18:27:16 +02:00
greatroar	d42c169458	Fix quadratic file reading in restic mount	2020-07-12 18:27:16 +02:00
greatroar	02bec13ef2	Fix repository_test.BenchmarkSaveAndEncrypt ... The benchmark was actually testing the speed of index lookups. name old time/op new time/op delta SaveAndEncrypt-8 101ns ± 2% 31505824ns ± 1% +31311591.31% (p=0.000 n=10+10) name old speed new speed delta SaveAndEncrypt-8 41.7TB/s ± 2% 0.0TB/s ± 1% -100.00% (p=0.000 n=10+10) name old alloc/op new alloc/op delta SaveAndEncrypt-8 1.00B ± 0% 20989508.40B ± 0% +2098950740.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta SaveAndEncrypt-8 0.00 123.00 ± 0% +Inf% (p=0.000 n=10+9) (The actual speed is ca. 131MiB/s.)	2020-07-05 17:41:42 +02:00
MichaelEischer	212607dc8a	Merge pull request #2760 from greatroar/backend-benchmark ... Fix backend benchmarks + a micro-optimization	2020-06-17 23:17:05 +02:00
greatroar	190d8e2f51	Flatten backend.LimitedReadCloser structure ... This inlines the io.LimitedReader into the LimitedReadCloser body to achieve fewer allocations. Results on linux/amd64: name old time/op new time/op delta Backend/BenchmarkLoadPartialFile-8 412µs ± 4% 413µs ± 4% ~ (p=0.634 n=17+17) Backend/BenchmarkLoadPartialFileOffset-8 455µs ±13% 441µs ±10% ~ (p=0.072 n=20+18) name old speed new speed delta Backend/BenchmarkLoadPartialFile-8 10.2GB/s ± 3% 10.2GB/s ± 4% ~ (p=0.817 n=16+17) Backend/BenchmarkLoadPartialFileOffset-8 9.25GB/s ±12% 9.54GB/s ± 9% ~ (p=0.072 n=20+18) name old alloc/op new alloc/op delta Backend/BenchmarkLoadPartialFile-8 888B ± 0% 872B ± 0% -1.80% (p=0.000 n=15+15) Backend/BenchmarkLoadPartialFileOffset-8 888B ± 0% 872B ± 0% -1.80% (p=0.000 n=15+15) name old allocs/op new allocs/op delta Backend/BenchmarkLoadPartialFile-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=15+15) Backend/BenchmarkLoadPartialFileOffset-8 18.0 ± 0% 17.0 ± 0% -5.56% (p=0.000 n=15+15)	2020-06-17 13:11:45 +02:00
greatroar	f4cd2a7120	Make backend benchmarks fairer by removing checks ... Checking whether the right data is returned takes up half the time in some benchmarks. Results for local backend benchmarks on linux/amd64: name old time/op new time/op delta Backend/BenchmarkLoadFile-8 4.89ms ± 0% 2.72ms ± 1% -44.26% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFile-8 936µs ± 6% 439µs ±15% -53.07% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFileOffset-8 940µs ± 1% 456µs ±10% -51.50% (p=0.008 n=5+5) Backend/BenchmarkSave-8 23.9ms ±14% 24.8ms ±41% ~ (p=0.690 n=5+5) name old speed new speed delta Backend/BenchmarkLoadFile-8 3.43GB/s ± 0% 6.16GB/s ± 1% +79.40% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFile-8 4.48GB/s ± 6% 9.63GB/s ±14% +114.78% (p=0.008 n=5+5) Backend/BenchmarkLoadPartialFileOffset-8 4.46GB/s ± 1% 9.22GB/s ±10% +106.74% (p=0.008 n=5+5) Backend/BenchmarkSave-8 706MB/s ±13% 698MB/s ±31% ~ (p=0.690 n=5+5)	2020-06-17 13:11:45 +02:00
Alexander Weiss	1361341c58	don't save duplicate packIDs when using internal/repository/Index.Store	2020-06-14 07:56:24 +02:00
Alexander Weiss	ce4a2f4ca6	save packIDs and duplicates separately ... A side remark to the definition of Index.blob: Another possibility would have been to use: blob map[restic.BlobHandle]*indexEntry This would have led to the following sizes: key: 32 + 1 = 33 bytes value: 8 bytes indexEntry: 8 + 4 + 4 = 16 bytes each packID: 32 bytes To save N index entries, we would therefore have needed: N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes More precicely, using a pointer instead of a direct entry is the better memory choice if: OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1) Under the assumption of OF=1.5, this means using pointers would have been the better choice if sizeof(indexEntry) > 24 bytes.	2020-06-14 07:56:21 +02:00
Alexander Weiss	cf979e2b81	make offset and length uint32	2020-06-14 07:50:19 +02:00
Michael Eischer	d92e2c5769	simplify index code	2020-06-14 07:50:19 +02:00
Alexander Weiss	7419844885	add changelog, benchmark, memory calculation	2020-06-14 07:50:15 +02:00
Alexander Weiss	d3c59d18e5	Fix inconsistency of saving/loading config file ... Fix saving/loading config file: Always set ID to a zero ID.	2020-06-13 16:30:23 +02:00
MichaelEischer	dd7b4f54f5	Merge pull request #2709 from greatroar/minio-sha256 ... Use Minio's optimized SHA-256	2020-06-12 23:32:58 +02:00
MichaelEischer	6896c6449b	Merge pull request #2779 from greatroar/archiver-comment ... Fix up comment on archiver.BlobSaver.Save	2020-06-12 23:04:21 +02:00
MichaelEischer	735a8074d5	Merge pull request #2773 from aawsome/index-uploads+knownblobs ... Fix non-intuitive repo behavior	2020-06-12 22:41:04 +02:00
Alexander Weiss	70347e95d5	disable index uploads for prune command ... + modifications of changelog	2020-06-12 09:24:38 +02:00
greatroar	0fa3091c78	Fix up comment on archiver.BlobSaver.Save	2020-06-11 13:40:30 +02:00
Alexander Weiss	91906911b0	Fix non-intuitive repository behavior ... - The SaveBlob method now checks for duplicates. - Moves handling of pending blobs to MasterIndex. -> also cleans up pending index entries when they are saved in the index -> when using SaveBlob no need to care about index any longer - Always check for full index and save it when storing packs. -> removes the need of an index uploader -> also removes the verbose "uploaded intermediate index" messages - The Flush method now also saves the index - Fix race condition when checking and saving full/non-finalized indexes	2020-06-11 13:05:23 +02:00
MichaelEischer	6856d1e422	Merge pull request #2749 from aawsome/fix-fullindex ... Change condition for full index	2020-06-10 20:40:19 +02:00
Alexander Weiss	8c1261ff02	changed condition for full index	2020-06-07 22:00:49 +02:00
MichaelEischer	84475aa3a8	Merge pull request #2730 from greatroar/mount-build-tags ... Simplify build tags for restic mount	2020-05-23 20:33:43 +02:00
Martin Michlmayr	5cc1760fdf	Fix typos	2020-05-16 14:05:26 +08:00
greatroar	649cbec6c5	Simplify build tags for restic mount ... This command can only be built on Darwin, FreeBSD and Linux (and if we upgrade bazil.org/fuse, only FreeBSD and Linux: https://github.com/bazil/fuse/issues/224). Listing the few supported operating systems explicitly here makes porting restic to new platforms easier.	2020-05-12 11:30:41 +02:00
greatroar	9f7cd69f13	Move Index.FindBlob to tests	2020-04-29 10:57:01 +02:00
greatroar	f97a680887	Fix repository benchmarks ... BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of their time measuring SHA-256 performance in their checks.	2020-04-28 07:57:29 +02:00
greatroar	42a3db05b0	Use Minio's optimized SHA-256 ... internal/repository benchmarks on an Intel i7-3770k: name old speed new speed delta PackerManager-8 209MB/s ± 1% 291MB/s ± 1% +38.94% (p=0.008 n=5+5) SaveAndEncrypt-8 112MB/s ± 1% 135MB/s ± 1% +20.25% (p=0.008 n=5+5)	2020-04-28 07:57:18 +02:00
greatroar	e7d7b85d59	Merge Repository.{LoadBlob,loadBlob} ... Pushing the allocation logic down into the former loadBlob body means that fewer allocations have to be performed: name old time/op new time/op delta LoadTree-8 478µs ± 1% 481µs ± 2% ~ (p=0.315 n=9+10) LoadBlob-8 11.6ms ± 1% 11.6ms ± 2% ~ (p=0.393 n=10+10) LoadAndDecrypt-8 13.3ms ± 3% 13.3ms ± 3% ~ (p=0.905 n=10+9) LoadIndex-8 33.6ms ± 2% 33.2ms ± 1% -1.15% (p=0.028 n=10+9) name old alloc/op new alloc/op delta LoadTree-8 41.2kB ± 0% 41.1kB ± 0% -0.23% (p=0.000 n=10+10) LoadBlob-8 2.28kB ± 0% 2.18kB ± 0% -4.21% (p=0.000 n=10+10) LoadAndDecrypt-8 2.10MB ± 0% 2.10MB ± 0% ~ (all equal) LoadIndex-8 5.22MB ± 0% 5.22MB ± 0% ~ (p=0.631 n=10+10) name old allocs/op new allocs/op delta LoadTree-8 652 ± 0% 651 ± 0% -0.15% (p=0.000 n=10+10) LoadBlob-8 24.0 ± 0% 23.0 ± 0% -4.17% (p=0.000 n=10+10) LoadAndDecrypt-8 30.0 ± 0% 30.0 ± 0% ~ (all equal) LoadIndex-8 30.2k ± 0% 30.2k ± 0% ~ (p=0.610 n=10+10) name old speed new speed delta LoadBlob-8 86.4MB/s ± 1% 85.9MB/s ± 2% ~ (p=0.393 n=10+10) LoadAndDecrypt-8 75.4MB/s ± 3% 75.4MB/s ± 3% ~ (p=0.858 n=10+9)	2020-04-23 10:04:20 +02:00