octoleo
/
restic
mirror of https://github.com/octoleo/restic.git
2
2
Fork 0
Code Issues Releases Wiki Activity
4,013 Commits 7 Branches 33 Tags 65 MiB
Commit Graph

435 Commits

Author SHA1 Message Date
Alexander Neumann 465700595c azure: Support uploading large files 
Closes #1822
 2018-06-01 14:52:16 +02:00
Jakob Unterwurzacher dd3b9910ee sftp: persist "ssh command exited" error 
If our ssh process has died, not only the next, but all subsequent
calls to clientError() should indicate the error.

restic output when the ssh process is killed with "kill -9":

  Save(<data/afb68adbf9>) returned error, retrying after 253.661803ms: Write: failed to send packet header: write |1: file already closed
  Save(<data/afb68adbf9>) returned error, retrying after 580.752212ms: ssh command exited: signal: killed
  Save(<data/afb68adbf9>) returned error, retrying after 790.150468ms: ssh command exited: signal: killed
  Save(<data/afb68adbf9>) returned error, retrying after 1.769595051s: ssh command exited: signal: killed
  [...]
  error in cleanup handler: ssh command exited: signal: killed

Before this patch:

  Save(<data/de698d934f>) returned error, retrying after 252.84163ms: Write: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 660.236963ms: OpenFile: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 568.049909ms: OpenFile: failed to send packet header: write |1: file already closed
  Save(<data/de698d934f>) returned error, retrying after 2.428813824s: OpenFile: failed to send packet header: write |1: file already closed
  [...]
  error in cleanup handler: failed to send packet header: write |1: file already closed
 2018-05-30 19:28:14 +02:00
Michael Kötter 589c23dc23 #1795 use unix.IoctlGetWinsize to get terminal size 2018-05-27 23:44:48 +02:00
Alexander Neumann 7d9642523b termstatus: Fix panic for non-terminal runs 
Closes #1803
 2018-05-27 12:52:01 +02:00
Alexander Neumann 2a976d795f b2: Remove extra error check 2018-05-26 10:12:30 +02:00
Alexander Neumann bfd923e81e rclone: Respect bandwith limits 2018-05-22 20:48:17 +02:00
Alexander Neumann adb682bc43 archiver: Don't open files with O_NONBLOCK 
This is not necessary any more, we're doing an lstat() before opening
an item, so we already known it's a file and not a pipe.
 2018-05-20 16:11:51 +02:00
Alexander Neumann 1e9744c9a4 archiver: Refuse to save an empty snapshot 2018-05-20 16:11:51 +02:00
Alexander Neumann 9a02f17cc2 archiver: Add tests for Save() for fs.Reader 2018-05-20 16:11:36 +02:00
Steve Kriss b358dd369b S3: rearrange credentials chain to be standard 
Signed-off-by: Steve Kriss <steve@heptio.com>
 2018-05-16 16:49:33 -07:00
Steve Kriss d67b9a32c6 S3: add file credentials to chain 
Signed-off-by: Steve Kriss <steve@heptio.com>
 2018-05-16 16:35:14 -07:00
Alexander Neumann 347a645450 Fix double error message 2018-05-15 11:03:33 +02:00
Alexander Neumann 9f5565b0fc Merge pull request #1735 from mholt/forget-max-age 
forget: Add --max-age policy to set hard cutoff for removing snapshots
 2018-05-14 21:18:06 +02:00
Alexander Neumann 060d8b57e0 Restructure TestApplyPolicy 2018-05-13 12:48:42 +02:00
Alexander Neumann cc627e832b Add custom Duration type 2018-05-13 12:48:42 +02:00
Alexander Neumann 5a0f0e3faa Add support for keeping a range of snapshots 2018-05-13 12:48:42 +02:00
Matthew Holt b52f2aa9a4 forget: Add policy to keep snapshots before a date 2018-05-13 12:48:10 +02:00
Alexander Neumann 60ea2435be Improve error message for readdir/readdirnames 
As mentioned in the forum[1], restic does not include the dir name when
readdir/readdirnames fails.

[1] https://forum.restic.net/t/readdirnames-readdirent-no-such-file-or-directory/653
 2018-05-13 10:34:50 +02:00
Alexander Neumann 159badf5ba Merge pull request #1778 from restic/fix-1771 
archiver: Improve error handling
 2018-05-13 00:13:54 +02:00
Alexander Neumann e43c9202a6 archiver: Make sure backend error is passed up 2018-05-12 23:55:59 +02:00
Alexander Neumann c5e75d1c98 archiver: Add test for early abort on unhandled error 2018-05-12 23:55:59 +02:00
Alexander Neumann 526956af35 archiver: Read files/dirs in order 2018-05-12 23:55:54 +02:00
Alexander Neumann 256104111d archiver: Clarify names 2018-05-12 23:55:54 +02:00
Alexander Neumann 21c83b1725 archiver: Add high-level documentation 2018-05-12 23:55:54 +02:00
Alexander Neumann 581c62ee72 archiver: Improve error handling 
This commit changes how the worker goroutines for saving e.g. blobs
interact. Before, it was possible to get stuck sending an instruction to
archive a file or dir when no worker goroutines were available any more.
This commit introduces a `done` channel for each of the worker pools,
which is set to the channel returned by `tomb.Dying()`, so it is closed
when the first worker returned an error.
 2018-05-12 23:55:54 +02:00
Gaige Lama a63989afcd
Ran gofmt on backend.go 
I ran gofmt on backend.go, this appears to valid edit.
 2018-05-11 11:07:16 -04:00
Alexander Neumann ca4af43c03 archiver: Return low-level errors 
This commit changes the archiver so that low-level errors saving data to
the repo are returned to the caller (instead of being handled by the
error callback function). This correctly bubbles up errors like a full
temp file system and makes restic abort early and makes all other worker
goroutines exit.
 2018-05-10 21:30:09 +02:00
Alexander Neumann 1f2463f42e archiver: Return correct error 2018-05-10 20:48:00 +02:00
Alexander Neumann 157c854d04 backup: Remove errored files from status display 2018-05-10 20:48:00 +02:00
Alexander Neumann b2208bb9c2 Rework termstatus 
This now keeps the cursor at the first column of the first status line
so that messages printed to stdout or stderr by some other part of the
progarm will still be visible. The message will overwrite the status
lines, but those are easily reprinted on the next status update.
 2018-05-07 21:40:07 +02:00
Alexander Neumann 4c25495d68 backup: Hide percent until total size is known 2018-05-06 20:20:25 +02:00
Alexander Neumann fd77646f8b Fix panic for nil-pointer dereference 
Closes #1755
 2018-05-03 20:49:30 +02:00
Alexander Neumann fca4fe4459 backup: Disable status output for --quiet 2018-05-02 21:24:18 +02:00
Alexander Neumann 26757ae2e5 Merge pull request #1750 from restic/archiver-fix-small-things 
backup: Tune user interface a bit
 2018-05-01 23:15:54 +02:00
Alexander Neumann 2218ecd049 archiver: Use lstat before open/fstat 
The previous code tried to be as efficient as possible and only do a
single open() on an item to save, and then fstat() on the fd to find out
what the item is (file, dir, other). For normal files, it would then
start reading the data without opening the file again, so it could not
be exchanged for e.g. a symlink.

This behavior starts the watchdog on my machine when /dev is saved
with restic, and after a few seconds, the machine reboots.

This commit reverts the behavior to the strategy the old archiver code
used: run lstat(), then decide what to do. For normal files, open the
file and then run fstat() on the fd to verify it's still a normal file,
then start reading the data.

The downside is that for normal files we now do two stat() calls
(lstat+fstat) instead of only one. On the upside, this does not start
the watchdog. :)
 2018-05-01 23:05:50 +02:00
Alexander Neumann d0974c155d backup: Tune user interface a bit 
Make non-verbose mode a bit more verbose.
 2018-05-01 22:18:32 +02:00
Alexander Neumann 8026e6fdfb Merge pull request #1749 from restic/add-cache-cmd 
Add 'cache' command to list and cleanup cache dirs
 2018-05-01 21:25:52 +02:00
Alexander Neumann f928aeec34 Add 'cache' command 2018-05-01 16:27:44 +02:00
Alexander Neumann eb6650b201 Add textfile 2018-05-01 14:38:59 +02:00
Alexander Neumann bc68d55e94 fs: Add TestTempFile 2018-05-01 14:38:41 +02:00
Alexander Neumann ecbbd851a1 Merge pull request #1744 from restic/fix-parent-detect 
backup: Clean target paths before finding parent
 2018-05-01 11:13:19 +02:00
Bryce Chidester e9f1721678 http backend: Parse the correct argument when loading --tls-client-cert 
Previously, the function read from ARGV[1] (hardcoded) rather than the
value passed to it, the command-line argument as it exists in globalOptions.

Resolves #1745
 2018-04-30 15:21:09 -07:00
Alexander Neumann 64d98945a6 backup: Clean target paths before finding parent 
This resolves an issue described in the forum where restic could not
find a parent snapshot if the target path ends with a slash:
https://forum.restic.net/t/new-archiver-code-please-test/623/23
 2018-04-30 22:03:11 +02:00
Alexander Neumann c83c03ed63 archiver: Fix blocking on pipes 2018-04-30 15:34:58 +02:00
Alexander Neumann 19b9c881ca fs: Add O_NONBLOCK 2018-04-30 15:34:58 +02:00
Alexander Neumann 4e34325035 archiver: Process dirs concurrently 2018-04-30 15:13:28 +02:00
Alexander Neumann 78bd591c7c archiver: Improve buffer pool 2018-04-30 15:13:28 +02:00
Alexander Neumann 39ac12f6ea archiver: Correct comment 2018-04-30 14:19:07 +02:00
Alexander Neumann 400730afca archiver: Improve memory usage, tune buffer pool 2018-04-30 14:19:07 +02:00
Alexander Neumann d80e108b03 backup: Clear status lines on finish 2018-04-30 14:19:07 +02:00
Alexander Neumann 846c2b6869 backup: Fix ETA calculation for >= 100% 2018-04-30 14:19:07 +02:00
Alexander Neumann 16c314ab7f termstatus: Don't print status if in background 2018-04-28 22:08:11 +02:00
Alexander Neumann 0e78ac92d8 Use new archiver code for backup 2018-04-28 22:08:11 +02:00
Alexander Neumann 1af96fc6dd Add termstatus 2018-04-27 21:42:15 +02:00
Alexander Neumann 38926d8576 Use new archiver code in tests 2018-04-25 14:42:45 +02:00
Alexander Neumann f279731168 Add new archiver code 2018-04-25 14:42:45 +02:00
Alexander Neumann 76b616451f Remove unneeded code 2018-04-23 21:40:33 +02:00
Alexander Neumann fd12a3af20 Remove old archiver code 2018-04-23 21:40:33 +02:00
Alexander Neumann a56b8fad87 repository: Improve buffer pooling 2018-04-22 11:37:05 +02:00
Alexander Neumann b6f98bdb02 node: Fill minimal info 2018-04-22 11:37:05 +02:00
Alexander Neumann c4b2486b7c fs: Add interface and FS implementations 
This adds two implementations of the new `FS` interface: One for the local
file system (`Local`) and one for a single file read from an
`io.Reader` (`Reader`).
 2018-04-22 11:37:05 +02:00
Alexander Neumann 83ca08245b checker: Check metadata size and blob sizes 2018-04-22 11:37:05 +02:00
Alexander Neumann 6a7c23d2ae tree: Add convenience functions 2018-04-22 11:37:05 +02:00
Alexander Neumann cc847a3d6d tree: Improve error for pre-existing node 2018-04-22 11:37:05 +02:00
Alexander Neumann baebf45e2e FindLatestSnapshot: Use absolute paths 2018-04-22 11:37:05 +02:00
Alexander Neumann fa4f438bc1 snapshot: Do not modify slice of paths 2018-04-22 11:37:05 +02:00
Alexander Neumann 4e0b2a8e3a snapshot: correct error handling for filepath.Abs 2018-04-22 11:37:05 +02:00
Alexander Neumann 0532f08048 Add test.Helper, also works with Go 1.8 2018-04-22 11:37:05 +02:00
Alexander Neumann a472868e06 fs: Add TestChdir() 2018-04-22 11:37:05 +02:00
Alexander Neumann e4fdc5eb76 fs: Add IsRegularFile() 2018-04-22 11:37:05 +02:00
Alexander Neumann 09bd924710 Do not restore sockets, correct error handling 
Closes #1730
 2018-04-20 13:53:11 +02:00
Alexander Neumann 577faa7570 local/sftp: Handling non-existing dirs in List() 2018-04-10 21:35:30 +02:00
Alexander Neumann 1c1fede399 Improve error message for orphaned pack files 2018-04-07 10:07:54 +02:00
Eri Bastos 901e1b129c Fixed issue #1608 - Use --time argument properly 
Backups via stdin will now handle --time argument and pass it down as
expected
 2018-04-03 14:40:42 -03:00
Alexander Neumann e68a7fea8a check: Allow filling the cache during check 
Closes #1665
 2018-04-01 13:59:27 +02:00
Alexander Neumann 2e7ec717c1 repository: Move cache preparation into function 2018-04-01 13:59:27 +02:00
Alexander Neumann 3f48e0e0f4 Add extra options to rclone 
For details see https://github.com/restic/restic/pull/1657#issuecomment-377707486
 2018-04-01 10:34:30 +02:00
Alexander Neumann 86f4b03730 Remove unneeded byte counters 2018-04-01 10:18:38 +02:00
Alexander Neumann c43c94776b rclone: Make concurrent connections configurable 2018-04-01 10:18:38 +02:00
Alexander Neumann 0b776e63e7 backend/rclone: Request random file name 
When `/` is requested, rclone returns the list of all files in the
remote, which is not what we want (and it can take quite some time).
 2018-04-01 10:18:38 +02:00
Alexander Neumann 737d93860a Extend first timeout to 60 seconds. 2018-04-01 10:18:38 +02:00
Alexander Neumann 17312d3a98 backend/rest: Ensure base URL ends with slash 
This makes it easier for rclone.
 2018-04-01 10:18:38 +02:00
Alexander Neumann 4d5c7a8749 backend/rclone: Make sure rclone terminates 2018-04-01 10:18:38 +02:00
Alexander Neumann fc0295016a Address code review comments 2018-04-01 10:18:38 +02:00
Alexander Neumann 99b62c11b8 backend/rclone: Stop rclone in case of errors 2018-04-01 10:18:38 +02:00
Alexander Neumann 6d9a029e09 backend/rclone: Prefix all error messages 2018-04-01 10:18:38 +02:00
Alexander Neumann 065fe1e54f backend/rclone: Skip test if binary is unavailable 2018-04-01 10:16:31 +02:00
Alexander Neumann 4dc0f24b38 backend/tests: Drain reader before returning error 2018-04-01 10:16:31 +02:00
Alexander Neumann fe99340e40 Add rclone backend 2018-04-01 10:16:31 +02:00
Alexander Neumann e377759c81 rest: Export Backend struct 2018-04-01 10:16:31 +02:00
Alexander Neumann cabbbd2b14 backend/rest: Export Content-Types 2018-04-01 10:16:31 +02:00
Alexander Neumann cf4cf94418 Move backend/sftp.StartForeground to backend/ 2018-04-01 10:16:31 +02:00
Alexander Neumann 34f27edc03 Refactor SplitShellStrings 2018-04-01 10:16:31 +02:00
Alexander Neumann 345b6c4694 Move backend/sftp.SplitShellArgs to backend/ 2018-04-01 10:16:31 +02:00
Alexander Neumann 594256bfa4 Merge pull request #1693 from restic/update-deps 
Update dependencies
 2018-03-30 17:19:08 +02:00
Alexander Neumann 673f0bbd6c Update vendored library github.com/cenkalti/backoff 2018-03-30 11:45:27 +02:00
Alexander Neumann d3f9c8b362 forget: Print policy 2018-03-30 10:24:26 +02:00
Alexander Neumann e5c929b793 Fix rest-server tests 
Since today, the rest-server needs to be explicitly told (via
`--no-auth`) that authentication is not necessary.
 2018-03-24 18:06:21 +01:00
Alexander Neumann 8206f85d2e Merge pull request #1552 from lawrencejones/use-auto-auth 
Automatically load Google auth
 2018-03-18 19:53:30 +01:00
Alexander Neumann 0279fd7212 Merge pull request #1669 from restic/make-tests-faster 
Reduce test runtime
 2018-03-13 19:50:34 +01:00