d8f0e7cbd1
This is subtle. A combination od fast client disk (read: SSD) with lots of files and fast network connection to restic-server would suddenly start getting lots of "dial tcp: connect: cannot assign requested address" errors during backup stage. Further inspection revealed that client machine was plagued with TCP sockets in TIME_WAIT state. When ephemeral port range was finally exhausted, no more sockets could be opened, so restic would freak out. To understand the magnitude of this problem, with ~18k ports and default timeout of 60 seconds, it means more than 300 HTTP connections per seconds were created and teared down. Yeah, restic-server is that fast. :) As it turns out, this behavior was product of 2 subtle issues: 1) The body of HTTP response wasn't read completely with io.ReadFull() at the end of the Load() function. This deactivated HTTP keepalive, so already open connections were not reused, but closed instead, and new ones opened for every new request. io.Copy(ioutil.Discard, resp.Body) before resp.Body.Close() remedies this. 2) Even with the above fix, somehow having MaxIdleConnsPerHost at its default value of 2 wasn't enough to stop reconnecting. It is hard to understand why this would be so detrimental, it could even be some subtle Go runtime bug. Anyhow, setting this value to match the connection limit, as set by connLimit global variable, finally nails this ugly bug. I fixed several other places where the response body wasn't read in full (or at all). For example, json.NewDecoder() is also known not to read the whole body of response. Unfortunately, this is not over yet. :( The check command is firing up to 40 simultaneous connections to the restic-server. Then, once again, MaxIdleConnsPerHost is too low to support keepalive, and sockets in the TIME_WAIT state pile up. But, as this kind of concurrency absolutely kill the poor disk on the server side, this is a completely different bug then. |
||
|---|---|---|
| .github | ||
| doc | ||
| src | ||
| vendor | ||
| .envrc | ||
| .gitignore | ||
| .hound.yml | ||
| .travis.yml | ||
| appveyor.yml | ||
| build.go | ||
| CONTRIBUTING.md | ||
| Dockerfile | ||
| LICENSE | ||
| Makefile | ||
| mkdocs.yml | ||
| README.md | ||
| run_integration_tests.go | ||
| Vagrantfile | ||
| VERSION | ||
Introduction
restic is a backup program that is fast, efficient and secure. Detailed information can be found in the documentation and the user manual. The design document lists the technical background and gives detailed information about the structure of the repository and the data saved therein.
The latest documentation can be viewed online at https://restic.readthedocs.io/en/latest. On the bottom left corner there is a menu that allows switching to the documentation and user manual for the latest released version.
Build restic
Install Go/Golang (at least version 1.6), then run go run build.go,
afterwards you'll find the binary in the current directory:
$ go run build.go
$ ./restic --help
Usage:
restic [OPTIONS] <command>
[...]
More documentation can be found in the user manual.
At the moment, the only tested compiler for restic is the official Go compiler. Building restic with gccgo may work, but is not supported.
Contribute and Documentation
Contributions are welcome! More information and a description of the
development environment can be found in CONTRIBUTING.md. A
document describing the design of restic and the data structures stored on the
back end is contained in doc/Design.md.
If you'd like to start contributing to restic, but don't know exactly what do
to, have a look at this great article by Dave Cheney:
Suggestions for contributing to an Open Source project
A few issues have been tagged with the label help wanted, you can start
looking at those: https://github.com/restic/restic/labels/help%20wanted
Contact
If you discover a bug, find something surprising or if you would like to discuss or ask something, please open a github issue. If you would like to chat about restic, there is also the IRC channel #restic on irc.freenode.net.
Important: If you discover something that you believe to be a possible critical security problem, please do not open a GitHub issue but send an email directly to alexander@bumpern.de. If possible, please encrypt your email using the following PGP key (0x91A6868BD3F7A907):
pub 4096R/91A6868BD3F7A907 2014-11-01
Key fingerprint = CF8F 18F2 8445 7597 3F79 D4E1 91A6 868B D3F7 A907
uid Alexander Neumann <alexander@bumpern.de>
sub 4096R/D5FC2ACF4043FDF1 2014-11-01
License
Restic is licensed under "BSD 2-Clause License". You can find the complete text
in the file LICENSE.