octoleo
/
restic
mirror of https://github.com/octoleo/restic.git
2
2
Fork 0
Code Issues Releases Wiki Activity
7,630 Commits 7 Branches 33 Tags 64 MiB
Commit Graph

126 Commits

Author SHA1 Message Date
Michael Pratt 9fa4f5eb6b gs: disable resumable uploads 
By default, the GCS Go packages have an internal "chunk size" of 8MB,
used for blob uploads.

Media().Do() will buffer a full 8MB from the io.Reader (or less if EOF
is reached) then write that full 8MB to the network all at once.

This behavior does not play nicely with --limit-upload, which only
limits the Reader passed to Media. While the long-term average upload
rate will be correctly limited, the actual network bandwidth will be
very spikey.

e.g., if an 8MB/s connection is limited to 1MB/s, Media().Do() will
spend 8s reading from the rate-limited reader (performing no network
requests), then 1s writing to the network at 8MB/s.

This is bad for network connections hurt by full-speed uploads,
particularly when writing 8MB will take several seconds.

Disable resumable uploads entirely by setting the chunk size to zero.
This causes the io.Reader to be passed further down the request stack,
where there is less (but still some) buffering.

My connection is around 1.5MB/s up, with nominal ~15ms ping times to
8.8.8.8.

Without this change, --limit-upload 1024 results in several seconds of
~200ms ping times (uploading), followed by several seconds of ~15ms ping
times (reading from rate-limited reader). A bandwidth monitor reports
this as several seconds of ~1.5MB/s followed by several seconds of
0.0MB/s.

With this change, --limit-upload 1024 results in ~20ms ping times and
the bandwidth monitor reports a constant ~1MB/s.

I've elected to make this change unconditional of --limit-upload because
the resumable uploads shouldn't be providing much benefit anyways, as
restic already uploads mostly small blobs and already has a retry
mechanism.

--limit-download is not affected by this problem, as Get().Download()
returns the real http.Response.Body without any internal buffering.

Updates #1216
 2017-10-17 21:12:04 -07:00
Alexander Neumann 980bb9059f Add entry to changelog 2017-10-16 20:15:19 +02:00
Alexander Neumann faadbd734b Add entry to CHANGELOG 2017-10-14 20:29:47 +02:00
Alexander Neumann 9ce2a73fc5 Add issue and PR to the changelog entry 2017-10-12 20:20:06 +02:00
rmdashrf 32637a0328 Basic rate limiting implementation. 
Added `--limit-upload` and `--limit-download` flags to rate limit
backups and restores.
 2017-10-11 20:01:20 -07:00
Alexander Neumann 8ceb22fe8a Add entry to CHANGELOG 2017-10-08 09:48:22 +02:00
Alexander Neumann a77c8cc5d2 Add entry to CHANGELOG 2017-10-07 10:23:56 +02:00
Alexander Neumann eb59d28154 Add entry to CHANGELOG 2017-10-03 18:54:07 +02:00
Alexander Neumann 2694def56a Add entry to CHANGELOG 2017-10-03 14:52:10 +02:00
Alexander Neumann 553ea812a7 Add entry to CHANGELOG 2017-10-03 12:38:00 +02:00
Michael Pratt 5ab9e12b46 Clarify cache location documentation 
PR #1287 changed the default cache location on darwin and windows.
Update the changelog and manual to reflect the new behavior.

Since the cache hasn't been included in an official release yet, I've
just changed the main cache changelog entry.

Fixes #1309
 2017-10-03 11:44:09 +02:00
Alexander Neumann 3afd974dea Add entry to CHANGELOG 2017-10-01 10:35:15 +02:00
Alexander Neumann ac92e2dd2d Add to CHANGELOG entry 2017-09-29 21:45:37 +02:00
Alexander Neumann eccc336319 Add entry to CHANGELOG 2017-09-26 14:18:37 +02:00
Michael Pratt fa0be82da8 gs: allow backend creation without storage.buckets.get 
If the service account used with restic does not have the
storage.buckets.get permission (in the "Storage Admin" role), Create
cannot use Get to determine if the bucket is accessible.

Rather than always trying to create the bucket on Get error, gracefully
fall back to assuming the bucket is accessible. If it is, restic init
will complete successfully. If it is not, it will fail on a later call.

Here is what init looks like now in different cases.

Service account without "Storage Admin":

Bucket exists and is accessible (this is the case that didn't work
before):

$ ./restic init -r gs:this-bucket-does-exist:/
enter password for new backend:
enter password again:
created restic backend c02e2edb67 at gs:this-bucket-does-exist:/

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

Bucket exists but is not accessible:

$ ./restic init -r gs:this-bucket-does-exist:/
enter password for new backend:
enter password again:
create key in backend at gs:this-bucket-does-exist:/ failed:
service.Objects.Insert: googleapi: Error 403:
my-service-account@myproject.iam.gserviceaccount.com does not have
storage.objects.create access to object this-bucket-exists/keys/0fa714e695c8ecd58cb467cdeb04d36f3b710f883496a90f23cae0315daf0b93., forbidden

Bucket does not exist:

$ ./restic init -r gs:this-bucket-does-not-exist:/
create backend at gs:this-bucket-does-not-exist:/ failed:
service.Buckets.Insert: googleapi: Error 403:
my-service-account@myproject.iam.gserviceaccount.com does not have storage.buckets.create access to bucket this-bucket-does-not-exist., forbidden

Service account with "Storage Admin":

Bucket exists and is accessible: Same

Bucket exists but is not accessible: Same. Previously this would fail
when Create tried to create the bucket. Now it fails when trying to
create the keys.

Bucket does not exist:

$ ./restic init -r gs:this-bucket-does-not-exist:/
enter password for new backend:
enter password again:
created restic backend c3c48b481d at gs:this-bucket-does-not-exist:/

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.
 2017-09-25 22:25:51 -07:00
Alexander Neumann bff1039e3a Add entry to CHANGELOG 2017-09-25 13:17:44 +02:00
Alexander Neumann 24c62e719a Add entry to CHANGELOG 2017-09-23 22:15:10 +02:00
Alexander Neumann 3a191f37cb Add entry to CHANGELOG 2017-09-23 14:05:55 +02:00
Alexander Neumann c40b3d3983 Add entry to CHANGELOG 2017-09-21 20:34:17 +02:00
Alexander Neumann e1a847e4d1 Add new version to CHANGELOG 2017-09-20 20:49:55 +02:00
Alexander Neumann 6df3d169b8 Add entry to CHANGELOG 2017-09-20 11:05:35 +02:00
Alexander Neumann 5b1e4df177 Add version to CHANGELOG 2017-09-13 17:14:43 +02:00
Alexander Neumann f3b49987f8 Add entry to CHANGELOG 2017-09-13 14:04:55 +02:00
Alexander Neumann 951a34dcbf Add entry to CHANGELOG 2017-09-11 19:02:36 +02:00
Alexander Neumann b1af544b1d Merge pull request #1224 from restic/improve-exclude-caches 
Allow multiple exclude-if-present
 2017-09-11 17:31:24 +02:00
Alexander Neumann 257a454515 Add entry to CHANGELOG 2017-09-10 20:55:02 +02:00
Alexander Neumann c8e05d1f2a Add entry to CHANGELOG 2017-09-10 20:29:08 +02:00
Fabian Wickborn dbda892542 Add option to exclude directories with a tagfile 
The option is named --exclude-if-present and accepts a parameter
filename[:content]. Directories are excluded and their contents is not
backed up if they contain a file with the specified name and,
optionally, that starts with the specified content. The tagfile itself
is never excluded.

There is also a shortcut --exclude-caches that works in the same way as
the likewise-named option of tar(1): Directories are recognized as cache
if they contain a file named "CACHEDIR.TAG.

Closes #317.
 2017-09-09 09:57:42 +02:00
Alexander Neumann c935d0558c Add entry to CHANGELOG 2017-09-05 21:48:13 +02:00
Alexander Neumann 8c40ae5a03 Add entry to CHANGELOG 2017-09-04 21:58:33 +02:00
Alexander Neumann 0e647417f3 Add entry to CHANGELOG 2017-09-03 09:49:37 +02:00
Alexander Neumann b8414b240c Add entry to CHANGELOG 2017-09-03 09:44:15 +02:00
Alexander Neumann 0230fa188f Add entry to CHANGELOG 2017-08-28 21:13:14 +02:00
Alexander Neumann 4e60156b45 Add entry to CHANGELOG 2017-08-18 19:50:32 +02:00
Alexander Neumann b67c178672 Merge pull request #1149 from restic/azure-support 
Add Azure blob storage as backend
 2017-08-09 21:30:35 +02:00
Alexander Neumann c4613c51d1 Add note about Go 1.8 2017-08-09 20:33:30 +02:00
Alexander Neumann 77bf17076b Add entry to CHANGELOG 2017-08-09 20:23:29 +02:00
Alexander Neumann a345386967 Add a section to the CHANGELOG 2017-08-09 20:15:08 +02:00
Alexander Neumann 14f5f6235a Add entry to CHANGELOG 2017-08-05 12:05:53 +02:00
Alexander Neumann 9bf3141893 Add entry to CHANGELOG 2017-08-01 22:02:08 +02:00
Alexander Neumann e8cc11ea34 Add entry to CHANGELOG 2017-07-26 22:29:39 +02:00
Alexander Neumann 94030a12cf Add entry to CHANGELOG 2017-07-23 16:54:56 +02:00
Alexander Neumann bc88a8bb03 Add entry to CHANGELOG 2017-07-22 11:52:16 +02:00
Alexander Neumann 02a245941a Adapt CHANGELOG for 0.7.1 2017-07-22 11:03:44 +02:00
Alexander Neumann 16340ce811 Merge pull request #1090 from middelink/fix-1081 
Update HasTags() and HasPaths() to follow #1081 feature request
 2017-07-19 17:11:18 +02:00
Alexander Neumann 2cf8153f4a Add entry to CHANGELOG 2017-07-19 17:09:02 +02:00
Alexander Neumann 0de17f64e9 Add entry to CHANGELOG 2017-07-19 17:02:53 +02:00
Alexander Neumann 6c05353086 Add entry to CHANGELOG 2017-07-17 22:02:06 +02:00
Alexander Neumann 7bb1a474df Add entry to CHANGELOG 2017-07-16 21:41:13 +02:00
Alexander Neumann 7cdcaadcf5 Add entry to CHANGELOG 2017-07-16 15:11:26 +02:00
Alexander Neumann 38e5640cda Add CHANGELOG entry 2017-07-09 21:43:05 +02:00
Alexander Neumann 04d27acd60 Add entry to CHANGELOG 2017-07-05 20:54:37 +02:00
Alexander Neumann 637de0149c Add entry to CHANGELOG 2017-07-03 19:49:18 +02:00
Alexander Neumann 8c30ae7c65 Add entry to CHANGELOG 2017-07-02 11:21:05 +02:00
Alexander Neumann e7577d7bb4 Add stub to CHANGELOG 2017-07-01 15:11:36 +02:00
Alexander Neumann b50fc08f39 Add entry to CHANGELOG 2017-06-30 22:15:00 +02:00
Alexander Neumann 216e2607ca Add entry to CHANGELOG 2017-06-18 21:18:11 +02:00
Alexander Neumann 1baaa778ee Add entry to CHANGELOG 2017-06-16 12:27:44 +02:00
Alexander Neumann 7b54f6e642 Add entry to CHANGELOG 2017-06-12 19:56:50 +02:00
Alexander Neumann 1f0916b01b Merge pull request #1004 from restic/add-migrate-s3 
Add 'migrate' command, change s3 layout
 2017-06-08 20:48:27 +02:00
Alexander Neumann eb7ddd6e11 Add entry to CHANGELOG 2017-06-08 19:21:52 +02:00
Alexander Neumann f2a51aa37c Add entry to CHANGELOG 2017-06-07 20:51:08 +02:00
Alexander Neumann 9cd664caa3 Add entry to CHANGELOG 2017-06-04 11:50:38 +02:00
Alexander Neumann 40a61b82ce Merge pull request #978 from restic/add-backblaze-backend 
Add Backblaze B2 backend
 2017-06-03 14:54:04 +02:00
Alexander Neumann 3a4727f0f5 Add entry to CHANGELOG.md 2017-06-03 14:28:29 +02:00
Alexander Neumann 3ed56f2192 Add entry to CHANGELOG.md 2017-06-03 14:24:59 +02:00
Alexander Neumann b5e0e3631b Addd nev version section 2017-06-03 14:10:28 +02:00
Alexander Neumann c2ce484e93 Add version to CHANGELOG 2017-05-31 23:50:54 +02:00
Alexander Neumann 997be9a036 Remove PR 2017-05-31 21:34:18 +02:00
Alexander Neumann 31fd8e98b9 Add Entry to CHANGELOG 2017-05-31 21:33:45 +02:00
Alexander Neumann a8386e7d71 Add entry to CHANGELOG 2017-05-31 19:53:54 +02:00
Alexander Neumann ed91cafce2 Add entry to CHANGELOG 2017-05-29 23:46:48 +02:00
Alexander Neumann 317d9c4559 Add entry to the changelog 2017-05-25 17:06:06 +02:00
Alexander Neumann 17ff41af47 Update version number in CHANGELOG.md 2017-05-25 15:42:44 +02:00
Alexander Neumann 5c4facd9dc Update CHANGELOG.md 2017-05-25 15:26:22 +02:00
Alexander Neumann 9e0207e534 Add CHANGELOG.md and instructions for contributors 2017-03-09 10:40:24 +01:00