restic/CHANGELOG.md

392 lines
16 KiB
Markdown
Raw Normal View History

This file describes changes relevant to all users that are made in each
released version of restic from the perspective of the user.
2017-09-21 18:34:17 +00:00
Important Changes in 0.X.Y
==========================
2017-09-25 11:17:44 +00:00
* We've added a local cache for metadata so that restic doesn't need to load
all metadata (snapshots, indexes, ...) from the repo each time it starts. By
default the cache is active, but there's a new global option `--no-cache`
that can be used to disable the cache. By deafult, the cache a standard
cache folder for the OS, which can be overridden with `--cache-dir`. The
cache will automatically populate, indexes and snapshots are saved as they
are loaded.
2017-09-25 11:17:44 +00:00
https://github.com/restic/restic/pull/1040
https://github.com/restic/restic/issues/29
https://github.com/restic/restic/issues/738
https://github.com/restic/restic/issues/282
2017-09-29 19:45:37 +00:00
https://github.com/restic/restic/pull/1287
2017-09-25 11:17:44 +00:00
* A related change was to by default create pack files in the repo that
contain either data or metadata, not both mixed together. This allows easy
caching of only the metadata files. The next run of `restic prune` will
untangle mixed files automatically.
https://github.com/restic/restic/pull/1265
2017-09-21 18:34:17 +00:00
gs: allow backend creation without storage.buckets.get If the service account used with restic does not have the storage.buckets.get permission (in the "Storage Admin" role), Create cannot use Get to determine if the bucket is accessible. Rather than always trying to create the bucket on Get error, gracefully fall back to assuming the bucket is accessible. If it is, restic init will complete successfully. If it is not, it will fail on a later call. Here is what init looks like now in different cases. Service account without "Storage Admin": Bucket exists and is accessible (this is the case that didn't work before): $ ./restic init -r gs:this-bucket-does-exist:/ enter password for new backend: enter password again: created restic backend c02e2edb67 at gs:this-bucket-does-exist:/ Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost. Bucket exists but is not accessible: $ ./restic init -r gs:this-bucket-does-exist:/ enter password for new backend: enter password again: create key in backend at gs:this-bucket-does-exist:/ failed: service.Objects.Insert: googleapi: Error 403: my-service-account@myproject.iam.gserviceaccount.com does not have storage.objects.create access to object this-bucket-exists/keys/0fa714e695c8ecd58cb467cdeb04d36f3b710f883496a90f23cae0315daf0b93., forbidden Bucket does not exist: $ ./restic init -r gs:this-bucket-does-not-exist:/ create backend at gs:this-bucket-does-not-exist:/ failed: service.Buckets.Insert: googleapi: Error 403: my-service-account@myproject.iam.gserviceaccount.com does not have storage.buckets.create access to bucket this-bucket-does-not-exist., forbidden Service account with "Storage Admin": Bucket exists and is accessible: Same Bucket exists but is not accessible: Same. Previously this would fail when Create tried to create the bucket. Now it fails when trying to create the keys. Bucket does not exist: $ ./restic init -r gs:this-bucket-does-not-exist:/ enter password for new backend: enter password again: created restic backend c3c48b481d at gs:this-bucket-does-not-exist:/ Please note that knowledge of your password is required to access the repository. Losing your password means that your data is irrecoverably lost.
2017-09-26 04:53:21 +00:00
* The Google Cloud Storage backend no longer requires the service account to
have the `storage.buckets.get` permission ("Storage Admin" role) in `restic
init` if the bucket already exists.
https://github.com/restic/restic/pull/1281
* Added support for rate limiting through `--limit-upload` and
`--limit-download` flags.
https://github.com/restic/restic/issues/1216
https://github.com/restic/restic/pull/1336
gs: disable resumable uploads By default, the GCS Go packages have an internal "chunk size" of 8MB, used for blob uploads. Media().Do() will buffer a full 8MB from the io.Reader (or less if EOF is reached) then write that full 8MB to the network all at once. This behavior does not play nicely with --limit-upload, which only limits the Reader passed to Media. While the long-term average upload rate will be correctly limited, the actual network bandwidth will be very spikey. e.g., if an 8MB/s connection is limited to 1MB/s, Media().Do() will spend 8s reading from the rate-limited reader (performing no network requests), then 1s writing to the network at 8MB/s. This is bad for network connections hurt by full-speed uploads, particularly when writing 8MB will take several seconds. Disable resumable uploads entirely by setting the chunk size to zero. This causes the io.Reader to be passed further down the request stack, where there is less (but still some) buffering. My connection is around 1.5MB/s up, with nominal ~15ms ping times to 8.8.8.8. Without this change, --limit-upload 1024 results in several seconds of ~200ms ping times (uploading), followed by several seconds of ~15ms ping times (reading from rate-limited reader). A bandwidth monitor reports this as several seconds of ~1.5MB/s followed by several seconds of 0.0MB/s. With this change, --limit-upload 1024 results in ~20ms ping times and the bandwidth monitor reports a constant ~1MB/s. I've elected to make this change unconditional of --limit-upload because the resumable uploads shouldn't be providing much benefit anyways, as restic already uploads mostly small blobs and already has a retry mechanism. --limit-download is not affected by this problem, as Get().Download() returns the real http.Response.Body without any internal buffering. Updates #1216
2017-10-18 04:04:35 +00:00
https://github.com/restic/restic/pull/1358
2017-10-14 18:29:47 +00:00
* Failed backend requests are now automatically retried.
https://github.com/restic/restic/pull/1353
2017-10-16 18:15:19 +00:00
* We've added the `dump` command which prints a file from a snapshot to
stdout. This can e.g. be used to restore files read with `backup --stdin`.
https://github.com/restic/restic/issues/510
https://github.com/restic/restic/pull/1346
2017-09-21 18:34:17 +00:00
Small changes
-------------
* The directory structure in the fuse mount now exposes a symlink `latest`
which points to the latest snapshot in that particular directory.
https://github.com/restic/restic/pull/1249
2017-09-23 12:05:55 +00:00
* The option `--compact` was added to the `forget` command to provide the same
compact view as the `snapshots` command.
https://github.com/restic/restic/pull/1269
* We've re-enabled a workaround for `minio-go` (the library we're using to
access s3 backends), this reduces memory usage.
https://github.com/restic/restic/issues/1256
https://github.com/restic/restic/pull/1267
2017-09-21 18:34:17 +00:00
2017-09-23 20:15:10 +00:00
* The sftp backend now prompts for the password if a password is necessary for
login.
https://github.com/restic/restic/issues/448
https://github.com/restic/restic/pull/1270
2017-09-26 12:18:37 +00:00
* The `generate` command has been added, which replaces the now removed
commands `manpage` and `autocomplete`. This release of restic contains the
most recent manpages in `doc/man` and the auto-completion files for bash and
zsh in `doc/bash-completion.sh` and `doc/zsh-completion.zsh`
https://github.com/restic/restic/issues/1274
https://github.com/restic/restic/pull/1282
2017-10-01 08:35:15 +00:00
* A bug was discovered in the library we're using to access Backblaze, it now
reuses already established TCP connections which should be a lot faster and
not cause network failures any more.
https://github.com/restic/restic/issues/1291
https://github.com/restic/restic/pull/1301
2017-10-03 10:38:00 +00:00
* Another bug in the `forget` command caused `prune` not to be run when
`--prune` was specified without a policy, e.g. when only snapshot IDs that
should be forgotten are listed manually. This is corrected now.
https://github.com/restic/restic/pull/1317
2017-10-03 12:52:10 +00:00
* The `check` command now explicetly prints `No errors were found` when no
errors could be found.
https://github.com/restic/restic/pull/1319
https://github.com/restic/restic/issues/1303
2017-10-03 16:54:07 +00:00
* The fuse mount now has an `ids` subdirectory which contains the snapshots
below their (short) IDs.
https://github.com/restic/restic/issues/1102
https://github.com/restic/restic/pull/1299
https://github.com/restic/restic/pull/1320
2017-10-07 08:23:56 +00:00
* The `backup` command was improved, it now caches the result of excludes for
a directory.
https://github.com/restic/restic/issues/1271
https://github.com/restic/restic/pull/1326
2017-10-08 07:48:22 +00:00
* We've added the `--cacert` option which can be used to pass one (or more) CA
certificates to restic. These are used in addition to the system CA
certificates to verify HTTPS certificates (e.g. for the REST backend).
https://github.com/restic/restic/issues/1114
https://github.com/restic/restic/pull/1276
2017-10-21 10:22:50 +00:00
* When the list of files/dirs to be saved is read from a file with
`--files-from`, comment lines (starting with `#`) are now ignored.
https://github.com/restic/restic/issues/1367
https://github.com/restic/restic/pull/1368
2017-09-20 18:49:55 +00:00
Important Changes in 0.7.3
2017-09-20 09:05:35 +00:00
==========================
* For large backups stored in Google Cloud Storage, the `prune` command fails
because listing only returns the first 1000 files. This has been corrected,
no data is lost in the process. In addition, a plausibility check was added
to `prune`.
https://github.com/restic/restic/issues/1246
https://github.com/restic/restic/pull/1247
2017-09-13 15:14:43 +00:00
Important Changes in 0.7.2
2017-07-22 09:52:16 +00:00
==========================
* We've added an official docker image and a Dockerfile to build this image in
`docker/`.
https://github.com/restic/restic/pull/1061
2017-07-23 14:54:56 +00:00
* The git repository layout was changed to resemble the layout typically used
in Go projects, we're not using `gb` for building restic any more and
vendoring the dependencies is now taken care of by `dep`.
https://github.com/restic/restic/pull/1126
2017-08-09 18:23:29 +00:00
* We now support saving backups on Google Cloud Storage.
https://github.com/restic/restic/pull/1134
https://github.com/restic/restic/pull/1052
https://github.com/restic/restic/issues/211
2017-08-09 18:15:08 +00:00
* We've added support for Microsoft Azure Blob Storage as a restic backend.
https://github.com/restic/restic/pull/1149
https://github.com/restic/restic/pull/1059
https://github.com/restic/restic/issues/609
2017-08-09 18:33:30 +00:00
* In the course of supporting Microsoft Azure Blobe Storage Go 1.8 is now a
requirement to build restic.
2017-09-04 19:58:33 +00:00
* The `restore` command has been improved: When dirs are excluded (or not
included) in a restore, they are not loaded from the repo any more.
https://github.com/restic/restic/pull/1044
2017-09-05 19:48:13 +00:00
* Name collisions are now resolved by appending a counter.
https://github.com/restic/restic/issues/1179
https://github.com/restic/restic/pull/1209
2017-08-09 18:23:29 +00:00
2017-07-22 09:52:16 +00:00
Small changes
-------------
2017-07-26 20:29:39 +00:00
* The `key` command now prompts for a password even if the original password
to access a repo has been specified via the `RESTIC_PASSWORD` environment
variable or a password file.
https://github.com/restic/restic/issues/1132
https://github.com/restic/restic/pull/1133
2017-08-01 20:02:08 +00:00
* Properly report errors when reading files with exclude patterns.
https://github.com/restic/restic/pull/1144
2017-08-05 09:59:14 +00:00
* We now automatically generate man pages for all restic commands, see the
subdir `doc/man`.
https://github.com/restic/restic/issues/697
https://github.com/restic/restic/pull/1147
2017-08-18 17:50:32 +00:00
* The `key remove` command was corrected and now works as documented.
https://github.com/restic/restic/pull/1164
2017-08-28 19:13:14 +00:00
* When a restic command other than `init` is used with a local repository and
the repository directory does not exist, restic creates the directory
structure. That's an error, only the `init` command should create the dir.
https://github.com/restic/restic/issues/1167
https://github.com/restic/restic/pull/1182
2017-09-03 07:44:15 +00:00
* Restic now prints stats on all BSD systems (not only on darwin) when SIGINFO
is received (usually when ctrl+t is pressed).
https://github.com/restic/restic/pull/1203
https://github.com/restic/restic/pull/1082#issuecomment-326279920
2017-07-22 09:52:16 +00:00
2017-09-03 07:49:37 +00:00
* Since a few releases restic had the ability to write profiling files for
memory and CPU usage when `debug` is enabled. It was discovered that when
restic is interrupted (ctrl+c is pressed), the proper shutdown hook is not
run. This is now corrected.
https://github.com/restic/restic/pull/1191
* A new option `--exclude-caches` was added that allows excluding cache
directories (that are tagged as such). This is a special case of a more
generic option `--exclude-if-present` which excludes a directory if a file
with a specific name (and contents) is present.
https://github.com/restic/restic/issues/317
https://github.com/restic/restic/pull/1170
2017-09-10 18:29:08 +00:00
https://github.com/restic/restic/pull/1224
2017-09-10 18:55:02 +00:00
* The `forget` command now has an option `--group-by` that allows flexible
grouping policies.
https://github.com/restic/restic/pull/1196
2017-09-11 17:02:36 +00:00
* The date and time restic records for a new backup can now be specified
externally by passing `--time` to the `backup` command.
https://github.com/restic/restic/pull/1205
2017-09-13 12:04:55 +00:00
* The option `--compact` was added to the `snapshots` command to get a better
overview of the snapshots in a repo. It limits each snapshot to a single
line.
https://github.com/restic/restic/issues/1218
https://github.com/restic/restic/pull/1223
2017-07-22 09:03:44 +00:00
Important Changes in 0.7.1
2017-06-03 12:10:28 +00:00
==========================
2017-07-02 09:17:07 +00:00
* The `migrate` command for chaning the `s3legacy` layout to the `default`
layout for s3 backends has been improved: It can now be restarted with
`restic migrate --force s3_layout` and automatically retries operations on
error.
https://github.com/restic/restic/issues/1073
https://github.com/restic/restic/pull/1075
2017-07-01 13:11:36 +00:00
2017-07-03 17:49:18 +00:00
Small changes
-------------
* The local and sftp backends now create the subdirs below `data/` on
open/init. This way, restic makes sure that they always exist. This is
connected to an issue for the sftp server:
https://github.com/restic/rest-server/pull/11#issuecomment-309879710
https://github.com/restic/restic/issues/1055
https://github.com/restic/restic/pull/1077
2017-07-16 13:11:26 +00:00
https://github.com/restic/restic/pull/1105
2017-07-03 17:49:18 +00:00
2017-07-05 18:54:37 +00:00
* When no S3 credentials are specified in the environment variables, restic
now tries to load credentials from an IAM instance profile when the s3
backend is used.
https://github.com/restic/restic/issues/1067
https://github.com/restic/restic/pull/1086
2017-07-09 19:43:05 +00:00
* On Darwin and FreeBSD, restic now prints stats when SIGINFO is received
(usually when ctrl+t is pressed).
https://github.com/restic/restic/pull/1082
2017-07-17 20:02:06 +00:00
* The dependencies have been updated.
https://github.com/restic/restic/pull/1108
2017-07-22 09:03:44 +00:00
https://github.com/restic/restic/pull/1124
2017-07-17 20:02:06 +00:00
2017-07-19 15:02:53 +00:00
* A bug was found (and corrected) in the index rebuilding after prune, which
led to indexes which include blobs that were not present in the repo any
more. There were already checks in place which detected this situation and
aborted with an error message. A new run of either `prune` or
`rebuild-index` corrected the index files. This is now fixed and a test has
been added to detect this.
https://github.com/restic/restic/pull/1115
2017-07-19 15:09:02 +00:00
* Errors for chmod() on Unix for filesystems which do not support it (e.g. smb
mounted via gvfs) are now ignored.
https://github.com/restic/restic/pull/1080
https://github.com/restic/restic/pull/1112
2017-07-03 17:49:18 +00:00
2017-07-16 13:24:27 +00:00
* The semantic for the `--tags` option to `forget` and `snapshots` was
clarified:
https://github.com/restic/restic/issues/1081
https://github.com/restic/restic/pull/1090
2017-07-03 17:49:18 +00:00
2017-07-01 13:11:36 +00:00
Important Changes in 0.7.0
==========================
2017-05-26 10:16:47 +00:00
* New "swift" backend: A new backend for the OpenStack Swift cloud storage
protocol has been added, https://wiki.openstack.org/wiki/Swift
https://github.com/restic/restic/pull/975
https://github.com/restic/restic/pull/648
2017-05-28 11:12:08 +00:00
* New "b2" backend: A new backend for Backblaze B2 cloud storage
service has been added, https://www.backblaze.com
https://github.com/restic/restic/issues/512
https://github.com/restic/restic/pull/978
2017-06-04 09:50:38 +00:00
* Improved performance for the `find` command: Restic recognizes paths it has
already checked for the files in question, so the number of backend requests
is reduced a lot.
https://github.com/restic/restic/issues/989
https://github.com/restic/restic/pull/993
2017-06-07 18:51:08 +00:00
* Improved performance for the fuse mount: Listing directories which contain
large files now is significantly faster.
https://github.com/restic/restic/pull/998
2017-06-08 17:21:52 +00:00
* The default layout for the s3 backend is now `default` (instead of
`s3legacy`). Also, there's a new `migrate` command to convert an existing
repo, it can be run like this: `restic migrate s3_layout`
https://github.com/restic/restic/issues/965
https://github.com/restic/restic/pull/1004
2017-06-18 19:18:11 +00:00
* The fuse mount now has two more directories: `tags` contains a subdir for
each tag, which in turn contains only the snapshots that have this tag. The
subdir `hosts` contains a subdir for each host that has a snapshot, and the
subdir contains the snapshots for that host.
https://github.com/restic/restic/issues/636
https://github.com/restic/restic/pull/1050
2017-06-12 17:56:50 +00:00
Small changes
-------------
* For the s3 backend we're back to using the high-level API the s3 client
library for uploading data, a few users reported dropped connections (which
the library will automatically retry now).
https://github.com/restic/restic/issues/1013
https://github.com/restic/restic/issues/1023
https://github.com/restic/restic/pull/1025
2017-06-16 10:27:44 +00:00
* The `prune` command has been improved and will now remove invalid pack
files, for example files that have not been uploaded completely because a
backup was interrupted.
https://github.com/restic/restic/issues/1029
https://github.com/restic/restic/pull/1036
2017-06-30 20:15:00 +00:00
* restic now tries to detect when an invalid/unknown backend is used and
returns an error message.
https://github.com/restic/restic/issues/1021
https://github.com/restic/restic/pull/1070
2017-05-31 21:50:48 +00:00
Important Changes in 0.6.1
2017-05-25 15:06:06 +00:00
==========================
2017-05-31 21:50:48 +00:00
This is mostly a bugfix release and only contains small changes:
2017-05-25 15:06:06 +00:00
2017-05-31 19:33:45 +00:00
* We've fixed a bug where `rebuild-index` would corrupt the index when used
with the s3 backend together with the `default` layout. This is not the
default setting.
2017-05-31 17:53:54 +00:00
* Backends based on HTTP now allow several idle connections in parallel. This
is especially important for the REST backend, which (when used with a local
server) may create a lot connections and exhaust available ports quickly.
https://github.com/restic/restic/issues/985
https://github.com/restic/restic/pull/986
2017-05-25 15:06:06 +00:00
* Regular status report: We've removed the status report that was printed
every 10 seconds when restic is run non-interactively. You can still force
reporting the current status by sending a `USR1` signal to the process.
https://github.com/restic/restic/pull/974
2017-05-29 21:46:48 +00:00
* The `build.go` now strips the temporary directory used for compilation from
the binary. This is the first step in enabling reproducible builds.
https://github.com/restic/restic/pull/981
2017-05-25 13:42:44 +00:00
Important Changes in 0.6.0
2017-05-25 13:26:22 +00:00
==========================
2017-05-25 13:26:22 +00:00
Consistent forget policy
------------------------
The `forget` command was corrected to be more consistent in which snapshots are
to be forgotten. It is possible that the new code removes more snapshots than
before, so please review what would be deleted by using the `--dry-run` option.
https://github.com/restic/restic/pull/957
https://github.com/restic/restic/issues/953
Unified repository layout
-------------------------
Up to now the s3 backend used a special repository layout. We've decided to
unify the repository layout and implemented the default layout also for the s3
backend. For creating a new repository on s3 with the default layout, use
`restic -o s3.layout=default init`. For further commands the option is not
necessary any more, restic will automatically detect the correct layout to use.
A future version will switch to the default layout for new repositories.
https://github.com/restic/restic/pull/966
https://github.com/restic/restic/issues/965
Memory and time improvements for the s3 backend
-----------------------------------------------
We've updated the library used for accessing s3, switched to using a lower
level API and added caching for some requests. This lead to a decrease in
memory usage and a great speedup. In addition, we added benchmark functions for
all backends, so we can track improvements over time. The Continuous
Integration test service we're using (Travis) now runs the s3 backend tests not
only against a Minio server, but also against the Amazon s3 live service, so we
should be notified of any regressions much sooner.
https://github.com/restic/restic/pull/962
https://github.com/restic/restic/pull/960
https://github.com/restic/restic/pull/946
https://github.com/restic/restic/pull/938
https://github.com/restic/restic/pull/883