octoleo
/
restic
mirror of https://github.com/octoleo/restic.git
2
2
Fork 0
Code Issues Releases Wiki Activity

Generate CHANGELOG.md for 0.14.0

This commit is contained in:
Alexander Neumann 2022-08-25 19:54:02 +02:00
parent a825e0d409
commit 193c62dfc3
1 changed files with 450 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

450
CHANGELOG.md
View File

@ -1,3 +1,453 @@
Changelog for restic 0.14.0 (2022-08-25)
=======================================
The following sections list the changes in restic 0.14.0 relevant to
restic users. The changes are ordered by importance.
Summary
-------
* Fix #2248: Support `self-update` on Windows
* Fix #3428: List snapshots in backend at most once to resolve snapshot IDs
* Fix #3432: Fix rare 'not found in repository' error for `copy` command
* Fix #3685: The `diff` command incorrectly listed some files as added
* Fix #3681: Fix rclone (shimmed by Scoop) and sftp not working on Windows
* Fix #3720: Directory sync errors for repositories accessed via SMB
* Fix #3736: The `stats` command miscalculated restore size for multiple snapshots
* Fix #3861: Yield error on invalid policy to `forget`
* Fix #3716: Print "wrong password" to stderr instead of stdout
* Fix #3772: Correctly rebuild index for legacy repositories
* Fix #3776: Limit number of key files tested while opening a repository
* Chg #1842: Support debug log creation in release builds
* Chg #3295: Deprecate `check --check-unused` and add further checks
* Chg #3680: Update dependencies and require Go 1.15 or newer
* Chg #3742: Replace `--repo2` option used by `init`/`copy` with `--from-repo`
* Enh #1153: Support pruning even when the disk is full
* Enh #21: Add compression support
* Enh #2162: Adaptive IO concurrency based on backend connections
* Enh #2291: Allow pack size customization
* Enh #2295: Allow use of SAS token to authenticate to Azure
* Enh #2696: Improve backup speed with many small files
* Enh #2907: Make snapshot directory structure of `mount` command customizable
* Enh #3114: Optimize handling of duplicate blobs in `prune`
* Enh #3465: Improve handling of temporary files on Windows
* Enh #3709: Validate exclude patterns before backing up
* Enh #3837: Improve SFTP repository initialization over slow links
* Enh #2351: Use config file permissions to control file group access
* Enh #3475: Allow limiting IO concurrency for local and SFTP backend
* Enh #3484: Stream data in `check` and `prune` commands
* Enh #2923: Improve speed of `copy` command
* Enh #3729: Display full IDs in `check` warnings
* Enh #3773: Optimize memory usage for directories with many files
* Enh #3819: Validate include/exclude patterns before restoring
Details
-------
* Bugfix #2248: Support `self-update` on Windows
Restic `self-update` would fail in situations where the operating system locks running
binaries, including Windows. The new behavior works around this by renaming the running file
and swapping the updated file in place.
https://github.com/restic/restic/issues/2248
https://github.com/restic/restic/pull/3675
* Bugfix #3428: List snapshots in backend at most once to resolve snapshot IDs
Many commands support specifying a list of snapshot IDs which are then used to determine the
snapshots to be processed by the command. To resolve snapshot IDs or `latest`, and check that
these exist, restic previously listed all snapshots stored in the repository. Depending on
the backend this could be a slow and/or expensive operation.
Restic now lists the snapshots only once and remembers the result in order to resolve all
further snapshot IDs swiftly.
https://github.com/restic/restic/issues/3428
https://github.com/restic/restic/pull/3570
https://github.com/restic/restic/pull/3395
* Bugfix #3432: Fix rare 'not found in repository' error for `copy` command
In rare cases `copy` (and other commands) would report that `LoadTree(...)` returned an `id
[...] not found in repository` error. This could be caused by a backup or copy command running
concurrently. The error was only temporary; running the failed restic command a second time as
a workaround did resolve the error.
This issue has now been fixed by correcting the order in which restic reads data from the
repository. It is now guaranteed that restic only loads snapshots for which all necessary data
is already available.
https://github.com/restic/restic/issues/3432
https://github.com/restic/restic/pull/3570
* Bugfix #3685: The `diff` command incorrectly listed some files as added
There was a bug in the `diff` command, causing it to always show files in a removed directory as
added. This has now been fixed.
https://github.com/restic/restic/issues/3685
https://github.com/restic/restic/pull/3686
* Bugfix #3681: Fix rclone (shimmed by Scoop) and sftp not working on Windows
In #3602 a fix was introduced to address the problem of `rclone` prematurely exiting when
Ctrl+C is pressed on Windows. The solution was to create the subprocess with its console
detached from the restic console.
However, this solution failed when using `rclone` installed by Scoop or using `sftp` with a
passphrase-protected private key. We've now fixed this by using a different approach to
prevent Ctrl-C from passing down too early.
https://github.com/restic/restic/issues/3681
https://github.com/restic/restic/issues/3692
https://github.com/restic/restic/pull/3696
* Bugfix #3720: Directory sync errors for repositories accessed via SMB
On Linux and macOS, accessing a repository via a SMB/CIFS mount resulted in restic failing to
save the lock file, yielding the following errors:
Save(<lock/071fe833f0>) returned error, retrying after 552.330144ms: sync /repo/locks:
no such file or directory Save(<lock/bf789d7343>) returned error, retrying after
552.330144ms: sync /repo/locks: invalid argument
This has now been fixed by ignoring the relevant error codes.
https://github.com/restic/restic/issues/3720
https://github.com/restic/restic/issues/3751
https://github.com/restic/restic/pull/3752
* Bugfix #3736: The `stats` command miscalculated restore size for multiple snapshots
Since restic 0.10.0 the restore size calculated by the `stats` command for multiple snapshots
was too low. The hardlink detection was accidentally applied across multiple snapshots and
thus ignored many files. This has now been fixed.
https://github.com/restic/restic/issues/3736
https://github.com/restic/restic/pull/3740
* Bugfix #3861: Yield error on invalid policy to `forget`
The `forget` command previously silently ignored invalid/unsupported units in the duration
options, such as e.g. `--keep-within-daily 2w`.
Specifying an invalid/unsupported duration unit now results in an error.
https://github.com/restic/restic/issues/3861
https://github.com/restic/restic/pull/3862
* Bugfix #3716: Print "wrong password" to stderr instead of stdout
If an invalid password was entered, the error message was printed on stdout and not on stderr as
intended. This has now been fixed.
https://github.com/restic/restic/pull/3716
https://forum.restic.net/t/4965
* Bugfix #3772: Correctly rebuild index for legacy repositories
After running `rebuild-index` on a legacy repository containing mixed pack files (that is,
pack files which store both metadata and file data), `check` printed warnings like `pack
12345678 contained in several indexes: ...`. This warning was not critical, but has now
nonetheless been fixed by properly handling mixed pack files while rebuilding the index.
Running `prune` for such legacy repositories will also fix the warning by reorganizing the
pack files which caused it.
https://github.com/restic/restic/pull/3772
https://github.com/restic/restic/pull/3884
https://forum.restic.net/t/5044/13
* Bugfix #3776: Limit number of key files tested while opening a repository
Previously, restic tested the password against every key in the repository when opening a
repository. The more keys there were in the repository, the slower this operation became.
Restic now tests the password against up to 20 key files in the repository. Alternatively, you
can use the `--key-hint=<key ID>` option to specify a specific key file to use instead.
https://github.com/restic/restic/pull/3776
* Change #1842: Support debug log creation in release builds
Creating a debug log was only possible in debug builds which required users to manually build
restic. We changed the release builds to allow creating debug logs by simply setting the
environment variable `DEBUG_LOG=logname.log`.
https://github.com/restic/restic/issues/1842
https://github.com/restic/restic/pull/3826
* Change #3295: Deprecate `check --check-unused` and add further checks
Since restic 0.12.0, it is expected to still have unused blobs after running `prune`. This made
the `--check-unused` option of the `check` command rather useless and tended to confuse
users. This option has been deprecated and is now ignored.
The `check` command now also warns if a repository is using either the legacy S3 layout or mixed
pack files with both tree and data blobs. The latter is known to cause performance problems.
https://github.com/restic/restic/issues/3295
https://github.com/restic/restic/pull/3730
* Change #3680: Update dependencies and require Go 1.15 or newer
We've updated most dependencies. Since some libraries require newer language features we're
dropping support for Go 1.14, which means that restic now requires at least Go 1.15 to build.
https://github.com/restic/restic/issues/3680
https://github.com/restic/restic/issues/3883
* Change #3742: Replace `--repo2` option used by `init`/`copy` with `--from-repo`
The `init` and `copy` commands can read data from another repository. However, confusingly
`--repo2` referred to the repository *from* which the `init` command copies parameters, but
for the `copy` command `--repo2` referred to the copy *destination*.
We've introduced a new option, `--from-repo`, which always refers to the source repository
for both commands. The old parameter names have been deprecated but still work. To create a new
repository and copy all snapshots to it, the commands are now as follows:
``` restic -r /srv/restic-repo-copy init --from-repo /srv/restic-repo
--copy-chunker-params restic -r /srv/restic-repo-copy copy --from-repo
/srv/restic-repo ```
https://github.com/restic/restic/pull/3742
https://forum.restic.net/t/5017
* Enhancement #1153: Support pruning even when the disk is full
When running out of disk space it was no longer possible to add or remove data from a repository.
To help with recovering from such a deadlock, the prune command now supports an
`--unsafe-recover-no-free-space` option to recover from these situations. Make sure to
read the documentation first!
https://github.com/restic/restic/issues/1153
https://github.com/restic/restic/pull/3481
* Enhancement #21: Add compression support
We've added compression support to the restic repository format. To create a repository using
the new format run `init --repository-version 2`. Please note that the repository cannot be
read by restic versions prior to 0.14.0.
You can configure whether data is compressed with the option `--compression`. It can be set to
`auto` (the default, which will compress very fast), `max` (which will trade backup speed and
CPU usage for better compression), or `off` (which disables compression). Each setting is
only applied for the current run of restic and does *not* apply to future runs. The option can
also be set via the environment variable `RESTIC_COMPRESSION`.
To upgrade in place run `migrate upgrade_repo_v2` followed by `prune`. See the documentation
for more details. The migration checks the repository integrity and upgrades the repository
format, but will not change any data. Afterwards, prune will rewrite the metadata to make use of
compression.
As an alternative you can use the `copy` command to migrate snapshots; First create a new
repository using `init --repository-version 2 --copy-chunker-params --repo2
path/to/old/repo`, and then use the `copy` command to copy all snapshots to the new
repository.
https://github.com/restic/restic/issues/21
https://github.com/restic/restic/issues/3779
https://github.com/restic/restic/pull/3666
https://github.com/restic/restic/pull/3704
https://github.com/restic/restic/pull/3733
* Enhancement #2162: Adaptive IO concurrency based on backend connections
Many commands used hard-coded limits for the number of concurrent operations. This prevented
speed improvements by increasing the number of connections used by a backend.
These limits have now been replaced by using the configured number of backend connections
instead, which can be controlled using the `-o <backend-name>.connections=5` option.
Commands will then automatically scale their parallelism accordingly.
To limit the number of CPU cores used by restic, you can set the environment variable
`GOMAXPROCS` accordingly. For example to use a single CPU core, use `GOMAXPROCS=1`.
https://github.com/restic/restic/issues/2162
https://github.com/restic/restic/issues/1467
https://github.com/restic/restic/pull/3611
* Enhancement #2291: Allow pack size customization
Restic now uses a target pack size of 16 MiB by default. This can be customized using the
`--pack-size size` option. Supported pack sizes range between 4 and 128 MiB.
It is possible to migrate an existing repository to _larger_ pack files using `prune
--repack-small`. This will rewrite every pack file which is significantly smaller than the
target size.
https://github.com/restic/restic/issues/2291
https://github.com/restic/restic/pull/3731
* Enhancement #2295: Allow use of SAS token to authenticate to Azure
Previously restic only supported AccountKeys to authenticate to Azure storage accounts,
which necessitates giving a significant amount of access.
We added support for Azure SAS tokens which are a more fine-grained and time-limited manner of
granting access. Set the `AZURE_ACCOUNT_NAME` and `AZURE_ACCOUNT_SAS` environment
variables to use a SAS token for authentication. Note that if `AZURE_ACCOUNT_KEY` is set, it
will take precedence.
https://github.com/restic/restic/issues/2295
https://github.com/restic/restic/pull/3661
* Enhancement #2696: Improve backup speed with many small files
We have restructured the backup pipeline to continue reading files while all upload
connections are busy. This allows the backup to already prepare the next data file such that the
upload can continue as soon as a connection becomes available. This can especially improve the
backup performance for high latency backends.
The upload concurrency is now controlled using the `-o <backend-name>.connections=5`
option.
https://github.com/restic/restic/issues/2696
https://github.com/restic/restic/pull/3489
* Enhancement #2907: Make snapshot directory structure of `mount` command customizable
We've added the possibility to customize the snapshot directory structure of the `mount`
command using templates passed to the `--snapshot-template` option. The formatting of
snapshots' timestamps is now controlled using `--time-template` and supports
subdirectories to for example group snapshots by year. Please see `restic help mount` for
further details.
Characters in tag names which are not allowed in a filename are replaced by underscores `_`. For
example a tag `foo/bar` will result in a directory name of `foo_bar`.
https://github.com/restic/restic/issues/2907
https://github.com/restic/restic/pull/2913
https://github.com/restic/restic/pull/3691
* Enhancement #3114: Optimize handling of duplicate blobs in `prune`
Restic `prune` always used to repack all data files containing duplicate blobs. This
effectively removed all duplicates during prune. However, as a consequence all these data
files were repacked even if the unused repository space threshold could be reached with less
work.
This is now changed and `prune` works nice and fast even when there are lots of duplicate blobs.
https://github.com/restic/restic/issues/3114
https://github.com/restic/restic/pull/3290
* Enhancement #3465: Improve handling of temporary files on Windows
In some cases restic failed to delete temporary files, causing the current command to fail.
This has now been fixed by ensuring that Windows automatically deletes the file. In addition,
temporary files are only written to disk when necessary, reducing disk writes.
https://github.com/restic/restic/issues/3465
https://github.com/restic/restic/issues/1551
https://github.com/restic/restic/pull/3610
* Enhancement #3709: Validate exclude patterns before backing up
Exclude patterns provided via `--exclude`, `--iexclude`, `--exclude-file` or
`--iexclude-file` previously weren't validated. As a consequence, invalid patterns
resulted in files that were meant to be excluded being backed up.
Restic now validates all patterns before running the backup and aborts with a fatal error if an
invalid pattern is detected.
https://github.com/restic/restic/issues/3709
https://github.com/restic/restic/pull/3734
* Enhancement #3837: Improve SFTP repository initialization over slow links
The `init` command, when used on an SFTP backend, now sends multiple `mkdir` commands to the
backend concurrently. This reduces the waiting times when creating a repository over a very
slow connection.
https://github.com/restic/restic/issues/3837
https://github.com/restic/restic/pull/3840
* Enhancement #2351: Use config file permissions to control file group access
Previously files in a local/SFTP repository would always end up with very restrictive access
permissions, allowing access only to the owner. This prevented a number of valid use-cases
involving groups and ACLs.
We now use the permissions of the config file in the repository to decide whether group access
should be given to newly created repository files or not. We arrange for repository files to be
created group readable exactly when the repository config file is group readable.
To opt-in to group readable repositories, a simple `chmod -R g+r` or equivalent on the config
file can be used. For repositories that should be writable by group members a tad more setup is
required, see the docs.
Posix ACLs can also be used now that the group permissions being forced to zero no longer masks
the effect of ACL entries.
https://github.com/restic/restic/issues/2351
https://github.com/restic/restic/pull/3419
https://forum.restic.net/t/1391
* Enhancement #3475: Allow limiting IO concurrency for local and SFTP backend
Restic did not support limiting the IO concurrency / number of connections for accessing
repositories stored using the local or SFTP backends. The number of connections is now limited
as for other backends, and can be configured via the the `-o local.connections=2` and `-o
sftp.connections=5` options. This ensures that restic does not overwhelm the backend with
concurrent IO operations.
https://github.com/restic/restic/pull/3475
* Enhancement #3484: Stream data in `check` and `prune` commands
The commands `check --read-data` and `prune` previously downloaded data files into
temporary files which could end up being written to disk. This could cause a large amount of data
being written to disk.
The pack files are now instead streamed, which removes the need for temporary files. Please
note that *uploads* during `backup` and `prune` still require temporary files.
https://github.com/restic/restic/issues/3710
https://github.com/restic/restic/pull/3484
https://github.com/restic/restic/pull/3717
* Enhancement #2923: Improve speed of `copy` command
The `copy` command could require a long time to copy snapshots for non-local backends. This has
been improved to provide a throughput comparable to the `restore` command.
Additionally, `copy` now displays a progress bar.
https://github.com/restic/restic/issues/2923
https://github.com/restic/restic/pull/3513
* Enhancement #3729: Display full IDs in `check` warnings
When running commands to inspect or repair a damaged repository, it is often necessary to
supply the full IDs of objects stored in the repository.
The output of `check` now includes full IDs instead of their shortened variant.
https://github.com/restic/restic/pull/3729
* Enhancement #3773: Optimize memory usage for directories with many files
Backing up a directory with hundreds of thousands or more files caused restic to require large
amounts of memory. We've now optimized the `backup` command such that it requires up to 30% less
memory.
https://github.com/restic/restic/pull/3773
* Enhancement #3819: Validate include/exclude patterns before restoring
Patterns provided to `restore` via `--exclude`, `--iexclude`, `--include` and
`--iinclude` weren't validated before running the restore. Invalid patterns would result in
error messages being printed repeatedly, and possibly unwanted files being restored.
Restic now validates all patterns before running the restore, and aborts with a fatal error if
an invalid pattern is detected.
https://github.com/restic/restic/pull/3819
Changelog for restic 0.13.0 (2022-03-26)
=======================================