If no specific Region is mentioned in RESTIC_REPOSITORY, AWS defaults to
us-east-1. For this reason, users that follow the tutorial and create
their S3 bucket in any other region get the following error:
"Fatal: create repository at [...] client.BucketExists"
Explicitly specifying the AWS region name fixes the issue.
The new option allows prune to operate with nearly no scratch space by only removing
no longer necessary pack files and first deleting the index before
rebuilding it. By first deleting the index it becomes safe to just
delete no longer necessary pack files. However, as a downside there's
now the risk that the repository becomes inaccessible if prune fails.
To recover from that problem a user might have to manually delete the
repository index and then run (a full) `rebuild-index` again.
Removing data based on a policy when the attacker had the opportunity to
add data to your repository comes with some considerations. This is
added to the 060_forget.rst documentation.
That document is also updated to reflect that restic now considers
the current system time while running "forget".
References to the security considerations section are added:
- In `restic forget --help`
- In the threat model (design.rst)
- In the (030) setup section where an append-only setup is referenced
A reference is also to be added to the `rest-server` readme's
append-only paragraph (see my fork).
This commit also resolves a typo (amount->number for countable noun),
changes a password length recommendation into the metric that
actually matters when creating passwords (entropy) since I was editing
these doc files anyway, and updates the outdated copyright year in
`conf.py`.
Some wording in 060_forget (line 21..22) was changed to clarify what
"forget" and "prune" do, to try and avoid the apparent misconception
that "forget" does not remove any data.
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.
Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:
~/work
!~/work/*.c
But:
~/work/*
!~/work/*.c
I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:
$HOME/**/*
!$HOME/Documents
!$HOME/code
!$HOME/.emacs.d
!$HOME/games
# [...]
node_modules
*~
*.o
*.lo
*.pyc
# [...]
$HOME/code/linux/*
!$HOME/code/linux/.git
# [...]
There are some limitations for this change:
- Patterns are not mixed accross methods: patterns from file are
handled first and if a file is excluded with this method, it's not
possible to reinclude it with `--exclude !something`.
- Patterns starting with `!` are now interpreted as a negative
pattern. I don't think anyone was relying on that.
- The whole list of patterns is walked for each match. We may
optimize later by exiting early if we know no pattern is starting
with `!`.
Fix #233
The keyfile provided by restic's own webserver (https://restic.net) should be
more stable than relying on public keyservers. So I changed the URL to the
GPG keyfile, as recommended by MichaelEischer.
Per Amazon's product page [1], S3 is officially called "Amazon S3". The
restic project uses the phrase "AWS S3" in some places. This patch
corrects the product name.
[1]:https://aws.amazon.com/s3/
Dependencies are fetched at build time and stored in the GOPATH. These paths end up being in the final binary.
Bump restic version to latest and go version to the 1.16.6, which was used to build restic 0.12.1.
This can be used to check how large a backup is or validate exclusions.
It does not actually write any data to the underlying backend. This is
implemented as a simple overlay backend that accepts writes without
forwarding them, passes through reads, and generally does the minimal
necessary to pretend that progress is actually happening.
Fixes #1542
Example usage:
$ restic -vv --dry-run . | grep add
new /changelog/unreleased/issue-1542, saved in 0.000s (350 B added)
modified /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added)
modified /cmd/restic/global.go, saved in 0.000s (0 B added)
new /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added)
new /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added)
modified /internal/backend/test/tests.go, saved in 0.000s (0 B added)
modified /internal/repository/repository.go, saved in 0.000s (20.707 KiB added)
modified /internal/ui/backup.go, saved in 0.000s (9.110 KiB added)
modified /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added)
modified /restic, saved in 0.131s (25.542 MiB added)
Would add to the repo: 25.892 MiB
Allow keeping hourly/daily/weekly/monthly/yearly snapshots for a given time period.
This adds the following flags/parameters to restic forget:
--keep-within-hourly duration
--keep-within-daily duration
--keep-within-weekly duration
--keep-within-monthly duration
--keep-within-yearly duration
Includes following changes:
- Add tests for --keep-within-hourly (and friends)
- Add documentation for --keep-within-hourly (and friends)
- Add changelog for --keep-within-hourly (and friends)
The `init` and `copy` commands can now use `--repository-file2` flag and
the `$RESTIC_REPOSITORY_FILE2` environment variable.
This also fixes the conflict with the `--repository-file` and `--repo2`
flag.
Tests are added for the initSecondaryGlobalOpts function.
This adds a NOK function to the test helper functions. This NOK tests if
err is not nil, and otherwise fail the test.
With the NOK function a couple of sad paths are tested in the
initSecondaryGlobalOpts function.
In total the tests checks wether the following are passed correct:
- Password
- PasswordFile
- Repo
- RepositoryFile
The following situation must return an error to pass the test:
- no Repo or RepositoryFile defined
- Repo and RepositoryFile defined both
I like the idea of verifying the integrity of applications, I download from the internet. So I was very happy to see that restic does provide SHA256-checksums which are signed with the maintainers PGP key.
The only thing I miss: I could not find a direct way to download the used PGP key and verify the keys fingerprint.
Doing some searches, I found:
* https://github.com/restic/rest-server/issues/121
* https://restic.net/blog/2015-09-16/verifying-code-archive-integrity/
To help other restic users, I think you should add information about your PGP key/fingerprint to this installation doc, too. To save you some precious time, I created a draft, how this doc might be expanded, in this pull-request. You are free to accept it or change the text to your liking.
I copied the key/fingerprint text from: ``restic/restic/master/doc/090_participating.rst``
Thank you for your work in restic!
This adds support for the following environment variables, which were
previously missing:
OS_USER_ID User ID for keystone v3 authentication
OS_USER_DOMAIN_ID User domain ID for keystone v3 authentication
OS_PROJECT_DOMAIN_ID Project domain ID for keystone v3 authentication
OS_TRUST_ID Trust ID for keystone v3 authentication
Add a callback to the PruneOptions struct which calculates the number of
bytes allowed to be unused after prune is done. This way, the logic is
closer to the option parsing code.
Also, add an explicit option `unlimited` for the use case when storage
does not matter but bandwidth and time do. Internally, this sets the
maximum number of unused bytes to MaxUint64.
Rework the documentation slightly so that no more "packs" are
mentioned and it talks about "files" instead.
Make it clear in the documentation that the percentage given to
`--max-unused` is relative to the whole repository size after pruning is
done. If specified, it must be below 100%, otherwise the repository
would contain 100% of unused data, which is pointless.
I had a hard time coming up with the correct formula to calculate the
maximum number of unused bytes based on the number of used bytes. For a
fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of
unused bytes `x` the following holds:
x ≤ p * (u+x)
⇔ x ≤ p*u + p*x
⇔ x - p*x ≤ p*u
⇔ x * (1-p) ≤ p*u
⇔ x ≤ p/(1-p) * u
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.
Snapshotting volumes also covers mountpoints but skips UNC paths.
Cache locations were documented inconsistently in three places.
The backup docs mentioned PATH being used to find fusermount, which is
never run by restic backup. It now mentions ssh and rclone, which are
used by backends.
The notion of a "system-wide" environment variable makes no sense.
TMPDIR is now mentioned because it allows for optimization and may
have security implications.
As an alternative to -r, this allows to read the repository URL
from a file in order to prevent certain types of information leaks,
especially for URLs containing credentials.
Fixes #1458, fixes #2900.
This allows creating multiple repositories with identical chunker
parameters which is required for working deduplication when copying
snapshots between different repositories.
The standard UNIX-style ordering of command-line arguments places
optional flags before other positional arguments. All of restic's
commands support this ordering, but some of the usage strings showed the
flags after the positional arguments (which restic also parses just
fine). This change updates the doc strings to reflect the standard
ordering.
Because the `restic help` command comes directly from Cobra, there does
not appear to be a way to update the argument ordering in its usage
string, so it maintains the non-standard ordering (positional arguments
before optional flags).
In the Google Cloud Storage backend, support specifying access tokens
directly, as an alternative to a credentials file. This is useful when
restic is used non-interactively by some other program that is already
authenticated and eliminates the need to store long lived credentials.
The access token is specified in the GOOGLE_ACCESS_TOKEN environment
variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.
The backup command used to return a zero exit code as long as a snapshot
could be created successfully, even if some of the source files could not
be read (in which case the snapshot would contain the rest of the files).
This made it hard for automation/scripts to detect failures/incomplete
backups by looking at the exit code. Restic now returns the following exit
codes for the backup command:
- 0 when the command was successful
- 1 when there was a fatal error (no snapshot created)
- 3 when some source data could not be read (incomplete snapshot created)
Changes proposed in #2763:
- Adding `RESTIC_CACHE_DIR` environment variables (introduced in #2425 for Unix and #2607 for Mac, Win).
- Adding used system-wide environment variables with links to the corresponding section.
That site might not have supported https:// when those links were
originally added. It does now.
Also dropping the _spec.html_ ending of the url, there being a `<link
rel="canonical" ...>` tag suggesting that that no longer being the
preferred address.
On Linux CIFS (SMB) seems to be incompatible with the async preemption
implementation of Go 1.14. CIFS seems not to restart syscalls (open,
read, chmod, readdir, ...) as expected by Go, which sets SA_RESTART for
its signal handler to have syscalls restarted automatically. This leads
to Go passing up lots of EINTR return codes to restic.
See https://github.com/restic/restic/issues/2659 for a detailed explanation.
The favicon on restic.readthedocs.io still contained the old
superman-style logo.
This changes it to the 32x32 gopher icon also used on https://forum.restic.net/ ,
just converted to .ico using Gimp.
Tested by building the documentation and opening index.html in Chrome.
The new favicon looks fine.
The environment variable RESTIC_PASSWORD_COMMAND works but has
not been documented yet. e.g. it could contain a command that
would fetch the password from a local user keyring
enhances: https://github.com/restic/restic/pull/2094
The help text for `restic stats` lists a number of modes in a list.
Make sure the "more info" text is a separate paragraph rather than
being part of the list.
With this change it is possible to dump a folder to stdout as a tar. The
It can be used just like the normal dump command:
`./restic dump fa97e6e1 "/data/test/" > test.tar`
Where `/data/test/` is a a folder instead of a file.
This patch makes it more explicit what is meant by the CACHEDIR.TAG file.
It not only has to have this particular name, but also a specific content
(described at http://bford.info/cachedir/spec.html), which is not immediately
obvious to the user.
I went pretty loud with this, but I think the performance is bad enough
that it's really worth highlighting, especially since it locks the index
during the prune.
Adds a small warning indicating that b2 bucket names need to be unique. It's an easy mistake to make, and it's surprising to get the following error if you're not accustomed to the way B2 works:
Fatal: create repository at b2:postgres failed: NewBucket: b2_create_bucket: 400: Bucket name is already in use
When I backup one of my filesystems which has a lot of Hard Links (Backup directory of burp) the live status shows me 4.5 TB but it only takes up 1.2 TB of space in the repository. This is confusing because my repo is on S3 and I feared a huge Bill. This change should clarify this.
