With this change it is possible to dump a folder to stdout as a tar. The
It can be used just like the normal dump command:
`./restic dump fa97e6e1 "/data/test/" > test.tar`
Where `/data/test/` is a a folder instead of a file.
This commit is a followup to the addition of the --group-by flag for the
snapshots command. Adding the grouping code there introduced duplicated
code (the forget command also does grouping). This commit refactors
boths sides to only use shared code.
This commit moves the code which is used to group snapshots in the
snapshots command into an own function to deduplicate code shared by the
snapshots command and forget command.
This commit will add json tags to the structs for json output, so all
json variables of the snapshot command output are lowercase and
snake-case.
Furthermore it adds some internal code changes based on the feedback in
the pull request #2087.
This commit adds a --group-by option to the snapshots command, which
behaves similar to the --group-by option of forget. Valid option values
are "host, paths, tags". If this option is given, the output of
snapshots will be divided into multiple tables, according to the value
given (i.e. "host" will create a table of snapshots for each host, that
has a snapshot in the list). Also the JSON output will be grouped.
The default behavior (when --group-by is not given) has not changed.
More to this discussion can be found in issue #2037.
Reading the password from non-terminal stdin used io.ReadFull with a
byte slice of length 1000.
We are now using a Scanner to read one line of input, independent of its
length.
Additionally, if stdin is not a terminal, the password is read only
once instead of twice (in an effort to detect typos).
Fixes #2203
Signed-off-by: Peter Schultz <peter.schultz@classmarkets.com>
This commit changes the signatures for repository.LoadAndDecrypt and
utils.LoadAll to allow passing in a []byte as the buffer to use. This
buffer is enlarged as needed, and returned back to the caller for
further use.
In later commits, this allows reducing allocations by reusing a buffer
for multiple calls, e.g. in a worker function.
This patch makes it more explicit what is meant by the CACHEDIR.TAG file.
It not only has to have this particular name, but also a specific content
(described at http://bford.info/cachedir/spec.html), which is not immediately
obvious to the user.
This adds a test of the json output of the forget command, by running it
once, asking it to keep one snapshot, and verifying that the output has
the right number of snapshots listed in the Keep and Remove fields of
the result.
This commit changes the logic slightly: checking the permissions in the
fuse mount when nobody else besides the current user can access the fuse
mount does not sense. The current user has access to the repo files in
addition to the password, so they can access all data regardless of what
the fuse mount does.
Enabling `--allow-root` allows the root user to access the files in the
fuse mount, for this user no permission checks will be done anyway.
The code now enables `DefaultPermissions` automatically when
`--allow-other` is set, it can be disabled with
`--no-default-permissions` to restore the old behavior.
This option restores the previous behavior of `mount` by disabling the "DefaultPermissions" FUSE option. This allows any user that can access the mountpoint to read any file from the snapshot. Normal FUSE rules apply, so `allow-root` or `allow-other` can be used to allow users besides the mounting user to access these files.
This enforces the Unix permissions of the snapshot files within the mounted filesystem, which will only allow users to access snapshot files if they had access to the file outside of the snapshot.
Make restic forget --keep-within accept time ranges measured in hours and choose
accordingly which snapshots to keep and which to forget. Add relative tests.
The default value of the `--host` flag was set to 'H' (the shorthand
version of the flag), this caused the snapshot lookup to fail.
Also add shorthand `-H` for `backup` command.
Closes #2040
Some time ago we changed the paths in the repo to always use a slash for
separation, it seems we missed that the `dump` command still uses the
`filepath` package, so on Windows backslashes are used.
Closes #2079
reworked restore error callback to use file location
path instead of much heavier Node. this reduced restore
memory usage by as much as 50% in some of my tests.
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
Sometimes, users run restic without retaining the local cache
directories. This was reported several times in the past.
Restic will now print a message whenever a new cache directory is
created from scratch (i.e. it did not exist before), so users have a
chance to recognize when the cache is not kept between different runs of
restic.
With --blob, --tree and --pack, the find command now lists the snapshots
that contain a specific tree or blob, or the snapshots that contain
blobs belonging to a given pack.
It also displays the pack ID a blob belongs to.
A list of IDs can be given, as long as the IDs are all of the same type.
This commit adds a command called `self-update` which downloads the
latest released version of restic from GitHub and replacing the current
binary with it. It does not rely on any external program (so it'll work
everywhere), but still verifies the GPG signature using the embedded GPG
public key.
By default, the `self-update` command is hidden behind the `selfupdate`
built tag, which is only set when restic is built using `build.go`. The
reason for this is that downstream distributions will then not include
the command by default, so users are encouraged to use the
platform-specific distribution mechanism.
This commit introduces two functions: withinDir() and
approachingMatchingTree()
Both bind the list of directories with a closure, so we don't need to
iterate over the list in the function passed to Walk(). This reduces the
indentation level and since we can just use return, we don't need the
breaks any more.
The case that len(dirs) == 0 can also be handled by the functions with a
return, which saves another indentation level.
The main function body of the function passed to Walk() was reduced to
three cases:
* Within one of the dirs: Print the node, and if recursive operation is
requested, directly return, so the walker continues recursive
traversal
* Approaching one of the dirs: don't print anything, but continue
recursive traversal.
* Nothing of the two: abort walking this branch of the tree.
Adds a SelectByName method to the archive and scanner which only require
the filename as input, and can thus be run before calling lstat on the
file. Can speed up scanning significantly if a lot of filename excludes
are used.
Before:
$ restic list
Fatal: type not specified
After:
$ restic list
Fatal: type not specified, usage: list [blobs|packs|index|snapshots|keys|locks]
Closes #1783
This patch should fix the following panic when trying to backup the
root filesystem with thre --one-file-system flag:
% restic backup --one-file-system /
(...)
panic: item /, device id 2082 not found, allowedDevs: map[/:2082]
This commit changes how the worker goroutines for saving e.g. blobs
interact. Before, it was possible to get stuck sending an instruction to
archive a file or dir when no worker goroutines were available any more.
This commit introduces a `done` channel for each of the worker pools,
which is set to the channel returned by `tomb.Dying()`, so it is closed
when the first worker returned an error.
This change removes the hardcoded Google auth mechanism for the GCS
backend, instead using Google's provided client library to discover and
generate credential material.
Google recommend that client libraries use their common auth mechanism
in order to authorise requests against Google services. Doing so means
you automatically support various types of authentication, from the
standard GOOGLE_APPLICATION_CREDENTIALS environment variable to making
use of Google's metadata API if running within Google Container Engine.
In #1590, it was mentioned that while lines read from exclude files via
`--exclude-file` have leading and trailing spaces stripped, this is not
the case for lines read via `--files-from`. This commit fixes that,
spaces are always stripped.
This commit removes the bandwidth displayed during backup process. It is
misleading and seldomly correct, because it's neither the "read
bandwidth" (only for the very first backup) nor the "upload bandwidth".
Many users are confused about (and rightly so), c.f. #1581, #1033, #1591
We'll eventually replace this display with something more relevant when
#1494 is done.
During the development of #1524 I discovered that the Google Cloud
Storage backend did not yet use the HTTP transport, so things such as
bandwidth limiting did not work. This commit does the necessary magic to
make the GS library use our HTTP transport.
When looking up a blob in the master index, with several
indexes present in the master index, a significant amount of time
is spent generating errors for each failed lookup. However, these
errors are often used to check if a blob is present, but the contents
are not inspected making the overhead of the error not useful.
Instead, change Index.Lookup (and Index.LookupSize) to instead return
a boolean denoting if the blob was found instead of an error. Also change
all the calls to these functions to handle the new function signature.
benchmark old ns/op new ns/op delta
BenchmarkMasterIndexLookupSingleIndex-6 820 897 +9.39%
BenchmarkMasterIndexLookupMultipleIndex-6 12821 2001 -84.39%
BenchmarkMasterIndexLookupSingleIndexUnknown-6 5378 492 -90.85%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6 17026 1649 -90.31%
benchmark old allocs new allocs delta
BenchmarkMasterIndexLookupSingleIndex-6 9 9 +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6 59 19 -67.80%
BenchmarkMasterIndexLookupSingleIndexUnknown-6 22 6 -72.73%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6 72 16 -77.78%
benchmark old bytes new bytes delta
BenchmarkMasterIndexLookupSingleIndex-6 160 160 +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6 3200 240 -92.50%
BenchmarkMasterIndexLookupSingleIndexUnknown-6 1232 48 -96.10%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6 4272 128 -97.00%
Is seems that #1307 is similar to #1087, which describes a comparable
observation on Apple's new filesystem. #1389 Has been committed and
fixes the problem on Darwin.
Although I'm not sure if the root cause of the issue is the same the
solution is similar for OpenBSD, and leverages #1389.
On Darwin, allow a 1μs difference in restored timestamps, because
macOS <10.13 cannot restore with nanosecond precision and the current
version of Go (1.9.2) does not yet support the new syscall required
for this. (#1087#1389)
Add --last flag to snapshots command to only show the last entry for any
(hostname, paths) combination.
This makes it easier to check when various paths were last backed up.
