Conceptually the backend configuration should be validated when creating
or opening the backend, but not when filling in information from
environment variables into the configuration.
This unified construction removes most backend-specific code from
global.go. The backend registry will also enable integration tests to
use custom backends if necessary.
In order to change the backend initialization in `global.go` to be able
to generically call cfg.ApplyEnvironment() for supported backends, the
`interface{}` returned by `ParseConfig` must contain a pointer to the
configuration.
An alternative would be to use reflection to convert the type from
`interface{}(Config)` to `interface{}(*Config)` (from value to pointer
type). However, this would just complicate the type mess further.
The index used by restic consumes a major part of the total memory
usage. This leads to an unnecessarily large amount of memory that
contains ephemeral objects that are only used for a short time.
Modifies format module to add options for human readable storage size formatting, using size parsing already in ui/format.
Cmd flag --human-readable added to ls and find commands.
Additional option added to formatNode to support printing size in regular or new human readable format
The tests are now split into individual files for each command. The
separation isn't perfect as many tests make use of multiple commands. In
particular `init`, `backup`, `check` and `list` are used by a larger
number of test cases.
Most tests now reside in files name cmd_<name>_integration_test.go. This
provides a certain indication which commands have significant test
coverage.
Use the logging methods from testing.TB to make use of tb.Helper(). This
allows the tests to log the filename and line number in which the test
helper was called. Previously the test helper was logged which is rarely
useful.
As the `Fatal` error type only includes a string, it becomes impossible
to inspect the contained error. This is for a example a problem for the
fuse implementation, which must be able to detect context.Canceled
errors.
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
The previous approach of rewriting all snapshots first, then flushing
the repository data and finally removing old snapshots has the downside
that an interrupted command execution leaves behind broken snapshots as
not all new data is already flushed.
This adds support for caching already rewritten trees, handling of load
errors and disabling the check that the serialization doesn't lead to
data loss.
The more generic RewriteNode callback replaces the SelectByName and
PrintExclude functions. The main part of this change is a preparation to
allow using the TreeRewriter for the `repair snapshots` command.
The files in a tree must be sorted in lexical order. However, this
cannot be guaranteed when appending a filename suffix. For two files
file, file.rep
where "file" is broken, this would result in
file.repaired, file.rep
which is no longer sorted.
In addition, adding a filename suffix is also prone to filename
collisions which would require a rather complex search for a
collision-free name in order to work reliably.
Simplify CLI options:
* Rename "DeleteSnapshots" to "Forget"
* Replace "AddTag" and "Append" with hardcoded values
Change output and snapshot modifications to be more in line with the
"rewrite" command.
The builtin mechanism to capture a stacktrace in Go is to send a SIGQUIT
to the running process. However, this mechanism is not avaiable on
Windows. Thus, tweak the SIGINT handler to dump a stacktrace if the
environment variable `RESTIC_DEBUG_STACKTRACE_SIGINT` is set.
When saving files to the local backend, in some cases the used fsync
calls are slow enough to cause the tests to time out. Thus, increase the
test timeouts as a stopgap measure until we can use the mem backend for
these tests.
The SemaphoreBackend now uniformly enforces the limit of concurrent
backend operations. In addition, it unifies the parameter validation.
The List() methods no longer uses a semaphore. Restic already never runs
multiple list operations in parallel.
By managing the semaphore in a wrapper backend, the sections that hold a
semaphore token grow slightly. However, the main bottleneck is IO, so
this shouldn't make much of a difference.
The key insight that enables the SemaphoreBackend is that all of the
complex semaphore handling in `openReader()` still happens within the
original call to `Load()`. Thus, getting and releasing the semaphore
tokens can be refactored to happen directly in `Load()`. This eliminates
the need for wrapping the reader in `openReader()` to release the token.
