Commit Graph

495 Commits

Author SHA1 Message Date
Alexander Neumann 18ec27a0da Merge pull request #2243 from treymo/ls-length-2104
backend find: Check prefix length < snapshot IDs when searching
2019-04-23 09:41:19 +02:00
Jan Niklas Richter 733519d895 Refactor duplicate code for grouping snapshots
This commit is a followup to the addition of the --group-by flag for the
snapshots command. Adding the grouping code there introduced duplicated
code (the forget command also does grouping). This commit refactors
boths sides to only use shared code.
2019-04-22 14:57:53 +02:00
Trey Moore 5ac9c1157a Prevent "slice bounds out of range" error if prefix is longer than snapshot IDs.
This includes tests as well as other tests for the backend_find.Find function. Issue #2104.
2019-04-16 20:26:20 -07:00
Alexander Neumann 6b700d02f5 Merge pull request #2217 from restic/improve-memory-usage
WIP: improve memory usage
2019-04-13 15:07:07 +02:00
Alexander Neumann 7304738872 check: Reduce default parallelism from 40 to 5 2019-04-13 13:38:39 +02:00
Alexander Neumann 66efa425bf Reuse buffer in worker functions 2019-04-13 13:38:39 +02:00
Alexander Neumann d51e9d1b98 Add []byte to repo.LoadAndDecrypt and utils.LoadAll
This commit changes the signatures for repository.LoadAndDecrypt and
utils.LoadAll to allow passing in a []byte as the buffer to use. This
buffer is enlarged as needed, and returned back to the caller for
further use.

In later commits, this allows reducing allocations by reusing a buffer
for multiple calls, e.g. in a worker function.
2019-04-13 13:38:39 +02:00
Alexander Neumann e046428c94 Replace FilesInParallel with an errgroup.Group 2019-04-13 13:38:39 +02:00
Alexander Neumann 75906edef5 mock: Remove unused repository type 2019-04-13 13:38:39 +02:00
Alexander Neumann 203d775190 restic: Make JSON unmarshal for ID more efficient
This commit reduces several allocations in UnmarshalJSON() by decoding
the hex string directly in a single step.
2019-04-13 13:38:39 +02:00
Benoît Knecht 3c112d9cae s3: Add config option to set storage class
The `s3.storage-class` option can be passed to restic (using `-o`) to
specify the storage class to be used for S3 objects created by restic.

The storage class is passed as-is to S3, so it needs to be understood by
the API. On AWS, it can be one of `STANDARD`, `STANDARD_IA`,
`ONEZONE_IA`, `INTELLIGENT_TIERING` and `REDUCED_REDUNDANCY`. If
unspecified, the default storage class is used (`STANDARD` on AWS).

You can mix storage classes in the same bucket, and the setting isn't
stored in the restic repository, so be sure to specify it with each
command that writes to S3.

Closes #706
2019-03-26 16:37:07 +01:00
Alexander Neumann c1058005c3 Fix build on FreeBSD for Go < 1.12 2019-03-16 20:52:37 +01:00
Alexander Neumann ca73808649 Merge pull request #2197 from dmgk/master
Fix build on FreeBSD with Go 1.12
2019-03-16 20:23:19 +01:00
Igor Fedorenko bf9a507148 Fix rare 'file already closed' during restore
Fixes #2183

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2019-03-16 17:01:20 +01:00
Alexander Neumann 65b476ead9 Fix gofmt 2019-03-16 13:29:05 +01:00
Alexander Neumann aaa1cc2c26 Merge pull request 2193 from restic/allow-empty-rclone-args
rclone: Rework backend option parsing
2019-03-16 12:17:38 +01:00
Alexander Neumann 95434cff16 Merge pull request 2156 from kayrus/swift-app-cred
Swift: introduce application credential auth support
2019-03-16 12:17:17 +01:00
Heiko Bornholdt db8f5864fc Add --ignore-inode option to backup cmd
revised version of https://github.com/restic/restic/pull/2047
2019-03-10 21:24:29 +01:00
Dmitri Goutnik 1d8b21cdad Fix build on FreeBSD with Go 1.12 2019-03-06 09:41:49 -05:00
Alexander Neumann 3865b59716 rclone: Rework backend option parsing
This change allows passing no arguments to rclone, using `-o
rclone.args=""`. It is helpful when running rclone remotely via SSH
using a key with a forced command (via `command=` in `authorized_keys`).
2019-03-02 10:36:42 +01:00
Alexander Neumann e277a92a2f self-update: Don't cancel download after 30 seconds 2019-02-23 11:15:18 +01:00
kayrus 6ebcfe7c18 Swift: introduce application credential auth support 2019-02-14 14:19:05 +01:00
Alexander Neumann fed25714a4 Merge pull request 2139 from restic/fail-zero-bytes-stdin
Return error when reading zero byte from stdin
2019-02-10 12:34:05 +01:00
Alexander Neumann 8906d85ab8 Merge pull request 2138 from restic/fix-2136
Use processed bytes for summary
2019-02-10 12:32:50 +01:00
Matthew Holt 6cf13483b5 Add snapshot ID to summary output 2019-02-10 12:09:09 +01:00
Matthew Holt f645306a18 Initial implementation of JSON progress reporter 2019-02-10 12:09:09 +01:00
Alexander Neumann 1f246c5309 Merge pull request 2090 from plumbeo/within-hours
Increase the granularity of the "keep within" snapshot retention policy
2019-01-06 15:04:55 +01:00
Alexander Neumann c4fbf2c779 Return error when reading zero byte from stdin
This commit changes the internal file system implementation for reading
data from stdin, it now returns an error when no bytes could be read. I
think it's worth failing in this case, the user instructed restic to
read some data from stdin, and no data was read at all. Maybe it was in
a pipe and some earlier stage failed.

See #2135 for a short discussion.
2019-01-06 14:52:20 +01:00
Alexander Neumann 7c084014fa Use processed bytes for summary
When restic reads the backup from stdin, the number of bytes processed
was always displayed as zero. The reason is that the UI for the archive
uses the total bytes as returned by the scanner, which is zero for
stdin. So instead we keep track of the real number of bytes processed
and print that at the end.

Closes #2136
2019-01-06 13:15:23 +01:00
Alexander Neumann 98526b8dbe Merge pull request 2100 from restic/fix-1989
gs: Respect bandwidth limiting
2018-11-28 21:39:42 +01:00
Alexander Neumann 0083680d33 Merge pull request 2099 from restic/fix-1229
cache: Don't return an error for truncated files
2018-11-28 21:39:35 +01:00
plumbeo 71891b340c Support time ranges expressed in hours in snapshot retention policies
Make restic forget --keep-within accept time ranges measured in hours and choose
accordingly which snapshots to keep and which to forget. Add relative tests.
2018-11-26 14:27:42 +01:00
Alexander Neumann c9745cd47e gs: Respect bandwidth limiting
In 0dfdc11ed9, accidentally we dropped
using the provided http.RoundTripper, this commits adds it back.

Closes #1989
2018-11-25 18:52:32 +01:00
Chris Howie 1688713400 Add key hinting (#2097) 2018-11-25 09:13:18 -05:00
Alexander Neumann 00597284de cache: Don't return an error for truncated files
Closes #1229
Closes #1328
2018-11-25 14:18:02 +01:00
plumbeo 7486bfea5b Extend custom Duration granularity to hours and add tests 2018-11-14 17:27:02 +01:00
Alexander Neumann 5b95bb7059 Fix error message on Windows 2018-11-11 14:06:32 +01:00
Alexander Neumann bfa18ee8ec DownloadAndHash: Check error returned by Load() 2018-10-28 21:28:56 +01:00
Alexander Neumann 0b600d6cef index.New: Test various error conditions 2018-10-28 14:02:31 +01:00
Alexander Neumann 3ae2a79bdf restic: Use local rand source instead of global one 2018-10-28 14:01:30 +01:00
Alexander Neumann f7c0893f76 index: Add tests for error conditions 2018-10-28 13:16:36 +01:00
Alexander Neumann c3de301fc8 index: Use interfaces instead of *repository.Repository 2018-10-28 12:58:34 +01:00
Alexander Neumann 944b446ac0 index: Cancel context for tests 2018-10-28 11:17:04 +01:00
Alexander Neumann b096fc7abf index: Correctly process errors listing all files
This also removes the now unused `list` and `worker` packages.
2018-10-28 11:16:29 +01:00
Alexander Neumann 7ac683c360 rclone: Inject debug logger for HTTP 2018-10-21 19:58:40 +02:00
Gábor Lipták e5d7879622
Correct ineffassign
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
2018-10-19 16:58:14 -04:00
Alexander Neumann c0fca3f50a restore: Close files tested with --verify
Before, the target files were opened once per blob and never closed,
this commit fixes that.
2018-10-14 21:00:14 +02:00
Alexander Neumann 4c2072d875 Merge pull request 1719 from ifedorenko/mt-restorer
new optimized multithreaded restore implementation
2018-10-14 20:29:16 +02:00
Igor Fedorenko bda8d7722e restorer: Optimize empty file restore
don't create fileInfo structs for empty files. this saves memory.
this also avoids extra serial scan of all fileInfo, which should
make restore faster and more consistent.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00
Igor Fedorenko c2bcb764cd restorer: Add a note on hardlink metadata
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:39:42 +02:00