2
2
mirror of https://github.com/octoleo/restic.git synced 2024-11-12 08:16:37 +00:00
Commit Graph

609 Commits

Author SHA1 Message Date
greatroar
e1969d1e33 Put host last in SSH command line
This is how the SSH manpage says the command line should look, and the
"--" prevents mistakes in hostnames from being interpreted as options.
2020-02-19 15:53:20 +01:00
greatroar
6ac6bca7a1 Support IPv6 in SFTP backend
The previous code was doing its own hostname:port splitting, which
caused IPv6 addresses to be misinterpreted.
2020-02-19 15:42:12 +01:00
greatroar
3a6feb0596 Replace sort.Stable by sort.Strings
Calling the slow, O(n lg² n) sort.Stable is equivalent to sort.Strings
for a slice of unique strings.
2020-02-18 19:41:06 +01:00
greatroar
2f8aa2ce30 Remove unused fs.FS from archiver.FileSaver 2020-02-18 10:39:14 +01:00
Michael Eischer
e1f722d266 archiver: Fix flaky TestArchiverAbortEarlyOnError
Each of the random test files was split into the same five blobs. The
test fails once the fifth blob is passed on to `SaveBlob`. That is for
certain interleavings of goroutine execution it would be possible for
the test to trigger the testErr just after storing the first file.

The fixed test uses a different file content for each of the nine files
and fails after writing the fourth blob. The file content is also small
enough to ensure that for each file only a single blob is saved. This
guarantees that the test cannot fail before reading the first four
files. FileReadConcurrency = 2 allows up to two files queued for
processing. Therefore the test can at most open the sixth file before it
has to save the fourth file / blob which triggers the testErr.
2020-02-14 23:16:13 +01:00
Michael Eischer
42a3292bcf Better name for jsonstatus package
internal/ui/jsonstatus and termstatus sound similar but are not related
in any way. Instead `internal/ui/backup` and `internal/ui/jsonstatus/status`
are the counterparts. Rename the latter to `internal/ui/json/backup` to
make this clear.
2020-02-13 21:14:20 +01:00
Michael Eischer
ef70a2fcb3 Fix mangled JSON output by backup command
jsonstatus wrote the JSON output without synchronization to the
stdio_wrapper which caused mangling between different status lines.

Use the Print and Error methods of termstatus instead which use a
central goroutine to synchronize output.
2020-02-13 21:14:20 +01:00
rawtaz
680a14afa1
Merge pull request #2530 from restic/fix-sftp-mkdirall
sftp: Use MkdirAll provided by the client
2020-02-13 01:12:59 +01:00
rawtaz
d8da9c4401
Merge pull request #2577 from alrs/fix-internal-errs
internal: Fix code and test dropped errors
2020-02-12 23:41:54 +01:00
rawtaz
299f5971f2
Merge pull request #2560 from brualan/master
Two small improvements to code quality
2020-02-12 22:40:20 +01:00
Lars Lehtonen
72734d59b5
internal/archiver: fix dropped error 2020-02-12 13:37:37 -08:00
Lars Lehtonen
3ed54e762e
internal/backend/sftp: fix dropped test error 2020-02-12 13:36:21 -08:00
Lars Lehtonen
fea835b4e2
internal/repository: fix dropped test error 2020-02-12 13:33:54 -08:00
Lars Lehtonen
16b321b140
internal/restic: fix dropped test error 2020-02-12 13:32:45 -08:00
Michael Eischer
760863e7f9 archiver: Fix TestRacyFileSwap on windows 2020-02-11 21:09:47 +01:00
Michael Eischer
a135699397 Close file if file type has changed after initial stat 2020-02-11 21:09:47 +01:00
Lars Lehtonen
57815d9cd6
restic/internal/ui: fix gofmt nit that popped up in Go 1.13 2020-02-10 11:06:47 -08:00
Daniel Hoffend
e7cdf2acbb fix backup --json total_bytes_processed output
Closes #2429
2020-02-09 02:02:45 +01:00
Alexander Bruyako
38ddfbc4d3 simpler error return 2020-01-27 18:41:46 +03:00
Alexander Bruyako
da48b925ff remove unnecessary error return
I was running "golangci-lint" and found this two warnings

internal/checker/checker.go:135:18: (*Checker).LoadIndex$3 - result 0 (error) is always nil (unparam)
        final := func() error {
                        ^
internal/repository/repository.go:457:18: (*Repository).LoadIndex$3 - result 0 (error) is always nil (unparam)
        final := func() error {
                        ^

It turns out that these functions are used only in "RunWorkers(...)",
which is used only two times in whole project right after this "final"
functions.
And because these "final" functions always return "nil", I've
descided, that it would be better to remove requriments for "final" func
to return error to avoid magick "return nil" at their end.
2020-01-27 18:28:21 +03:00
rawtaz
d70a4a9350
Merge pull request #2373 from vrusinov/issue-2372
Ignore username difference in TestMetadataChanged
2020-01-05 21:07:34 +01:00
Alexander Neumann
2cd9c7ef16 sftp: Use MkdirAll provided by the client
Closes #2518
2020-01-01 17:26:38 +01:00
Ward Vandewege
69a0d0ee90
fix typos 2019-12-21 21:00:28 -05:00
streambinder
97e5ce4344 internal: backend: sftp: support user@domain parsing as user 2019-12-19 13:15:37 +01:00
rawtaz
da4473aba6
Merge pull request #2391 from kastenhq/refresh-lock-time
Update Lock.Time in lock.Refresh()
2019-11-26 21:44:29 +01:00
rawtaz
6e85a58045
Merge pull request #2465 from whs-dot-hk/fix-test-metadata-changed
Fix test metadata changed
2019-11-26 21:42:42 +01:00
rawtaz
e14c4b1737
Merge pull request #2484 from restic/add-s3-region
s3: Allow specifying region
2019-11-22 15:51:17 +01:00
Alexander Neumann
409909a7f5 Add option description for Region 2019-11-22 15:09:09 +01:00
mdauphin
df500a372d Add AWS_REGION env var to specify s3 region 2019-11-22 15:04:48 +01:00
Alexander Neumann
a6e8af7e0f Update minio-go 2019-11-22 14:50:46 +01:00
rawtaz
41fe9318b1
Merge pull request #2425 from thiell/restic_cache_dir_env
Add support for $RESTIC_CACHE_DIR
2019-11-20 21:18:57 +01:00
Leo R. Lundgren
4d2aa18273 Accept both singular and plural for --group-by (#2330) 2019-11-20 19:25:48 +01:00
rawtaz
26d1f9f4ba
Merge pull request #2307 from gary-kim/fix-2306-password-retries
Allow multiple retries for interactive password input
2019-11-20 18:30:20 +01:00
whs
7cacba0394 Assume WithAtime default to false 2019-11-06 16:38:46 +08:00
Stephane Thiell
0e897ef7b8 Add support for $RESTIC_CACHE_DIR
Add support for restic-specific $RESTIC_CACHE_DIR environment variable
to override the cache directory like --cache-dir would have.
2019-09-26 15:59:56 -07:00
Eric Hamilton
1596d06f8e Update Lock.Time in lock.Refresh() 2019-09-04 11:38:35 -07:00
Vladimir Rusinov
db20c0b8d0 Ignore username difference in TestMetadataChanged.
In some (rare) cases "fake" UID 51234 may exist in a system running a
test. When this is the case, `cmp.Equal(want, node3)` will fail based on
difference between empty string and an actual username present in a
system.

Fixes github issue #2372
2019-08-13 22:25:00 +01:00
Alexander Neumann
01c51b3449 Merge pull request #2321 from restic/fix-index-error-handling
Check errors returned by LoadIndex()
2019-07-27 09:37:18 +02:00
Alexandr Bruyako
e19622e4b1 start using fileMode const 2019-07-01 00:26:00 +03:00
Alexandr Bruyako
38ea7ed4f6 remove unused code 2019-07-01 00:24:45 +03:00
Alexandr Bruyako
76d1866444 avoiding unnecessary type conversions 2019-06-30 23:58:00 +03:00
Alexandr Bruyako
8b22fe29cf improved slice copying 2019-06-30 23:56:36 +03:00
Alexandr Bruyako
02014be76c simplified prefix removal, removed unnecessary if-else statements 2019-06-30 23:34:47 +03:00
Alexandr Bruyako
16eeed2ad5 simplified string sorting by using a more suitable function 2019-06-30 23:20:32 +03:00
Alexander Neumann
88716794e3 Check errors returned by LoadIndex()
Bug was reported in the forum here: https://forum.restic.net/t/check-rebuild-index-prune/1848/13
2019-06-30 21:34:53 +02:00
Gary Kim
fea2464d4d
Allow multiple retries for interactive password input
Restic used to quit if the repository password was typed incorrectly once.
Restic will now ask the user again for the repository password if typed incorrectly.
The user will now get three tries to input the correct password before restic quits.
2019-06-13 20:11:02 +08:00
Garry McNulty
8066195e6e fs: Handle absolute pathname for --stdin-filename
Return valid directory info from Lstat() for parent directories of the
specified filename. Previously only "/" and "." were valid directories.

Also set directory mode as this is checked by archiver.

Closes #2063
2019-05-08 15:21:27 +02:00
Garry McNulty
f7f14cf8c9 fs: Add file info base name check in reader tests (#2063) 2019-05-08 15:21:27 +02:00
Garry McNulty
5096f3b491 fs: Update directory check in reader tests (#2063) 2019-05-08 15:21:27 +02:00
Alexander Neumann
920d458a4a archiver: Use untyped constants for testing FileInfo 2019-05-05 14:57:38 +02:00
Alexander Neumann
b016dc2ff0 archiver/Windows: Skip test new-content-same-filestamp 2019-05-05 14:02:11 +02:00
Alexander Neumann
355db0bc29 windows: Use LastWriteTime for ctime and mtime
Windows does not have a concept of a `change time` in the sense as Unix
has it: the field `CreationTime` of the `Win32FileAttributeData` struct
is not updated when attributes or content is changed. So from now on
we're using the `LastWriteTime` as the `change time` on Windows.
2019-05-05 14:02:11 +02:00
Alexander Neumann
6e2fe73189 archiver: Move tests back into the same file
Move all Archiver tests back into `archiver_test.go` and add some tiny
helpers to mock what `lstat` returns (for Windows and Unix separately).
2019-05-05 14:02:11 +02:00
Alexander Neumann
303a5dab6a archiver: Clarify value in test struct
Since I could not remember what the value for `Check` means this commit
renames it to `SameFile`: when set to true, the test should make sure
that `FileChanged` should return false (=file is unmodified).
2019-05-05 12:57:00 +02:00
Gábor Lipták
bce6438d22
Don't run TestMetadataChanged test on Windows
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
2019-04-27 21:23:47 -04:00
Alexander Neumann
919dd2ac84 Merge pull request #2252 from restic/fix-2249
Read fresh metadata for unmodified files
2019-04-25 09:15:50 +02:00
Alexander Neumann
870bc5108e Merge pull request #2253 from restic/fix-2174
Make sure timestamps are valid
2019-04-25 09:14:43 +02:00
Alexander Neumann
a6481b3707 Merge pull request #2212 from cbane/check-ctime
Examine file ctime when checking if files have changed.
2019-04-25 08:58:27 +02:00
Courtney Bane
0ebfc55ee3 Use existing setTimestamp function for ctime test and improve error checking. 2019-04-24 20:53:08 -05:00
Courtney Bane
35b7607802 Don't check ctime when ignoring inode. 2019-04-24 20:53:08 -05:00
Alexander Neumann
939f3e972c node: Make sure year of all timestamps is valid
Sometimes restic gets bogus timestamps which cannot be converted to
JSON, because the stdlib JSON encoder returns an error if the year is
not within [0, 9999]. We now make sure that we at least record _some_
timestamp and cap the year either to 0000 or 9999. Before, restic would
refuse to save the file at all, so this improves the status quo.

This fixes #2174 and #1173
2019-04-24 16:39:36 +02:00
Alexander Neumann
389067fb8b Only use list of blobs for old node
Closes #2249
2019-04-24 15:07:26 +02:00
Alexander Neumann
4b0ca9ddab Add test for #2249 2019-04-24 15:07:23 +02:00
Courtney Bane
b8c2544dcb Examine file ctime when checking if files have changed. 2019-04-23 21:54:35 -05:00
Alexander Neumann
18ec27a0da Merge pull request #2243 from treymo/ls-length-2104
backend find: Check prefix length < snapshot IDs when searching
2019-04-23 09:41:19 +02:00
Jan Niklas Richter
733519d895 Refactor duplicate code for grouping snapshots
This commit is a followup to the addition of the --group-by flag for the
snapshots command. Adding the grouping code there introduced duplicated
code (the forget command also does grouping). This commit refactors
boths sides to only use shared code.
2019-04-22 14:57:53 +02:00
Trey Moore
5ac9c1157a Prevent "slice bounds out of range" error if prefix is longer than snapshot IDs.
This includes tests as well as other tests for the backend_find.Find function. Issue #2104.
2019-04-16 20:26:20 -07:00
Alexander Neumann
6b700d02f5 Merge pull request #2217 from restic/improve-memory-usage
WIP: improve memory usage
2019-04-13 15:07:07 +02:00
Alexander Neumann
7304738872 check: Reduce default parallelism from 40 to 5 2019-04-13 13:38:39 +02:00
Alexander Neumann
66efa425bf Reuse buffer in worker functions 2019-04-13 13:38:39 +02:00
Alexander Neumann
d51e9d1b98 Add []byte to repo.LoadAndDecrypt and utils.LoadAll
This commit changes the signatures for repository.LoadAndDecrypt and
utils.LoadAll to allow passing in a []byte as the buffer to use. This
buffer is enlarged as needed, and returned back to the caller for
further use.

In later commits, this allows reducing allocations by reusing a buffer
for multiple calls, e.g. in a worker function.
2019-04-13 13:38:39 +02:00
Alexander Neumann
e046428c94 Replace FilesInParallel with an errgroup.Group 2019-04-13 13:38:39 +02:00
Alexander Neumann
75906edef5 mock: Remove unused repository type 2019-04-13 13:38:39 +02:00
Alexander Neumann
203d775190 restic: Make JSON unmarshal for ID more efficient
This commit reduces several allocations in UnmarshalJSON() by decoding
the hex string directly in a single step.
2019-04-13 13:38:39 +02:00
Benoît Knecht
3c112d9cae s3: Add config option to set storage class
The `s3.storage-class` option can be passed to restic (using `-o`) to
specify the storage class to be used for S3 objects created by restic.

The storage class is passed as-is to S3, so it needs to be understood by
the API. On AWS, it can be one of `STANDARD`, `STANDARD_IA`,
`ONEZONE_IA`, `INTELLIGENT_TIERING` and `REDUCED_REDUNDANCY`. If
unspecified, the default storage class is used (`STANDARD` on AWS).

You can mix storage classes in the same bucket, and the setting isn't
stored in the restic repository, so be sure to specify it with each
command that writes to S3.

Closes #706
2019-03-26 16:37:07 +01:00
Alexander Neumann
c1058005c3 Fix build on FreeBSD for Go < 1.12 2019-03-16 20:52:37 +01:00
Alexander Neumann
ca73808649 Merge pull request #2197 from dmgk/master
Fix build on FreeBSD with Go 1.12
2019-03-16 20:23:19 +01:00
Igor Fedorenko
bf9a507148 Fix rare 'file already closed' during restore
Fixes #2183

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2019-03-16 17:01:20 +01:00
Alexander Neumann
65b476ead9 Fix gofmt 2019-03-16 13:29:05 +01:00
Alexander Neumann
aaa1cc2c26 Merge pull request 2193 from restic/allow-empty-rclone-args
rclone: Rework backend option parsing
2019-03-16 12:17:38 +01:00
Alexander Neumann
95434cff16 Merge pull request 2156 from kayrus/swift-app-cred
Swift: introduce application credential auth support
2019-03-16 12:17:17 +01:00
Heiko Bornholdt
db8f5864fc Add --ignore-inode option to backup cmd
revised version of https://github.com/restic/restic/pull/2047
2019-03-10 21:24:29 +01:00
Dmitri Goutnik
1d8b21cdad Fix build on FreeBSD with Go 1.12 2019-03-06 09:41:49 -05:00
Alexander Neumann
3865b59716 rclone: Rework backend option parsing
This change allows passing no arguments to rclone, using `-o
rclone.args=""`. It is helpful when running rclone remotely via SSH
using a key with a forced command (via `command=` in `authorized_keys`).
2019-03-02 10:36:42 +01:00
Alexander Neumann
e277a92a2f self-update: Don't cancel download after 30 seconds 2019-02-23 11:15:18 +01:00
kayrus
6ebcfe7c18 Swift: introduce application credential auth support 2019-02-14 14:19:05 +01:00
Alexander Neumann
fed25714a4 Merge pull request 2139 from restic/fail-zero-bytes-stdin
Return error when reading zero byte from stdin
2019-02-10 12:34:05 +01:00
Alexander Neumann
8906d85ab8 Merge pull request 2138 from restic/fix-2136
Use processed bytes for summary
2019-02-10 12:32:50 +01:00
Matthew Holt
6cf13483b5 Add snapshot ID to summary output 2019-02-10 12:09:09 +01:00
Matthew Holt
f645306a18 Initial implementation of JSON progress reporter 2019-02-10 12:09:09 +01:00
Alexander Neumann
1f246c5309 Merge pull request 2090 from plumbeo/within-hours
Increase the granularity of the "keep within" snapshot retention policy
2019-01-06 15:04:55 +01:00
Alexander Neumann
c4fbf2c779 Return error when reading zero byte from stdin
This commit changes the internal file system implementation for reading
data from stdin, it now returns an error when no bytes could be read. I
think it's worth failing in this case, the user instructed restic to
read some data from stdin, and no data was read at all. Maybe it was in
a pipe and some earlier stage failed.

See #2135 for a short discussion.
2019-01-06 14:52:20 +01:00
Alexander Neumann
7c084014fa Use processed bytes for summary
When restic reads the backup from stdin, the number of bytes processed
was always displayed as zero. The reason is that the UI for the archive
uses the total bytes as returned by the scanner, which is zero for
stdin. So instead we keep track of the real number of bytes processed
and print that at the end.

Closes #2136
2019-01-06 13:15:23 +01:00
Alexander Neumann
98526b8dbe Merge pull request 2100 from restic/fix-1989
gs: Respect bandwidth limiting
2018-11-28 21:39:42 +01:00
Alexander Neumann
0083680d33 Merge pull request 2099 from restic/fix-1229
cache: Don't return an error for truncated files
2018-11-28 21:39:35 +01:00
plumbeo
71891b340c Support time ranges expressed in hours in snapshot retention policies
Make restic forget --keep-within accept time ranges measured in hours and choose
accordingly which snapshots to keep and which to forget. Add relative tests.
2018-11-26 14:27:42 +01:00
Alexander Neumann
c9745cd47e gs: Respect bandwidth limiting
In 0dfdc11ed9, accidentally we dropped
using the provided http.RoundTripper, this commits adds it back.

Closes #1989
2018-11-25 18:52:32 +01:00
Chris Howie
1688713400 Add key hinting (#2097) 2018-11-25 09:13:18 -05:00
Alexander Neumann
00597284de cache: Don't return an error for truncated files
Closes #1229
Closes #1328
2018-11-25 14:18:02 +01:00
plumbeo
7486bfea5b Extend custom Duration granularity to hours and add tests 2018-11-14 17:27:02 +01:00