68608a89ad
restic: add comment about StreamTrees shutdown
2021-01-28 11:10:50 +01:00
ddb7697d29
restic: Test progress reporting of StreamTrees
2021-01-28 11:10:50 +01:00
eda8c67616
restic: let FindUsedBlobs handle multiple snapshots at once
2021-01-28 11:08:43 +01:00
258ce0c1e5
parallel: report progress for StreamTrees
...
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
2021-01-28 11:08:43 +01:00
f2a1b125cb
restic: Actually parallelize FindUsedBlobs
2021-01-28 11:08:43 +01:00
6e03f80ca2
check: Split the parallelized tree loader into a reusable component
...
The actual code change is minimal
2021-01-28 11:08:43 +01:00
31e56f1ad5
Merge pull request #3197 from SkYNewZ/fix/3183
...
Fix tag handling for multiple tag lists
2020-12-29 18:44:38 +01:00
334d8ce724
feat(tags): Create Flatten() method
2020-12-29 10:59:46 +01:00
3b09ae9074
AIX port
2020-12-29 01:35:01 +01:00
18531e3d6f
Portability fixes to internal/restic
...
syscall.Mknod is not available on AIX.
2020-12-29 01:35:01 +01:00
68b74e359e
Count packs directly in RebuildIndexFiles
2020-12-22 23:01:58 +01:00
34b6130a0e
restic: parallelize lock file loading
2020-12-18 20:46:16 +01:00
f7784bddb3
Don't retry when "no space left on device" in local backend
...
Also adds relevant documentation to the restic.Backend interface.
2020-12-17 12:43:09 +01:00
e329623771
Remove LoadAllSnapshots
2020-12-06 05:22:27 +01:00
26f85779be
Parallelize ForAllSnapshots
2020-12-06 05:09:58 +01:00
5b9ee56335
Add ForAllSnapshots
2020-12-06 05:04:21 +01:00
aa7a5f19c2
Use BlobHandle in index methods
2020-11-22 20:41:12 +01:00
92bd448691
Make BlobHandle substruct of Blob
2020-11-22 20:41:10 +01:00
ce5d630681
Add MasterIndex.PackSize()
2020-11-21 22:13:54 +01:00
c3ddde9e7d
Return hdrSize in ListPack
2020-11-21 22:13:54 +01:00
c45f8ee075
Replace restic.statT interface by concrete types
...
name old time/op new time/op delta
NodeFillUser-8 1.81µs ± 9% 1.50µs ± 5% -17.07% (p=0.000 n=19+20)
NodeFromFileInfo-8 1.76µs ± 4% 1.49µs ± 6% -15.63% (p=0.000 n=20+19)
name old alloc/op new alloc/op delta
NodeFillUser-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
NodeFromFileInfo-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
NodeFillUser-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
NodeFromFileInfo-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
63e32c44c0
Improve error reporting from restic diff
...
Instead of a stacktrace, restic diff 111 222 now reports:
Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
407843c5f9
Merge pull request #3034 from hoyho/gh_master
...
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
9df52327cc
bugfix: omit ENOTDATA for extended attributes
...
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
ddca699cd2
Replace restic.Progress with new progress.Counter
...
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
38cc4393f6
Add Masterindex.Save(); Add Index.Packs()
2020-11-06 20:23:30 +01:00
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command
...
Reimplementation of prune
2020-11-05 10:12:19 +01:00
c2f3eee5af
Don't separately allocate sync.{Mutex,Once} if not necessary
...
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
2020-11-04 09:55:43 +01:00
b2f5381737
Make realistic forget --prune --dryrun
2020-11-03 16:42:21 +01:00
ce7d613749
Add Blob.Handle()
2020-11-03 16:42:21 +01:00
41a45ae908
Merge pull request #3016 from greatroar/uid-gid-lookup
...
Simplify/optimize cached UID/GID lookups
2020-10-12 23:24:55 +02:00
f0cd16e5ea
Cache uint32-typed ids in lookup{Username,Group}
...
NodeFillUser-8 1.92µs ± 5% 1.75µs ± 3% -8.89% (p=0.000 n=10+10)
NodeFromFileInfo-8 1.89µs ± 7% 1.76µs ± 4% -6.69% (p=0.001 n=10+10)
name old alloc/op new alloc/op delta
NodeFillUser-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10)
NodeFromFileInfo-8 504B ± 0% 496B ± 0% -1.59% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
NodeFillUser-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)
NodeFromFileInfo-8 5.00 ± 0% 3.00 ± 0% -40.00% (p=0.000 n=10+10)
2020-10-12 15:08:15 +02:00
a03fe4562e
Remove unused error return in lookup{Username,Group}
2020-10-12 14:48:23 +02:00
e638b46a13
Embed context into ReaderAt
...
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
2020-10-09 22:39:07 +02:00
4b0fcaed45
unlock: use proper context for locks cleanup
...
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
2020-10-09 22:37:56 +02:00
c458e114d4
pass context to Find / FindSnapshot
...
This allows proper interruption of restic while it searches for
snapshots or key files.
2020-10-09 22:37:56 +02:00
028c9a5343
backup: Correct keep policy text
...
Makes the following corrections to the "Applying Policy:" output:
- keep the last 1 snapshots snapshots => keep 1 latest snapshots
- keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots
2020-09-28 14:26:53 +02:00
23fcbb275a
Remove restic.Cache interface
...
It was used in one code path, which then asserted its concrete type as
*cache.Cache.
Privatised some of the interface methods.
2020-09-18 10:48:13 +02:00
b77e933d80
Convert underscore variable names to camelCase
2020-09-05 10:07:16 +02:00
f6f11400c2
restic: Remove unused variable value
2020-09-05 10:07:16 +02:00
e2dc5034d3
restic/file: Use FileType for all File constants
...
The data type of a variable is only carried on when using iota.
2020-09-05 10:07:16 +02:00
37113282ca
Merge a few variable declaration and initializations
2020-09-05 10:05:34 +02:00
337725c354
Use sort.Strings
2020-09-05 10:05:34 +02:00
0fed6a8dfc
Use "pack file" instead of "data file" ( #2885 )
...
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
af66a62c04
FindUsedBlobs: Test that seen blobs are skipped
...
This also copies the TreeLoader interface from internal/walker to allow
stubbing the repository in the call to `FindUsedBlobs`.
2020-08-01 12:29:16 +02:00
9ea1a78bd4
FindUsedBlobs: Check for seen blobs before loading trees
...
The only effective change in behavior is that that toplevel nodes can
also be skipped.
2020-08-01 12:29:16 +02:00
184103647a
FindUsedBlobs: merge seen into blobs BlobSet
...
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.
Suggested-by: Alexander Weiss <alex@weissfam.de>
2020-08-01 12:29:16 +02:00
c847aace35
Rename Index interface to MasterIndex
...
The interface is now only implemented by repository.MasterIndex.
2020-07-25 21:19:46 +02:00
9d1fb94c6c
make Lookup() return all blobs
...
+ simplify syntax
2020-07-25 21:18:34 +02:00
5d8d70542f
Merge pull request #2852 from MichaelEischer/drop-go-1.11
...
Drop support for Go version 1.11
2020-07-25 01:17:36 +02:00
7c23381a2b
Drop support for Go version 1.11
2020-07-24 18:52:39 +02:00
34181b13a2
Merge pull request #2328 from MichaelEischer/no-repeated-checks
...
Fix duplicate tree checks within `restic check`
2020-07-22 22:08:02 +02:00
ddf0b8cd0b
checker: Properly distinguish between data and tree blobs
...
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
2020-07-20 22:58:39 +02:00
7bda28f31f
Chaining hash table for repository.Index
...
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.
Benchmark results on linux/amd64, -benchtime=3s -count=20:
name old time/op new time/op delta
PackerManager-8 178ms ± 0% 178ms ± 0% ~ (p=0.231 n=20+20)
DecodeIndex-8 4.54s ± 0% 4.30s ± 0% -5.20% (p=0.000 n=18+17)
DecodeIndexParallel-8 4.54s ± 0% 4.30s ± 0% -5.22% (p=0.000 n=19+18)
IndexHasUnknown-8 44.4ns ± 5% 50.5ns ±11% +13.82% (p=0.000 n=19+17)
IndexHasKnown-8 48.3ns ± 0% 51.5ns ±12% +6.68% (p=0.001 n=16+20)
IndexAlloc-8 758ms ± 1% 616ms ± 1% -18.69% (p=0.000 n=19+19)
IndexAllocParallel-8 234ms ± 3% 204ms ± 2% -12.60% (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8 122ns ± 0% 145ns ± 9% +18.44% (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8 369ns ± 2% 429ns ± 8% +16.27% (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8 68.4ns ± 5% 74.9ns ±13% +9.47% (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8 315ns ± 3% 369ns ±11% +17.14% (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8 743ns ± 1% 816ns ± 2% +9.87% (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8 238ns ± 1% 260ns ± 2% +9.14% (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8 1.01µs ± 3% 1.11µs ± 2% +9.79% (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8 222ns ± 0% 269ns ± 2% +20.83% (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8 1.06µs ± 2% 1.19µs ± 2% +12.95% (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8 413ns ± 1% 530ns ± 1% +28.19% (p=0.000 n=18+20)
SaveAndEncrypt-8 30.2ms ± 1% 30.4ms ± 0% +0.71% (p=0.000 n=19+19)
LoadTree-8 540µs ± 1% 576µs ± 1% +6.73% (p=0.000 n=20+20)
LoadBlob-8 5.64ms ± 0% 5.64ms ± 0% ~ (p=0.883 n=18+17)
LoadAndDecrypt-8 5.93ms ± 0% 5.95ms ± 1% ~ (p=0.247 n=20+19)
LoadIndex-8 25.1ms ± 0% 24.5ms ± 1% -2.54% (p=0.000 n=18+17)
name old speed new speed delta
PackerManager-8 296MB/s ± 0% 296MB/s ± 0% ~ (p=0.229 n=20+20)
SaveAndEncrypt-8 139MB/s ± 1% 138MB/s ± 0% -0.71% (p=0.000 n=19+19)
LoadBlob-8 177MB/s ± 0% 177MB/s ± 0% ~ (p=0.890 n=18+17)
LoadAndDecrypt-8 169MB/s ± 0% 168MB/s ± 1% ~ (p=0.227 n=20+19)
name old alloc/op new alloc/op delta
PackerManager-8 91.8kB ± 0% 91.8kB ± 0% ~ (p=0.772 n=12+19)
IndexAlloc-8 786MB ± 0% 400MB ± 0% -49.04% (p=0.000 n=20+18)
IndexAllocParallel-8 786MB ± 0% 401MB ± 0% -49.04% (p=0.000 n=19+15)
SaveAndEncrypt-8 21.0MB ± 0% 21.0MB ± 0% +0.00% (p=0.000 n=19+19)
name old allocs/op new allocs/op delta
PackerManager-8 1.41k ± 0% 1.41k ± 0% ~ (all equal)
IndexAlloc-8 977k ± 0% 907k ± 0% -7.18% (p=0.000 n=20+20)
IndexAllocParallel-8 977k ± 0% 907k ± 0% -7.17% (p=0.000 n=19+15)
SaveAndEncrypt-8 73.0 ± 0% 73.0 ± 0% ~ (all equal)
2020-07-19 13:58:22 +02:00
dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256
...
Use Minio's optimized SHA-256
2020-06-12 23:32:58 +02:00
91906911b0
Fix non-intuitive repository behavior
...
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
-> also cleans up pending index entries when they are saved in the index
-> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
-> removes the need of an index uploader
-> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
2020-06-11 13:05:23 +02:00
42a3db05b0
Use Minio's optimized SHA-256
...
internal/repository benchmarks on an Intel i7-3770k:
name old speed new speed delta
PackerManager-8 209MB/s ± 1% 291MB/s ± 1% +38.94% (p=0.008 n=5+5)
SaveAndEncrypt-8 112MB/s ± 1% 135MB/s ± 1% +20.25% (p=0.008 n=5+5)
2020-04-28 07:57:18 +02:00
be5a0ff59f
Centralize buffer allocation and size checking in Repository.LoadBlob
...
Benchmark results for internal/repository:
name old time/op new time/op delta
LoadTree-8 479µs ± 2% 478µs ± 1% ~ (p=0.780 n=10+9)
LoadBlob-8 11.6ms ± 2% 11.6ms ± 1% ~ (p=0.631 n=10+10)
LoadAndDecrypt-8 13.2ms ± 2% 13.3ms ± 3% ~ (p=0.631 n=10+10)
name old alloc/op new alloc/op delta
LoadTree-8 41.2kB ± 0% 41.2kB ± 0% ~ (all equal)
LoadBlob-8 2.28kB ± 0% 2.28kB ± 0% ~ (all equal)
LoadAndDecrypt-8 2.10MB ± 0% 2.10MB ± 0% ~ (all equal)
name old allocs/op new allocs/op delta
LoadTree-8 652 ± 0% 652 ± 0% ~ (all equal)
LoadBlob-8 24.0 ± 0% 24.0 ± 0% ~ (all equal)
LoadAndDecrypt-8 30.0 ± 0% 30.0 ± 0% ~ (all equal)
name old speed new speed delta
LoadBlob-8 86.2MB/s ± 2% 86.4MB/s ± 1% ~ (p=0.594 n=10+10)
LoadAndDecrypt-8 75.7MB/s ± 2% 75.4MB/s ± 3% ~ (p=0.617 n=10+10)
2020-04-23 10:04:20 +02:00
2e31120f89
Remove unused argument to restic.fakeFile
2020-04-18 17:40:13 +02:00
b7c0d4d8bf
Merge pull request #2644 from greatroar/signal-notify-buffered
...
Make all signal.Notify channels buffered
2020-04-18 11:31:02 +02:00
9efbe98879
Merge pull request #2623 from alrs/internal-restic-err-before-close
...
internal/restic: close os.File after checking for error
2020-03-18 22:23:20 +01:00
7d9300efca
Merge pull request #2637 from greatroar/unused
...
Remove Go 1.5 compatibility code
2020-03-18 22:22:15 +01:00
47d4d5bf1b
Make all signal.Notify channels buffered
2020-03-12 20:59:39 +01:00
4de12bf593
Remove restic.RandReader
...
math/rand.Rand has implemented Reader since Go 1.6. The repacking tests
are not deterministic, but they weren't before, either.
2020-03-09 10:00:28 +01:00
4a2156d3f0
internal/restic: close os.File after checking for error
2020-03-05 16:22:46 -08:00
58bd165253
Merge pull request #2581 from aawsome/multiple-hostnames
...
Allow multiple hostnames tags
2020-02-27 08:35:23 +01:00
9a9101d144
Support specifying multiple host flags for various commands
...
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`,
`stats` and `tag` commands will now take into account multiple
`--host` and `-H` flags.
2020-02-26 22:17:59 +01:00
16b321b140
internal/restic: fix dropped test error
2020-02-12 13:32:45 -08:00
69a0d0ee90
fix typos
2019-12-21 21:00:28 -05:00
da4473aba6
Merge pull request #2391 from kastenhq/refresh-lock-time
...
Update Lock.Time in lock.Refresh()
2019-11-26 21:44:29 +01:00
4d2aa18273
Accept both singular and plural for --group-by ( #2330 )
2019-11-20 19:25:48 +01:00
1596d06f8e
Update Lock.Time in lock.Refresh()
2019-09-04 11:38:35 -07:00
38ea7ed4f6
remove unused code
2019-07-01 00:24:45 +03:00
355db0bc29
windows: Use LastWriteTime for ctime and mtime
...
Windows does not have a concept of a `change time` in the sense as Unix
has it: the field `CreationTime` of the `Win32FileAttributeData` struct
is not updated when attributes or content is changed. So from now on
we're using the `LastWriteTime` as the `change time` on Windows.
2019-05-05 14:02:11 +02:00
939f3e972c
node: Make sure year of all timestamps is valid
...
Sometimes restic gets bogus timestamps which cannot be converted to
JSON, because the stdlib JSON encoder returns an error if the year is
not within [0, 9999]. We now make sure that we at least record _some_
timestamp and cap the year either to 0000 or 9999. Before, restic would
refuse to save the file at all, so this improves the status quo.
This fixes #2174 and #1173
2019-04-24 16:39:36 +02:00
18ec27a0da
Merge pull request #2243 from treymo/ls-length-2104
...
backend find: Check prefix length < snapshot IDs when searching
2019-04-23 09:41:19 +02:00
733519d895
Refactor duplicate code for grouping snapshots
...
This commit is a followup to the addition of the --group-by flag for the
snapshots command. Adding the grouping code there introduced duplicated
code (the forget command also does grouping). This commit refactors
boths sides to only use shared code.
2019-04-22 14:57:53 +02:00
5ac9c1157a
Prevent "slice bounds out of range" error if prefix is longer than snapshot IDs.
...
This includes tests as well as other tests for the backend_find.Find function. Issue #2104 .
2019-04-16 20:26:20 -07:00
d51e9d1b98
Add []byte to repo.LoadAndDecrypt and utils.LoadAll
...
This commit changes the signatures for repository.LoadAndDecrypt and
utils.LoadAll to allow passing in a []byte as the buffer to use. This
buffer is enlarged as needed, and returned back to the caller for
further use.
In later commits, this allows reducing allocations by reusing a buffer
for multiple calls, e.g. in a worker function.
2019-04-13 13:38:39 +02:00
203d775190
restic: Make JSON unmarshal for ID more efficient
...
This commit reduces several allocations in UnmarshalJSON() by decoding
the hex string directly in a single step.
2019-04-13 13:38:39 +02:00
c1058005c3
Fix build on FreeBSD for Go < 1.12
2019-03-16 20:52:37 +01:00
1d8b21cdad
Fix build on FreeBSD with Go 1.12
2019-03-06 09:41:49 -05:00
71891b340c
Support time ranges expressed in hours in snapshot retention policies
...
Make restic forget --keep-within accept time ranges measured in hours and choose
accordingly which snapshots to keep and which to forget. Add relative tests.
2018-11-26 14:27:42 +01:00
7486bfea5b
Extend custom Duration granularity to hours and add tests
2018-11-14 17:27:02 +01:00
3ae2a79bdf
restic: Use local rand source instead of global one
2018-10-28 14:01:30 +01:00
da57302fca
restore: Removed legacy restore implementation
...
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-10-14 17:37:30 +02:00
e42d2d1da8
restore: suppress lchown errors when not running as root
...
Like "cp -a" and "rsync -a" do, only report lchown errors if we run
as root.
Like cp from GNU coreutils does, we check Geteuid() to determine if
we are running as root
( http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/copy.c#n3012 ).
On Windows, lchown errors are always reported.
Fixes https://github.com/restic/restic/issues/1766
2018-09-09 15:39:10 +02:00
9fd3796d93
forget: Display reasons why snapshots are kept
...
This change displays the reasons for keeping a snapshot in the table,
unless `--compact` is specified.
2018-08-20 21:47:51 +02:00
6f5b0f3622
stat: Improve error message for bad snapshot ID ( fixes #1933 )
2018-08-08 16:49:36 -06:00
abb18a830c
Fix test
2018-07-14 11:51:34 +02:00
7ddf91b65c
Merge pull request #1772 from ifedorenko/restore-verify
...
Restore verify
2018-06-23 14:50:29 +02:00
f080142137
Support NetBSD without fuse
2018-06-16 15:55:04 +09:00
5fa6dc53cb
Refactor: introduced restorer tree visitor
...
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-06-11 07:57:22 -04:00
26be094f28
Refactor: moved restorer to separate package
...
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-06-11 07:57:22 -04:00
060d8b57e0
Restructure TestApplyPolicy
2018-05-13 12:48:42 +02:00
cc627e832b
Add custom Duration type
2018-05-13 12:48:42 +02:00
5a0f0e3faa
Add support for keeping a range of snapshots
2018-05-13 12:48:42 +02:00
b52f2aa9a4
forget: Add policy to keep snapshots before a date
2018-05-13 12:48:10 +02:00
64d98945a6
backup: Clean target paths before finding parent
...
This resolves an issue described in the forum where restic could not
find a parent snapshot if the target path ends with a slash:
https://forum.restic.net/t/new-archiver-code-please-test/623/23
2018-04-30 22:03:11 +02:00
76b616451f
Remove unneeded code
2018-04-23 21:40:33 +02:00
b6f98bdb02
node: Fill minimal info
2018-04-22 11:37:05 +02:00
6a7c23d2ae
tree: Add convenience functions
2018-04-22 11:37:05 +02:00
cc847a3d6d
tree: Improve error for pre-existing node
2018-04-22 11:37:05 +02:00
baebf45e2e
FindLatestSnapshot: Use absolute paths
2018-04-22 11:37:05 +02:00
fa4f438bc1
snapshot: Do not modify slice of paths
2018-04-22 11:37:05 +02:00
4e0b2a8e3a
snapshot: correct error handling for filepath.Abs
2018-04-22 11:37:05 +02:00
a472868e06
fs: Add TestChdir()
2018-04-22 11:37:05 +02:00
09bd924710
Do not restore sockets, correct error handling
...
Closes #1730
2018-04-20 13:53:11 +02:00
e68a7fea8a
check: Allow filling the cache during check
...
Closes #1665
2018-04-01 13:59:27 +02:00
d3f9c8b362
forget: Print policy
2018-03-30 10:24:26 +02:00
b9ada91054
Reduce data for TestCreateSnapshot
2018-03-11 21:42:39 +01:00
c67a8452f7
Disable polynomial check for chunker for tests
2018-03-11 20:59:40 +01:00
d8dcbc89d1
lock: Ignore invalid lock file
...
This commit fixes a bug introduced in
e9ea26884739a218bbab0248b634d2821be9c3ea: When an invalid lock is
encountered (e.g. if the file is empty), the code used to ignore that,
but now returns the error.
Now, invalid files are ignored for the normal lock check, and removed
when `restic unlock --remove-all` is run.
Closes #1652
2018-03-05 20:19:57 +01:00
be0a5b7f06
Merge pull request #1649 from jasperla/solaris
...
Minimal set of patches to get restic working on Solaris
2018-03-05 20:00:17 +01:00
96311d1a2b
Add support for illumos/Solaris
...
This does come without xattr/fuse support at this point.
NB: not hooking up the integration tests as restic won't compile without
cgo with Go < 1.10.
2018-03-04 20:11:29 +00:00
929afc63d5
Use int64 for the length in the RewindReader
2018-03-04 10:40:42 +01:00
99f7fd74e3
backend: Improve Save()
...
As mentioned in issue [#1560 ](https://github.com/restic/restic/pull/1560#issuecomment-364689346 )
this changes the signature for `backend.Save()`. It now takes a
parameter of interface type `RewindReader`, so that the backend
implementations or our `RetryBackend` middleware can reset the reader to
the beginning and then retry an upload operation.
The `RewindReader` interface also provides a `Length()` method, which is
used in the backend to get the size of the data to be saved. This
removes several ugly hacks we had to do to pull the size back out of the
`io.Reader` passed to `Save()` before. In the `s3` and `rest` backend
this is actively used.
2018-03-03 15:49:44 +01:00
93210614f4
backend/retry: return worker function error and abort
...
This is a bug fix: Before, when the worker function fn in List() of the
RetryBackend returned an error, the operation is retried with the next
file. This is not consistent with the documentation, the intention was
that when fn returns an error, this is passed on to the caller and the
List() operation is aborted. Only errors happening on the underlying
backend are retried.
The error leads to restic ignoring exclusive locks that are present in
the repo, so it may happen that a new backup is written which references
data that is going to be removed by a concurrently running `prune`
operation.
The bug was reported by a user here:
https://forum.restic.net/t/restic-backup-returns-0-exit-code-when-already-locked/484
2018-02-24 13:26:13 +01:00
be02008025
Merge pull request #1611 from qbit/master
...
On OpenBSD only root can set sticky bit
2018-02-18 12:57:54 +01:00
d58ae43317
Reworked Backend.Load API to retry errors during ongoing download
...
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-02-16 21:12:14 -05:00
99d88ad297
Disable the 'testSticky' test on OpenBSD. Only root can set sticky.
2018-02-11 07:46:31 -07:00
aa333f4d49
Implement RetryBackend.List()
...
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-29 22:14:12 -05:00
663c57ab4d
debug: Remove manual Str() call Log()
2018-01-25 20:49:41 +01:00
9c55e8d69c
Merge pull request #1549 from MJDSys/more_index_lookup_avoids
...
More optimizations to avoid calling Index.Lookup()
2018-01-24 20:53:30 +01:00
0084e42cb6
Optimize Repository.ListPack()
...
Use pack file size returned by Backend.List() to avoid extra per-pack
Backend.Stat() requests
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2018-01-23 22:39:51 -05:00
df2c03a6a4
repository/master_index: Optimize Index.Lookup()
...
When looking up a blob in the master index, with several
indexes present in the master index, a significant amount of time
is spent generating errors for each failed lookup. However, these
errors are often used to check if a blob is present, but the contents
are not inspected making the overhead of the error not useful.
Instead, change Index.Lookup (and Index.LookupSize) to instead return
a boolean denoting if the blob was found instead of an error. Also change
all the calls to these functions to handle the new function signature.
benchmark old ns/op new ns/op delta
BenchmarkMasterIndexLookupSingleIndex-6 820 897 +9.39%
BenchmarkMasterIndexLookupMultipleIndex-6 12821 2001 -84.39%
BenchmarkMasterIndexLookupSingleIndexUnknown-6 5378 492 -90.85%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6 17026 1649 -90.31%
benchmark old allocs new allocs delta
BenchmarkMasterIndexLookupSingleIndex-6 9 9 +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6 59 19 -67.80%
BenchmarkMasterIndexLookupSingleIndexUnknown-6 22 6 -72.73%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6 72 16 -77.78%
benchmark old bytes new bytes delta
BenchmarkMasterIndexLookupSingleIndex-6 160 160 +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6 3200 240 -92.50%
BenchmarkMasterIndexLookupSingleIndexUnknown-6 1232 48 -96.10%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6 4272 128 -97.00%
2018-01-23 22:25:56 -05:00
685ce719ad
Add comments for List()
2018-01-23 23:14:05 +01:00
b0c6e53241
Fix calls to repo/backend.List() everywhere
2018-01-21 21:15:09 +01:00
52230b8f07
backend: Rework List()
...
For a discussion see #1567
2018-01-21 18:35:37 +01:00
c31a5e7e5c
Add argument to Skipf()
2018-01-17 23:14:37 +01:00
b8620429e2
restorer: Set directory mode as last step
...
Closes #1512
2018-01-07 15:13:24 +01:00
aab414b368
Add test case for #1512
2018-01-07 14:50:47 +01:00
6c2b2a58ad
backend: Retry deletes
2017-12-22 22:41:28 +01:00
3a5e040b7e
Merge pull request #1439 from armhold/propagate-context
...
replace ad-hoc context.TODO() with gopts.ctx
2017-12-08 20:27:36 +01:00
82ded35706
node: Correctly compare times
2017-12-03 18:33:35 +01:00
88607fc625
Correctly format all node types
2017-12-03 17:38:55 +01:00
20d78ab0d9
add context to repo.Flush() call in test code
...
gh-1434
2017-12-03 07:32:50 -05:00
d886cb5c27
replace ad-hoc context.TODO() with gopts.ctx, so that cancellation
...
can properly trickle down from cmd_*.
gh-1434
2017-12-03 07:22:14 -05:00
0f5e38609f
restore: Fix restore to relative path
2017-11-26 18:36:48 +01:00
27d29b9853
restorer: Make sure node names are clean
2017-11-26 15:28:45 +01:00
8a171731ba
restorer: Add tests for invalid node names
2017-11-26 15:28:37 +01:00
8ce5d35543
Merge pull request #1390 from armhold/lint
...
small cleanup
2017-10-28 09:54:35 +02:00
eea96f652d
go fmt
2017-10-26 16:22:10 -04:00
f5fa602482
detect and return error from file Close() in Node.createFileAt()
...
gh-1385
2017-10-26 13:53:31 -04:00
5b96885c6d
Control progress rate with RESTIC_PROGRESS_FPS env
...
Add a RESTIC_PROGRESS_FPS environment variable to limit the interval
at which the progress indicator updates (allowed values: 1-60).
The default rate of 60 FPS can cause high terminal CPU load on some
systems, like iTerm2 on macOS with font anti-aliasing enabled.
Usage:
RESTIC_PROGRESS_FPS=1 restic ...
RESTIC_PROGRESS_FPS=60 restic ...
2017-10-26 14:46:56 +08:00
bcdebfb84e
small cleanup:
...
- be explicit when discarding returned errors from .Close(), etc.
- remove named return values from funcs when naked return not used
- fix some "err" shadowing when redeclaration not needed
2017-10-25 12:03:55 -04:00
e56370eb5b
Remove Deleter interface
2017-10-14 16:04:29 +02:00
0e722efb09
backend: Add Delete() to restic.Backend interface
2017-10-14 15:56:25 +02:00
1dcfd64028
Merge pull request #1325 from antonlindstrom/lookup-group
...
Add group name in fillUser
2017-10-04 21:32:46 +02:00
26df48b2aa
Add group name in fillUser
...
This does a lookup of the group name from the GID and adds it to the
Node.
2017-10-04 20:11:54 +02:00
3473c3f7b6
Remove all dot-imports
2017-10-02 15:06:39 +02:00
2e72b57f2f
Correct debug message
2017-09-25 14:35:37 +02:00
176bfa6529
backend: Improve ReadAt
2017-09-24 23:11:23 +02:00
610b676444
Automatically exclude current restic cache
2017-09-24 21:54:53 +02:00
9be24a1c9f
Add cache
...
This commits adds rudimentary support for a cache directory, enabled by
default. The cache directory is created if it does not exist. The cache
is used if there's anything in it, newly created snapshot and index
files are written to the cache automatically.
2017-09-24 21:54:53 +02:00
3f6e11d26e
Allow sorting nodes in trees
2017-09-22 12:37:05 +02:00
a60e751217
Use .Equal() instead of == for time.Time
...
Closes #1238
2017-09-15 20:57:35 +02:00
36eaa22ed0
Merge pull request #1205 from mungomat/backup_time
...
Backup time
2017-09-11 19:01:41 +02:00
62df316356
Merge pull request #1194 from mungomat/bugfix_bucker_always
...
fix: bucker
2017-09-11 19:01:23 +02:00
9fa909ccd6
Update golden files
2017-09-11 17:52:22 +02:00
9924c311c9
added test cases
2017-09-10 12:23:28 +02:00
a677f1139a
removed unnacessary line
2017-09-10 10:41:07 +02:00
087c3fe1dc
tests updated
2017-09-09 13:26:35 +02:00
43ff971dfd
new sub-option for backup: time
...
New option to specify the timestamp for a backup
2017-09-09 13:26:35 +02:00
fa2ee78a5c
Merge pull request #1044 from lloeki/982-improve-restore
...
Improve restore
2017-09-04 21:51:12 +02:00
2d73a273af
saving a variable
2017-09-03 17:09:55 +02:00
761af08889
fix: bucker
...
bucker "always" does not return a unique id in case of exact same timestamps
2017-09-03 17:09:55 +02:00
3fbdd12b04
Handle SIGINFO on all supported platforms
...
Not just darwin
2017-09-02 22:06:31 +03:00
f880ff21aa
Fixing restore with excluded
...
An exclude filter is basically a 'wildcard but foo', so even if a
childMayMatch, other children of a dir may not, therefore childMayMatch
does not matter, but we should not go down unless the dir is selected
for restore.
2017-08-16 15:25:02 +02:00
4a36993c19
Smarter filter when children won't match
...
This improves restore performance by several orders of magniture by not
going through the whole tree recursively when we can anticipate that no
match will ever occur.
2017-08-16 15:25:02 +02:00
c2ff7150aa
internal: check error before deferring file Close()
...
If there is an error, file will be `nil`. We should check the returned error before deferring file `Close()`.
2017-08-13 19:28:13 +02:00
23c903074c
Move restic package to internal/restic
2017-07-24 17:43:32 +02:00
Michael Eischer
Quentin Lemaire
greatroar
Alexander Weiss
hoyho
Alexander Neumann
Leo R. Lundgren
aawsome
rawtaz
Lars Lehtonen
Ward Vandewege
Eric Hamilton
Alexandr Bruyako
Alexander Neumann
Jan Niklas Richter
Trey Moore
Dmitri Goutnik
plumbeo
Igor Fedorenko
Jakob Unterwurzacher
Matthew Holt
Iku Iwasa
Jasper Lievisse Adriaanse
Aaron Bieber
Matthew Dawson
George Armhold
Konrad Wojas
Anton Lindstrom
Herbert
Tobias Klein
Greg V
Loic Nageleisen
Emil Hessman