octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-19 19:45:18 +00:00
Code Issues Releases Wiki Activity
5,518 Commits 7 Branches 33 Tags 66 MiB
77551597b2
Commit Graph

5518 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andreas Olsson
0fcef2ec23
Use https:// for linking CACHEDIR.TAG specs 
That site might not have supported https:// when those links were
originally added. It does now.

Also dropping the _spec.html_ ending of the url, there being a `<link
rel="canonical" ...>` tag suggesting that that no longer being the
preferred address.
 2020-06-21 19:36:26 +02:00
MichaelEischer
212607dc8a
Merge pull request #2760 from greatroar/backend-benchmark 
Fix backend benchmarks + a micro-optimization
 2020-06-17 23:17:05 +02:00
greatroar
55c3a90a0d Clarify rclone-over-SSH docs 
Also added a link to S. Ruderich's blog post explaining append-only
repos using rclone and SSH.
 2020-06-17 15:22:20 +02:00
greatroar
190d8e2f51 Flatten backend.LimitedReadCloser structure 
This inlines the io.LimitedReader into the LimitedReadCloser body to
achieve fewer allocations. Results on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadPartialFile-8           412µs ± 4%     413µs ± 4%    ~     (p=0.634 n=17+17)
Backend/BenchmarkLoadPartialFileOffset-8     455µs ±13%     441µs ±10%    ~     (p=0.072 n=20+18)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadPartialFile-8        10.2GB/s ± 3%  10.2GB/s ± 4%    ~     (p=0.817 n=16+17)
Backend/BenchmarkLoadPartialFileOffset-8  9.25GB/s ±12%  9.54GB/s ± 9%    ~     (p=0.072 n=20+18)

name                                      old alloc/op   new alloc/op   delta
Backend/BenchmarkLoadPartialFile-8            888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)

name                                      old allocs/op  new allocs/op  delta
Backend/BenchmarkLoadPartialFile-8            18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
 2020-06-17 13:11:45 +02:00
greatroar
f4cd2a7120 Make backend benchmarks fairer by removing checks 
Checking whether the right data is returned takes up half the time in
some benchmarks. Results for local backend benchmarks on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadFile-8                 4.89ms ± 0%    2.72ms ± 1%   -44.26%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8           936µs ± 6%     439µs ±15%   -53.07%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8     940µs ± 1%     456µs ±10%   -51.50%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                     23.9ms ±14%    24.8ms ±41%      ~     (p=0.690 n=5+5)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadFile-8               3.43GB/s ± 0%  6.16GB/s ± 1%   +79.40%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8        4.48GB/s ± 6%  9.63GB/s ±14%  +114.78%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8  4.46GB/s ± 1%  9.22GB/s ±10%  +106.74%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                    706MB/s ±13%   698MB/s ±31%      ~     (p=0.690 n=5+5)
 2020-06-17 13:11:45 +02:00
MichaelEischer
aba270df7e
Merge pull request #2786 from aawsome/memory-list-blobs 
Loop over index files for 'list blobs'
 2020-06-15 22:27:29 +02:00
Alexander Weiss
b5543cff5d Loop over index files for 'list blobs' 
=> reduces memory consumption a lot!
 2020-06-14 17:25:34 +02:00
Alexander Neumann
285b5236c2
Merge pull request #2684 from MichaelEischer/cleanup-output 
Cleanup command output code
 2020-06-14 11:25:40 +02:00
Michael Eischer
bb1e258bb7 cat/stats: Get stdout from globalOptions 2020-06-14 11:08:11 +02:00
Michael Eischer
182655bc88 Replace fmt.Printf/Println/Fprintf with wrapper functions 
cmd/restic/globals.go already provides Printf, Println and Warnf wrapper
which get their output streams from the globalOptions object. This
allows for stream replacements when testing.
 2020-06-14 11:08:11 +02:00
MichaelEischer
74bc7141c1
Merge pull request #2781 from aawsome/reduce-index-memory 
Reduce index memory
 2020-06-14 10:58:28 +02:00
Alexander Weiss
1361341c58 don't save duplicate packIDs when using internal/repository/Index.Store 2020-06-14 07:56:24 +02:00
Alexander Weiss
ce4a2f4ca6 save packIDs and duplicates separately 
A side remark to the definition of Index.blob:

Another possibility would have been to use:
blob       map[restic.BlobHandle]*indexEntry

This would have led to the following sizes:
key: 32 + 1 = 33 bytes
value: 8 bytes
indexEntry:  8 + 4 + 4 = 16 bytes
each packID: 32 bytes

To save N index entries, we would therefore have needed:
N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes

More precicely, using a pointer instead of a direct entry is the better memory choice if:
OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1)
Under the assumption of OF=1.5, this means using pointers would have been the better choice
if sizeof(indexEntry) > 24 bytes.
 2020-06-14 07:56:21 +02:00
Alexander Weiss
cf979e2b81 make offset and length uint32 2020-06-14 07:50:19 +02:00
Michael Eischer
d92e2c5769 simplify index code 2020-06-14 07:50:19 +02:00
Alexander Weiss
7419844885 add changelog, benchmark, memory calculation 2020-06-14 07:50:15 +02:00
rawtaz
1d66bb9e62
Merge pull request #2784 from MichaelEischer/minio-sha256-changelog 
Add changelog for #2709
 2020-06-13 18:33:54 +02:00
Michael Eischer
0b2c31b05b Add changelog for #2709 2020-06-13 18:13:36 +02:00
Alexander Weiss
d3c59d18e5 Fix inconsistency of saving/loading config file 
Fix saving/loading config file: Always set ID to a zero ID.
 2020-06-13 16:30:23 +02:00
MichaelEischer
dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256 
Use Minio's optimized SHA-256
 2020-06-12 23:32:58 +02:00
MichaelEischer
6896c6449b
Merge pull request #2779 from greatroar/archiver-comment 
Fix up comment on archiver.BlobSaver.Save
 2020-06-12 23:04:21 +02:00
MichaelEischer
735a8074d5
Merge pull request #2773 from aawsome/index-uploads+knownblobs 
Fix non-intuitive repo behavior
 2020-06-12 22:41:04 +02:00
Alexander Weiss
70347e95d5 disable index uploads for prune command 
+ modifications of changelog
 2020-06-12 09:24:38 +02:00
greatroar
0fa3091c78 Fix up comment on archiver.BlobSaver.Save 2020-06-11 13:40:30 +02:00
Alexander Weiss
91906911b0 Fix non-intuitive repository behavior 
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
 2020-06-11 13:05:23 +02:00
MichaelEischer
fae7f78057
Merge pull request #2755 from greatroar/solaris-workaround 
Remove Solaris build workaround in integration tests
 2020-06-10 23:30:51 +02:00
MichaelEischer
ac9ec4b990
Merge pull request #2778 from MichaelEischer/fix-unlock-hint 
Revive hint to the unlock command if a repository is locked
 2020-06-10 21:26:38 +02:00
Michael Eischer
087c770161 Revive hint to the unlock command if a repository is locked 
errors.Fatalf wraps a error and just keeps an error message as a string.
This prevents the `restic.IsAlreadyLocked(err)` check from working as
the error is no longer an ErrAlreadyLocked.

Just add an additional remark to the error using `errors.WithMessage`.
 2020-06-10 20:58:56 +02:00
MichaelEischer
6856d1e422
Merge pull request #2749 from aawsome/fix-fullindex 
Change condition for full index
 2020-06-10 20:40:19 +02:00
Alexander Weiss
8c1261ff02 changed condition for full index 2020-06-07 22:00:49 +02:00
rawtaz
26704be17f
Merge pull request #2776 from nunoperalta/patch-1 
Fix #2774 - make unable to umount message clearer
 2020-06-07 17:19:50 +02:00
nunoperalta
2c3360db98
Fix #2774 - make unable to umount message clearer 2020-06-07 15:57:39 +01:00
MichaelEischer
cba6ad8d8e
Merge pull request #2711 from greatroar/update-x-net 
Update golang.org/x/net for fewer allocations in http2
 2020-05-25 20:43:29 +02:00
greatroar
2a3312ac35 Remove Solaris build workaround in integration tests 
This was introduced in #1821 because of issues with Go 1.9, but that
compiler version is no longer supported.
 2020-05-25 10:33:36 +02:00
greatroar
c35c4e0cbf Update golang.org/x/net for fewer allocations in http2 
name                                          old allocs/op  new allocs/op  delta
BackendREST/BenchmarkLoadFile-8                  2.15k ± 0%     2.14k ± 0%  -0.27%  (p=0.002 n=20+20)
BackendREST/BenchmarkLoadPartialFile-8             587 ± 0%       582 ± 0%  -0.88%  (p=0.000 n=20+20)
BackendREST/BenchmarkLoadPartialFileOffset-8       584 ± 0%       579 ± 0%  -0.90%  (p=0.000 n=19+16)

The throughput isn't significantly higher or lower.
 2020-05-24 16:39:17 +02:00
MichaelEischer
84475aa3a8
Merge pull request #2730 from greatroar/mount-build-tags 
Simplify build tags for restic mount
 2020-05-23 20:33:43 +02:00
MichaelEischer
f12f9ae240
Merge pull request #2741 from tbm/typos 
Fix typos
 2020-05-21 23:40:12 +02:00
Martin Michlmayr
5cc1760fdf
Fix typos 2020-05-16 14:05:26 +08:00
MichaelEischer
32ac5486e9
Merge pull request #2732 from restic/rawtaz-update-contributing 
Reword parts of Providing Patches in CONTRIBUTING.md
 2020-05-12 21:37:08 +02:00
rawtaz
c4336978eb
Reword parts of Providing Patches in CONTRIBUTION.md 
Primarily makes the request that contributors create a new branch for their changes stand out more.
 2020-05-12 14:03:57 +02:00
greatroar
649cbec6c5 Simplify build tags for restic mount 
This command can only be built on Darwin, FreeBSD and Linux
(and if we upgrade bazil.org/fuse, only FreeBSD and Linux:
https://github.com/bazil/fuse/issues/224).

Listing the few supported operating systems explicitly here makes
porting restic to new platforms easier.
 2020-05-12 11:30:41 +02:00
MichaelEischer
b17bd7f860
Merge pull request #2719 from greatroar/update-x-crypto 
Update x/crypto to stop poly1305 segfaulting on ARM
 2020-05-09 00:02:18 +02:00
greatroar
68f1e9c524 Update x/crypto to fix poly1305 on ARM 
The broken poly1305 implementation for arm64 was removed, fixing the
segfault issue on arm64 (gh-2618). The version for amd64 has improved
performance, which shows up in the internal/repository benchmark:

name              old speed      new speed      delta
SaveAndEncrypt-8   110MB/s ± 2%   113MB/s ± 1%  +2.24%  (p=0.000 n=20+20)
 2020-05-05 11:05:35 +02:00
MichaelEischer
1ee2306033
Merge pull request #2717 from MichaelEischer/fix-flaky-password-prompt 
backup: Avoid race between password prompt and open repository message
 2020-05-01 23:58:55 +02:00
Michael Eischer
c882a92cd6 backup: Avoid race between password prompt and open repository message 
`term.Print` sends the output via a channel to a goroutine which
actually prints the message. This may race with the password prompt
printed by `OpenRepository` resulting in a missing prompt.
 2020-05-01 23:42:16 +02:00
MichaelEischer
f54db5d796
Merge pull request #2713 from greatroar/unused 
Move Index.FindBlob to tests
 2020-05-01 20:41:59 +02:00
MichaelEischer
843e7f404e
Merge pull request #2716 from MichaelEischer/revert-apfs-test-workaround 
Revert "Darwin test fix: allow 1μs timestamp difference"
 2020-05-01 18:13:38 +02:00
Michael Eischer
d465b5b9ad Revert "Darwin test fix: allow 1μs timestamp difference" 
This reverts commit f3016a9096.
 2020-05-01 17:11:30 +02:00
greatroar
9f7cd69f13 Move Index.FindBlob to tests 2020-04-29 10:57:01 +02:00
greatroar
f97a680887 Fix repository benchmarks 
BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of
their time measuring SHA-256 performance in their checks.
 2020-04-28 07:57:29 +02:00