octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-19 19:45:18 +00:00
Code Issues Releases Wiki Activity
5,161 Commits 7 Branches 33 Tags 66 MiB
d6b3859e48
Commit Graph

5161 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexander Weiss
b5543cff5d Loop over index files for 'list blobs' 
=> reduces memory consumption a lot!
 2020-06-14 17:25:34 +02:00
Alexander Neumann
285b5236c2
Merge pull request #2684 from MichaelEischer/cleanup-output 
Cleanup command output code
 2020-06-14 11:25:40 +02:00
Michael Eischer
bb1e258bb7 cat/stats: Get stdout from globalOptions 2020-06-14 11:08:11 +02:00
Michael Eischer
182655bc88 Replace fmt.Printf/Println/Fprintf with wrapper functions 
cmd/restic/globals.go already provides Printf, Println and Warnf wrapper
which get their output streams from the globalOptions object. This
allows for stream replacements when testing.
 2020-06-14 11:08:11 +02:00
MichaelEischer
74bc7141c1
Merge pull request #2781 from aawsome/reduce-index-memory 
Reduce index memory
 2020-06-14 10:58:28 +02:00
Alexander Weiss
1361341c58 don't save duplicate packIDs when using internal/repository/Index.Store 2020-06-14 07:56:24 +02:00
Alexander Weiss
ce4a2f4ca6 save packIDs and duplicates separately 
A side remark to the definition of Index.blob:

Another possibility would have been to use:
blob       map[restic.BlobHandle]*indexEntry

This would have led to the following sizes:
key: 32 + 1 = 33 bytes
value: 8 bytes
indexEntry:  8 + 4 + 4 = 16 bytes
each packID: 32 bytes

To save N index entries, we would therefore have needed:
N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes

More precicely, using a pointer instead of a direct entry is the better memory choice if:
OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1)
Under the assumption of OF=1.5, this means using pointers would have been the better choice
if sizeof(indexEntry) > 24 bytes.
 2020-06-14 07:56:21 +02:00
Alexander Weiss
cf979e2b81 make offset and length uint32 2020-06-14 07:50:19 +02:00
Michael Eischer
d92e2c5769 simplify index code 2020-06-14 07:50:19 +02:00
Alexander Weiss
7419844885 add changelog, benchmark, memory calculation 2020-06-14 07:50:15 +02:00
rawtaz
1d66bb9e62
Merge pull request #2784 from MichaelEischer/minio-sha256-changelog 
Add changelog for #2709
 2020-06-13 18:33:54 +02:00
Michael Eischer
0b2c31b05b Add changelog for #2709 2020-06-13 18:13:36 +02:00
MichaelEischer
dd7b4f54f5
Merge pull request #2709 from greatroar/minio-sha256 
Use Minio's optimized SHA-256
 2020-06-12 23:32:58 +02:00
MichaelEischer
6896c6449b
Merge pull request #2779 from greatroar/archiver-comment 
Fix up comment on archiver.BlobSaver.Save
 2020-06-12 23:04:21 +02:00
MichaelEischer
735a8074d5
Merge pull request #2773 from aawsome/index-uploads+knownblobs 
Fix non-intuitive repo behavior
 2020-06-12 22:41:04 +02:00
Alexander Weiss
70347e95d5 disable index uploads for prune command 
+ modifications of changelog
 2020-06-12 09:24:38 +02:00
greatroar
0fa3091c78 Fix up comment on archiver.BlobSaver.Save 2020-06-11 13:40:30 +02:00
Alexander Weiss
91906911b0 Fix non-intuitive repository behavior 
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
 2020-06-11 13:05:23 +02:00
MichaelEischer
fae7f78057
Merge pull request #2755 from greatroar/solaris-workaround 
Remove Solaris build workaround in integration tests
 2020-06-10 23:30:51 +02:00
MichaelEischer
ac9ec4b990
Merge pull request #2778 from MichaelEischer/fix-unlock-hint 
Revive hint to the unlock command if a repository is locked
 2020-06-10 21:26:38 +02:00
Michael Eischer
087c770161 Revive hint to the unlock command if a repository is locked 
errors.Fatalf wraps a error and just keeps an error message as a string.
This prevents the `restic.IsAlreadyLocked(err)` check from working as
the error is no longer an ErrAlreadyLocked.

Just add an additional remark to the error using `errors.WithMessage`.
 2020-06-10 20:58:56 +02:00
MichaelEischer
6856d1e422
Merge pull request #2749 from aawsome/fix-fullindex 
Change condition for full index
 2020-06-10 20:40:19 +02:00
Alexander Weiss
8c1261ff02 changed condition for full index 2020-06-07 22:00:49 +02:00
rawtaz
26704be17f
Merge pull request #2776 from nunoperalta/patch-1 
Fix #2774 - make unable to umount message clearer
 2020-06-07 17:19:50 +02:00
nunoperalta
2c3360db98
Fix #2774 - make unable to umount message clearer 2020-06-07 15:57:39 +01:00
MichaelEischer
cba6ad8d8e
Merge pull request #2711 from greatroar/update-x-net 
Update golang.org/x/net for fewer allocations in http2
 2020-05-25 20:43:29 +02:00
greatroar
2a3312ac35 Remove Solaris build workaround in integration tests 
This was introduced in #1821 because of issues with Go 1.9, but that
compiler version is no longer supported.
 2020-05-25 10:33:36 +02:00
greatroar
c35c4e0cbf Update golang.org/x/net for fewer allocations in http2 
name                                          old allocs/op  new allocs/op  delta
BackendREST/BenchmarkLoadFile-8                  2.15k ± 0%     2.14k ± 0%  -0.27%  (p=0.002 n=20+20)
BackendREST/BenchmarkLoadPartialFile-8             587 ± 0%       582 ± 0%  -0.88%  (p=0.000 n=20+20)
BackendREST/BenchmarkLoadPartialFileOffset-8       584 ± 0%       579 ± 0%  -0.90%  (p=0.000 n=19+16)

The throughput isn't significantly higher or lower.
 2020-05-24 16:39:17 +02:00
MichaelEischer
84475aa3a8
Merge pull request #2730 from greatroar/mount-build-tags 
Simplify build tags for restic mount
 2020-05-23 20:33:43 +02:00
MichaelEischer
f12f9ae240
Merge pull request #2741 from tbm/typos 
Fix typos
 2020-05-21 23:40:12 +02:00
Martin Michlmayr
5cc1760fdf
Fix typos 2020-05-16 14:05:26 +08:00
MichaelEischer
32ac5486e9
Merge pull request #2732 from restic/rawtaz-update-contributing 
Reword parts of Providing Patches in CONTRIBUTING.md
 2020-05-12 21:37:08 +02:00
rawtaz
c4336978eb
Reword parts of Providing Patches in CONTRIBUTION.md 
Primarily makes the request that contributors create a new branch for their changes stand out more.
 2020-05-12 14:03:57 +02:00
greatroar
649cbec6c5 Simplify build tags for restic mount 
This command can only be built on Darwin, FreeBSD and Linux
(and if we upgrade bazil.org/fuse, only FreeBSD and Linux:
https://github.com/bazil/fuse/issues/224).

Listing the few supported operating systems explicitly here makes
porting restic to new platforms easier.
 2020-05-12 11:30:41 +02:00
MichaelEischer
b17bd7f860
Merge pull request #2719 from greatroar/update-x-crypto 
Update x/crypto to stop poly1305 segfaulting on ARM
 2020-05-09 00:02:18 +02:00
greatroar
68f1e9c524 Update x/crypto to fix poly1305 on ARM 
The broken poly1305 implementation for arm64 was removed, fixing the
segfault issue on arm64 (gh-2618). The version for amd64 has improved
performance, which shows up in the internal/repository benchmark:

name              old speed      new speed      delta
SaveAndEncrypt-8   110MB/s ± 2%   113MB/s ± 1%  +2.24%  (p=0.000 n=20+20)
 2020-05-05 11:05:35 +02:00
MichaelEischer
1ee2306033
Merge pull request #2717 from MichaelEischer/fix-flaky-password-prompt 
backup: Avoid race between password prompt and open repository message
 2020-05-01 23:58:55 +02:00
Michael Eischer
c882a92cd6 backup: Avoid race between password prompt and open repository message 
`term.Print` sends the output via a channel to a goroutine which
actually prints the message. This may race with the password prompt
printed by `OpenRepository` resulting in a missing prompt.
 2020-05-01 23:42:16 +02:00
MichaelEischer
f54db5d796
Merge pull request #2713 from greatroar/unused 
Move Index.FindBlob to tests
 2020-05-01 20:41:59 +02:00
MichaelEischer
843e7f404e
Merge pull request #2716 from MichaelEischer/revert-apfs-test-workaround 
Revert "Darwin test fix: allow 1μs timestamp difference"
 2020-05-01 18:13:38 +02:00
Michael Eischer
d465b5b9ad Revert "Darwin test fix: allow 1μs timestamp difference" 
This reverts commit f3016a9096.
 2020-05-01 17:11:30 +02:00
greatroar
9f7cd69f13 Move Index.FindBlob to tests 2020-04-29 10:57:01 +02:00
greatroar
f97a680887 Fix repository benchmarks 
BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of
their time measuring SHA-256 performance in their checks.
 2020-04-28 07:57:29 +02:00
greatroar
42a3db05b0 Use Minio's optimized SHA-256 
internal/repository benchmarks on an Intel i7-3770k:

name               old speed      new speed       delta
PackerManager-8     209MB/s ± 1%    291MB/s ± 1%  +38.94%  (p=0.008 n=5+5)
SaveAndEncrypt-8    112MB/s ± 1%    135MB/s ± 1%  +20.25%  (p=0.008 n=5+5)
 2020-04-28 07:57:18 +02:00
MichaelEischer
070d43e290
Merge pull request #2704 from azak-azkaran/patch-1 
Update 030_preparing_a_new_repo.rst
 2020-04-23 20:48:34 +02:00
MichaelEischer
d4bd32a37e
Merge pull request #2640 from greatroar/simplify-loadblob-usage 
Simplify Repository.LoadBlob usage
 2020-04-23 20:23:35 +02:00
greatroar
e7d7b85d59 Merge Repository.{LoadBlob,loadBlob} 
Pushing the allocation logic down into the former loadBlob body means
that fewer allocations have to be performed:

name              old time/op    new time/op    delta
LoadTree-8           478µs ± 1%     481µs ± 2%    ~     (p=0.315 n=9+10)
LoadBlob-8          11.6ms ± 1%    11.6ms ± 2%    ~     (p=0.393 n=10+10)
LoadAndDecrypt-8    13.3ms ± 3%    13.3ms ± 3%    ~     (p=0.905 n=10+9)
LoadIndex-8         33.6ms ± 2%    33.2ms ± 1%  -1.15%  (p=0.028 n=10+9)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.1kB ± 0%  -0.23%  (p=0.000 n=10+10)
LoadBlob-8          2.28kB ± 0%    2.18kB ± 0%  -4.21%  (p=0.000 n=10+10)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%    ~     (all equal)
LoadIndex-8         5.22MB ± 0%    5.22MB ± 0%    ~     (p=0.631 n=10+10)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       651 ± 0%  -0.15%  (p=0.000 n=10+10)
LoadBlob-8            24.0 ± 0%      23.0 ± 0%  -4.17%  (p=0.000 n=10+10)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%    ~     (all equal)
LoadIndex-8          30.2k ± 0%     30.2k ± 0%    ~     (p=0.610 n=10+10)

name              old speed      new speed      delta
LoadBlob-8        86.4MB/s ± 1%  85.9MB/s ± 2%    ~     (p=0.393 n=10+10)
LoadAndDecrypt-8  75.4MB/s ± 3%  75.4MB/s ± 3%    ~     (p=0.858 n=10+9)
 2020-04-23 10:04:20 +02:00
greatroar
be5a0ff59f Centralize buffer allocation and size checking in Repository.LoadBlob 
Benchmark results for internal/repository:

name              old time/op    new time/op    delta
LoadTree-8           479µs ± 2%     478µs ± 1%   ~     (p=0.780 n=10+9)
LoadBlob-8          11.6ms ± 2%    11.6ms ± 1%   ~     (p=0.631 n=10+10)
LoadAndDecrypt-8    13.2ms ± 2%    13.3ms ± 3%   ~     (p=0.631 n=10+10)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.2kB ± 0%   ~     (all equal)
LoadBlob-8          2.28kB ± 0%    2.28kB ± 0%   ~     (all equal)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%   ~     (all equal)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       652 ± 0%   ~     (all equal)
LoadBlob-8            24.0 ± 0%      24.0 ± 0%   ~     (all equal)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%   ~     (all equal)

name              old speed      new speed      delta
LoadBlob-8        86.2MB/s ± 2%  86.4MB/s ± 1%   ~     (p=0.594 n=10+10)
LoadAndDecrypt-8  75.7MB/s ± 2%  75.4MB/s ± 3%   ~     (p=0.617 n=10+10)
 2020-04-23 10:04:20 +02:00
azak-azkaran
c5100d5632
Update 030_preparing_a_new_repo.rst 
fixed markdown for wasabi region link
 2020-04-23 09:24:19 +02:00
MichaelEischer
956a1b0f96
Merge pull request #2294 from BenWiederhake/debugsyms 
build: Preserve debug symbols in debug build
 2020-04-19 17:26:03 +02:00