Alexander Neumann
ba892e1ec2
Refactor backup pipeline
2015-03-08 14:16:01 +01:00
Alexander Neumann
1516cb5996
Bump Go version to 1.3
2015-03-07 22:31:44 +01:00
Alexander Neumann
798ac97a52
Re-enable dedup, add test ( closes #95 )
2015-03-07 12:05:33 +01:00
Alexander Neumann
03bdbb064f
Allow cancelling walker
2015-03-02 19:58:11 +01:00
Alexander Neumann
f167366be5
Add walk for trees, restructure
2015-03-02 19:58:06 +01:00
Alexander Neumann
c8be54564f
Add local cache
2015-03-02 10:04:01 +01:00
Alexander Neumann
c056382c9c
Remove zlib compression
2015-03-02 10:02:55 +01:00
Alexander Neumann
b4cb75693e
Do not close file if it could not be opened
2015-02-22 00:21:56 +01:00
Alexander Neumann
ef498a987b
Allow configuring temp dir for tests
2015-02-21 16:53:14 +01:00
Alexander Neumann
a59b0ec1f6
Add progress report for loading blobs
2015-02-21 15:32:48 +01:00
Alexander Neumann
58cded6b75
Refactor progress reporting
2015-02-21 14:25:06 +01:00
Alexander Neumann
97b963ef1d
Report files and dirs
2015-02-21 14:05:02 +01:00
Alexander Neumann
2f3aa344af
Make decryptReader fulfill flate.Reader
...
benchmark old ns/op new ns/op delta
BenchmarkLoadJSONID 50232171 48596972 -3.26%
benchmark old allocs new allocs delta
BenchmarkLoadJSONID 43643 42884 -1.74%
benchmark old bytes new bytes delta
BenchmarkLoadJSONID 5773048 3785517 -34.43%
2015-02-18 23:30:59 +01:00
Alexander Neumann
a0fea201d9
Reenable Preloading
2015-02-18 23:10:59 +01:00
Alexander Neumann
f214dce87c
Add tests for Archiver.Preload() and a few more
2015-02-18 22:48:59 +01:00
Alexander Neumann
3bb2aba141
Remove test files
2015-02-18 21:50:59 +01:00
Alexander Neumann
68141c738b
Correct length of plaintext before decryption
2015-02-18 20:44:21 +01:00
Alexander Neumann
2462f93533
Disable Tests when benchdir is unset
2015-02-17 23:40:37 +01:00
Alexander Neumann
f8e47181b0
Add more debug for preloading
2015-02-17 23:38:40 +01:00
Alexander Neumann
65aae5f8c1
Disable preloading for now
2015-02-17 23:38:15 +01:00
Alexander Neumann
d3fe4a7f6d
Close Reader after read
2015-02-17 23:37:45 +01:00
Alexander Neumann
215fb982c6
Do not run archiver tests by default
2015-02-17 23:23:12 +01:00
Alexander Neumann
677a5038c5
Increase concurrency for loading blobs
2015-02-17 23:15:08 +01:00
Alexander Neumann
b6f25aa690
Remove Each(), add basic stats
2015-02-17 23:11:56 +01:00
Alexander Neumann
4bb7f2f2ed
Add Preloader for blobs
2015-02-17 22:39:44 +01:00
Alexander Neumann
c631b5c738
Add Len() functions for IDSet and Map
2015-02-17 22:39:16 +01:00
Alexander Neumann
4d1e7b9f24
Change parameters of LoadTree()
2015-02-17 22:37:34 +01:00
Alexander Neumann
5a16a66973
Add 'plaintext' parameter to key.Decrypt()
...
% benchcmp /tmp/{before,after}
benchmark old ns/op new ns/op delta
BenchmarkChunkEncrypt 261144414 260252046 -0.34%
BenchmarkChunkEncryptParallel 261201438 261267029 +0.03%
BenchmarkArchiveDirectory 0.00 0.00 +0.00%
BenchmarkEncryptWriter 88297245 87526529 -0.87%
BenchmarkEncrypt 87406446 87917897 +0.59%
BenchmarkDecryptReader 89948630 90042541 +0.10%
BenchmarkEncryptDecryptReader 178374144 179477306 +0.62%
BenchmarkDecrypt 88289705 87937073 -0.40%
BenchmarkSaveJSON 213917 213571 -0.16%
BenchmarkSaveFrom 74881361 75111148 +0.31%
benchmark old MB/s new MB/s speedup
BenchmarkChunkEncrypt 40.15 40.29 1.00x
BenchmarkChunkEncryptParallel 40.14 40.13 1.00x
BenchmarkEncryptWriter 95.00 95.84 1.01x
BenchmarkEncrypt 95.97 95.41 0.99x
BenchmarkDecryptReader 93.26 93.16 1.00x
BenchmarkEncryptDecryptReader 47.03 46.74 0.99x
BenchmarkDecrypt 95.01 95.39 1.00x
BenchmarkSaveFrom 56.01 55.84 1.00x
benchmark old allocs new allocs delta
BenchmarkChunkEncrypt 113 112 -0.88%
BenchmarkChunkEncryptParallel 104 103 -0.96%
BenchmarkArchiveDirectory 0 0 +0.00%
BenchmarkEncryptWriter 20 20 +0.00%
BenchmarkEncrypt 14 14 +0.00%
BenchmarkDecryptReader 18 18 +0.00%
BenchmarkEncryptDecryptReader 40 45 +12.50%
BenchmarkDecrypt 17 16 -5.88%
BenchmarkSaveJSON 125 125 +0.00%
BenchmarkSaveFrom 116 119 +2.59%
benchmark old bytes new bytes delta
BenchmarkChunkEncrypt 8515750 118956 -98.60%
BenchmarkChunkEncryptParallel 8515766 118972 -98.60%
BenchmarkArchiveDirectory 0 0 +0.00%
BenchmarkEncryptWriter 28927 28927 +0.00%
BenchmarkEncrypt 422313 2473 -99.41%
BenchmarkDecryptReader 527827 527827 +0.00%
BenchmarkEncryptDecryptReader 4100824 4100870 +0.00%
BenchmarkDecrypt 8391127 2509 -99.97%
BenchmarkSaveJSON 9208 9208 +0.00%
BenchmarkSaveFrom 39694 40541 +2.13%
2015-02-17 21:10:26 +01:00
Alexander Neumann
43ae0b69ab
Fix benchmarks for ChunkEncrypt
2015-02-17 20:02:43 +01:00
Alexander Neumann
af57fb86d7
Reduce memory usage of decryptReader
...
benchmark old ns/op new ns/op delta
BenchmarkChunkEncrypt 260007360 261144414 +0.44%
BenchmarkChunkEncryptParallel 262839697 261201438 -0.62%
BenchmarkArchiveDirectory 0.00 0.00 +0.00%
BenchmarkEncryptWriter 86994839 88297245 +1.50%
BenchmarkEncrypt 87414849 87406446 -0.01%
BenchmarkDecryptReader 90354651 89948630 -0.45%
BenchmarkEncryptDecryptReader 184533845 178374144 -3.34%
BenchmarkDecrypt 88153894 88289705 +0.15%
BenchmarkSaveJSON 213906 213917 +0.01%
BenchmarkSaveFrom 75263853 74881361 -0.51%
benchmark old MB/s new MB/s speedup
BenchmarkChunkEncrypt 40.33 40.15 1.00x
BenchmarkChunkEncryptParallel 39.89 40.14 1.01x
BenchmarkEncryptWriter 96.43 95.00 0.99x
BenchmarkEncrypt 95.96 95.97 1.00x
BenchmarkDecryptReader 92.84 93.26 1.00x
BenchmarkEncryptDecryptReader 45.46 47.03 1.03x
BenchmarkDecrypt 95.16 95.01 1.00x
BenchmarkSaveFrom 55.73 56.01 1.01x
benchmark old allocs new allocs delta
BenchmarkChunkEncrypt 113 113 +0.00%
BenchmarkChunkEncryptParallel 104 104 +0.00%
BenchmarkArchiveDirectory 0 0 +0.00%
BenchmarkEncryptWriter 20 20 +0.00%
BenchmarkEncrypt 14 14 +0.00%
BenchmarkDecryptReader 18 18 +0.00%
BenchmarkEncryptDecryptReader 55 40 -27.27%
BenchmarkDecrypt 17 17 +0.00%
BenchmarkSaveJSON 125 125 +0.00%
BenchmarkSaveFrom 119 116 -2.52%
benchmark old bytes new bytes delta
BenchmarkChunkEncrypt 8515750 8515750 +0.00%
BenchmarkChunkEncryptParallel 8515766 8515766 +0.00%
BenchmarkArchiveDirectory 0 0 +0.00%
BenchmarkEncryptWriter 28927 28927 +0.00%
BenchmarkEncrypt 422313 422313 +0.00%
BenchmarkDecryptReader 527827 527827 +0.00%
BenchmarkEncryptDecryptReader 35814894 4100824 -88.55%
BenchmarkDecrypt 8391127 8391127 +0.00%
BenchmarkSaveJSON 9208 9208 +0.00%
BenchmarkSaveFrom 40541 39694 -2.09%
2015-02-17 18:14:56 +01:00
Alexander Neumann
be03a31bdd
Use correct buffer size for benchmark
2015-02-17 18:05:58 +01:00
Alexander Neumann
4fd8222da6
Remove unneeded buffers from Archiver.SaveFile()
2015-02-16 23:47:32 +01:00
Alexander Neumann
d4cfa7b797
Remove scanner (not needed any more)
2015-02-16 23:47:14 +01:00
Alexander Neumann
0a164ad5e0
Add test and benchmark for Scanner
2015-02-16 23:13:23 +01:00
Alexander Neumann
a5e13f1280
Switch SaveFrom() to Reader/Writer
...
benchcmp:
benchmark old ns/op new ns/op delta
BenchmarkSaveFrom 76159512 74795081 -1.79%
benchmark old MB/s new MB/s speedup
BenchmarkSaveFrom 55.07 56.08 1.02x
benchmark old allocs new allocs delta
BenchmarkSaveFrom 97 117 +20.62%
benchmark old bytes new bytes delta
BenchmarkSaveFrom 433826 40117 -90.75%
benchmark old ns/op new ns/op delta
BenchmarkArchiveDirectory-4 29539886128 29262480891 -0.94%
benchmark old allocs new allocs delta
BenchmarkArchiveDirectory-4 540048 709662 +31.41%
benchmark old bytes new bytes delta
BenchmarkArchiveDirectory-4 404573416 443098816 +9.52%
2015-02-16 22:42:11 +01:00
Alexander Neumann
874b29b91f
Improve memory usage (and speed) of crypto reader/writer
...
benchcmp:
benchmark old ns/op new ns/op delta
BenchmarkChunkEncrypt 260811933 261294510 +0.19%
BenchmarkChunkEncryptParallel 262787948 259394601 -1.29%
BenchmarkEncryptWriter 89769328 87108804 -2.96%
BenchmarkEncrypt 88586645 87489151 -1.24%
BenchmarkDecryptReader 99669425 90019414 -9.68%
BenchmarkEncryptDecryptReader 189263157 185185733 -2.15%
BenchmarkDecrypt 89631109 89172417 -0.51%
benchmark old MB/s new MB/s speedup
BenchmarkChunkEncrypt 40.20 40.13 1.00x
BenchmarkChunkEncryptParallel 39.90 40.42 1.01x
BenchmarkEncryptWriter 93.45 96.30 1.03x
BenchmarkEncrypt 94.69 95.88 1.01x
BenchmarkDecryptReader 84.16 93.19 1.11x
BenchmarkEncryptDecryptReader 44.32 45.30 1.02x
BenchmarkDecrypt 93.59 94.07 1.01x
benchmark old allocs new allocs delta
BenchmarkChunkEncrypt 113 113 +0.00%
BenchmarkChunkEncryptParallel 104 104 +0.00%
BenchmarkEncryptWriter 20 20 +0.00%
BenchmarkEncrypt 14 14 +0.00%
BenchmarkDecryptReader 38 18 -52.63%
BenchmarkEncryptDecryptReader 61 55 -9.84%
BenchmarkDecrypt 17 17 +0.00%
benchmark old bytes new bytes delta
BenchmarkChunkEncrypt 8515750 8515750 +0.00%
BenchmarkChunkEncryptParallel 8515766 8515766 +0.00%
BenchmarkEncryptWriter 8391305 28927 -99.66%
BenchmarkEncrypt 2475 2475 +0.00%
BenchmarkDecryptReader 33563550 527827 -98.43%
BenchmarkEncryptDecryptReader 50348456 35814894 -28.87%
BenchmarkDecrypt 8391127 8391127 +0.00%
2015-02-16 22:35:46 +01:00
Alexander Neumann
d94d003165
Add test and benchmark for SaveFrom()
2015-02-16 22:35:46 +01:00
Alexander Neumann
731e81ef06
Re-use zlib writers to reduce memory usage
...
benchcmp:
benchmark old ns/op new ns/op delta
BenchmarkArchiveDirectory-4 29624960475 29511001504 -0.38%
BenchmarkSaveJSON-4 379833 225609 -40.60%
benchmark old allocs new allocs delta
BenchmarkArchiveDirectory-4 546736 540642 -1.11%
BenchmarkSaveJSON-4 150 126 -16.00%
benchmark old bytes new bytes delta
BenchmarkArchiveDirectory-4 1372476952 438519336 -68.05%
BenchmarkSaveJSON-4 1462773 9087 -99.38%
2015-02-16 22:35:26 +01:00
Alexander Neumann
06ed5c12b8
Remove Compress() and Uncompress() from backend
2015-02-16 00:30:51 +01:00
Alexander Neumann
b762a342d3
SaveJSON: use streams
2015-02-16 00:24:58 +01:00
Alexander Neumann
fb2cd7b485
HashingWriter: Track size
2015-02-16 00:24:43 +01:00
Alexander Neumann
b816192d1c
backend: Throw error when Close() is called twice
2015-02-16 00:24:02 +01:00
Alexander Neumann
259389f7d1
EncryptWriter: do not close underlying writer
2015-02-16 00:22:46 +01:00
Alexander Neumann
68911ef11d
server: use streams for LoadJSON*()
2015-02-15 23:49:06 +01:00
Alexander Neumann
907d401e32
backend: use HashingWriter
2015-02-15 23:46:21 +01:00
Alexander Neumann
3dbe02182b
Remove EncryptFrom*() methods, add Tests for EncryptTo()
2015-02-15 23:38:45 +01:00
Alexander Neumann
5c724b92b8
Add HashingWriter and HashAppendWriter
2015-02-15 23:21:35 +01:00
Alexander Neumann
5801d9f42f
Add EncryptTo() methods
2015-02-15 18:13:42 +01:00
Alexander Neumann
72c951a6ec
Simply backend interface
...
Rename CreateBlob() method to Create(), remove old Create* methods
2015-02-15 17:26:20 +01:00
Alexander Neumann
35636a9d92
Introduce CreateBlob() method for backend
2015-02-15 17:09:49 +01:00
