octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-12 00:06:35 +00:00
Code Issues Releases Wiki Activity
657 Commits 7 Branches 33 Tags 66 MiB
cf0531a2b1
Commit Graph

35 Commits

Author SHA1 Message Date
Alexander Neumann
f79e530e18 chunker: Further cleanup 2015-05-05 00:56:07 +02:00
Alexander Neumann
ea22b2dfb1 chunker: move buffer pool to chunker package 2015-05-05 00:51:47 +02:00
Alexander Neumann
589dbaaed2 chunker: Remove unused constant 2015-05-05 00:06:44 +02:00
Alexander Neumann
64a290c8db chunker: Add buffer to New() 2015-05-04 23:25:07 +02:00
Florian Weingarten
282c4379db Remove chunker.Reset() and chunker pool 2015-05-04 16:49:49 -04:00
Alexander Neumann
f8804d4265 chunker: Add benchmark for reducibility test 2015-05-04 21:41:57 +02:00
Alexander Neumann
ba1125df10 Cleanup commets with feedback from @fw42 2015-05-02 17:01:31 +02:00
Alexander Neumann
6fe38b0cd8 fix comments to have the standard form 2015-05-02 16:22:43 +02:00
Alexander Neumann
cf62eb3e15 chunker: unexport WindowSize and AverageBits 2015-05-02 16:19:16 +02:00
Florian Weingarten
ebae478631 Prefer mixedCaps over underscore 2015-04-24 19:39:32 -04:00
Alexander Neumann
4f4f3c421a Use package "restic/test" 2015-04-09 21:15:48 +02:00
Alexander Neumann
f02865a205 chunker: Use polynomial functions 2015-04-06 20:45:06 +02:00
Alexander Neumann
8986c75091 Add more benchmarks 2015-04-06 20:08:09 +02:00
Alexander Neumann
3d058929e3 Fix chunker tests 2015-04-06 00:31:31 +02:00
Alexander Neumann
f924dc1dc6 Remove debug output 2015-04-06 00:26:58 +02:00
Alexander Neumann
b2307cafa2 Seed chunker with random per-repository polynomial 2015-04-06 00:22:19 +02:00
Alexander Neumann
c969a42e8d chunker: Allow resetting polynomial 2015-04-05 22:53:06 +02:00
Alexander Neumann
25e3ac40ee chunker: Cache computations 2015-04-05 22:53:06 +02:00
Alexander Neumann
367cc75970 chunker: Refactor to take polynomial on creation 2015-04-05 22:52:55 +02:00
Alexander Neumann
3cdf3a25b9 chunker: Require a random irreducible polynomial 
This also implements the necessary polynomial arithmetics in F_2[X].
 2015-04-05 22:52:43 +02:00
Alexander Neumann
59cfc4bd03 Slightly improve performance for chunker 
benchcmp output:

benchmark                      old ns/op     new ns/op     delta
BenchmarkChunkerWithSHA256     151752169     148072829     -2.42%
BenchmarkChunkerWithMD5        97166433      93357090      -3.92%
BenchmarkChunker               77265146      73642723      -4.69%

benchmark                      old MB/s     new MB/s     speedup
BenchmarkChunkerWithSHA256     69.10        70.81        1.02x
BenchmarkChunkerWithMD5        107.92       112.32       1.04x
BenchmarkChunker               135.71       142.39       1.05x

benchmark                      old allocs     new allocs     delta
BenchmarkChunkerWithSHA256     15             13             -13.33%
BenchmarkChunkerWithMD5        14             12             -14.29%
BenchmarkChunker               8              6              -25.00%

benchmark                      old bytes     new bytes     delta
BenchmarkChunkerWithSHA256     262924        26821         -89.80%
BenchmarkChunkerWithMD5        262774        13554         -94.84%
BenchmarkChunker               262678        13458         -94.88%
 2015-02-13 20:03:40 +01:00
Alexander Neumann
3bf66d24fa chunker: Add test, only pass hash.Hash to New() 2015-02-11 13:12:36 +01:00
Alexander Neumann
3daeee8e2c Make chunker resettable 2015-02-09 23:37:33 +01:00
Alexander Neumann
bda33e612c Implement streaming chunker using io.Reader 2015-02-09 00:28:54 +01:00
Alexander Neumann
a5c33d80d8 Chunker: remove pool, buf and make bufsize an option 2015-02-08 19:32:12 +01:00
Alexander Neumann
bdcdcdea7d Optimize chunker 
Skip MinSize bytes at the beginning of each chunk. This increases
throughput about 30% percent.

Before:
    $ go test -v -bench . ./chunker
    === RUN TestChunker
    --- PASS: TestChunker (0.60s)
    === RUN TestChunkerReuse
    --- PASS: TestChunkerReuse (2.18s)
    PASS
    BenchmarkChunker          10     111229642 ns/op      94.27 MB/s
    --- BENCH: BenchmarkChunker
        chunker_test.go:168: 6 chunks, average chunk size: 1747626 bytes
        chunker_test.go:168: 6 chunks, average chunk size: 1747626 bytes
    ok      github.com/restic/restic/chunker    4.120s

After:
    $ go test -v -bench . ./chunker
    === RUN TestChunker
    --- PASS: TestChunker (0.48s)
    === RUN TestChunkerReuse
    --- PASS: TestChunkerReuse (1.75s)
    PASS
    BenchmarkChunker          20      81468596 ns/op     128.71 MB/s
    --- BENCH: BenchmarkChunker
        chunker_test.go:168: 6 chunks, average chunk size: 1747626 bytes
        chunker_test.go:168: 6 chunks, average chunk size: 1747626 bytes
    ok      github.com/restic/restic/chunker    4.061s
 2015-01-14 16:33:41 +01:00
Alexander Neumann
e2fea0d088 Rename khepri -> restic 2014-12-05 21:45:49 +01:00
Alexander Neumann
9dc0bf6378 Fix reusing chunker 2014-12-03 23:20:45 +01:00
Alexander Neumann
f95788ed90 Limit memory usage, add several sync.Pool 2014-11-23 21:26:20 +01:00
Alexander Neumann
d594cd89b7 Chunker: remove unneeded return 2014-11-19 22:56:52 +01:00
Alexander Neumann
83ea81d8c3 Fix special case with null byte input 2014-10-03 21:44:39 +02:00
Alexander Neumann
878a96e761 Fix missing byte 2014-09-18 22:34:08 +02:00
Alexander Neumann
1c03bd7f7b Clarify behaviour of Next() 2014-09-18 22:22:44 +02:00
Alexander Neumann
8e24a8b811 Add more documentation and error handling 2014-09-18 22:20:12 +02:00
Alexander Neumann
18131f88bf Add code for content defined chunking (CDC) 2014-09-18 21:10:20 +02:00