Philipp Serr
e7bf936d2b
Increase number of chunks and test repetitions
2015-12-28 18:33:28 +01:00
Philipp Serr
3d7f72311a
Provoke unreferenced packs using fewer goroutines
...
TestParallelSaveWithDuplication has been reworked to provoke
unreferenced packs using fewer goroutines than before and create
only one bytes.Reader per blob. This reduces memory usage
significantly.
The following actions have been taken to keep the chance of provoking
unreferenced packs due to #358 high:
* Interweaved processing of subsequent chunks
* Delaying each goroutine by a few pseudo-randomly chosen nanoseconds
(depending on the platform this will most probably only make the os
yield execution to another thread): together with the interweaved
processing of subsequent chunks, this ensures a minimalistic delay
between processing of (some) duplicated chunks
* Repeating the test 5 times with different seeds
On my test machine, the modified test provoked unreferenced packs 60
times in a row.
2015-12-28 18:33:26 +01:00
Philipp Serr
6a548336ec
Add a test concurrently saving duplicated chunks
...
This commit adds an integration test, that calls Archiver.Save from
many goroutines processing several duplicated chunks concurrently.
The test asserts, that after all chunks have been saved, there are no
unreferenced packs in the repository.
The test has been checked to give the expected results:
1) Running the test with maxParallel=1 (all chunks are processed
sequentially) has been verified not to produce any unreferenced
packs. Consequently the test passes.
2) Running the test with unbounded parallelism (maxParallel=
math.MaxInt32) has been verified to produce unreferenced packs
all the time (at least 25 test runs). Consequently the test fails
due to #358 .
references: #358
2015-12-28 18:33:22 +01:00
Alexander Neumann
86fcd170f6
Add and use MasterIndex
2015-10-12 22:34:12 +02:00
Florian Weingarten
36a62cf77d
Extract chunker
2015-07-08 16:58:23 -04:00
Alexander Neumann
26e4d2e019
tests: Remove more flags
2015-06-28 13:52:22 +02:00
Alexander Neumann
189a33730a
tests: Standardize use of SetupRepo/Teardown
2015-06-28 13:52:22 +02:00
Alexander Neumann
d9b5832034
Rename variables
2015-05-09 13:47:21 +02:00
Alexander Neumann
f79e530e18
chunker: Further cleanup
2015-05-05 00:56:07 +02:00
Alexander Neumann
ea22b2dfb1
chunker: move buffer pool to chunker package
2015-05-05 00:51:47 +02:00
Alexander Neumann
bdbb3ab329
Remove pools.go
2015-05-05 00:14:07 +02:00
Alexander Neumann
64a290c8db
chunker: Add buffer to New()
2015-05-04 23:25:07 +02:00
Florian Weingarten
282c4379db
Remove chunker.Reset() and chunker pool
2015-05-04 16:49:49 -04:00
Alexander Neumann
c57c4f0b8f
Merge branch 'add_config'
2015-05-04 22:13:59 +02:00
Alexander Neumann
ab3da5c8af
archiver: Remove trees and packs from dedup test
2015-05-04 22:11:53 +02:00
Alexander Neumann
1213d87b1a
server: Only save crypto.Key
...
At the moment, the server doesn't need the full server.Key (master and
user key), just the master key.
2015-05-03 18:56:20 +02:00
Alexander Neumann
1cedff0e9e
tests: Simplify key handling
2015-05-03 16:36:52 +02:00
Alexander Neumann
d753a5ffa3
archiver: correct benchmark
2015-05-01 22:58:50 +02:00
Florian Weingarten
0d9360a815
more cleanup
2015-04-29 22:33:34 -04:00
Alexander Neumann
ec108fb708
Remove backend.Tree
2015-04-29 23:27:36 +02:00
Alexander Neumann
60a0fe8349
Major restructure, bundle blobs
...
This commit also breaks the repository format.
2015-04-29 23:27:32 +02:00
Alexander Neumann
d19b23d4f1
Move Server and Key to new sub-package
2015-04-26 14:46:15 +02:00
Alexander Neumann
387082f983
Fix test BenchmarkChunkEncrypt
2015-04-25 00:54:15 +02:00
Alexander Neumann
3a2525809c
Reorganize crypto code
...
Move all crypto functions to package "crypto", move random generators
for tests into helper package.
2015-04-12 09:36:14 +02:00
Alexander Neumann
4f4f3c421a
Use package "restic/test"
2015-04-09 21:15:48 +02:00
Alexander Neumann
b2307cafa2
Seed chunker with random per-repository polynomial
2015-04-06 00:22:19 +02:00
Alexander Neumann
f157f775da
Add checks for Server.Load(), use Blob for load
2015-03-28 15:07:08 +01:00
Alexander Neumann
5e69788eac
Refactor backends
2015-03-28 14:36:50 +01:00
Alexander Neumann
91578fb2a3
Fix counting for tests for Preload
2015-03-10 15:58:23 +01:00
Alexander Neumann
6d5f1c0197
Add test for incremental backup
2015-03-10 14:17:54 +01:00
Alexander Neumann
14298fe232
Cache blobs for each snapshot
2015-03-09 23:50:32 +01:00
Alexander Neumann
f167366be5
Add walk for trees, restructure
2015-03-02 19:58:06 +01:00
Alexander Neumann
c8be54564f
Add local cache
2015-03-02 10:04:01 +01:00
Alexander Neumann
a59b0ec1f6
Add progress report for loading blobs
2015-02-21 15:32:48 +01:00
Alexander Neumann
58cded6b75
Refactor progress reporting
2015-02-21 14:25:06 +01:00
Alexander Neumann
f214dce87c
Add tests for Archiver.Preload() and a few more
2015-02-18 22:48:59 +01:00
Alexander Neumann
2462f93533
Disable Tests when benchdir is unset
2015-02-17 23:40:37 +01:00
Alexander Neumann
215fb982c6
Do not run archiver tests by default
2015-02-17 23:23:12 +01:00
Alexander Neumann
4bb7f2f2ed
Add Preloader for blobs
2015-02-17 22:39:44 +01:00
Alexander Neumann
43ae0b69ab
Fix benchmarks for ChunkEncrypt
2015-02-17 20:02:43 +01:00
Alexander Neumann
0a164ad5e0
Add test and benchmark for Scanner
2015-02-16 23:13:23 +01:00
Alexander Neumann
f8f8107d55
wip
2015-02-15 15:17:24 +01:00
Alexander Neumann
93abaf204a
Fix get_random for count != 4k, add randomReader()
2015-02-11 17:38:51 +01:00
Alexander Neumann
869ba83c6d
Add Benchmarks for Scanner and Archiver
2015-02-10 21:59:18 +01:00
Alexander Neumann
9fcd23bd38
Add parallel benchmark for chunk+encrypt
2015-02-09 23:39:16 +01:00
Alexander Neumann
bda33e612c
Implement streaming chunker using io.Reader
2015-02-09 00:28:54 +01:00
Alexander Neumann
e2fea0d088
Rename khepri -> restic
2014-12-05 21:45:49 +01:00
Alexander Neumann
719e121c74
Use large, dynamic buffer for encrypting maps
2014-11-23 22:58:41 +01:00
Alexander Neumann
f95788ed90
Limit memory usage, add several sync.Pool
2014-11-23 21:26:20 +01:00
Alexander Neumann
1ac4f92299
Add benchmark for Chunk+Encrypt
2014-11-17 23:52:31 +01:00
