Laurent Etiemble
c8b6e6fd9b
Increase maximum allowed file size to 10 Mblocks
...
Upgrade FileInfo up to 10000000 blocks. 1310 GB files can be shared.
Increase limit when unmarshaling XDR.
Increase the size of message.
2016-03-04 16:24:54 +01:00
Jakob Borg
e1ac740ac4
Use v2 of XDR package (actual changes)
2016-02-02 15:33:46 +01:00
Jakob Borg
4feeaf1641
Use v2 of XDR package (auto generated)
2016-02-02 12:44:33 +01:00
Jakob Borg
79680b1d5e
Benchmark for single database update
2016-01-28 09:12:01 +01:00
Jakob Borg
ac190b2e39
Change DB label format (index folders, devices)
2016-01-03 19:32:40 +01:00
Jakob Borg
fa4b4dece1
Compact database on startup (ref #2400 )
2015-11-24 13:17:30 +01:00
Jakob Borg
2f12d41d9d
Don't dirty blockmap key between lookups ( fixes #2455 )
2015-11-13 15:44:30 +01:00
Mike Boone
342036408e
Fix typos.
2015-11-11 21:20:34 -05:00
Jakob Borg
36f6a9347c
Benchmark must use *db.Instance
2015-11-05 17:46:53 +00:00
Jakob Borg
2a4fc28318
We should pass around db.Instance instead of leveldb.DB
...
We're going to need the db.Instance to keep some state, and for that to
work we need the same one passed around everywhere. Hence this moves the
leveldb-specific file opening stuff into the db package and exports the
dbInstance type.
2015-10-31 12:35:30 +01:00
Jakob Borg
313485e406
Remove file that snuck in by mistake
2015-10-31 11:38:59 +01:00
Jakob Borg
faf4267c73
Refactor: the various db key functions should be instance methods
2015-10-31 11:27:04 +01:00
Jakob Borg
c4ad97136f
Move leveldb instance and transactions into separate files
2015-10-29 08:07:51 +01:00
Jakob Borg
fd7b8ec77e
Neater transaction handling
2015-10-28 22:04:00 +01:00
Jakob Borg
e28c991331
Create an instance type to tie database methods to
2015-10-28 21:03:05 +01:00
Jakob Borg
83131103cf
Don't load block list in ...Truncated methods
...
Speeds up and reduces allocations on those operations, at the price of
having a manually tweaked XDR decoder for FileInfoTruncated.
benchmark old ns/op new ns/op delta
BenchmarkReplaceAll-8 1868198122 1880206886 +0.64%
BenchmarkUpdateOneChanged-8 231852 172695 -25.51%
BenchmarkUpdateOneUnchanged-8 230624 179341 -22.24%
BenchmarkNeedHalf-8 104601744 109461427 +4.65%
BenchmarkHave-8 29102480 34105026 +17.19%
BenchmarkGlobal-8 150547687 172778045 +14.77%
BenchmarkNeedHalfTruncated-8 102471355 76564986 -25.28%
BenchmarkHaveTruncated-8 28758368 14277481 -50.35%
BenchmarkGlobalTruncated-8 151192913 106070136 -29.84%
benchmark old allocs new allocs delta
BenchmarkReplaceAll-8 555577 557554 +0.36%
BenchmarkUpdateOneChanged-8 1135 587 -48.28%
BenchmarkUpdateOneUnchanged-8 1135 587 -48.28%
BenchmarkNeedHalf-8 374780 374775 -0.00%
BenchmarkHave-8 151992 152085 +0.06%
BenchmarkGlobal-8 530033 530135 +0.02%
BenchmarkNeedHalfTruncated-8 374699 22160 -94.09%
BenchmarkHaveTruncated-8 151834 4904 -96.77%
BenchmarkGlobalTruncated-8 530037 30536 -94.24%
benchmark old bytes new bytes delta
BenchmarkReplaceAll-8 1765116216 1765305376 +0.01%
BenchmarkUpdateOneChanged-8 135085 93043 -31.12%
BenchmarkUpdateOneUnchanged-8 134976 92928 -31.15%
BenchmarkNeedHalf-8 44758752 44751791 -0.02%
BenchmarkHave-8 11845052 11967172 +1.03%
BenchmarkGlobal-8 80431136 80431065 -0.00%
BenchmarkNeedHalfTruncated-8 46526459 18243543 -60.79%
BenchmarkHaveTruncated-8 11348357 418998 -96.31%
BenchmarkGlobalTruncated-8 80977672 43116991 -46.75%
2015-10-21 23:49:10 +02:00
Jakob Borg
c1591a5efd
Only run benchmarks with -tags benchmark
...
Avoids creating temp database and stuff on a normal test run
2015-10-21 23:19:26 +02:00
Jakob Borg
918ef4dff8
Use batches in blockmap, speeds up and reduces memory usage on large Replace and Update ops
...
benchmark old ns/op new ns/op delta
BenchmarkReplaceAll-8 2880834572 1868198122 -35.15%
BenchmarkUpdateOneChanged-8 236596 231852 -2.01%
BenchmarkUpdateOneUnchanged-8 227326 230624 +1.45%
BenchmarkNeedHalf-8 105151538 104601744 -0.52%
BenchmarkHave-8 28827492 29102480 +0.95%
BenchmarkGlobal-8 150768724 150547687 -0.15%
BenchmarkNeedHalfTruncated-8 104434216 102471355 -1.88%
BenchmarkHaveTruncated-8 27860093 28758368 +3.22%
BenchmarkGlobalTruncated-8 149972888 151192913 +0.81%
benchmark old allocs new allocs delta
BenchmarkReplaceAll-8 555451 555577 +0.02%
BenchmarkUpdateOneChanged-8 1135 1135 +0.00%
BenchmarkUpdateOneUnchanged-8 1135 1135 +0.00%
BenchmarkNeedHalf-8 374779 374780 +0.00%
BenchmarkHave-8 151996 151992 -0.00%
BenchmarkGlobal-8 530066 530033 -0.01%
BenchmarkNeedHalfTruncated-8 374702 374699 -0.00%
BenchmarkHaveTruncated-8 151834 151834 +0.00%
BenchmarkGlobalTruncated-8 530049 530037 -0.00%
benchmark old bytes new bytes delta
BenchmarkReplaceAll-8 5018351912 1765116216 -64.83%
BenchmarkUpdateOneChanged-8 135085 135085 +0.00%
BenchmarkUpdateOneUnchanged-8 134976 134976 +0.00%
BenchmarkNeedHalf-8 44769400 44758752 -0.02%
BenchmarkHave-8 11930612 11845052 -0.72%
BenchmarkGlobal-8 81523668 80431136 -1.34%
BenchmarkNeedHalfTruncated-8 46692342 46526459 -0.36%
BenchmarkHaveTruncated-8 11348357 11348357 +0.00%
BenchmarkGlobalTruncated-8 81843956 80977672 -1.06%
2015-10-21 23:05:23 +02:00
Jakob Borg
0d9a04c713
Reuse blockkey, speeds up large Update and Replace calls
...
benchmark old ns/op new ns/op delta
BenchmarkReplaceAll-8 2866418930 2880834572 +0.50%
BenchmarkUpdateOneChanged-8 226635 236596 +4.40%
BenchmarkUpdateOneUnchanged-8 229090 227326 -0.77%
BenchmarkNeedHalf-8 104483393 105151538 +0.64%
BenchmarkHave-8 29288220 28827492 -1.57%
BenchmarkGlobal-8 159269126 150768724 -5.34%
BenchmarkNeedHalfTruncated-8 108235000 104434216 -3.51%
BenchmarkHaveTruncated-8 28945489 27860093 -3.75%
BenchmarkGlobalTruncated-8 149355833 149972888 +0.41%
benchmark old allocs new allocs delta
BenchmarkReplaceAll-8 1054944 555451 -47.35%
BenchmarkUpdateOneChanged-8 1135 1135 +0.00%
BenchmarkUpdateOneUnchanged-8 1135 1135 +0.00%
BenchmarkNeedHalf-8 374777 374779 +0.00%
BenchmarkHave-8 151995 151996 +0.00%
BenchmarkGlobal-8 530063 530066 +0.00%
BenchmarkNeedHalfTruncated-8 374699 374702 +0.00%
BenchmarkHaveTruncated-8 151834 151834 +0.00%
BenchmarkGlobalTruncated-8 530021 530049 +0.01%
benchmark old bytes new bytes delta
BenchmarkReplaceAll-8 5074297112 5018351912 -1.10%
BenchmarkUpdateOneChanged-8 135097 135085 -0.01%
BenchmarkUpdateOneUnchanged-8 134976 134976 +0.00%
BenchmarkNeedHalf-8 44759436 44769400 +0.02%
BenchmarkHave-8 11911138 11930612 +0.16%
BenchmarkGlobal-8 81609867 81523668 -0.11%
BenchmarkNeedHalfTruncated-8 46588024 46692342 +0.22%
BenchmarkHaveTruncated-8 11348354 11348357 +0.00%
BenchmarkGlobalTruncated-8 79485168 81843956 +2.97%
2015-10-21 23:05:23 +02:00
Jakob Borg
0c0c69f0cf
The GC runs are legacy and slows things down quite a bit
...
benchmark old ns/op new ns/op delta
BenchmarkReplaceAll-8 2942370526 2866418930 -2.58%
BenchmarkUpdateOneChanged-8 7402489 226635 -96.94%
BenchmarkUpdateOneUnchanged-8 7298777 229090 -96.86%
BenchmarkNeedHalf-8 113608416 104483393 -8.03%
BenchmarkHave-8 29834263 29288220 -1.83%
BenchmarkGlobal-8 162773699 159269126 -2.15%
BenchmarkNeedHalfTruncated-8 111943400 108235000 -3.31%
BenchmarkHaveTruncated-8 29490369 28945489 -1.85%
BenchmarkGlobalTruncated-8 165841081 149355833 -9.94%
benchmark old allocs new allocs delta
BenchmarkReplaceAll-8 1054942 1054944 +0.00%
BenchmarkUpdateOneChanged-8 1149 1135 -1.22%
BenchmarkUpdateOneUnchanged-8 1135 1135 +0.00%
BenchmarkNeedHalf-8 374774 374777 +0.00%
BenchmarkHave-8 151995 151995 +0.00%
BenchmarkGlobal-8 530042 530063 +0.00%
BenchmarkNeedHalfTruncated-8 374697 374699 +0.00%
BenchmarkHaveTruncated-8 151834 151834 +0.00%
BenchmarkGlobalTruncated-8 530050 530021 -0.01%
benchmark old bytes new bytes delta
BenchmarkReplaceAll-8 5074294728 5074297112 +0.00%
BenchmarkUpdateOneChanged-8 141048 135097 -4.22%
BenchmarkUpdateOneUnchanged-8 134976 134976 +0.00%
BenchmarkNeedHalf-8 44734813 44759436 +0.06%
BenchmarkHave-8 11911634 11911138 -0.00%
BenchmarkGlobal-8 80436854 81609867 +1.46%
BenchmarkNeedHalfTruncated-8 46514673 46588024 +0.16%
BenchmarkHaveTruncated-8 11348357 11348354 -0.00%
BenchmarkGlobalTruncated-8 81730740 79485168 -2.75%
2015-10-21 23:05:22 +02:00
Jakob Borg
943e80e26c
Make benchmarks more realistic
2015-10-21 23:04:29 +02:00
Jakob Borg
1eca4170f7
Add test for LocalSize/GlobalSize results
2015-10-21 21:58:48 +02:00
Jakob Borg
c268e4ad1b
Also keep GlobalSize in RAM
2015-10-21 21:58:48 +02:00
Jakob Borg
d4f81e8791
Keep LocalSize data in RAM
2015-10-21 21:58:48 +02:00
Jakob Borg
76af9ba53d
Implement facility based logger, debugging via REST API
...
This implements a new debug/trace infrastructure based on a slightly
hacked up logger. Instead of the traditional "if debug { ... }" I've
rewritten the logger to have no-op Debugln and Debugf, unless debugging
has been enabled for a given "facility". The "facility" is just a
string, typically a package name.
This will be slightly slower than before; but not that much as it's
mostly a function call that returns immediately. For the cases where it
matters (the Debugln takes a hex.Dump() of something for example, and
it's not in a very occasional "if err != nil" branch) there is an
l.ShouldDebug(facility) that is fast enough to be used like the old "if
debug".
The point of all this is that we can now toggle debugging for the
various packages on and off at runtime. There's a new method
/rest/system/debug that can be POSTed a set of facilities to enable and
disable debug for, or GET from to get a list of facilities with
descriptions and their current debug status.
Similarly a /rest/system/log?since=... can grab the latest log entries,
up to 250 of them (hardcoded constant in main.go) plus the initial few.
Not implemented in this commit (but planned) is a simple debug GUI
available on /debug that shows the current log in an easily pasteable
format and has checkboxes to enable the various debug facilities.
The debug instructions to a user then becomes "visit this URL, check
these boxes, reproduce your problem, copy and paste the log". The actual
log viewer on the hypothetical /debug URL can poll regularly for new log
entries and this bypass the 250 line limit.
The existing STTRACE=foo variable is still obeyed and just sets the
start state of the system.
2015-10-03 18:09:53 +02:00
Jakob Borg
4581c57478
Fix import paths
2015-09-22 19:38:46 +02:00
Jakob Borg
4a9997e449
lib/db need not depend on lib/config
2015-09-04 12:01:00 +02:00
Jakob Borg
7705a6c1f1
mv internal lib
2015-08-09 09:35:26 +02:00
