Alexander Neumann
4f1f03cdb9
Move testing for known blobs to Archiver
...
This removes the list of in-flight blobs from the master index and
instead keeps a list of "known" blobs in the Archiver. "known" here
means: either already processed, or included in an index. This property
is tested atomically, when the blob is not in the list of "known" blobs,
it is added to the list and the caller is responsible to make this
happen (i.e. save the blob).
2016-02-01 23:50:56 +01:00
Alexander Neumann
382c766983
Move test for #405 : Test Archiver instead of Repo
2016-02-01 23:50:41 +01:00
Alexander Neumann
f5f6e9cf37
Add test to reproduce #405
2016-02-01 23:35:01 +01:00
Alexander Neumann
f2371db2a9
repository: remove decryptReadCloser
2016-01-27 22:47:09 +01:00
Alexander Neumann
ea29ad6f96
Remove last ocurrence of Create()
2016-01-24 19:30:14 +01:00
Alexander Neumann
ac2fe4e04f
Remove BlobWriter
2016-01-24 18:53:39 +01:00
Alexander Neumann
cfdd3a853d
Remove usage of CreateEncryptedBlob()
2016-01-24 18:52:11 +01:00
Alexander Neumann
01e40e62bf
repo: Use Save() instead of Create()
2016-01-24 18:50:41 +01:00
Alexander Neumann
fe565e17c3
Key: Use Save() instead of Create()
2016-01-24 17:52:44 +01:00
Alexander Neumann
782a1bf7b0
repository: remove GetDecryptReader()
2016-01-24 00:12:17 +01:00
Alexander Neumann
3191778d33
repository: Use Load() instead of GetReader()
2016-01-24 00:12:09 +01:00
Alexander Neumann
9bfa633187
repository/key: Use Load() instead of GetReader()
2016-01-23 23:48:19 +01:00
Alexander Neumann
10b03eee27
Add comment
2016-01-23 23:27:40 +01:00
Alexander Neumann
8b7bf8691d
backend: Remove Get()
...
This is the first commit that removes the (redundant) Get() method of
the backend interface. Get(x, y) is equivalent to GetReader(x, y, 0, 0).
2016-01-23 13:13:05 +01:00
Alexander Neumann
f53008d916
Allow saving duplicate blobs in the repacker
...
This adds code to the master index to allow saving duplicate blobs
within the repacker. In this mode, only the list of currently in flight
blobs is consulted, and not the index. This correct because while
repacking, a unique list of blobs is saved again to the index.
2016-01-17 21:14:55 +01:00
Alexander Neumann
34c1056efc
Merge pull request #358 from episource/iss358_pack_not_referened_add_test
...
Closes #365
Closes #358
2016-01-17 20:07:56 +01:00
Alexander Neumann
6a56d5b87b
Repo: Add more debug
2016-01-17 18:48:05 +01:00
Philipp Serr
0fde09a866
Lock MasterIndex and InFlight store together
...
fixes: #358
2015-12-28 18:40:43 +01:00
Alexander Neumann
8209bb309b
split out decryptReader and packerManager
2015-11-29 14:29:59 +01:00
Alexander Neumann
d4b873ca76
Repo: add documentation
2015-11-29 14:29:57 +01:00
Alexander Neumann
5b601f00b1
Add error checking
2015-11-29 14:25:57 +01:00
Alexander Neumann
567de35df4
Save new packs in index atomically
...
This commit fixes a situation reported by a user where two indexes
contained information about the same pack without overlap, e.g.:
Index 3e6a32 contained:
{
"id": "c02e3b",
"blobs": [
{
"id": "8114b1",
"type": "data",
"offset": 0,
"length": 530107
}
]
}
And index 62da5f contained:
{
"id": "c02e3b",
"blobs": [
{
"id": "e344f8",
"type": "data",
"offset": 1975848,
"length": 3426468
},
{
"id": "939ed9",
"type": "data",
"offset": 530107,
"length": 1445741
}
]
}
This commit adds all blobs in a pack in one atomic operation so that
intermediate such as these do not happen.
2015-11-20 22:56:56 +01:00
Alexander Neumann
96061d2a2f
Fix debug log message
2015-11-13 23:47:53 +01:00
Alexander Neumann
5776b8f01c
remove ConvertIndex
2015-11-08 22:27:13 +01:00
Alexander Neumann
6c54d3fa82
index: also mark old index as final on decode
2015-11-08 22:21:29 +01:00
Alexander Neumann
db41102bfa
Finalize repacker
2015-11-02 19:28:30 +01:00
Alexander Neumann
1fc0d78913
Refactor Index.Store() to take a PackedBlob
2015-11-02 19:07:03 +01:00
Alexander Neumann
f3f84b1544
Add ID handling for index
2015-11-02 18:52:13 +01:00
Alexander Neumann
60a34087c9
Move LoadIndexWithDecoder to index.go
2015-11-02 18:52:13 +01:00
Alexander Neumann
89a77ab2f9
Add Index.ListPack()
2015-11-02 18:52:13 +01:00
Alexander Neumann
fccde030d5
Refactor Index.Lookup() to return struct PackedBlob
2015-10-31 14:47:42 +01:00
Alexander Neumann
7711fcda69
use new index format for repository tests
2015-10-27 20:51:55 +01:00
Alexander Neumann
7717ea5cca
Add benchmark for LoadJSONPack
2015-10-27 20:51:55 +01:00
Alexander Neumann
88849c06a6
rebuild-index: Refactor a bit
2015-10-25 17:53:02 +01:00
Alexander Neumann
9074c923ea
index: add AddToSupersedes()
2015-10-25 17:06:56 +01:00
Alexander Neumann
461d54e43c
Refactor repository.SaveIndex()
2015-10-25 17:05:54 +01:00
Alexander Neumann
04614c7527
Add test for packs in duplicate indexes
2015-10-25 15:35:33 +01:00
Alexander Neumann
f7ff5b766c
Mark written indexes as finalized
2015-10-25 15:35:18 +01:00
Alexander Neumann
d9f9b77d68
Add Index.Packs() and IDSet.Equals()
2015-10-25 15:28:01 +01:00
Alexander Neumann
4b1a2caea7
Allow overwriting the IndexFull function for tests
2015-10-25 15:05:22 +01:00
Alexander Neumann
af0d6f58b9
Remove unneeded pointer to pack id
2015-10-25 14:35:08 +01:00
Alexander Neumann
2710d6399a
Cleanup index code
...
The selectFn wasn't used any more, so remove it from generatePackList().
2015-10-25 14:26:04 +01:00
Alexander Neumann
1dd731fdb8
Handle concurrent access to the inFlight list
2015-10-14 20:50:54 +02:00
Alexander Neumann
6fa4be5af2
Regularly save intermediate indexes
2015-10-12 23:59:17 +02:00
Alexander Neumann
941b7025b6
Delete Index.Remove()
2015-10-12 22:51:11 +02:00
Alexander Neumann
4b2a4b03ec
Remove Index.StoreInProgress()
2015-10-12 22:49:31 +02:00
Alexander Neumann
86fcd170f6
Add and use MasterIndex
2015-10-12 22:34:12 +02:00
Alexander Neumann
64fa89d406
Add error checks
2015-10-12 22:07:56 +02:00
Alexander Neumann
eb73182fcf
Rework index decode and handling old format
2015-10-12 20:53:07 +02:00
Alexander Neumann
356bb62243
Add CreateEncryptedBlob and GetDecryptReader
2015-10-12 20:53:07 +02:00
