This adds code to the master index to allow saving duplicate blobs
within the repacker. In this mode, only the list of currently in flight
blobs is consulted, and not the index. This correct because while
repacking, a unique list of blobs is saved again to the index.
This commit fixes a situation reported by a user where two indexes
contained information about the same pack without overlap, e.g.:
Index 3e6a32 contained:
{
"id": "c02e3b",
"blobs": [
{
"id": "8114b1",
"type": "data",
"offset": 0,
"length": 530107
}
]
}
And index 62da5f contained:
{
"id": "c02e3b",
"blobs": [
{
"id": "e344f8",
"type": "data",
"offset": 1975848,
"length": 3426468
},
{
"id": "939ed9",
"type": "data",
"offset": 530107,
"length": 1445741
}
]
}
This commit adds all blobs in a pack in one atomic operation so that
intermediate such as these do not happen.
... by first adding a preliminary index entry and making this fail if
an index entry for the same blob already exists.
A preliminary index entry is characterized by not yet being associated
with a pack. Until now, these entries where added to the index just
like final index entries using index.Store, which silently overwrites
existing index entries.
This commit adds a new method index.StoreInProgress which refuses to
overwrite existing index entries and allows for creating preliminary
index entries only. The existing method index.Store has not been
changed and continues to silently overwrite existing index entries.
This distinction is important, as otherwise, it would be impossible to
update a preliminary index entry after the blob has been written to a
pack.
Resolves: restic#292
