TestParallelSaveWithDuplication has been reworked to provoke
unreferenced packs using fewer goroutines than before and create
only one bytes.Reader per blob. This reduces memory usage
significantly.
The following actions have been taken to keep the chance of provoking
unreferenced packs due to #358 high:
* Interweaved processing of subsequent chunks
* Delaying each goroutine by a few pseudo-randomly chosen nanoseconds
(depending on the platform this will most probably only make the os
yield execution to another thread): together with the interweaved
processing of subsequent chunks, this ensures a minimalistic delay
between processing of (some) duplicated chunks
* Repeating the test 5 times with different seeds
On my test machine, the modified test provoked unreferenced packs 60
times in a row.
This commit adds an integration test, that calls Archiver.Save from
many goroutines processing several duplicated chunks concurrently.
The test asserts, that after all chunks have been saved, there are no
unreferenced packs in the repository.
The test has been checked to give the expected results:
1) Running the test with maxParallel=1 (all chunks are processed
sequentially) has been verified not to produce any unreferenced
packs. Consequently the test passes.
2) Running the test with unbounded parallelism (maxParallel=
math.MaxInt32) has been verified to produce unreferenced packs
all the time (at least 25 test runs). Consequently the test fails
due to #358.
references: #358
A user discovered that restic does not restore setuid/setgid/sticky file
attributes. This commit fixes that. The mode is stored in the Go format
as an uint32: https://golang.org/pkg/os/#FileMode
This commit fixes a situation reported by a user where two indexes
contained information about the same pack without overlap, e.g.:
Index 3e6a32 contained:
{
"id": "c02e3b",
"blobs": [
{
"id": "8114b1",
"type": "data",
"offset": 0,
"length": 530107
}
]
}
And index 62da5f contained:
{
"id": "c02e3b",
"blobs": [
{
"id": "e344f8",
"type": "data",
"offset": 1975848,
"length": 3426468
},
{
"id": "939ed9",
"type": "data",
"offset": 530107,
"length": 1445741
}
]
}
This commit adds all blobs in a pack in one atomic operation so that
intermediate such as these do not happen.
