Lorenz Bausch
9fb81c4246
Validate exclude patterns
2022-05-07 21:12:47 +02:00
Lorenz Bausch
e7fd200237
Keep original pattern for later use
2022-05-07 21:08:09 +02:00
Alexander Neumann
fb5b9370f3
Merge pull request #3728 from MichaelEischer/debug-examine-upload
...
Add support to reupload blobs in `debug examine`
2022-04-30 20:39:19 +02:00
Alexander Neumann
ffbd48c0c6
Merge pull request #3481 from MichaelEischer/recover-enospace
...
Recover from no free space errors
2022-04-30 20:22:26 +02:00
Michael Eischer
95bcc9ea31
debug: Support pack ID prefixes in debug examine
2022-04-30 20:20:31 +02:00
Michael Eischer
2d6a943911
debug: Add switch to upload blobs extracted by debug examine
...
This simplifies salvaging a damaged pack file. Reuploading tree blobs
was previously not possible.
2022-04-30 20:20:31 +02:00
Alexander Neumann
9af499d8a4
Tidy go.sum
2022-04-30 20:16:09 +02:00
Alexander Neumann
2e3d23c1d7
Update github.com/klauspost/compress
2022-04-30 20:03:21 +02:00
Michael Eischer
dbbeac7174
prune: Add unsafe option to recover from no free space
...
The new option allows prune to operate with nearly no scratch space by only removing
no longer necessary pack files and first deleting the index before
rebuilding it. By first deleting the index it becomes safe to just
delete no longer necessary pack files. However, as a downside there's
now the risk that the repository becomes inaccessible if prune fails.
To recover from that problem a user might have to manually delete the
repository index and then run (a full) `rebuild-index` again.
2022-04-30 19:21:07 +02:00
Michael Eischer
cf5cb673fb
repository: Use existing method to collect pack ids
2022-04-30 19:14:21 +02:00
Michael Eischer
b335cb6285
repository: Refactor index IDs collection
2022-04-30 19:14:21 +02:00
MichaelEischer
9c047f170a
Merge pull request #3419 from DanielG/relax-file-modes
...
Enable admin to decide group access to repository files
2022-04-30 16:39:37 +02:00
Daniel Gröber
f31b4f29c1
Use config file modes to derive new dir/file modes
...
Fixes #2351
2022-04-30 15:59:51 +02:00
Alexander Neumann
71c653f9e0
Merge pull request #3727 from MichaelEischer/changelog-3475
...
Add changelog for local/sftp connection limit
2022-04-30 14:05:21 +02:00
Michael Eischer
29a8f92967
Add changelog for local/sftp connection limit
2022-04-30 13:36:47 +02:00
MichaelEischer
ac9324aeaf
Merge pull request #3666 from MichaelEischer/compression
...
Implement compression support
2022-04-30 11:49:05 +02:00
Alexander Neumann
dc5adef255
Add documentation for --repository-version
2022-04-30 11:34:10 +02:00
Michael Eischer
4b01b06f2f
repository: Test compressed blobs in StreamPack
2022-04-30 11:34:10 +02:00
Michael Eischer
bcab548617
pack: slightly expand testing of compressed blobs
2022-04-30 11:34:10 +02:00
Michael Eischer
ec2b25565a
repository: test uncompressedLength field and index example
2022-04-30 11:34:10 +02:00
Michael Eischer
9ffb8920f1
repository: run blackbox tests using old and new repo version
2022-04-30 11:34:10 +02:00
Michael Eischer
abe5935693
repository: unify repository version-specific initialization
...
Mark the master index as compressed also when initializing a new
repository. This is only relevant for testing.
2022-04-30 11:34:10 +02:00
Alexander Neumann
8776031f96
Leave allocating slices to the decompress code
2022-04-30 11:34:10 +02:00
Alexander Neumann
5eb05a0afe
Configure zstd encoder/decoder
2022-04-30 11:34:10 +02:00
Michael Eischer
2f36e044db
Cleanup pack header check
2022-04-30 11:34:10 +02:00
Alexander Neumann
94dc9a0fa7
Amend changelog
2022-04-30 11:34:10 +02:00
Alexander Neumann
8b11b86383
Add option global --compression
2022-04-30 11:34:10 +02:00
Michael Eischer
f38f457a64
Add basic changelog for compression support
2022-04-30 11:34:10 +02:00
Michael Eischer
ba27d29d58
Print repository version when opening a repo
2022-04-30 11:34:10 +02:00
Michael Eischer
7132df529e
repository: Increase index size for repo version 2
...
A compressed index is only about one third the size of an uncompressed
one. Thus increase the number of entries in an index to avoid cluttering
the repository with small indexes.
2022-04-30 11:34:10 +02:00
Michael Eischer
2535524132
debug: Add support for compressed blobs
2022-04-30 11:34:10 +02:00
Michael Eischer
fda7bb0f09
debug: Reduce code duplication
2022-04-30 11:34:10 +02:00
Michael Eischer
66f9048bce
repository: Alloc zstd encoder/decoder on demand
2022-04-30 11:34:10 +02:00
Michael Eischer
fd05037e1a
repository: recalibrate index batch allocation size
2022-04-30 11:34:10 +02:00
Michael Eischer
6fb408d90e
repository: implement pack compression
2022-04-30 11:34:10 +02:00
Michael Eischer
362ab06023
init: Add flag to specify created repository version
2022-04-30 10:07:42 +02:00
Michael Eischer
4b957e7373
repository: Implement index/snapshot/lock compression
...
The config file is not compressed as it should remain readable by older
restic versions such that these can return a proper error.
As the old format for unpacked data does not include a version header,
make use of a trick: The old data is always encoded as JSON. Thus it can
only start with '{' or '['. For any other value the first byte indicates
a versioned format. The version is set to 2 for now. Then the zstd
compressed data follows.
2022-04-30 10:07:42 +02:00
Michael Eischer
0957b74887
Misc design.rst cleanups
2022-04-30 10:07:42 +02:00
Alexander Neumann
270ed00d1f
doc: Add repository compression support documentation
...
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2022-04-30 10:07:42 +02:00
Alexander Neumann
4e1ef7804a
Merge pull request #3717 from MichaelEischer/fix-stuck-repack
...
Fix stuck repack step
2022-04-30 09:50:43 +02:00
Alexander Neumann
e4780d3956
Merge pull request #3718 from MichaelEischer/sftp-docs-fix
...
doc: sftp with password actually works
2022-04-23 20:27:27 +02:00
Alexander Neumann
c183e35b5a
Merge pull request #3719 from MichaelEischer/read-write-order
...
Describe repository read/write order required by repository format
2022-04-23 15:23:00 +02:00
Michael Eischer
6f9e20a1bb
doc: Describe repository read/write order
2022-04-23 15:21:02 +02:00
Michael Eischer
f9219e8608
doc: sftp with password actually works
2022-04-23 11:57:36 +02:00
Michael Eischer
3b630d9998
add missing streamPacks changelog
2022-04-23 11:50:19 +02:00
Michael Eischer
566ac11c65
fix changelog name
2022-04-23 11:37:00 +02:00
Michael Eischer
f5609d1d3c
prune: Fail early if too few backend connections
2022-04-23 11:32:52 +02:00
Michael Eischer
e597b99b55
repository: Reduce repack workers to prevent deadlock
...
As repack streams packs these occupy one backend connection. Uploading a
new pack also requires a backend connection. To prevent a deadlock
during repack when reaching the backend connections limit, simply limit
the repackWorker count to always leave one connection for uploading.
2022-04-23 11:28:18 +02:00
Michael Eischer
ee627cd832
backend/mem: Actually enforce connection limit
...
This will allow tests to detect deadlocks related to the connections
limit.
2022-04-23 11:22:00 +02:00
Michael Eischer
4f97492d28
Backend: Expose connections parameter
2022-04-23 11:13:08 +02:00
