Michael Eischer
b335cb6285
repository: Refactor index IDs collection
2022-04-30 19:14:21 +02:00
Daniel Gröber
f31b4f29c1
Use config file modes to derive new dir/file modes
...
Fixes #2351
2022-04-30 15:59:51 +02:00
Michael Eischer
4b01b06f2f
repository: Test compressed blobs in StreamPack
2022-04-30 11:34:10 +02:00
Michael Eischer
bcab548617
pack: slightly expand testing of compressed blobs
2022-04-30 11:34:10 +02:00
Michael Eischer
ec2b25565a
repository: test uncompressedLength field and index example
2022-04-30 11:34:10 +02:00
Michael Eischer
9ffb8920f1
repository: run blackbox tests using old and new repo version
2022-04-30 11:34:10 +02:00
Michael Eischer
abe5935693
repository: unify repository version-specific initialization
...
Mark the master index as compressed also when initializing a new
repository. This is only relevant for testing.
2022-04-30 11:34:10 +02:00
Alexander Neumann
8776031f96
Leave allocating slices to the decompress code
2022-04-30 11:34:10 +02:00
Alexander Neumann
5eb05a0afe
Configure zstd encoder/decoder
2022-04-30 11:34:10 +02:00
Michael Eischer
2f36e044db
Cleanup pack header check
2022-04-30 11:34:10 +02:00
Alexander Neumann
8b11b86383
Add option global --compression
2022-04-30 11:34:10 +02:00
Michael Eischer
7132df529e
repository: Increase index size for repo version 2
...
A compressed index is only about one third the size of an uncompressed
one. Thus increase the number of entries in an index to avoid cluttering
the repository with small indexes.
2022-04-30 11:34:10 +02:00
Michael Eischer
66f9048bce
repository: Alloc zstd encoder/decoder on demand
2022-04-30 11:34:10 +02:00
Michael Eischer
fd05037e1a
repository: recalibrate index batch allocation size
2022-04-30 11:34:10 +02:00
Michael Eischer
6fb408d90e
repository: implement pack compression
2022-04-30 11:34:10 +02:00
Michael Eischer
362ab06023
init: Add flag to specify created repository version
2022-04-30 10:07:42 +02:00
Michael Eischer
4b957e7373
repository: Implement index/snapshot/lock compression
...
The config file is not compressed as it should remain readable by older
restic versions such that these can return a proper error.
As the old format for unpacked data does not include a version header,
make use of a trick: The old data is always encoded as JSON. Thus it can
only start with '{' or '['. For any other value the first byte indicates
a versioned format. The version is set to 2 for now. Then the zstd
compressed data follows.
2022-04-30 10:07:42 +02:00
Michael Eischer
e597b99b55
repository: Reduce repack workers to prevent deadlock
...
As repack streams packs these occupy one backend connection. Uploading a
new pack also requires a backend connection. To prevent a deadlock
during repack when reaching the backend connections limit, simply limit
the repackWorker count to always leave one connection for uploading.
2022-04-23 11:28:18 +02:00
Michael Eischer
ee627cd832
backend/mem: Actually enforce connection limit
...
This will allow tests to detect deadlocks related to the connections
limit.
2022-04-23 11:22:00 +02:00
Michael Eischer
4f97492d28
Backend: Expose connections parameter
2022-04-23 11:13:08 +02:00
Alexander Neumann
89ee1cf9ee
Merge pull request #3610 from MichaelEischer/windows-temp-files
...
Improve handling of temporary files on windows
2022-04-10 20:29:04 +02:00
Alexander Neumann
a059ef90f8
Merge pull request #3702 from MichaelEischer/extend-config-error
...
Print used key name if config fails to load
2022-04-10 20:25:24 +02:00
Michael Eischer
4077a81b34
Add simple test for fs.TempFile on windows
2022-04-09 23:37:58 +02:00
Michael Eischer
9a3f1a9703
Simplify and comment TempFile implementation for windows
2022-04-09 23:37:58 +02:00
Michael Eischer
c2aabb2686
Print used key name if config fails to load
2022-04-09 22:38:18 +02:00
MichaelEischer
c60a5f00c9
Merge pull request #3675 from ItsMattL/update
...
Refactor file handing for self-update.
2022-04-09 21:55:56 +02:00
Matt LaPlante
0ba9d4ced7
Refactor file handing for self-update.
...
* Write new file payload to a temp file before touching the original
binary. Minimizes the possibility of failing mid-write and corrupting
the binary.
* On Windows, move the original binary out to a temp file rather than
removing it as the running binary is locked. Fixes issue #2248 .
2022-04-09 21:40:33 +02:00
Alexander Neumann
04e054465a
Merge pull request #3475 from MichaelEischer/local-sftp-conn-limit
...
Limit concurrent operations for local / sftp backend
2022-04-09 21:33:00 +02:00
Michael Eischer
7b9ae91e04
copy: Load snapshots before indexes
2022-04-09 12:27:25 +02:00
Michael Eischer
9e12159230
Fix O(n) backend list calls in FindFilteredSnapshots
...
When resolving snapshotIDs in FindFilteredSnapshots either
FindLatestSnapshot or FindSnapshot is called. Both operations issue a
list operation to the backend. When for example passing a long list of
snapshot ids to `forget` this could lead to a large number of list
operations.
2022-04-09 12:26:31 +02:00
Michael Eischer
3d29083e60
copy/find/ls/recover/stats: Memorize snapshot listing before index
...
These commands filter the snapshots according to some criteria which
essentially requires loading the index before filtering the snapshots.
Thus create a copy of the snapshots list beforehand and use it later on.
2022-04-09 12:26:30 +02:00
Michael Eischer
ece06f125e
sftp: Limit concurrent backend operations
2022-04-09 12:21:38 +02:00
Michael Eischer
cd783358d3
local: Limit concurrent backend operations
...
Use a limit of 2 similar to the filereader concurrency in the archiver.
2022-04-09 12:21:38 +02:00
Michael Eischer
0b258cc054
backends: clean reader closing
2022-04-09 12:21:38 +02:00
Alex Duchesne
9e34c791c9
Better temp file cleanup on Windows.
2022-04-09 12:00:22 +02:00
Charlie Jiang
d9c9415cfd
Fix rclone (scoop shim) and sftp issue due to detached console on Windows
2022-04-03 17:53:17 +08:00
greatroar
c23c0f7c14
Cast unix.Statfs_t.Type to int64 when checking for btrfs
...
Fixes #3687 . Uses the cast suggested by @MichaelEischer, except that the
contant isn't cast along, because it's untyped and will be converted by
the compiler as necessary.
2022-03-31 22:30:45 +02:00
Alexander Neumann
db8a958991
Merge pull request #3683 from MichaelEischer/fix-golangci-lint-warnings
...
Fix golangci lint warnings
2022-03-29 11:45:10 +02:00
Michael Eischer
af31266b7d
golangci-lint: replace deprecated golint with revive
2022-03-28 22:33:17 +02:00
Michael Eischer
2f81af6afa
bloblru: Fix comment for New function
2022-03-28 22:25:25 +02:00
Michael Eischer
61e179ee78
switch to golang.org/x/term
2022-03-28 22:24:15 +02:00
Michael Eischer
c60540b196
add go:build headers everywhere
2022-03-28 22:23:47 +02:00
Michael Eischer
fefe9f5c0e
pack: Hide more implementation details
2022-03-28 22:12:16 +02:00
Michael Eischer
a773cb6527
pack: cleanup header size calculation
2022-03-28 22:09:49 +02:00
Michael Eischer
6408686973
repository: Simplify Blob equality check
2022-03-28 22:09:49 +02:00
Michael Eischer
243698680a
crypto: Use helpers for size calculations
2022-03-28 22:09:49 +02:00
Michael Eischer
d6db5a1fc2
archiver: Fix test
...
The test relied on an undeocumented sideeffect of the LoadBlob implementation
2022-03-28 22:09:49 +02:00
Michael Eischer
f78bd14e28
repository: Remove pack implementation details from MasterIndex
2022-03-28 22:09:49 +02:00
Michael Eischer
dc3d77dacc
repository: make saveAndEncrypt private
2022-03-28 22:09:49 +02:00
Michael Eischer
6877e7edbb
repository: Rename LoadAndDecrypt to LoadUnpacked
...
The method is the complement for SaveUnpacked and not for
SaveAndEncrypt. The latter assembles blobs into pack files.
2022-03-28 22:09:49 +02:00
