Alexander Weiss
2a1add7538
check: remove file size counter
2020-12-23 02:34:31 +01:00
tWido
7dab113035
Don't retry when "Permission denied" occurs in local backend
2020-12-22 23:41:12 +01:00
Michael Eischer
de99207046
repository: tweak comment for packs method
2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e
Count packs directly in RebuildIndexFiles
2020-12-22 23:01:58 +01:00
Michael Eischer
b9f5d3fe13
repository: Add test for ForAllIndexes
2020-12-22 22:36:18 +01:00
Michael Eischer
a12c5f1d37
repository: move otherwise unused LoadIndex to tests
2020-12-22 22:36:18 +01:00
Michael Eischer
24474a36f4
repository: deduplicate index loading implementation
2020-12-22 22:36:18 +01:00
Michael Eischer
96904f8972
check: extract parallel index loading
2020-12-22 22:36:18 +01:00
Alexander Neumann
6c514adb8a
ui/progress: Use mutex instead of atomic
...
The counter value needs to be aligned to 64 bit in memory for the
atomic functions to work on some platform (such as 32 bit ARM).
The atomic package says in its documentation:
> These functions require great care to be used correctly. Except for
> special, low-level applications, synchronization is better done with
> channels or the facilities of the sync package.
This commit replaces the atomic functions with a simple sync.Mutex, so
we don't have to care about alignment.
2020-12-22 21:03:27 +01:00
DRON-666
83b10dbb12
Deduplicate dumper closing logic
2020-12-19 02:42:46 +03:00
DRON-666
e136dd8696
Deduplicate test code
2020-12-19 02:06:54 +03:00
DRON-666
33adb58817
Minor fixes and linter suggestions
2020-12-19 02:04:17 +03:00
DRON-666
89ab6d557e
dump: Add zip dumper
2020-12-19 01:09:47 +03:00
DRON-666
0256f95994
dump: Split tar and walk logic
2020-12-19 01:09:47 +03:00
Michael Eischer
34b6130a0e
restic: parallelize lock file loading
2020-12-18 20:46:16 +01:00
greatroar
66d904c905
Make invalid handles permanent errors
2020-12-17 12:47:53 +01:00
greatroar
746dbda413
Mark "ssh exited" errors in SFTP as permanent
2020-12-17 12:43:09 +01:00
greatroar
f7784bddb3
Don't retry when "no space left on device" in local backend
...
Also adds relevant documentation to the restic.Backend interface.
2020-12-17 12:43:09 +01:00
Alexander Neumann
e96677cafb
Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables
...
swift: Add support for id based keystone v3 auth parameters
2020-12-12 16:27:38 +01:00
Michael Eischer
1d69341e88
swift: Add support for id based keystone v3 auth parameters
...
This adds support for the following environment variables, which were
previously missing:
OS_USER_ID User ID for keystone v3 authentication
OS_USER_DOMAIN_ID User domain ID for keystone v3 authentication
OS_PROJECT_DOMAIN_ID Project domain ID for keystone v3 authentication
OS_TRUST_ID Trust ID for keystone v3 authentication
2020-12-11 19:22:34 +01:00
Alexander Neumann
36c5d39c2c
Fix issues reported by semgrep
2020-12-11 09:41:59 +01:00
Alexander Neumann
7facc8ccc1
Merge pull request #2505 from aawsome/fix-repo-configfile
...
Fix repo configfile
2020-12-07 07:52:37 +01:00
Alexander Neumann
ba31c6fdaa
Merge pull request #3150 from MichaelEischer/fix-windows-redir-output
...
termstatus: Fix canUpdateStatus detection for redirected output on windows
2020-12-06 21:17:27 +01:00
Alexander Neumann
b58799d83a
Merge pull request #3152 from MichaelEischer/fix-backup-background-hang
...
backup: Fix shutdown hang when running in the background on linux
2020-12-06 21:16:00 +01:00
Alexander Neumann
0d5b764f90
Merge pull request #3130 from aawsome/snapshots-parallel
...
Make loading snapshots parallel
2020-12-06 21:08:18 +01:00
Michael Eischer
b48f579530
termstatus: Fix canUpdateStatus detection for redirected output
...
The canUpdateStatus check was simplified in #2608 , but it accidentally flipped
the condition. The correct check is as follows: If the output is a pipe then
restic probably runs in mintty/cygwin. In that case it's possible to
update the output status. In all other cases it isn't.
This commit inverts to condition again to offer the previous and correct
behavior.
2020-12-06 19:02:42 +01:00
Michael Eischer
401ef92c5f
backup: Fix shutdown hang when running in the background
...
On shutdown the backup commands waits for the terminal output goroutine
to stop. However while running in the background the goroutine ignored
the canceled context.
2020-12-06 18:53:41 +01:00
Alexander Weiss
e329623771
Remove LoadAllSnapshots
2020-12-06 05:22:27 +01:00
Alexander Weiss
26f85779be
Parallelize ForAllSnapshots
2020-12-06 05:09:58 +01:00
Alexander Weiss
5b9ee56335
Add ForAllSnapshots
2020-12-06 05:04:21 +01:00
Michael Eischer
d05c88a5d6
fuse: Properly set uid/gid for directories
...
In #2584 this was changed to use the uid/gid of the root node. This
would be okay for the top-level directory of a snapshot, however, this
change also applied to normal directories within a snapshot. This
change reverts the problematic part and adds a test that directory
attributes are represented correctly.
2020-11-30 23:41:49 +01:00
MichaelEischer
f2959127b6
Merge pull request #3065 from greatroar/local-subdirs
...
Don't recurse in local backend's List if not required
2020-11-29 19:03:59 +01:00
Alexander Weiss
aa7a5f19c2
Use BlobHandle in index methods
2020-11-22 20:41:12 +01:00
Alexander Weiss
e3013271a6
Harmonize naming
2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691
Make BlobHandle substruct of Blob
2020-11-22 20:41:10 +01:00
Alexander Weiss
a851c53cbe
Use PackSize in checker
2020-11-21 22:13:54 +01:00
Alexander Weiss
ce5d630681
Add MasterIndex.PackSize()
2020-11-21 22:13:54 +01:00
Alexander Weiss
c3ddde9e7d
Return hdrSize in ListPack
2020-11-21 22:13:54 +01:00
greatroar
8e213e82fc
backend/local: replace fs.Walk with custom walker
...
This code is more strict in what it expects to find in the backend:
depending on the layout, either a directory full of files or a directory
full of such directories.
2020-11-19 16:46:42 +01:00
Alexander Neumann
75eff92b56
Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init
...
do not require gs bucket permissions to init repository
2020-11-18 16:53:45 +01:00
eleith
a24e986b2b
do not require gs bucket permissions to init repository
...
a gs service account may only have object permissions on an existing
bucket but no bucket create/get permissions.
these service accounts currently are blocked from initialization a
restic repository because restic can not determine if the bucket exists.
this PR updates the logic to assume the bucket exists when the bucket
attribute request results in a permissions denied error.
this way, restic can still initialize a repository if the service
account does have object permissions
fixes: https://github.com/restic/restic/issues/3100
2020-11-18 06:14:11 -08:00
rawtaz
1b6e8c888f
Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param
...
Remove `--drive-use-trash=false` from rclone param
2020-11-17 11:21:20 +01:00
Michael Eischer
1f43cac12d
check: Only track data blobs when unused blobs should be reported
...
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00
Michael Eischer
6da66c15d8
check: Simplify referenced blob tracking
...
The result is identical as long as the context in not canceled. However,
in that case the result is incomplete anyways.
2020-11-15 18:42:55 +01:00
Michael Eischer
3500f9490c
check: Simplify blob status tracking
...
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
2020-11-15 18:42:42 +01:00
Michael Eischer
b8c7543a55
check: Merge 'size could not be found' and 'not found in index' errors
...
By construction these two errors always show up in pairs: 'size could
not be found' is printed when the blob is not found in the repository
index. That blob is also part of the `blobs` array. Later on, check
iterates over that array and checks whether the blob is marked as
existing. Which cannot be the case as that mark is generated by
iterating over the repository index.
The merged warning no longer reports the blob index within a file. That
information could also be derived by printing the affected tree using
`cat` and searching for the blob.
2020-11-15 18:41:50 +01:00
Alexander Neumann
3c0c0c132b
Merge pull request #3006 from aawsome/new-rebuild-index
...
Reimplement rebuild-index and remove /internal/index
2020-11-15 17:48:43 +01:00
Alexander Neumann
9968220652
Merge pull request #2850 from greatroar/packer-malloc
...
Decrease allocation rate in internal/pack
2020-11-15 17:19:49 +01:00
greatroar
ab2b7d7f9a
Decrease allocation rate in internal/pack
...
internal/repository benchmark results:
name old time/op new time/op delta
PackerManager-8 179ms ± 1% 181ms ± 1% +0.78% (p=0.009 n=10+10)
name old speed new speed delta
PackerManager-8 294MB/s ± 1% 292MB/s ± 1% -0.77% (p=0.009 n=10+10)
name old alloc/op new alloc/op delta
PackerManager-8 91.3kB ± 0% 72.2kB ± 0% -20.92% (p=0.000 n=9+7)
name old allocs/op new allocs/op delta
PackerManager-8 1.38k ± 0% 0.76k ± 0% -45.20% (p=0.000 n=10+7)
2020-11-15 16:51:47 +01:00
greatroar
9a8a2cae4c
Move pack testing logic to test file
2020-11-15 16:45:49 +01:00
Alexander Weiss
9607cad267
Remove internal/index
2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a
Reimplement rebuild-index
2020-11-15 07:05:09 +01:00
Alexander Weiss
187c8fb259
Parallelize MasterIndex.Save()
2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5
Add extraObsolete to MasterIndex.Save
2020-11-15 07:05:09 +01:00
Alexander Weiss
5898cb341f
Use CreateIndexFromPacks() in test
2020-11-15 07:05:05 +01:00
Alexander Weiss
43732bb885
Add CreateIndexFromPacks()
2020-11-15 07:04:51 +01:00
greatroar
c45f8ee075
Replace restic.statT interface by concrete types
...
name old time/op new time/op delta
NodeFillUser-8 1.81µs ± 9% 1.50µs ± 5% -17.07% (p=0.000 n=19+20)
NodeFromFileInfo-8 1.76µs ± 4% 1.49µs ± 6% -15.63% (p=0.000 n=20+19)
name old alloc/op new alloc/op delta
NodeFillUser-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
NodeFromFileInfo-8 496B ± 0% 352B ± 0% -29.03% (p=0.000 n=20+20)
name old allocs/op new allocs/op delta
NodeFillUser-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
NodeFromFileInfo-8 3.00 ± 0% 2.00 ± 0% -33.33% (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
MichaelEischer
3601a9b6cd
Merge pull request #2690 from SkYNewZ/master
...
Fix #2688 : Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
MichaelEischer
333c5a19d4
Merge pull request #3082 from aawsome/check-sizes
...
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
Quentin Lemaire
ae441d3134
fix(backup): Switch tags cobra type to handle comma-separated list
2020-11-14 15:48:56 +00:00
Alexander Weiss
17bb77b1f9
check: Also check blob length and offset
2020-11-14 00:42:49 +01:00
Alexander Weiss
80dcfca191
check: Check sizes computed from index and pack header
2020-11-14 00:42:49 +01:00
tofran
94a154c7ca
Remove --drive-use-trash=false
from rclone param
...
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
2020-11-13 22:58:48 +00:00
Alexander Neumann
04d1983800
Merge pull request #3090 from fgma/vss-fix-386
...
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-13 21:30:34 +01:00
Alexander Neumann
8ad9f88993
helpers: Improve error message
2020-11-12 20:38:31 +01:00
fgma
f9223cd827
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-12 19:31:00 +01:00
rawtaz
dfb9326b1b
Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag
...
Extended option to select V1 API for ListObjects on S3 backend
2020-11-11 20:52:44 +01:00
Alexander Neumann
4a0b7328ec
s3: Remove dots for config description
2020-11-11 20:20:35 +01:00
greatroar
63e32c44c0
Improve error reporting from restic diff
...
Instead of a stacktrace, restic diff 111 222 now reports:
Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
Nick Douma
829959390a
Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1
2020-11-11 11:54:38 +01:00
Nick Douma
ccd55d529d
Add s3.list-objects-v1 extended option and default to false
2020-11-11 11:54:36 +01:00
Nick Douma
4ddcc17135
Add support for boolean extended options
2020-11-11 11:54:27 +01:00
MichaelEischer
407843c5f9
Merge pull request #3034 from hoyho/gh_master
...
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
MichaelEischer
46d31ab86d
Merge pull request #3058 from greatroar/counter
...
Replace restic.Progress with new progress.Counter (fixes two race conditions)
2020-11-09 22:19:09 +01:00
Alexander Neumann
c986823d3f
Merge pull request #3048 from aawsome/check-pack-index
...
check: check index for packs that are read
2020-11-09 20:12:32 +01:00
Alexander Weiss
239931578c
check: check index for packs that are read
2020-11-09 17:28:14 +01:00
hoyho
9df52327cc
bugfix: omit ENOTDATA for extended attributes
...
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
greatroar
21b787a4d1
Stop Counters where they're constructed and started
2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2
Replace restic.Progress with new progress.Counter
...
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
Alexander Neumann
3bf43d7951
Merge pull request #2475 from restic/use-github-actions
...
Use GitHub Actions for CI
2020-11-08 17:30:12 +01:00
Alexander Neumann
5cf42884c8
Run tests on GitHub Actions
2020-11-08 16:04:57 +01:00
Alexander Weiss
826cfa0533
fix context in archiver tests
2020-11-08 08:24:24 +01:00
Alexander Weiss
fef408a8bd
Return context error in mem backend
2020-11-08 00:05:53 +01:00
greatroar
a2d4209322
Don't recurse in local backend's List if not required
...
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.
Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9 ).
2020-11-07 08:54:13 +01:00
Alexander Weiss
fd33030556
Use in-memory index to rebuild index in prune
2020-11-06 20:23:30 +01:00
Alexander Weiss
38cc4393f6
Add Masterindex.Save(); Add Index.Packs()
2020-11-06 20:23:30 +01:00
Alexander Weiss
aaf1c44362
Fix #3062
2020-11-05 17:05:42 +01:00
Ivan Andreev
ab2790d9de
Fix http2 stream reset between restic and rest backends #3014
2020-11-05 15:57:40 +03:00
Nick Craig-Wood
86b5d8ffaa
s3: add bucket-lookup parameter to select path or dns style bucket lookup
...
This is to enable restic working with Alibaba cloud
Fixes #2528
2020-11-05 12:20:10 +01:00
Alexander Neumann
ae5302c7a8
Add comment that keepBlobs is modified
2020-11-05 10:33:38 +01:00
Alexander Neumann
866a52ad4e
Remove unneeded seek
...
The file returned from DownloadAndHash() is already seeked to the start
of the file.
2020-11-05 10:31:49 +01:00
Alexander Neumann
a4507610a0
Fix typo
2020-11-05 10:31:49 +01:00
Alexander Neumann
7def2d8ea7
Use a non-constant seed
2020-11-05 10:31:49 +01:00
Alexander Neumann
ee0112ab3b
Clarify message about expected error
2020-11-05 10:31:49 +01:00
Michael Eischer
b373f164fe
prune: Parallelize repack command
2020-11-05 10:31:49 +01:00
Alexander Neumann
5144141321
Merge pull request #2718 from aawsome/new-cleanup-command
...
Reimplementation of prune
2020-11-05 10:12:19 +01:00
fgma
916b2d303b
vss: fix potential crash (not reachable in restic) ( #3045 )
...
HasSufficientPrivilegesForVSS() now returns an error
2020-11-04 22:14:18 +01:00
Alexander Neumann
a06f5c28c0
Merge pull request #3055 from greatroar/cleanup
...
Don't separately allocate sync.{Mutex,Once} if not necessary
2020-11-04 11:11:21 +01:00
greatroar
c2f3eee5af
Don't separately allocate sync.{Mutex,Once} if not necessary
...
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
2020-11-04 09:55:43 +01:00
Alexander Weiss
b2f5381737
Make realistic forget --prune --dryrun
2020-11-03 16:42:21 +01:00