2
2
mirror of https://github.com/octoleo/restic.git synced 2024-11-29 16:23:59 +00:00
Commit Graph

1174 Commits

Author SHA1 Message Date
Alex Duchesne
9e34c791c9 Better temp file cleanup on Windows. 2022-04-09 12:00:22 +02:00
Charlie Jiang
d9c9415cfd Fix rclone (scoop shim) and sftp issue due to detached console on Windows 2022-04-03 17:53:17 +08:00
greatroar
c23c0f7c14 Cast unix.Statfs_t.Type to int64 when checking for btrfs
Fixes #3687. Uses the cast suggested by @MichaelEischer, except that the
contant isn't cast along, because it's untyped and will be converted by
the compiler as necessary.
2022-03-31 22:30:45 +02:00
Alexander Neumann
db8a958991
Merge pull request #3683 from MichaelEischer/fix-golangci-lint-warnings
Fix golangci lint warnings
2022-03-29 11:45:10 +02:00
Michael Eischer
af31266b7d golangci-lint: replace deprecated golint with revive 2022-03-28 22:33:17 +02:00
Michael Eischer
2f81af6afa bloblru: Fix comment for New function 2022-03-28 22:25:25 +02:00
Michael Eischer
61e179ee78 switch to golang.org/x/term 2022-03-28 22:24:15 +02:00
Michael Eischer
c60540b196 add go:build headers everywhere 2022-03-28 22:23:47 +02:00
Michael Eischer
fefe9f5c0e pack: Hide more implementation details 2022-03-28 22:12:16 +02:00
Michael Eischer
a773cb6527 pack: cleanup header size calculation 2022-03-28 22:09:49 +02:00
Michael Eischer
6408686973 repository: Simplify Blob equality check 2022-03-28 22:09:49 +02:00
Michael Eischer
243698680a crypto: Use helpers for size calculations 2022-03-28 22:09:49 +02:00
Michael Eischer
d6db5a1fc2 archiver: Fix test
The test relied on an undeocumented sideeffect of the LoadBlob implementation
2022-03-28 22:09:49 +02:00
Michael Eischer
f78bd14e28 repository: Remove pack implementation details from MasterIndex 2022-03-28 22:09:49 +02:00
Michael Eischer
dc3d77dacc repository: make saveAndEncrypt private 2022-03-28 22:09:49 +02:00
Michael Eischer
6877e7edbb repository: Rename LoadAndDecrypt to LoadUnpacked
The method is the complement for SaveUnpacked and not for
SaveAndEncrypt. The latter assembles blobs into pack files.
2022-03-28 22:09:49 +02:00
Michael Eischer
2e1613d4c6 errors: Ensure that errors.IsFatal(errors.Fatal("err")) == true
This fixes a few cases where restic output "Fatal: Fatal: [...]"
2022-03-28 22:09:49 +02:00
Michael Eischer
537b4c310a copy: Implement by reusing repack
The repack operation copies all selected blobs from a set of pack files
into new pack files. For prune the source and destination repositories
are identical. To implement copy, just use a different source and
destination repository.
2022-03-26 20:47:15 +01:00
Alexander Neumann
4d5db61bd0
Merge pull request #3484 from MichaelEischer/stream-check-repack
Stream packs in `check --read-data` and during repacking
2022-03-26 20:46:17 +01:00
Alexander Neumann
e682f7c0d6 Add tests for StreamPack 2022-03-21 21:15:03 +01:00
Alexander Neumann
0b8b524f12
Merge pull request #3512 from MichaelEischer/cleaner-lock-refresh
Prevent lock refresh from leaving behind lots of stale locks
2022-03-21 11:10:37 +01:00
Alexander Neumann
a350625554
Merge pull request #3524 from MichaelEischer/atomic-sftp
sftp: Implement atomic uploads
2022-03-21 11:08:22 +01:00
Alexander Neumann
8388f66c4c
Merge pull request #3668 from greatroar/symlink-size
Report symlink sizes from FUSE mount
2022-03-21 11:02:32 +01:00
Alexander Neumann
0937008648
Merge pull request #3654 from MichaelEischer/limit-huge-tree-streams
Limit number of large tree blobs loaded in parallel by StreamTrees
2022-03-21 11:01:04 +01:00
Michael Eischer
cd190bee14 filter: short circuit if no negative patterns 2022-03-20 13:33:08 +01:00
Vincent Bernat
2ee07ded2b filter: ability to use negative patterns
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix #233
2022-03-20 13:33:08 +01:00
Michael Eischer
12606b575f filter: Cleanup variable naming 2022-03-20 13:33:08 +01:00
Michael Eischer
5f145f0c7e filter: introduce pattern struct 2022-03-20 13:33:08 +01:00
Vincent Bernat
13c40d4199 filter: additional tests for filter.List() 2022-03-20 13:33:08 +01:00
Jason Lenz
2b1932a258 Report symlink sizes from FUSE mount for snapshot dir
Fixes #3667.
2022-03-17 22:21:47 -05:00
greatroar
fdc738fb70 Report symlink sizes from FUSE mount
Fixes #3667.
2022-03-13 16:48:35 +01:00
Michael Eischer
254c8743fc Limit number of large tree blobs loaded in parallel by StreamTrees
Load tree blobs with more than 50MB only from a single goroutine. Very
large tree blobs with for example 400 MB size can otherwise require
roughly 1GB * streamTreeParallelism memory.
2022-02-19 12:26:09 +01:00
Michael Eischer
27524979e8 restorer: Remove dead code 2022-02-13 11:43:09 +01:00
gum3ng
dd30083c2b [#issue 3127] Add xattr support for Solaris 2022-02-13 14:24:37 +05:30
MichaelEischer
fb4c5af5c4
Merge pull request #3642 from gco/master
Fix test failures on Solaris
2022-02-12 22:07:14 +01:00
Michael Eischer
5ec312ca06 sftp: Implement atomic uploads
Create a temporary file with a sufficiently random name to essentially
avoid any chance of conflicts. Once the upload has finished remove the
temporary suffix. Interrupted upload thus will be ignored by restic.
2022-02-12 20:30:49 +01:00
Michael Eischer
c6fd13425b remember the refreshed lock file even if removal failed
This ensures that restic won't create lots of new lock files without
deleting them later on.

In some cases a Delete operation on a backend can return a "File does
not exist" error even though the Delete operation succeeded. This can
for example be caused by request retries. This caused restic to forget
about the new lock file and continue trying to remove the old (already
deleted) lock file.
2022-02-12 20:21:58 +01:00
Michael Eischer
bba8ba7a5b repository: cancel streampack context after error 2022-02-12 20:18:25 +01:00
Michael Eischer
47554a3428 repository: Fix error handling in repack
When storing a blob fails, this is a fatal error which must not be
retried.
2022-02-12 20:18:25 +01:00
Michael Eischer
4b3dc415ef checker: cleanup header extraction 2022-02-12 20:18:25 +01:00
Michael Eischer
930a00ad54 checker: reuse bufio reader 2022-02-12 20:18:25 +01:00
Michael Eischer
34ebafb8b6 repository: don't crash if blob size is too short 2022-02-12 20:18:25 +01:00
Michael Eischer
becebf5d88 repository: remove unused DownloadAndHash 2022-02-12 20:18:25 +01:00
Michael Eischer
f1e58e7c7f checker: rewrite ReadData to stream packs 2022-02-12 20:18:25 +01:00
Michael Eischer
f40abd92fa restorer: convert to use StreamPack 2022-02-12 20:18:25 +01:00
Michael Eischer
f00f690658 repository: stream packs during repacking 2022-02-12 20:18:25 +01:00
Michael Eischer
c4a2bfcb39 repository: Add StreamPacks function
The function supports efficiently loading a specified list of blobs from
a single pack in a streaming fashion. That is there's no need for
temporary files independent of the pack size.
2022-02-12 20:18:25 +01:00
Michael Eischer
153e2ba859 repository: Implement lisiting blobs per pack file 2022-02-12 20:18:24 +01:00
MichaelEischer
cc90f2ba6b
Merge pull request #2816 from greatroar/noatime
Set O_NOATIME flag on Linux
2022-02-07 21:38:31 +01:00
Michael Eischer
6b17a7110c backup: Set O_NOATIME in the right place
The archiver uses FS.OpenFile, where FS is an instance of the FS
interface. This is different from fs.OpenFile, which uses the OpenFile
method provided by the fs package.
2022-02-06 15:00:37 +01:00