octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-24 05:37:37 +00:00
Code Issues Releases Wiki Activity
4,671 Commits 7 Branches 33 Tags 66 MiB
0fed6a8dfc
Commit Graph

757 Commits

Author SHA1 Message Date
Alexander Neumann
4c2072d875 Merge pull request 1719 from ifedorenko/mt-restorer 
new optimized multithreaded restore implementation
 2018-10-14 20:29:16 +02:00
Igor Fedorenko
9e24154ec9 restore: significantly reduce memory footprint 
reworked restore error callback to use file location
path instead of much heavier Node. this reduced restore
memory usage by as much as 50% in some of my tests.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-10-14 17:39:42 +02:00
Igor Fedorenko
da57302fca restore: Removed legacy restore implementation 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-10-14 17:37:30 +02:00
Igor Fedorenko
1869930d95 restore: New optimized multithreaded implementation 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-10-14 17:37:30 +02:00
Alexander Neumann
a432b42c81 self-update: Check current version before download 2018-10-14 17:29:46 +02:00
Alexander Neumann
7d0f2eaf24 self-update: Use correct path to restic binary 
Closes #2041
 2018-10-14 17:29:19 +02:00
Alexander Neumann
41a4d67d93 Merge pull request 1962 from mholt/lsjson 
ls: Stream output when using --json option
 2018-10-14 15:51:38 +02:00
Alexander Neumann
d7baa67acb snapshots: Don't print empty line for --compact 
Closes #2038
 2018-10-13 20:17:34 +02:00
Alexander Neumann
167397c18c Set development version for 0.9.3 2018-10-13 13:48:31 +02:00
Alexander Neumann
be36c5f150 Add version for 0.9.3 2018-10-13 13:48:31 +02:00
Johannes Hertenstein
277cba4b32 Catch errors when walking cache directories 2018-10-08 15:47:34 +02:00
Johannes Hertenstein
ed651df19b Use correct method for joining paths 2018-10-08 15:47:08 +02:00
Johannes Hertenstein
641dc65e6e Output directory size in cache command 2018-10-07 14:37:51 +02:00
Daniel Bretoi
966e5a5575 use latest instead of newest in forget language 2018-10-06 12:02:22 -07:00
Alexander Neumann
5dbef3712e Merge pull request #2027 from j6s/feature.cache-dir-information 
Print base directory with cache command
 2018-10-06 12:40:48 +02:00
Johannes Hertenstein
9b8deb51ba Print base directory with cache command 2018-10-05 21:23:57 +02:00
Johannes Hertenstein
0916ff71bd Add defaults to command line help of --cache-dir 2018-10-04 19:50:05 +02:00
Alexander Neumann
585a5e3416 Use --host for all commands, deprecate --hostname 2018-10-03 14:12:35 +02:00
Alexander Neumann
fb31d66951 backup: Improve error message for invalid pattern 2018-10-03 11:12:51 +02:00
Stefano Fancello
e2da0a416c Expand Glob (wildcards character) in paths in file in --files-from 2018-10-03 10:37:35 +02:00
Alexander Neumann
6d53e767d5 cache: Print message when new cache is created 
Sometimes, users run restic without retaining the local cache
directories. This was reported several times in the past.

Restic will now print a message whenever a new cache directory is
created from scratch (i.e. it did not exist before), so users have a
chance to recognize when the cache is not kept between different runs of
restic.
 2018-08-29 22:01:10 +02:00
Quentin Bourgeois
a065ada46a
Make generated man pages more easy to read with items 
Allow Cobra to output a more list items friendly format, this make
reading of the documentation more easily using information from
manpages.
 2018-08-27 17:42:54 +02:00
Alexander Neumann
896089976a Fix comment 2018-08-26 19:54:17 +02:00
Alexander Neumann
de307ea2ab Merge pull request #1876 from restic/forget-explain 
forget: Add --explain
 2018-08-25 21:48:44 +02:00
Alexander Neumann
76387b6cd0 Remove old text table implementation 2018-08-20 22:00:23 +02:00
Alexander Neumann
9fd3796d93 forget: Display reasons why snapshots are kept 
This change displays the reasons for keeping a snapshot in the table,
unless `--compact` is specified.
 2018-08-20 21:47:51 +02:00
Alexander Neumann
15ad0e5bc7 walk: Pass parent tree ID to WalkFunc 2018-08-19 23:28:04 +02:00
Mikael Berthe
1f27d17c0d walker.Walk: Pass parent tree-id to WalkFunc 2018-08-19 23:28:04 +02:00
Mikael Berthe
8af918a1e4 find: Add support for multiple patterns or objects 2018-08-19 23:28:04 +02:00
Mikael Berthe
bb5425a1d8 find: Add support for blobs, IDs and packs lookups 
With --blob, --tree and --pack, the find command now lists the snapshots
that contain a specific tree or blob, or the snapshots that contain
blobs belonging to a given pack.
It also displays the pack ID a blob belongs to.

A list of IDs can be given, as long as the IDs are all of the same type.
 2018-08-19 22:32:30 +02:00
Matthew Holt
22475729ce Select specific Node fields for listing 2018-08-19 09:03:47 -06:00
Matthew Holt
04c67d700d ls: Stream output when using --json option 2018-08-19 00:18:43 -06:00
kitone
48cc2f2188 fix: switch struct_type value to lower case 2018-08-18 14:57:50 +02:00
kitone
7925217e25 ls: Add JSON output support for restic ls cmd 2018-08-18 14:57:50 +02:00
Alexander Neumann
401a564486 Merge pull request #1961 from mholt/statsoutput 
stats: Show what was scanned and scanning mode used
 2018-08-18 14:29:14 +02:00
Matthew Holt
2d89311d49 stats: Show what was scanned and scanning mode used 2018-08-17 17:15:30 -06:00
Alexander Neumann
5a25ad1972 Update version in build.go 2018-08-17 21:17:26 +02:00
Alexander Neumann
1ab5703404 prune: Fix calculation for removed bytes 2018-08-14 22:06:05 +02:00
Alexander Neumann
49d95e9a50 Merge pull request #1949 from restic/add-self-update 
Add command self-update
 2018-08-13 22:02:34 +02:00
Alexander Neumann
5fee36fa84 Merge pull request #1941 from mholt/lsfilter 
ls: Implement directory filter, optionally subfolders
 2018-08-13 21:58:32 +02:00
Alexander Neumann
6e1a3987b7 Add 'self-update' command 
This commit adds a command called `self-update` which downloads the
latest released version of restic from GitHub and replacing the current
binary with it. It does not rely on any external program (so it'll work
everywhere), but still verifies the GPG signature using the embedded GPG
public key.

By default, the `self-update` command is hidden behind the `selfupdate`
built tag, which is only set when restic is built using `build.go`. The
reason for this is that downstream distributions will then not include
the command by default, so users are encouraged to use the
platform-specific distribution mechanism.
 2018-08-12 23:34:47 +02:00
Alexander Neumann
9630398e3b ls: Rework and simplify logic 
This commit introduces two functions: withinDir() and
approachingMatchingTree()

Both bind the list of directories with a closure, so we don't need to
iterate over the list in the function passed to Walk(). This reduces the
indentation level and since we can just use return, we don't need the
breaks any more.

The case that len(dirs) == 0 can also be handled by the functions with a
return, which saves another indentation level.

The main function body of the function passed to Walk() was reduced to
three cases:

 * Within one of the dirs: Print the node, and if recursive operation is
   requested, directly return, so the walker continues recursive
   traversal

 * Approaching one of the dirs: don't print anything, but continue
   recursive traversal.

 * Nothing of the two: abort walking this branch of the tree.
 2018-08-12 23:13:34 +02:00
Alexander Neumann
7e34de4c29 ls: Add comments 2018-08-12 22:18:44 +02:00
Alexander Neumann
ace5cc4ed3 ls: Only skip directory nodes 
Special case for Walk(): When SkipDir is returned for a non-dir node,
the remaining nodes for the current tree are skipped. We don't want
that.
 2018-08-12 22:02:59 +02:00
Alexander Neumann
7f617cfd7f ls: Use nodepath for filter 2018-08-12 22:01:38 +02:00
Alexander Neumann
0deb4e5994 ls: Check dirs before opening the repository 
Users get feedback instantly, and before any expensive network calls
have been made.
 2018-08-12 21:59:57 +02:00
Andreas Skielboe
b07bb3d8c3 Reject files excluded by name before calling lstat to improve scan speed 
Adds a SelectByName method to the archive and scanner which only require
the filename as input, and can thus be run before calling lstat on the
file. Can speed up scanning significantly if a lot of filename excludes
are used.
 2018-08-12 17:51:12 +02:00
Matthew Holt
11ce572894 Fix bug where some folder listings were empty 2018-08-11 17:17:43 -06:00
Matthew Holt
7a468d1226 Speed up nonrecursive queries; include exact filter match 2018-08-11 16:18:09 -06:00
Matthew Holt
00e2fd8b5f Apply feedback and use SkipNode 2018-08-11 15:25:22 -06:00
Matthew Holt
ac3bd6b2eb Replace Exitf with errors.Fatalf 2018-08-10 22:15:33 -06:00
Matthew Holt
156d85a29b Minor fixes/tweaks; add docs 2018-08-10 22:10:02 -06:00
Matthew Holt
8c146eac4b ls: Implement directory filter, optionally subfolders 2018-08-10 21:41:38 -06:00
Matthew Holt
6f5b0f3622 stat: Improve error message for bad snapshot ID (fixes #1933) 2018-08-08 16:49:36 -06:00
Alexander Neumann
3422c1ca83 Merge pull request #1729 from mholt/stats 
Implement `restic stats` command to get more info about a repository
 2018-07-31 23:24:36 +02:00
Matthew Holt
f6b2731aa5 stats: Add manual doc, improve -h doc 
Also rename files-by-content to files-by-contents, once and for all
 2018-07-31 22:54:10 +02:00
Alexander Neumann
3678ec9ad8 Print message for password being read from stdin 
Closes #1911
 2018-07-31 20:21:18 +02:00
Alexander Neumann
12c797700e make statsWalkSnapshot return a function 2018-07-27 21:44:59 +02:00
Matthew Holt
daca9d6815 Consolidate mode flags; use new Walk function 2018-07-27 21:27:40 +02:00
Matthew Holt
930602a444 Update comment now that question was answered 2018-07-27 21:27:39 +02:00
Matthew Holt
acb05e7855 Fix filepath uniqueness bug for blobs-per-file mode 2018-07-27 21:27:39 +02:00
Matthew Holt
a7b95d716a Implement four counting modes 2018-07-27 21:27:39 +02:00
Matthew Holt
925b542eb0 Count unique files by blob sequence rather than tree ID 2018-07-27 21:27:39 +02:00
Matthew Holt
f7659bd8b0 stats: Initial implementation of stats command 2018-07-27 21:27:39 +02:00
Alexander Neumann
f715bef82f Merge pull request #1899 from garrmcnu/check-cache-dir 
check: Use --cache-dir argument
 2018-07-22 21:03:52 +02:00
Garry McNulty
7603ab7ac1 check: Update --cache-dir argument handling based on code review comments 
The temporary cache directory is created in the specified directory, or
if not specified in the default temporary directory.
 2018-07-22 18:24:11 +01:00
Garry McNulty
f1f69bc648 check: Use --cache-dir argument 
Closes #1880
 2018-07-20 20:51:20 +01:00
Alexander Neumann
fb74de6360 Return an error when exclude files cannot be read 2018-07-18 21:39:07 +02:00
Alexander Neumann
b511f4dce2 Improve help message for check 2018-07-05 22:19:08 +02:00
Mikael Berthe
fbf8073dfc Fix find -i (case-insensitive search) 2018-06-24 19:35:11 +02:00
Alexander Neumann
7ddf91b65c Merge pull request #1772 from ifedorenko/restore-verify 
Restore verify
 2018-06-23 14:50:29 +02:00
Alexander Neumann
8dae2de2ce Merge pull request #1858 from restic/fix-1857 
Allow excluding files with $ via --exclude-file
 2018-06-22 21:22:23 +02:00
Alexander Neumann
025ec9dff5 Allow excluding files with $ via --exclude-file 
Previously it wasn't possbile to exclude files with a literal dollar
sign (`$`) via exclude files, now users can write `$$` for that.
 2018-06-22 20:46:04 +02:00
David Roman
2384c1cee7
Ignore ExcludeOtherFs if Stdin is true 
Closes: #1807
 2018-06-21 23:12:38 +02:00
Steve Kriss
5088905502 add JSON output support for restic key list cmd 
Signed-off-by: Steve Kriss <steve@heptio.com>
 2018-06-18 15:26:26 -07:00
Iku Iwasa
f080142137 Support NetBSD without fuse 2018-06-16 15:55:04 +09:00
Igor Fedorenko
e206680947 restore: New --verify flag to verify restored files content 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-06-11 07:57:22 -04:00
Igor Fedorenko
5fa6dc53cb Refactor: introduced restorer tree visitor 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-06-11 07:57:22 -04:00
Igor Fedorenko
26be094f28 Refactor: moved restorer to separate package 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-06-11 07:57:22 -04:00
Alexander Neumann
321efec60c Fix integration tests on Windows 2018-06-10 00:00:22 +02:00
Alexander Neumann
8f26fe271c ls: Use walker for ls 2018-06-09 23:35:20 +02:00
Alexander Neumann
081743d0a5 find: Use walker.Walk 2018-06-09 23:35:20 +02:00
Alexander Neumann
ce01ca30d6 find: Correct tree pruning optimization 
The `find` command will now take care to only mark trees as "not found"
when the pattern couldn't be found within any subtree.

Closes #1825, #1823
 2018-06-09 18:59:13 +02:00
Alexander Neumann
e2d347a698 find: Use OS independent slash-based format 2018-06-09 18:58:13 +02:00
Alexander Neumann
42ebb0a0a6 backup: Parse timestamp earlier 2018-06-09 18:21:12 +02:00
Alexander Neumann
bfd923e81e rclone: Respect bandwith limits 2018-05-22 20:48:17 +02:00
Alexander Neumann
8d9d218d1c list: Improve error message 
Before:

    $ restic list
    Fatal: type not specified

After:

    $ restic list
    Fatal: type not specified, usage: list [blobs|packs|index|snapshots|keys|locks]

Closes #1783
 2018-05-17 19:41:56 +02:00
Mikael Berthe
a868a30f4d Fix restic backup --one-file-system / 
This patch should fix the following panic when trying to backup the
root filesystem with thre --one-file-system flag:

    % restic backup --one-file-system /
    (...)
    panic: item /, device id 2082 not found, allowedDevs: map[/:2082]
 2018-05-16 13:44:14 +02:00
Alexander Neumann
cc627e832b Add custom Duration type 2018-05-13 12:48:42 +02:00
Alexander Neumann
5a0f0e3faa Add support for keeping a range of snapshots 2018-05-13 12:48:42 +02:00
Matthew Holt
b52f2aa9a4 forget: Add policy to keep snapshots before a date 2018-05-13 12:48:10 +02:00
Alexander Neumann
159badf5ba Merge pull request #1778 from restic/fix-1771 
archiver: Improve error handling
 2018-05-13 00:13:54 +02:00
Alexander Neumann
903a3a31dc Merge pull request #1776 from restic/fix-1775 
Always use cleaned path for excludes
 2018-05-13 00:13:52 +02:00
Alexander Neumann
581c62ee72 archiver: Improve error handling 
This commit changes how the worker goroutines for saving e.g. blobs
interact. Before, it was possible to get stuck sending an instruction to
archive a file or dir when no worker goroutines were available any more.
This commit introduces a `done` channel for each of the worker pools,
which is set to the channel returned by `tomb.Dying()`, so it is closed
when the first worker returned an error.
 2018-05-12 23:55:54 +02:00
Alexander Neumann
ef7747313d backup: Use absolute paths for allowed devices 2018-05-11 21:32:44 +02:00
Alexander Neumann
18d4ac2fd9 backup: Always use cleaned path for excludes 2018-05-11 21:32:41 +02:00
Alexander Neumann
9180e2c48a Remove unneeded file excludes 2018-05-11 21:26:10 +02:00
Gaige Lama
d3c0bd6d0e
Fix small typo 
"explicitely" should be "explicitly"
 2018-05-11 10:55:12 -04:00
Mikael Berthe
ffc276a603 cat: Do not display a stack trace 
Don't show a stack trace when "restic cat snapshot" is invoked with
invalid/nonexistent IDs.
 2018-05-10 19:22:46 +02:00
Alexander Neumann
fca4fe4459 backup: Disable status output for --quiet 2018-05-02 21:24:18 +02:00
Alexander Neumann
d0974c155d backup: Tune user interface a bit 
Make non-verbose mode a bit more verbose.
 2018-05-01 22:18:32 +02:00
Alexander Neumann
8026e6fdfb Merge pull request #1749 from restic/add-cache-cmd 
Add 'cache' command to list and cleanup cache dirs
 2018-05-01 21:25:52 +02:00
Alexander Neumann
f928aeec34 Add 'cache' command 2018-05-01 16:27:44 +02:00
Alexander Neumann
f77bc0fae8 Use textfile.Read() to read files 
This converts the text to UTF-8 and removes a Byte Order Mark.
 2018-05-01 16:25:53 +02:00
Alexander Neumann
d8bbe5dc84 Print repository ID after opening 2018-04-30 14:19:07 +02:00
Alexander Neumann
d926b9fd80 Add profile build flag 2018-04-30 14:19:04 +02:00
Alexander Neumann
4ba8d40282 Add block profile option 2018-04-30 14:18:45 +02:00
Alexander Neumann
4fb1401266 Fix --cacert help text 2018-04-29 13:17:33 +02:00
Alexander Neumann
1449d7dc29 Remove background checking code 2018-04-28 22:08:11 +02:00
Alexander Neumann
0e78ac92d8 Use new archiver code for backup 2018-04-28 22:08:11 +02:00
Alexander Neumann
9fac2ca832 Add flags to set verbosity 2018-04-25 14:42:45 +02:00
Alexander Neumann
a5c0cf2324 Add workaround for symlinked temp dir on darwin 
Chdir to the tempdir, then use os.Getwd() to get the name that
filepath.Abs() uses (and stores in the Snapshot).
 2018-04-25 14:42:45 +02:00
Alexander Neumann
38926d8576 Use new archiver code in tests 2018-04-25 14:42:45 +02:00
Alexander Neumann
b804279fe8 integration tests: Don't print anything to stdout 2018-04-22 11:37:05 +02:00
Alexander Neumann
a069467e72 ls: Improve output 2018-04-22 11:37:05 +02:00
Alexander Neumann
09365cc4ea Add --trace-profile 2018-04-22 11:37:05 +02:00
Eri Bastos
2aa6b49651 Return exit code 130 when SIGINT is received 2018-04-20 21:09:50 +02:00
Matthew Holt
c5829e9ffc key: Add flag for non-interactive password changes 2018-04-20 14:29:39 +02:00
Alexander Neumann
1c1fede399 Improve error message for orphaned pack files 2018-04-07 10:07:54 +02:00
Eri Bastos
901e1b129c Fixed issue #1608 - Use --time argument properly 
Backups via stdin will now handle --time argument and pass it down as
expected
 2018-04-03 14:40:42 -03:00
Alexander Neumann
01fe719aff check: Make sure temp cache dir is removed 2018-04-01 18:09:53 +02:00
Alexander Neumann
2c964df3e2 Merge pull request #1699 from restic/fix-incremental-backup-test 
Slightly increment size for TestIncrementalBackup
 2018-04-01 14:33:12 +02:00
Alexander Neumann
a95eb33616 check: Use cache in temporary directory if possible 
Closes #1694
 2018-04-01 13:59:27 +02:00
Alexander Neumann
4544a77172 Slightly increment size for TestIncrementalBackup 
This should make the test more reliable, it should hit the accidental
"repo is has grown too much" way less often.
 2018-04-01 13:49:42 +02:00
Alexander Neumann
fe99340e40 Add rclone backend 2018-04-01 10:16:31 +02:00
Alexander Neumann
d3f9c8b362 forget: Print policy 2018-03-30 10:24:26 +02:00
Leo R. Lundgren
923be90906 Make version output one line. 2018-03-21 21:49:03 +01:00
Bowen Song
34671d7c9b Skip archiver.Scan before backup when --quiet is set 2018-03-20 20:44:10 +00:00
Alexander Neumann
8206f85d2e Merge pull request #1552 from lawrencejones/use-auto-auth 
Automatically load Google auth
 2018-03-18 19:53:30 +01:00
Alexander Neumann
dfb6d0fced Reduce data for TestIncrementalBackup 2018-03-11 21:17:27 +01:00
Alexander Neumann
c6c1dccc53 Reduce data set for TestRestore 2018-03-11 21:10:37 +01:00
Alexander Neumann
279566bafe Reduce dataset for integration tests 2018-03-11 21:07:47 +01:00
Alexander Neumann
c67a8452f7 Disable polynomial check for chunker for tests 2018-03-11 20:59:40 +01:00
Lawrence Jones
0dfdc11ed9
Automatically load Google auth 
This change removes the hardcoded Google auth mechanism for the GCS
backend, instead using Google's provided client library to discover and
generate credential material.

Google recommend that client libraries use their common auth mechanism
in order to authorise requests against Google services. Doing so means
you automatically support various types of authentication, from the
standard GOOGLE_APPLICATION_CREDENTIALS environment variable to making
use of Google's metadata API if running within Google Container Engine.
 2018-03-11 17:11:25 +00:00
Eri Bastos
b9d643358a Skip checking for password file existence if command is 'version' - Issue #1632 2018-03-08 17:55:03 -04:00
Jasper Lievisse Adriaanse
96311d1a2b Add support for illumos/Solaris 
This does come without xattr/fuse support at this point.

NB: not hooking up the integration tests as restic won't compile without
    cgo with Go < 1.10.
 2018-03-04 20:11:29 +00:00
Alexander Neumann
7d59df1ab8 mount: Ignore non-existing locks 
Closes #1637
 2018-02-25 13:11:03 +01:00
Igor Fedorenko
07d080830e Add --read-data-subset flag to check command 
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
 2018-02-18 23:31:27 -05:00
Alexander Neumann
c5575c7ed9 Strip spaces from items read via --files-from 
In #1590, it was mentioned that while lines read from exclude files via
`--exclude-file` have leading and trailing spaces stripped, this is not
the case for lines read via `--files-from`. This commit fixes that,
spaces are always stripped.
 2018-02-11 20:56:11 +01:00
Alexander Neumann
cbfa516575 Remove archiver progress "data processed" bandwith 
This commit removes the bandwidth displayed during backup process. It is
misleading and seldomly correct, because it's neither the "read
bandwidth" (only for the very first backup) nor the "upload bandwidth".
Many users are confused about (and rightly so), c.f. #1581, #1033, #1591

We'll eventually replace this display with something more relevant when
 #1494 is done.
 2018-01-28 09:30:26 +01:00
Alexander Neumann
5dc8d3588d GS: Use generic http transport 
During the development of #1524 I discovered that the Google Cloud
Storage backend did not yet use the HTTP transport, so things such as
bandwidth limiting did not work. This commit does the necessary magic to
make the GS library use our HTTP transport.
 2018-01-27 20:12:34 +01:00
Alexander Neumann
c34db983d8 Read TLS client cert and key from the same file 2018-01-27 14:02:01 +01:00
Bryce Chidester
e805b968b1 Support for TLS client certificate authentication 
This adds --tls-client-cert and --tls-client-key parameters and enables use
of that certificate/key pair when connecting to https servers.
 2018-01-27 13:18:22 +01:00
Alexander Neumann
ff3de66ddf Merge pull request #1582 from restic/optimize-debug-log 
Optimize debug logs
 2018-01-26 21:57:18 +01:00
Alexander Neumann
f63c7048c7 index: Limit index files to 3000 pack files 2018-01-26 21:07:16 +01:00
Alexander Neumann
663c57ab4d debug: Remove manual Str() call Log() 2018-01-25 20:49:41 +01:00
Alexander Neumann
9c55e8d69c Merge pull request #1549 from MJDSys/more_index_lookup_avoids 
More optimizations to avoid calling Index.Lookup()
 2018-01-24 20:53:30 +01:00
Matthew Dawson
df2c03a6a4
repository/master_index: Optimize Index.Lookup() 
When looking up a blob in the master index, with several
indexes present in the master index, a significant amount of time
is spent generating errors for each failed lookup.  However, these
errors are often used to check if a blob is present, but the contents
are not inspected making the overhead of the error not useful.

Instead, change Index.Lookup (and Index.LookupSize) to instead return
a boolean denoting if the blob was found instead of an error.  Also change
all the calls to these functions to handle the new function signature.

benchmark                                            old ns/op     new ns/op     delta
BenchmarkMasterIndexLookupSingleIndex-6              820           897           +9.39%
BenchmarkMasterIndexLookupMultipleIndex-6            12821         2001          -84.39%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       5378          492           -90.85%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     17026         1649          -90.31%

benchmark                                            old allocs     new allocs     delta
BenchmarkMasterIndexLookupSingleIndex-6              9              9              +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            59             19             -67.80%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       22             6              -72.73%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     72             16             -77.78%

benchmark                                            old bytes     new bytes     delta
BenchmarkMasterIndexLookupSingleIndex-6              160           160           +0.00%
BenchmarkMasterIndexLookupMultipleIndex-6            3200          240           -92.50%
BenchmarkMasterIndexLookupSingleIndexUnknown-6       1232          48            -96.10%
BenchmarkMasterIndexLookupMultipleIndexUnknown-6     4272          128           -97.00%
 2018-01-23 22:25:56 -05:00
Alexander Neumann
b0c6e53241 Fix calls to repo/backend.List() everywhere 2018-01-21 21:15:09 +01:00
Alexander Neumann
663e8284b2 init: Correct text (backend vs. repository) 
Closes #1565
 2018-01-20 09:51:49 +01:00
Alexander Neumann
0bdb131521 Remove SuspendSignalHandler 2018-01-17 23:14:47 +01:00
Alexander Neumann
97565d68a2 fuse/mount: Add option for snapshot template 2018-01-15 14:47:17 +01:00
Alexander Neumann
92316a9853 restore: Improve error message for more than one ID 2018-01-06 22:44:18 +01:00
Alexander Neumann
c912b38bf0 forget: Also run prune when only IDs are forgotten 2018-01-01 21:27:40 +01:00
Alexander Neumann
18ecd9df30 Improve limiting HTTP based backends 2017-12-29 12:43:49 +01:00
Gábor Lipták
32505c3916
Correct golint warning on return 2017-12-26 17:09:41 -05:00
Alexander Neumann
84e493beba b2: Warn when account ID or key is empty 
Closes #1490
 2017-12-19 21:12:38 +01:00
Alexander Neumann
2f26fb8834 Only print cache warning for terminals 2017-12-13 19:58:10 +01:00
Alexander Neumann
d3ebe1311f backup: Reject filenames with paths read from stdin 2017-12-13 19:55:53 +01:00
Alexander Neumann
3a5e040b7e Merge pull request #1439 from armhold/propagate-context 
replace ad-hoc context.TODO() with gopts.ctx
 2017-12-08 20:27:36 +01:00
George Armhold
1695c8ed55 use global context for check, debug, dump, find, forget, init, key, 
list, mount, tag, unlock commands

gh-1434
 2017-12-06 07:02:55 -05:00
George Armhold
366622f09a use global context for cat and ls 
gh-1434
 2017-12-06 05:48:39 -05:00
Alexander Neumann
0405e67f8b Change diff output slightly 2017-12-05 22:26:56 +01:00
Alexander Neumann
df350e1f6e Improve diff 2017-12-05 22:05:22 +01:00
Alexander Neumann
2424012d75 Add 'diff' command 2017-12-03 18:33:35 +01:00
Alexander Neumann
88607fc625 Correctly format all node types 2017-12-03 17:38:55 +01:00
Alexander Neumann
ad82781743 Reenable cache 
In 6341c7d72c, the cache was accidentally
disabled due to a bug, this commit reenables the cache.
 2017-12-03 15:52:57 +01:00
George Armhold
d886cb5c27 replace ad-hoc context.TODO() with gopts.ctx, so that cancellation 
can properly trickle down from cmd_*.

gh-1434
 2017-12-03 07:22:14 -05:00
Alexander Neumann
2579fe6b7b Ignore SIGPIPE 
Handling SIGPIPE made restic abort when a TCP connection was reset by a
server. This happened on DigitalOcean Spaces, which uses the s3 backend.
 2017-11-30 21:23:43 +01:00
Alexander Neumann
eddb8549ef backup: By default, do not save the access time 
This can be re-enabled with `--with-atime`.
 2017-11-28 21:31:35 +01:00
Alexander Neumann
2567026ccb Merge pull request #1447 from fawick/fixMultipleRejectIfPresent 
Fix multiple rejectIfPresent cancelling each other
 2017-11-28 19:20:29 +01:00
Alexander Neumann
0cc8fc6f18 Merge pull request #1431 from n0npax/master 
fix #1411
 2017-11-27 21:54:06 +01:00
Fabian Wickborn
27fadd2c6e Document approach for multiple reject-if-present test 2017-11-27 21:38:15 +01:00
Alexander Neumann
dc38265b54 Merge pull request #1436 from restic/remove-old-cache 
Remove old cache directories
 2017-11-27 21:37:05 +01:00
Fabian Wickborn
1ea518d5ef cmd/restic: Use a dedicated cache for each rejectIfPresent 2017-11-27 17:33:53 +01:00
Fabian Wickborn
901cd5edef cmd/restic: Add test for rejectIfPresent bug 
All RejectFuncs returned by rejectIfPresent share the same rejection
cache and hence might cancel each other out.
 2017-11-27 17:26:19 +01:00
Alexander Neumann
35a5307db3 Merge pull request #1415 from armhold/signals 
also handle SIGPIPE in cleanup routines
 2017-11-24 21:28:08 +01:00
Alexander Neumann
6341c7d72c cache: Add option to remove old cache dirs 2017-11-24 20:53:28 +01:00
Alexander Neumann
fa893ee477 cache: Add detection code for old cache dirs 2017-11-24 20:53:26 +01:00
n0npax
b12bba4e2a fix #1411 
replace panic during index save with fatal error
 2017-11-19 11:40:47 +01:00
George Armhold
0ed2401711 exit 1 if received signal is other than SIGINT 
send cleanup msg to stderr, not stdout
gh-1413
 2017-11-09 07:16:01 -05:00
George Armhold
c347431907 also handle SIGPIPE in cleanup routines 
fixes gh-1413: restic fails to cleanup locks when bash pipeline fails
 2017-11-05 07:14:27 -05:00
Björn Ketelaars
f63d7048f9 Fix test failing on OpenBSD #1307 
Is seems that #1307 is similar to #1087, which describes a comparable
observation on Apple's new filesystem. #1389 Has been committed and
fixes the problem on Darwin.

Although I'm not sure if the root cause of the issue is the same the
solution is similar for OpenBSD, and leverages #1389.
 2017-11-05 07:27:58 +01:00
Alexander Neumann
526aaca6f5 Merge pull request #1389 from wojas/fix-modtime-tests-on-apfs 
Fix modtime tests on APFS: allow 1us difference
 2017-10-29 14:20:40 +01:00
Konrad Wojas
f3016a9096 Darwin test fix: allow 1μs timestamp difference 
On Darwin, allow a 1μs difference in restored timestamps, because
macOS <10.13 cannot restore with nanosecond precision and the current
version of Go (1.9.2) does not yet support the new syscall required
for this. (#1087 #1389)
 2017-10-29 20:48:07 +08:00
Zlatko Čalušić
134f834c60 Use lowercase in messages from check/tag commands, too 2017-10-27 21:06:34 +02:00
Konrad Wojas
c5da90a5b7 Add --last flag to snapshots command 
Add --last flag to snapshots command to only show the last entry for any
(hostname, paths) combination.

This makes it easier to check when various paths were last backed up.
 2017-10-26 14:02:29 +08:00
Alexander Neumann
8d37b723ca Upadte vendored dependencies 2017-10-22 10:07:36 +02:00
TobyLL
9e3dde8ec7 Fix whitespace 2017-10-19 15:52:06 +01:00
TobyLL
9dba182e51 Ignore comments (lines starting with #) in the --files-from file 2017-10-19 15:48:22 +01:00
Alexander Neumann
beaf55f1fc dump: Allow absolute paths 2017-10-16 20:22:01 +02:00
Alexander Neumann
0e7281eb71 Merge pull request #1346 from fawick/cmdCatFile 
Add dump sub-command for dumping files to stdout
 2017-10-16 20:13:44 +02:00
Alexander Neumann
897c923cc9 Retry failed backend requests 2017-10-14 15:56:25 +02:00
Alexander Neumann
67193e3deb Improve error message when creating lock failed 2017-10-14 15:56:25 +02:00
Fabian Wickborn
d1d9c3f9d7 Renamed cmd_catfile.go to cmd_dump.go 2017-10-14 13:55:21 +02:00
Fabian Wickborn
cd5cbe0910 Rename debug dump related variable and run function 2017-10-14 13:55:00 +02:00
Fabian Wickborn
814e992c0b Rename subcommand catfile to dump 2017-10-14 11:34:04 +02:00
Fabian Wickborn
87d084e18c Add subcommand dump 2017-10-12 20:24:34 +02:00
Alexander Neumann
f2314b26ba Move 'dump' to 'debug dump' 2017-10-12 20:18:45 +02:00
Alexander Neumann
74dcf41f25 Disable 'dump' for non-debug builds 2017-10-12 20:14:48 +02:00