octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-02 11:46:36 +00:00
Code Issues Releases Wiki Activity
2,623 Commits 7 Branches 33 Tags 66 MiB
d1bd160b0a
Commit Graph

848 Commits

Author SHA1 Message Date
Alexander Neumann
698ba57597 backend/tests: Print error stacktrace if available 2017-04-10 22:42:33 +02:00
Alexander Neumann
27ce6a85e9 sftp: Rework Open/Create 2017-04-10 22:42:33 +02:00
Alexander Neumann
ae290ab374 sftp: Rename Dir -> Path 2017-04-10 22:42:33 +02:00
Alexander Neumann
ab602c9d14 sftp: Add Layout 2017-04-10 22:40:24 +02:00
Alexander Neumann
2e53af1b75 sftp: Rename Open/Create 2017-04-10 22:39:13 +02:00
Alexander Neumann
a725e065d9 Merge pull request #903 from restic/fix-hardlinks 
Add new field DeviceID and tests
 2017-04-07 21:05:20 +02:00
Alexander Neumann
db7e23b423 Skip /dev/null on darwin 2017-04-07 20:37:20 +02:00
Alexander Neumann
10a395ca33 Make tests runnable on os x/darwin 2017-04-06 20:36:09 +02:00
Alexander Neumann
522c7ade91 Add test for double defined flags 2017-04-06 19:44:53 +02:00
Alexander Neumann
a3d6099892 Remove short option conflict 2017-04-06 19:14:38 +02:00
Alexander Neumann
280028290e Disable tests on darwin 2017-04-05 21:35:29 +02:00
Alexander Neumann
c195139d31 Only run tests on unix 2017-04-05 20:51:26 +02:00
Alexander Neumann
6f1b03415c Fix hardlinks 2017-04-05 20:45:24 +02:00
Alexander Neumann
ffcb015581 Add new field DeviceID and tests 2017-04-05 20:42:15 +02:00
jgfrm
6674b2a70c Moved assignment of device id outside the switch 2017-04-05 11:33:08 +02:00
jgfrm
c855d6bb9a storage of device id of files 2017-04-04 21:28:25 +02:00
Alexander Neumann
1086528ab7 sftp: Fix errors import 2017-04-03 21:42:41 +02:00
Alexander Neumann
c26dd6b76f sftp: Integrate command 2017-04-03 21:05:42 +02:00
Alexander Neumann
d3b6f75848 sftp: Add SplitShellArgs 2017-04-03 08:57:33 +02:00
Alexander Neumann
d1efdcd78e Add integration test for layouts 2017-04-02 20:35:17 +02:00
Alexander Neumann
95ab5adda1 local: Expose layout as extended option 2017-04-02 20:29:00 +02:00
Alexander Neumann
c5eb36fe9d layout: improve error message for ParseLayout 2017-04-02 20:28:42 +02:00
Alexander Neumann
e3e3a8a695 local: Add layout tests 2017-04-02 20:01:40 +02:00
Alexander Neumann
24ebf95f33 local: Automatically detect layout 2017-04-02 20:01:40 +02:00
Alexander Neumann
54465c92cc layout: Allow passing in a default layout 2017-04-02 20:01:36 +02:00
Alexander Neumann
f7c4b3a922 Fix layout detection 2017-04-02 19:18:03 +02:00
Alexander Neumann
50dfa64a54 Add layout name parser 2017-04-02 17:57:28 +02:00
Alexander Neumann
c6b8ffbb61 Add layout auto detection 2017-04-02 17:25:22 +02:00
Alexander Neumann
3e81dcdfc2 Add cloud and s3 layout 2017-03-26 22:20:10 +02:00
Alexander Neumann
782b740c95 local: Remove unused code 2017-03-26 22:14:37 +02:00
Alexander Neumann
3fd6fa6f86 local: Use Layout for filename generation 2017-03-26 21:53:26 +02:00
Alexander Neumann
6a201f7962 backend: Add Layout 2017-03-26 21:52:49 +02:00
Alexander Neumann
80a864c52c test: Add TempDir() helper 2017-03-26 20:40:45 +02:00
Alexander Neumann
c8eea49909 debug: Allow creating insecure repositories 
Uses low-security KDF parameters for scrypt(). Do not use in production!
 2017-03-26 10:55:08 +02:00
Alexander Neumann
719bb18316 Parse extended options 2017-03-25 17:52:16 +01:00
Alexander Neumann
a8a7701f60 options: Add namespace to Apply() 2017-03-25 17:52:16 +01:00
Alexander Neumann
2924ebc124 options: Add Apply() 2017-03-25 17:52:16 +01:00
Alexander Neumann
f587a5f4f0 options: fail when key is configured twice 2017-03-25 17:52:16 +01:00
Alexander Neumann
946b4f4b86 Add extended options via -o/--option 2017-03-25 17:52:12 +01:00
Alexander Neumann
d0a5e86da1 Add free-form options parsing 2017-03-25 17:45:11 +01:00
Alexander Neumann
2e0b19f63f location: Move to backend/location 2017-03-25 17:45:11 +01:00
Alexander Neumann
9861f3d435 location: Reformat tests, use sub tests 2017-03-25 17:45:10 +01:00
Alexander Neumann
482a6e9840 local: Add Config struct 2017-03-25 17:45:10 +01:00
Alexander Neumann
1733b24f9d doc: Add section about how to change the sftp port 
Closes #895
Related to #785
 2017-03-25 09:07:50 +01:00
Alexander Neumann
6f76a6db66 rest: Make backend honor the REST protocol 2017-03-16 21:50:26 +01:00
trbs
f1ba45723c introduce RWLock for caching obj.Stat() 2017-03-14 23:05:51 +01:00
trbs
b523eef294 Cache size of last ReadAt on S3 for performance 
Each obj.Stat() call adds another request to the S3 endpoint
for some commands a lot of ReadAt calls are made for the same
object in S3. This patch essentially cuts the number of calls
to S3 in this case in half. Speeding up the progress and lowering
costs to S3.
 2017-03-14 14:02:50 +01:00
Pauline Middelink
642cd3bebf Fix restoring files as non-root user 
As we cannot reliably detect in advance if we can set ownership, permissions,
timestamps or ext attributes, execute ALL the requested changes before
returning the first error we found.

Report total errors at end of restore and stop printing entire stacktraces
where just the error message is sufficient.

Fixes #655
 2017-03-12 16:51:58 +01:00
Alexander Neumann
887e81188f Merge pull request #877 from middelink/fix-875 
Fix crash at restoring pre-existing hardlinked files
 2017-03-11 10:13:43 +01:00
Pauline Middelink
b56e16acd0 Fix crash at restoring pre-existing hardlinked files 
Remove target file, ignore non existing file errors.

Small memory saving: Only keep inodes around for files with a link count > 1.
(We will/can never be asked to restore a hardlinked file with one of the
files having a link count of 1.)

Closes #836
 2017-03-11 00:06:58 +01:00
Pauline Middelink
b75186533d Restore does not truncate files it restores 
Missing O_TRUNC when doing O_CREAT.

Closes #875
 2017-03-10 22:46:13 +01:00
Alexander Neumann
8958efba60 Merge pull request #871 from middelink/fix-869 
Refactor output of `find` to allow for json and normal output.
 2017-03-10 20:03:26 +01:00
Pauline Middelink
a9707a5728 Refactor output of find to allow for json and normal output. 
Rather complicated solution becaused I wanted to retain the streaming
character of the output, which means for json I have to manually add
headers and footers per snapshot scanned + a list around the whole
set.

As the json ouput is now partly handcrafted, add proper testing to catch
unintentional changes to the output, making it non-json compliant.

Closes #869
 2017-03-10 16:23:58 +01:00
Alexander Neumann
deddedd88f Merge pull request #872 from middelink/fix-787 
Suppress (non error) output of `forget` during -q
 2017-03-09 21:23:36 +01:00
Pauline Middelink
b38294f236 Suppress (non error) output of forget during -q 
Closes #787
 2017-03-09 20:20:17 +01:00
Pauline Middelink
8a05de537f Refactor prune and rebuild_index 
Factor out and reuse `rebuildIndex()` in cmd_rebuild_index and cmd_prune.

Use contexts.
 2017-03-08 20:30:52 +01:00
Pauline Middelink
8a92687d9a Refactor find and ls commands 
Implement filtering by using `FindFilteredSnapshots()` to iterate over the snapshots

Refactor cmd_ls' `PrintNode()` into format.go, reuse its pretty printing in both `find`
and `ls` commands.

Use contexts.
 2017-03-08 20:29:31 +01:00
Pauline Middelink
3432e7edcd Refactor tag to use FindFilteredSnapshots() 2017-03-08 20:28:44 +01:00
Pauline Middelink
3c6c17abcd Refactor forget and snapshots command 
Implement filtering by using `FindFilteredSnapshots()` to iterate over the snapshots

Refactor cmd_snapshots' `PrintSnapshots()` so its pretty printing can be used from
both `forget` and `snapshots`.

Use contexts.
 2017-03-08 20:24:58 +01:00
Pauline Middelink
11d237c252 New helper function FindFilteredSnapshots to iterate over snapshots 
This helper function takes a set of filters and/or a list of snapshots
from the commandline. It returns a channel of *Snapshot.

When snapshot ids are given, they are checked for validity and their
corresponding Snapshots returned. The snapshot id "latest" is handled
special to return either the last snapshot (no filters) or the last
snapshot matching the filters.

When no arguments are given, the filters are applied over all available
snapshots and these are returned.
 2017-03-08 20:19:12 +01:00
Pauline Middelink
0f7b6ec5ac Adapt key command to context world. 2017-03-08 20:17:30 +01:00
Pauline Middelink
b4526c4e6e Enable the use of context in restic 
Set up a cancelble context in global options, hook it into the ctrl-C handler
for proper cancel propegation.

Bump up minimal requirement for Go to version 1.7 in documentation
and test-build files.
 2017-03-08 20:12:16 +01:00
Pauline Middelink
3eaaa0f286 Correct some typo's in comments. 2017-03-08 20:09:24 +01:00
Pauline Middelink
b1c8071163 Add filtering to mount command 2017-03-08 19:59:19 +01:00
Pauline Middelink
3468108d4c Implement --tag processing to backup and restore command 
Add `tags` argument to `FindLatestSnapshot()`
 2017-03-08 19:55:58 +01:00
Alexander Neumann
33c8dd4ee5 Merge pull request #867 from middelink/optimize 
Optimize List() pipeline
 2017-03-08 11:50:47 +01:00
Pauline Middelink
7c989ca487 Optimize List() pipeline 2017-03-07 20:58:36 +01:00
Pauline Middelink
e9a2982ecd Add --path for snapshot filtering by path. 
Add `--group-by-tags` for grouping on host,tags,dirs instead of host,dirs.

Borrow the snapshot printing from cmd_snapshot.

Closes #841
 2017-03-07 19:28:42 +01:00
Alexander Neumann
f5a55a81f7 Merge pull request #865 from restic/handle-empty-snapshots 
Refuse to create empty snapshots
 2017-03-07 11:21:48 +01:00
Alexander Neumann
340f2c80a0 Merge pull request #864 from restic/find-case-insensitive 
find: Add option to ignore case
 2017-03-07 11:20:33 +01:00
Alexander Neumann
bb144436c7 Add test for empty snapshot 2017-03-07 11:17:15 +01:00
Alexander Neumann
b50d3ba805 Refuse to create empty snapshots 
Closes #862
 2017-03-07 11:12:34 +01:00
Alexander Neumann
221bef48c0 find: Add option to ignore case 
Closes #859
 2017-03-07 10:58:09 +01:00
Pauline Middelink
afcc1ba706 Create a helper function to get the terminal width 
Rationale: contain terminal access to one file.
 2017-03-06 11:23:00 +01:00
Alexander Neumann
bf88a62a16 Merge pull request #858 from middelink/snapshot-layout 
Fix layout issue in cmd_snapshot "ascii art"
 2017-03-06 09:10:35 +01:00
Pauline Middelink
354e8ffb82 Fix layout issue in cmd_snapshot "ascii art" 
The layouter does not account for multi tags when determining the
need for ascii art.

36fd8178  2017-03-03 21:35:04  abuseio.polyware.nl    NL          /
                                                      A       └──

vs

36fd8178  2017-03-03 21:35:04  abuseio.polyware.nl    NL      ┌── /
                                                      A       └──
 2017-03-06 02:49:15 +01:00
Pauline Middelink
e1c828be3e Fix SamePaths() and make it into a receiver function 
Add `HasPath(paths []string) bool` to Snapshot for testing if the
snapshot has at least the paths given to the function.

Reimplemented SamePaths(paths []string) so it does what the name implies,
compare if all given paths are in the snapshot.
 2017-03-06 02:21:58 +01:00
Pauline Middelink
1fa2313aef Snapshot: Add Original ID 
The Original ID is used when the snapshot is modified (e.g. by `tag`
command). Adjust integration testing to assert correctness.
 2017-03-05 19:55:23 +01:00
Pauline Middelink
26e266a951 Fix type of ID field in cmd_snapshots type Snapshot 2017-03-05 19:55:22 +01:00
Pauline Middelink
208edaa3d1 Snapshot: Add AddTags() and RemoveTags() 
Both prevent duplicate tags.
 2017-03-05 19:55:22 +01:00
Pauline Middelink
f6a258b4a8 Add tag: Manipulate tags on existing snapshots 
Add integration testing.
 2017-03-05 19:55:22 +01:00
Pauline Middelink
45e9f35654 Make columns for host and tags size width dynamicly on their content. 2017-03-05 05:32:01 +01:00
Pauline Middelink
edd5c8b44d Add integration test to make sure cmd_backup adds tags when required. 2017-03-05 05:24:40 +01:00
Pauline Middelink
7238a3ee89 Changed cmd_snapshots to be testable (no more using os.Stdout) 2017-03-05 05:24:11 +01:00
Alexander Neumann
bbcab800c9 Merge pull request #850 from middelink/fix-848 
Add progressbar to repack and blob remove phases of prune cmd.
 2017-03-04 18:47:03 +01:00
Pauline Middelink
5564c78e53 English typo: rewriten > rewritten. 2017-03-04 17:43:58 +01:00
Pauline Middelink
792b81725e Add progressbar to repack and blob remove phases of prune cmd. 2017-03-04 17:38:34 +01:00
Pauline Middelink
1273c6f3d4 Display the proper amount of bytes we will be pruning from the repo. 2017-03-04 15:17:44 +01:00
Alexander Neumann
90bc187355 Merge pull request #840 from middelink/master 
Display absolute paths when displaying the output of ls and find.
 2017-03-03 12:09:20 +01:00
Pauline Middelink
5ecaaea90b Really use absolute pathnames, not all systems use /. 2017-03-03 11:14:39 +01:00
Pauline Middelink
039e81b04b Fix unit test, we need to check for absolute paths now. 2017-03-02 23:30:56 +01:00
Pauline Middelink
efb4315a1e Display absolute paths when displaying the output of ls and find. 2017-03-02 22:41:11 +01:00
Alexander Neumann
f53d33ba34 Make ArchiveReader a struct 2017-03-02 15:45:35 +01:00
Alexander Neumann
9b776dc7ab Use new Index implementation for rebuild-index 2017-03-02 15:23:59 +01:00
Alexander Neumann
7c92994f10 Clarify variable name 2017-03-02 14:52:18 +01:00
Alexander Neumann
bf97cc7efa Allow filtering absolute paths 
Before, the restorer called the filter function with a relative path,
this prevented anchoring absolute patterns (which just never matched).
Now call the restore function with an absolute virtual path, starting at
the filepath separator.

Closes #834
 2017-03-02 14:50:54 +01:00
Alexander Neumann
31ff506309 Ignore empty lines in --files-from 
Closes #822
 2017-02-27 19:42:00 +01:00
Alexander Neumann
685f5ebbd1 Add --prune switch to forget 2017-02-21 10:58:30 +01:00
Alexander Neumann
b8ce1b4e69 Correct archiver behavior in case of errors 2017-02-18 17:46:06 +01:00
Alexander Neumann
132afbe83b Correct error check for ENOTSUP, add errors.Wrap() 2017-02-18 14:36:37 +01:00
Alexander Neumann
ef52d15edd Continue if extended attribute cannot be read 2017-02-18 14:35:11 +01:00
Alexander Neumann
6df2f9e5ba Add support for extended attributes on FreeBSD 2017-02-18 14:35:11 +01:00
Jaap Gordijn
49cae0904f Add support for extended attributes (e.g. ACL) 2017-02-16 11:44:51 +01:00
Alexander Neumann
40685a0e61 Merge pull request #803 from ibib/adds-json-support 
Adds JSON support for the snapshots command
 2017-02-16 11:26:14 +01:00
Alexander Neumann
4772a4986b Merge pull request #808 from restic/fix-807 
restore: Make sure buffer is large enough
 2017-02-15 19:58:03 +01:00
Alexander Neumann
50d066befb restore: Make sure buffer is large enough 2017-02-15 15:19:28 +01:00
Øystein Olsen
c35e48291d Fix correct number of arguments for key command 2017-02-13 16:16:34 +01:00
Øystein Olsen
511278b66a Fix wrong description of rebuild-index command in help text 2017-02-13 16:07:29 +01:00
Øystein Olsen
514a11346d Add long description of list command in help text 2017-02-13 16:06:27 +01:00
Øystein Olsen
2eb75bb941 Consistently refer to 'the' instead of 'a' repository in help text 2017-02-13 16:05:25 +01:00
Øystein Olsen
9922ce97bf Use lowercase consistently in help text 2017-02-13 16:02:47 +01:00
ibib
3ed4127297 Adds JSON support for the snapshots command 2017-02-12 21:43:39 +01:00
Jaap Gordijn
3047702ded Correct hardlinks for fuse directories 2017-02-11 21:54:23 +01:00
Alexander Neumann
8c34eaad15 Improve error message 2017-02-11 14:28:15 +01:00
Alexander Neumann
0492eabff1 Improve error messages 2017-02-11 14:24:11 +01:00
Alexander Neumann
7797e084f9 checker: Pass on error loading an index 2017-02-11 14:22:14 +01:00
Alexander Neumann
b40aa66985 errors: Add method Wrapf 2017-02-11 14:22:04 +01:00
Alexander Neumann
76c06c5f2a Add check for modified index 2017-02-11 14:13:58 +01:00
Alexander Neumann
83538c745a Merge pull request #792 from restic/fix-791 
s3: Increase MaxIdleConnsPerHost
 2017-02-11 11:20:20 +01:00
Alexander Neumann
f266741f40 s3/rest: raise connection limit to 40 2017-02-11 10:40:51 +01:00
Alexander Neumann
4795a5c5d1 Merge pull request #795 from restic/fix-767 
fuse: Add options --allow-other and --allow-root
 2017-02-11 10:38:33 +01:00
Alexander Neumann
c14cb62cd7 fuse: Add options --allow-other and --allow-root 
Closes #767
 2017-02-10 21:58:10 +01:00
Alexander Neumann
b3ec01521e Merge pull request #794 from restic/fix-789 
Use non-formatting functions of errors for strings
 2017-02-10 21:42:59 +01:00
Alexander Neumann
6483df5ee4 Merge pull request #793 from restic/add-host-parameter 
backup: Add `--hostname` parameter
 2017-02-10 21:42:56 +01:00
Alexander Neumann
6275d69a36 fuse: Show link count 2017-02-10 21:16:48 +01:00
Alexander Neumann
6300c8df56 Merge pull request #763 from jgfrm/issue25 
Support hard links
 2017-02-10 20:58:39 +01:00
Jaap Gordijn
366bf4eb0c Support hard links 
Closes #152
 2017-02-10 20:58:19 +01:00
Alexander Neumann
21b358c742 backend tests: Always close reader 2017-02-10 20:49:46 +01:00
Alexander Neumann
7a0303f7ae s3: Make sure to return connection token 2017-02-10 20:49:37 +01:00
Alexander Neumann
64165ea4c8 s3: Hold connection semaphore until Close() 2017-02-10 20:17:52 +01:00
Alexander Neumann
c8fc789393 Use non-formatting functions of errors for strings 
Commands used:

    $ gofmt -w -r 'errors.Fatalf(x) -> errors.Fatal(x)' src
    $ gofmt -w -r 'errors.Errorf(x) -> errors.New(x)' src

Closes #789
 2017-02-10 19:39:49 +01:00
Alexander Neumann
f145e1de0f backup: Add --hostname parameter 2017-02-10 19:37:33 +01:00
Alexander Neumann
36dee7d892 s3: Increase MaxIdleConnsPerHost 2017-02-10 19:25:42 +01:00
Alexander Neumann
073edd914d Merge pull request #783 from opennota/master 
Fix some typos
 2017-02-09 10:43:27 +01:00
opennota
25e459659a Fix some typos 2017-02-09 06:43:10 +07:00
Alexander Neumann
fdebb022e4 Make sure backups read from stdin have a file name 2017-02-08 22:37:02 +01:00
Alexander Neumann
ed1739acbd Merge pull request #779 from restic/benchmark-checker 
checker: Reduce memory usage
 2017-02-07 11:03:49 +01:00
Alexander Neumann
1f81919d4a checker: Reduce memory usage 
benchmark              old bytes     new bytes     delta
    BenchmarkChecker-4     25551348      4288037       -83.22%
 2017-02-06 21:19:27 +01:00
Alexander Neumann
436b5dc20c Add Blob.String() 2017-02-06 19:50:27 +01:00
Alexander Neumann
0c867b21ff Add benchmark for checker 2017-02-06 19:34:40 +01:00
opennota
24acb09a2a Allow --files-from to take a dash for stdin (fixes #769) 2017-02-06 18:43:44 +07:00
Alexander Neumann
4ca134a41c prune: Close backend reader after download 
Closes #777
 2017-02-05 15:40:30 +01:00
Alexander Neumann
aee58a8c17 Add more tests for reading backups from stdin 2017-02-04 16:38:33 +01:00
Alexander Neumann
22f3e21266 Start error message with a lower case character 2017-02-03 17:06:06 +01:00
Alexander Neumann
807fcf07d9 Return an error if password is not set for stdin 
Closes #770
 2017-02-03 15:53:07 +01:00
Alexander Neumann
79e198451c Don't set GOMAXPROCS 
This was a temporary fix for Go earlier than 1.5 to run code on all
avaialble cores. We don't need that any more since we require at least
Go 1.6.
 2017-02-02 10:18:07 +01:00
Alexander Neumann
c7d60279f7 Merge pull request #762 from restic/fix-759 
Add 'index' to list of options for 'cat' command
 2017-01-30 18:26:32 +01:00
Alexander Neumann
7d49c65dd0 Merge pull request #761 from restic/fix-758 
Remove inconsistencies regarding the `cat` command
 2017-01-30 18:25:49 +01:00
Alexander Neumann
abd44ca7c5 Add 'index' to list of options for 'cat' command 2017-01-30 10:53:17 +01:00
Alexander Neumann
44f4ff9d37 Remove 'tree' from help text for 'cat' command 2017-01-30 10:50:52 +01:00
Alexander Neumann
524ce01423 Remove 'cat tree' command 2017-01-30 10:48:50 +01:00
Alexander Neumann
41f59ffc78 Fix 'cat' command for tree blobs 2017-01-30 10:48:21 +01:00
Alexander Neumann
d7f52fd7e5 Allow listing blobs 
Closes #756
 2017-01-30 10:28:17 +01:00
Alexander Neumann
1f9db97d33 Merge pull request #728 from cit/improve-snapshot-output 
Add box-drawing character to the snapshots command
 2017-01-29 11:00:13 +01:00
Alexander Neumann
31f6093513 Fix tests 2017-01-27 12:47:34 +01:00
Alexander Neumann
b9bddeff39 Normalise the backend API 
This makes the following changes, before:

    type backend interface {
        // Test a boolean value whether a File with the name and type exists.
        Test(t FileType, name string) (bool, error)

        // Remove removes a File with type t and name.
        Remove(t FileType, name string) error
    }

After:

    type backend interface {
        // Test a boolean value whether a File with the name and type exists.
        Test(h Handle) (bool, error)

        // Remove removes a File with type t and name.
        Remove(h Handle) error
    }
 2017-01-26 22:02:22 +01:00
Florian Adamsky
54c2f622a4 Add box-drawing character to the snapshots command 
Remove underscore of variable name

Format code with gofmt

Change snapshot output according to the discussion
 2017-01-26 21:54:27 +01:00
Alexander Neumann
8d6fdb7a3e Fix packer manager test on Windows 2017-01-25 17:51:34 +01:00
Alexander Neumann
dceaae33ed PackerManager: Remove unused Finalize() method 2017-01-25 17:12:06 +01:00
Alexander Neumann
e8995b85b8 rest backend: Do not close the reader 
Closes #747
 2017-01-25 17:12:06 +01:00
Alexander Neumann
925a3cfad7 backend: Check that backends do not close the reader 2017-01-25 17:12:03 +01:00
Alexander Neumann
c99a44b122 Fix Random() function 2017-01-25 13:26:16 +01:00
Alexander Neumann
afc593676a fuse: Improve memory usage 
Discard blobs that aren't in use any more. This greatly reduces memory
usage and will probably only trigger on sequential read (e.g. for
restore via fuse).

Closes #480
 2017-01-24 12:38:44 +01:00
Alexander Neumann
0b982d3316 Add restore test with larger files 2017-01-24 11:51:21 +01:00
Alexander Neumann
26da14f315 fuse: improve tests 2017-01-24 11:42:50 +01:00
Alexander Neumann
2de7e03698 repository.LoadBlob: Read correct number of bytes 2017-01-24 11:42:50 +01:00
Alexander Neumann
3d2fbed55f Add test for LoadBlob 2017-01-24 11:42:50 +01:00
Alexander Neumann
0e445ec0f5 checker: Use TestRepository 2017-01-24 11:42:50 +01:00
Alexander Neumann
31055d88a5 Add debug messages 2017-01-24 11:42:42 +01:00
Alexander Neumann
0d125725bc Merge pull request #741 from restic/rework-backend-api 
Rework backend API
 2017-01-23 20:18:31 +01:00
Alexander Neumann
8b09b5b3cd Merge pull request #740 from restic/add-debug-profiles 
Add debug memory/cpu profile options
 2017-01-23 20:18:28 +01:00
Alexander Neumann
0d95507909 Fix test for PackerManager 2017-01-23 19:00:15 +01:00
Alexander Neumann
8e722d8fee Fix saving pack: close temp file before removing 2017-01-23 18:45:15 +01:00
Alexander Neumann
03292d10cc backend: Rename Get() -> Load() 2017-01-23 18:11:10 +01:00
Alexander Neumann
cfc9e8b2fa backends: Remove Load() 2017-01-23 17:54:12 +01:00
Alexander Neumann
f382696ccf repository: Use ReadAt() instead of Load() 2017-01-23 17:54:12 +01:00
Alexander Neumann
e8fcc7e74c repack: Use Get() instead of Load() 
In addition, use a tempfile instead of a buffer.
 2017-01-23 17:54:12 +01:00
Alexander Neumann
2bd9c9247c checker: Remove Load() from test error backend 2017-01-23 17:54:12 +01:00
Alexander Neumann
fc235317fe backend: Use Get instead of Load for ReaderAt 2017-01-23 17:54:12 +01:00
Alexander Neumann
82d9163955 backend: Ensure Reader is closed on error 2017-01-23 17:54:12 +01:00
Alexander Neumann
4a354befe5 Fix checker test 2017-01-23 17:54:12 +01:00
Alexander Neumann
212936eb52 Make backend.LoadAll() similar to ioutil.ReadAll() 2017-01-23 17:54:12 +01:00
Alexander Neumann
05afedd950 Add backend.Get() 2017-01-23 17:54:11 +01:00
Alexander Neumann
a36c01372d Use streaming functions for saving data in repo 2017-01-23 17:54:11 +01:00
Alexander Neumann
9b48da5b4e Change backend Save() function signature 2017-01-23 17:54:11 +01:00
Alexander Neumann
c93f79f0f3 Add hashing package 2017-01-23 17:54:11 +01:00
Alexander Neumann
89a5152f7d Fix Archiver test: Clean up temp files 2017-01-23 17:53:59 +01:00
Alexander Neumann
47bd9cdf2f Add options for creating a memory or CPU profile 2017-01-23 17:52:26 +01:00
Alexander Neumann
668a36a652 Add option for debug pprof service 2017-01-23 17:27:42 +01:00
Alexander Neumann
af1cc0717b Add integration test for forget and prune 2017-01-22 22:23:30 +01:00
Alexander Neumann
5e3365d233 Index: Store pack ID 2017-01-22 22:10:36 +01:00
Alexander Neumann
4f780a01f9 Index: Test pack ID 2017-01-22 22:09:56 +01:00
Alexander Neumann
dc6a832cc3 Correct BenchmarkIndexSave 2017-01-22 09:59:19 +01:00
Alexander Neumann
164ba823e5 Merge pull request #731 from restic/improve-memory-usage 
Improve memory usage
 2017-01-20 15:56:31 +01:00
Alexander Neumann
8dd7fe82ff Add TestIndexSave 2017-01-20 14:46:14 +01:00
Mirko Dziadzka
d47758a540 fix typo 2017-01-18 10:46:04 +01:00
Alexander Neumann
dac18e3bf8 Improve BenchmarkIndexSave 2017-01-17 13:00:59 +01:00
Alexander Neumann
c4f44c7bcb Reduce memory consuption of TestCreateSnapshot 2017-01-17 12:56:20 +01:00
Alexander Neumann
73ad3d418d Index: Remove unneeded allocation 2017-01-17 12:46:41 +01:00
Alexander Neumann
36276c41b2 Add Benchmark for IndexSave 2017-01-17 10:40:58 +01:00
Alexander Neumann
d40f566e41 Index: Use slices instead of maps, reduce data 2017-01-17 10:40:58 +01:00
Alexander Neumann
cd9b526203 Preallocate pack entries list 2017-01-17 10:40:58 +01:00
Alexander Neumann
caabc4ec44 Reduce memory usage while decoding index 2017-01-17 10:40:58 +01:00
Alexander Neumann
73e7a2bea8 Add BenchmarkLoadIndex 2017-01-17 10:40:58 +01:00
Alexander Neumann
e463587bad Add BenchmarkDecodeIndex 2017-01-17 10:40:57 +01:00
Alexander Neumann
e571b6a656 Use the same buffer for decryption 2017-01-17 10:40:57 +01:00
Alexander Neumann
710499cf46 Add benchmark for LoadAndDecrypt 2017-01-17 10:40:57 +01:00
Alexander Neumann
32a5c2c1f6 Add a few functions to calculate Blob buffer len 2017-01-17 10:40:57 +01:00
Alexander Neumann
91dcb958e0 Fix tests 2017-01-17 10:40:57 +01:00
Alexander Neumann
9a5b9253c4 LoadBlob: use buffer as scratch space 
benchmark               old bytes     new bytes     delta
    BenchmarkLoadBlob-4     1010128       2256          -99.78%
 2017-01-17 10:40:57 +01:00
Alexander Neumann
215af5c60a Add LoadBlob benchmark 2017-01-17 10:40:57 +01:00
Alexander Neumann
8734c2466c Fix call to debug.Log() 2017-01-17 10:40:57 +01:00
Alexander Neumann
0556687584 Reduce memory usage for prune 2017-01-17 10:40:57 +01:00
Alexander Neumann
094e80f4a4 Merge pull request #723 from ulziibuyan/ls-latest-cmd 
Added latest keyword in ls command.
 2017-01-15 12:40:58 +01:00
Uzi
c13a0953c8 User interface inconsistency fixed 2017-01-14 11:19:47 +08:00
Uzi
02c02283cf Added latest keyword in ls command. 2017-01-12 19:24:08 +08:00
Matthew Holt
0a34a2d5d8 Consider the environment 2017-01-02 12:21:30 -07:00
Matthew Holt
a394b675b0 CLI options now override env vars 2017-01-02 11:14:22 -07:00
Alexander Neumann
f9501e97a2 Only add entries to indexes inside PackerManager 
This was a nasty bug. Users reported that restic aborts with panic:

    panic: store new item in finalized index

The code calling panic() is in the Store() method of an index and guards
the failure case that an index is to be modified while it has already
been saved in the repo.

What happens here (at least that's what I suspect): PackerManager calls
Current() on a MasterIndex, which yields one index A. Concurrently,
another goroutine calls Repository.SaveFullIndex(), which in turn calls
MasterIndex.FullIndexes(), which (among others) yields the index A. Then
all indexes are marked as final. Then the other goroutine is executed
which adds an entry to the index A, which is now marked as final. Then
the panic occurs.

The commit solves this by removing MasterIndex.Current() and adding a
Store() method that stores the entry in one non-finalized index. This
method uses the same RWMutex as the other methods (e.g. FullIndexes()),
thereby ensuring that the full indexes can only be processed before or
after Store() is called.

Closes #367
 2017-01-02 14:14:51 +01:00
Zlatko Čalušić
2267aca296 Rest server moved to https://github.com/restic/rest-server 2017-01-01 16:22:46 +01:00
Alexander Neumann
80457018d7 Make sure cleanup is executed before exiting 
Closes #708
 2016-12-28 10:53:31 +01:00
Alexander Neumann
b0997d05fb Merge pull request #704 from restic/remove-timestamp 
Remove timestamp from `version` command
 2016-12-19 22:22:43 +01:00
Alexander Neumann
166d1811a1 Remove timestamp from version command 
This enables reproducible builds, for details see
https://reproducible-builds.org/docs/timestamps/
 2016-12-19 21:14:12 +01:00
Sjoerd Simons
e1fc455079 Avoid duplicate backup paths 
Target directories from the from-files argument get added to the command
line args, after which all command line args were appended to the same
variable again causing duplicates. Split the used variables to avoid
this.

Signed-off-by: Sjoerd Simons <sjoerd@luon.net>
 2016-12-18 23:23:57 +01:00
Alexander Neumann
9885aeac3b Make sure SaveFile always returns a node 2016-12-14 18:56:11 +01:00
Alexander Neumann
e6a40af06d Treat changed files as a warning, not an error 2016-12-10 17:14:13 +01:00
Alexander Neumann
3fcbb4ac25 Use new Node if file has changed 
Closes #604
 2016-12-10 16:54:20 +01:00
Alexander Neumann
7d71bad4eb Test if modified files are correctly saved 2016-12-10 16:36:58 +01:00
Alexander Neumann
dbdfed6343 Merge pull request #690 from zcalusic/master 
Even if file changes size during backup, still save it
 2016-12-10 12:36:56 +01:00
Alexander Neumann
5e48c1fadc Merge pull request #688 from restic/fix-686 
Save snapshot after saving all pack files
 2016-12-10 12:33:58 +01:00
Zlatko Čalušić
deb6dd7f72 Even if file changes size during backup, still save it 
Previously such files (typically log files) wouldn't be backed up at
all!

The proper behaviour is to backup what we can, and warn the operator
that file is possibly not complete. But it is a warning, not an error.

Closes #689
 2016-12-10 12:24:45 +01:00
Alexander Neumann
c265673c8e Save snapshot after saving all pack files 
Closes #686
 2016-12-10 11:49:09 +01:00
Jan Niehusmann
c5897e0d62 Update debug message 
Since client.BucketExists was changed to return a separate 'found' value, instead of reporting an error when the bucket doesn't exist, the error code path does no longer imply a call to client.MakeBucket. So the second part of the debug message, "...trying to create the bucket" doesn't apply any more.
Also, changed the name of the return value from 'ok' to 'found', matching the API documentation at https://docs.minio.io/docs/golang-client-api-reference#BucketExists.
 2016-12-05 23:12:30 +01:00
Jan Niehusmann
9267c25aa0 Omit "archived as %v" messages in quiet mode. 2016-12-03 10:28:49 +01:00
JP Mens
5996d671a0 Small typo in dry-run of remove snapshot 2016-12-02 17:33:05 +01:00
Alexander Neumann
e5617b5fd1 Merge pull request #675 from restic/parent-check-hostname 
Use the hostname filter to find a parent snasphot
 2016-11-19 12:42:40 +01:00
Alexander Neumann
2828003d60 Test that existing files and dirs are restored 2016-11-15 21:41:41 +01:00
Alexander Neumann
16cef3b4c6 Use the hostname filter to find a parent snasphot 
Closes #674
 2016-11-15 21:04:51 +01:00
Alexander Neumann
699f39e3cf FindLatestSnapshot: Rename parameter to clarify meaning 2016-11-15 21:03:54 +01:00
Seb Patane
33b6a7381b Don't consider a pre-existing directory in the restore path to be a failure 
* When a directory already exists, CreateDirAt returns an error stating so
  * This means that the restoreMetadata step is skipped, so for directories which already exist no file permissions, owners, groups, etc will be restored on them
* Not returning the error if it's a "directory exists" error means the metadata will get restored
  * It also removes the superfluous "error for ...: mkdir ...: file exists" messages
* This makes the behaviour of directories consistent with that of files (which always have their content & metadata restored, regardless of whether they existed or not)
 2016-11-14 17:53:09 +10:00
Alexander Thaller
b7b03dbd4a Added new flag to backup subcommand that reads the files to backup from a file 2016-11-12 15:45:32 +01:00
Alexander Neumann
b56bde3f61 Remove fadvise 
This commit removes the use of FADV_DONTNEED, which also purges active
cached pages for other processes.
 2016-11-10 22:21:22 +01:00
Zlatko Čalušić
d8f0e7cbd1 Fix REST backend HTTP keepalive 
This is subtle.  A combination od fast client disk (read: SSD) with lots
of files and fast network connection to restic-server would suddenly
start getting lots of "dial tcp: connect: cannot assign requested
address" errors during backup stage.  Further inspection revealed that
client machine was plagued with TCP sockets in TIME_WAIT state.  When
ephemeral port range was finally exhausted, no more sockets could be
opened, so restic would freak out.

To understand the magnitude of this problem, with ~18k ports and default
timeout of 60 seconds, it means more than 300 HTTP connections per
seconds were created and teared down.  Yeah, restic-server is that
fast. :)

As it turns out, this behavior was product of 2 subtle issues:

1) The body of HTTP response wasn't read completely with io.ReadFull()
   at the end of the Load() function.  This deactivated HTTP keepalive,
   so already open connections were not reused, but closed instead, and
   new ones opened for every new request.  io.Copy(ioutil.Discard,
   resp.Body) before resp.Body.Close() remedies this.

2) Even with the above fix, somehow having MaxIdleConnsPerHost at its
   default value of 2 wasn't enough to stop reconnecting.  It is hard to
   understand why this would be so detrimental, it could even be some
   subtle Go runtime bug.  Anyhow, setting this value to match the
   connection limit, as set by connLimit global variable, finally nails
   this ugly bug.

I fixed several other places where the response body wasn't read in
full (or at all).  For example, json.NewDecoder() is also known not to
read the whole body of response.

Unfortunately, this is not over yet. :( The check command is firing up
to 40 simultaneous connections to the restic-server.  Then, once again,
MaxIdleConnsPerHost is too low to support keepalive, and sockets in the
TIME_WAIT state pile up.  But, as this kind of concurrency absolutely
kill the poor disk on the server side, this is a completely different
bug then.
 2016-11-10 09:32:07 +01:00
Alexander Neumann
51322a1055 selectFunc: handle nil 2016-11-05 12:38:33 +01:00
Alexander Neumann
c5bc802ff0 fs.DeviceID(): Return errors when fi is nil 2016-11-05 12:38:17 +01:00
Justin Clift
ecc1f92787 Remove redundant check of error var e 
As per #649
 2016-10-25 18:10:53 +01:00
Alexander Neumann
1dd72693f9 forget: Remove unneeded index loading 2016-10-24 14:01:23 +02:00
Alexander Neumann
fe1013e779 cmds/ls: Format timestamp 2016-10-19 22:11:37 +02:00
David
5f8a6cea6f don't print status info if running in the background 
clean

fix OS issues & format code

fix issues
 2016-10-15 18:12:19 +00:00
Alexander Neumann
250b36eeb1 Improve error message for 'forget' 
$ bin/restic forget /d 7 /w 4 /m 12
    argument "/d" is not a snapshot ID, ignoring
    argument "7" is not a snapshot ID, ignoring
    argument "/w" is not a snapshot ID, ignoring
    argument "4" is not a snapshot ID, ignoring
    argument "/m" is not a snapshot ID, ignoring
    cound not find a snapshot for ID "12", ignoring
 2016-10-10 20:55:02 +02:00
Daniel Örn
ba8d960c8f using backtics instead of doublequotes 2016-10-05 08:26:32 +02:00
Daniel Örn
84421a7c68 structured file with gofmt 2016-10-05 07:30:46 +02:00
Daniel Örn
5c7325f44a Added long paths fix for samba network shares 2016-10-05 07:09:56 +02:00
Christian Kemper
a4261dcc9c Use path.Join to create the s3 object name within the bucket. 
path.Join already automatically skips empty path segments when
joining, so this simplifies the s3Path code.
 2016-10-02 16:56:07 -07:00
Alexander Neumann
22f5fc5739 Improve help text for slice options 2016-09-29 20:39:55 +02:00
Alexander Neumann
e994cacbfe Fix short-hand option clash 2016-09-29 20:37:45 +02:00
Alexander Neumann
feed54caef Remove timing, simplify function matching 2016-09-28 20:10:40 +02:00
Alexander Neumann
4eddcb344e Update calls to debug.Log() 2016-09-28 19:56:03 +02:00
Alexander Neumann
2ae06a7a01 Rework debug log function 2016-09-28 19:56:03 +02:00
Alexander Neumann
25945718a1 Fix recursive call to debug.Log 2016-09-28 19:56:03 +02:00
Alexander Neumann
3601c39177 Add comments 2016-09-27 20:22:01 +02:00
Alexander Neumann
565d72ef36 Use cobra for all commands 2016-09-27 19:53:03 +02:00
Alexander Neumann
a257a613d7 Fix debug log 2016-09-27 19:52:48 +02:00
Alexander Neumann
0a752b9fab test helpers: Always print stack trace 2016-09-27 19:50:26 +02:00
Alexander Neumann
04d6b5da2f Remove more unused bits 2016-09-21 20:45:18 +02:00
Alexander Neumann
1dfd3b8aa3 Remove unused bits and pieces 
Reported by https://github.com/dominikh/go-unused
 2016-09-21 20:22:32 +02:00
Alexander Neumann
0a9cbd47c7 Merge pull request #626 from rfjakob/master 
Add "-x", "--one-file-system" option
 2016-09-18 20:03:58 +02:00
Jakob Unterwurzacher
53701891a1 Add "-x", "--one-file-system" option 
Equivalent to rsync's "-x" option.

Notes to the naming:

"--exclude-other-filesystems"
is used by Duplicity,

"--one-file-system"
is used rsync and tar.

This latter should be more familiar to the user.
 2016-09-18 18:52:30 +02:00
Alexander Neumann
68b462d057 fuse: Add test for same timestamps 2016-09-18 18:30:25 +02:00
Alexander Neumann
649f789190 fuse: Fix test for timestamps with same second 2016-09-18 18:13:39 +02:00
Alexander Neumann
5494c1858e fuse: correctly handle snapshots 
The fuse code kept adding snapshots to the top-level dir "snapshots". In
addition, snapshots with the same timestamp (same second) were not added
correctly, they will now be suffixed by an incrementing counter, e.g.:

    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:44+02:00
    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:48+02:00
    dr-xr-xr-x 1 fd0 users 0 Sep 18 15:01 2016-09-18T15:01:48+02:00-1

Closes #624
 2016-09-18 15:04:39 +02:00
Alexander Neumann
b090c73bd4 Remove wrapper functions in errors package 
This way, our own errors package does not appear in the stack traces.
 2016-09-18 13:28:59 +02:00
Alexander Neumann
2b9a408ccc Return a fatal for location.Parse 2016-09-18 13:28:41 +02:00
Alexander Neumann
83c35bd6b5 Do not print stack trace when open repo failed 
Closes #622
 2016-09-18 13:24:46 +02:00
Alexander Neumann
a9af896ddd Add verbose error when marshalling a node fails 
This code is introduced to watch for issue #529, in which two users
describe that restic failed encoding a time in a node to JSON with the
error message:

    panic: json: error calling MarshalJSON for type *restic.Node: json: error calling MarshalJSON for type time.Time: Time.MarshalJSON: year outside of range [0,9999]

The error message now is:

    panic: Marshal: json: error calling MarshalJSON for type *restic.Node: node /home/fd0/shared/work/restic/restic/.git/hooks/applypatch-msg.sample has invalid ModTime year -1: -0001-01-02 03:04:05.000000006 +0053 LMT
 2016-09-17 10:43:04 +02:00
Alexander Neumann
64fe9ec048 Update github.com/jessevdk/go-flags 2016-09-15 22:29:49 +02:00
Alexander Neumann
d4a2d70089 Retry umount for integration tests 2016-09-15 21:37:50 +02:00
Alexander Neumann
4ffca0f4b4 Improve integration tests for fuse 2016-09-15 21:17:20 +02:00
Alexander Neumann
a0f3e94655 fuse: handle duplicate timestamps for snapshots 
This closes #606, which fails because several snapshots are created with
exactly the same timestamp, and the code checks that for each snapshot
there is a dir in the fuse mount. This fails for colliding timestamps,
so we now add a suffix "-1", "-2" etc for each duplicate timestamp.
 2016-09-15 21:15:49 +02:00
Alexander Neumann
6485a6cdc0 Simplify mount logic 2016-09-15 19:59:07 +02:00
Alexander Neumann
3975d76f23 Correct filenames for expire policy tests 2016-09-13 21:20:41 +02:00
Alexander Neumann
bf6602bc1b Update golden file 2016-09-13 21:19:57 +02:00
Alexander Neumann
828267aaa3 Fix status for stdin archiver 2016-09-13 21:01:29 +02:00
Alexander Neumann
a77c615909 Fix 'forget' command with tags 2016-09-13 20:56:18 +02:00
Alexander Neumann
cfdf4c92f7 Add --keep-tag to forget command 2016-09-13 20:37:11 +02:00
Alexander Neumann
0f9fb37c78 Add tags to forget command 2016-09-13 20:20:55 +02:00
Alexander Neumann
673bce936e Add tags to 'backup' and 'snapshots' commands 2016-09-13 20:20:52 +02:00
Alexander Neumann
1f83635267 Add tags to snapshots and filter 2016-09-13 20:12:55 +02:00
Alexander Neumann
ceb4a3ecc0 Merge pull request #613 from restic/read-password-from-file 
Read password from file
 2016-09-12 20:37:08 +02:00
Alexander Neumann
223dc78acb Improve statistics for prune 
Sample:

    counting files in repo
    building new index for repo
    [0:00] 100.00%  22 / 22 packs
    repository contains 22 packs (1377 blobs) with 90.610 MiB bytes
    processed 1377 blobs: 0 duplicate blobs, 0B duplicate
    load all snapshots
    find data that is still in use for 1 snapshots
    [0:00] 100.00%  1 / 1 snapshots
    found 409 of 1377 data blobs still in use, removing 968 blobs
    will delete 10 packs and rewrite 10 packs, this frees 64.232 MiB
    creating new index
    [0:00] 100.00%  7 / 7 packs
    saved new index as df467c6e
    done

Closes #581
 2016-09-12 14:26:47 +02:00
Alexander Neumann
65afeba19a Add option to read the password from a file 2016-09-12 14:09:22 +02:00
Alexander Neumann
e443454c4b Add OS and Arch to 'version' output 2016-09-04 15:46:50 +02:00
Alexander Neumann
b628bcee27 Remove redundant ParseID 2016-09-04 14:38:18 +02:00
Alexander Neumann
dfc0cbf3a8 Use one test password 2016-09-04 14:30:14 +02:00
Alexander Neumann
512a92895f Rename WithTestEnvironment -> Env 2016-09-04 14:29:04 +02:00
Alexander Neumann
6ab425f130 Remove SetupRepo 2016-09-04 13:24:51 +02:00
Alexander Neumann
f5b9ee53a3 Fix mock.Repository 2016-09-04 13:18:25 +02:00
Alexander Neumann
ea073f58cf Correct comment 2016-09-04 13:08:09 +02:00
Alexander Neumann
bef5c4acb8 Add mock.Repository, Rework SetupRepo 2016-09-04 12:52:43 +02:00
Alexander Neumann
b5b3c0eaf8 Add repository.SaveTree 2016-09-03 21:10:25 +02:00