octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-12-25 12:09:59 +00:00
Code Issues Releases Wiki Activity
1,231 Commits 7 Branches 33 Tags 66 MiB
74ce027924
Commit Graph

185 Commits

Author SHA1 Message Date
Alexander Neumann
5cdcc99eba Use array instead of hash for backend.ID 
Since backend.ID is always a slice of constant length, use an array
instead of a slice. Mostly, arrays behave as slices, except that an
array cannot be nil, so use `*backend.ID` insteaf of `backend.ID` in
places where the absence of an ID is possible (e.g. for the Subtree of a
Node, which may not present when the node is a file node).

This change allows to directly use backend.ID as the the key for a map,
so that arbitrary data structures (e.g. a Set implemented as a
map[backend.ID]struct{}) can easily be formed.
 2015-07-25 18:01:57 +02:00
Alexander Neumann
1c14be8993 backend: Remove IDSet 
This can easily be replaced by a map[{32}byte]struct{}, and this is much
faster.
 2015-07-11 16:12:50 +02:00
Alexander Neumann
281eea9c05 sftp: Use own process group for ssh subprocess 2015-07-05 11:07:10 +02:00
Emil Hessman
52b6cfbbfa backend: fix doc typos 2015-06-28 09:44:06 +02:00
Alexander Neumann
6c2d6bfd11 Merge pull request #184 from howeyc/backend-s3 
Add Amazon S3 backend.
 2015-06-23 20:14:47 +02:00
Alexander Neumann
9853fbcf48 Remove more flags from tests 2015-06-21 17:40:21 +02:00
Alexander Neumann
030f08a410 Remove flags from tests 2015-06-21 17:40:21 +02:00
Alexander Neumann
12677b4f8a Use flag instead of build tag to run integration tests 2015-06-21 17:40:21 +02:00
Alexander Neumann
9b7db4df24 travis: Add new integration tests 2015-06-21 17:40:21 +02:00
Chris Howey
cf0531a2b1 Pass buffer instead of byte slice 
s3 library creates internal buffer pass in byte slice
 2015-06-14 09:05:20 -05:00
Chris Howey
12596c6841 Fix test to use new type name on backend 2015-06-14 08:43:47 -05:00
Chris Howey
57ee1db8aa Rename backend type to S3Backend 2015-06-14 08:28:50 -05:00
Chris Howey
e963faf846 Place all restic data under a key prefix. 
Deleting a restic repository now only deletes restic keys, leaving the
rest of the bucket in tact.
 2015-06-14 08:28:50 -05:00
Chris Howey
d30a95e8ad Return bucket readers instead of buffering a key on read. 2015-06-14 08:28:50 -05:00
Chris Howey
aaae7f33d3 Have number of connections limited by channel 
Removes previous limit of 1 connection
 2015-06-14 08:28:50 -05:00
Chris Howey
2350419f59 Modify test suite to use s3 test server. 
Fix s3 backend to return error on attempting to write to already
existing key.
 2015-06-14 08:28:50 -05:00
Chris Howey
4aa75994b0 Add ability to open bucket using S3 object for testing purposes. 2015-06-14 08:28:50 -05:00
Chris Howey
d1554d1133 Add backend test for s3 2015-06-14 08:28:50 -05:00
Chris Howey
bfe221e71c Handle long list responses from s3. 
Also change naming of backend struct.
 2015-06-14 08:28:50 -05:00
Chris Howey
47186806cf Add Amazon S3 backend. 2015-06-14 08:28:50 -05:00
Alexander Neumann
6e38a8a033 Move FindSnapshot, make Repository.List() return IDs 2015-05-17 20:58:22 +02:00
Alexander Neumann
4e5094b7c9 Fix loading config on sftp backend 2015-05-14 18:08:45 +02:00
Alexander Neumann
9ac6a1d232 Add minimal documentation for backend/{local,sftp} 2015-05-09 12:54:45 +02:00
Alexander Neumann
e4e502959f Clarify documentation on backend 2015-05-09 12:53:04 +02:00
Alexander Neumann
c57c4f0b8f Merge branch 'add_config' 2015-05-04 22:13:59 +02:00
Alexander Neumann
5399358272 Fix spelling errors in comments 2015-05-04 20:39:45 +02:00
Alexander Neumann
250080db9c sftp: Annotate a few errors 2015-05-03 21:06:20 +02:00
Alexander Neumann
2fb1783885 backend/server: remove id and version from backend 2015-05-03 18:56:16 +02:00
Alexander Neumann
6fe38b0cd8 fix comments to have the standard form 2015-05-02 16:22:43 +02:00
Alexander Neumann
1f94ac724f backend: rename LimitReader to LimitReadCloser 2015-05-02 16:13:56 +02:00
Alexander Neumann
98dc811536 Add nil-receiver awareness to all Reader/Writer 2015-05-01 17:32:42 +02:00
Alexander Neumann
ec108fb708 Remove backend.Tree 2015-04-29 23:27:36 +02:00
Alexander Neumann
60a0fe8349 Major restructure, bundle blobs 
This commit also breaks the repository format.
 2015-04-29 23:27:32 +02:00
Alexander Neumann
87ce38b13a Add type 'Index' to backend 2015-04-29 23:27:08 +02:00
Alexander Neumann
edfd86624c Fix "cleanup" flag 2015-04-26 15:40:25 +02:00
Florian Weingarten
ebae478631 Prefer mixedCaps over underscore 2015-04-24 19:39:32 -04:00
Alexander Neumann
4f4f3c421a Use package "restic/test" 2015-04-09 21:15:48 +02:00
Alexander Neumann
24b14e21cc Remove pools for nodes and IDs 
This removes the allocation pools for nodes and IDs. I feel they aren't
really needed. Benchmarks:

    benchmark                         old ns/op      new ns/op      delta
    BenchmarkChunkEncrypt             197890867      198616293      +0.37%
    BenchmarkChunkEncryptParallel     196127004      198819818      +1.37%
    BenchmarkArchiveDirectory         1098848419     1087237723     -1.06%
    BenchmarkPreload                  30464455       29910239       -1.82%
    BenchmarkLoadTree                 3265092        3088543        -5.41%
    BenchmarkEncryptWriter            37213511       37134683       -0.21%
    BenchmarkEncrypt                  36037879       36166546       +0.36%
    BenchmarkDecryptReader            38165659       38556734       +1.02%
    BenchmarkEncryptDecryptReader     77027044       77194987       +0.22%
    BenchmarkDecrypt                  36017602       35937888       -0.22%
    BenchmarkSaveJSON                 47906          50270          +4.93%
    BenchmarkSaveFrom                 49775973       50520969       +1.50%
    BenchmarkLoadJSONID               105290245      107281849      +1.89%
    BenchmarkChunkerWithSHA256        151501430      148264078      -2.14%
    BenchmarkChunkerWithMD5           93606346       94036392       +0.46%
    BenchmarkChunker                  74285431       75933882       +2.22%
    BenchmarkPipelineWalker           387689         346467         -10.63%

    benchmark                         old MB/s     new MB/s     speedup
    BenchmarkChunkEncrypt             52.99        52.79        1.00x
    BenchmarkChunkEncryptParallel     53.46        52.74        0.99x
    BenchmarkEncryptWriter            225.42       225.90       1.00x
    BenchmarkEncrypt                  232.77       231.94       1.00x
    BenchmarkDecryptReader            219.79       217.57       0.99x
    BenchmarkEncryptDecryptReader     108.90       108.67       1.00x
    BenchmarkDecrypt                  232.90       233.42       1.00x
    BenchmarkSaveFrom                 84.26        83.02        0.99x
    BenchmarkChunkerWithSHA256        69.21        70.72        1.02x
    BenchmarkChunkerWithMD5           112.02       111.51       1.00x
    BenchmarkChunker                  141.15       138.09       0.98x

    benchmark                         old allocs     new allocs     delta
    BenchmarkChunkEncrypt             110            110            +0.00%
    BenchmarkChunkEncryptParallel     100            100            +0.00%
    BenchmarkArchiveDirectory         475591         476635         +0.22%
    BenchmarkPreload                  28059          24182          -13.82%
    BenchmarkLoadTree                 3124           2889           -7.52%
    BenchmarkEncryptWriter            19             19             +0.00%
    BenchmarkEncrypt                  13             13             +0.00%
    BenchmarkDecryptReader            16             15             -6.25%
    BenchmarkEncryptDecryptReader     39             39             +0.00%
    BenchmarkDecrypt                  11             11             +0.00%
    BenchmarkSaveJSON                 74             74             +0.00%
    BenchmarkSaveFrom                 109            112            +2.75%
    BenchmarkLoadJSONID               103630         97849          -5.58%
    BenchmarkChunkerWithSHA256        13             13             +0.00%
    BenchmarkChunkerWithMD5           12             12             +0.00%
    BenchmarkChunker                  6              6              +0.00%
    BenchmarkPipelineWalker           212            165            -22.17%

    benchmark                         old bytes     new bytes     delta
    BenchmarkChunkEncrypt             64697         64697         +0.00%
    BenchmarkChunkEncryptParallel     64681         64681         +0.00%
    BenchmarkArchiveDirectory         193385504     193790864     +0.21%
    BenchmarkPreload                  4064701       3942000       -3.02%
    BenchmarkLoadTree                 344954        325396        -5.67%
    BenchmarkEncryptWriter            12793         12793         +0.00%
    BenchmarkEncrypt                  1950          1950          +0.00%
    BenchmarkDecryptReader            3120          2774          -11.09%
    BenchmarkEncryptDecryptReader     1528036       1528036       +0.00%
    BenchmarkDecrypt                  1919          1919          +0.00%
    BenchmarkSaveJSON                 5524          5524          +0.00%
    BenchmarkSaveFrom                 31353         40804         +30.14%
    BenchmarkLoadJSONID               12872020      16010968      +24.39%
    BenchmarkChunkerWithSHA256        26821         26821         +0.00%
    BenchmarkChunkerWithMD5           13554         13554         +0.00%
    BenchmarkChunker                  13458         13458         +0.00%
    BenchmarkPipelineWalker           58584         55560         -5.16%
 2015-03-28 16:51:37 +01:00
Alexander Neumann
702b6cfc1e Make ID.Str() handle nil ids correctly 2015-03-28 16:46:21 +01:00
Alexander Neumann
2abda75404 Refactor mockBackend for test 2015-03-28 16:42:35 +01:00
Alexander Neumann
223108c6ae Remove unused vars and const 2015-03-28 15:28:08 +01:00
Alexander Neumann
5e69788eac Refactor backends 2015-03-28 14:36:50 +01:00
Alexander Neumann
12c8522b33 Fix file mode for data blobs 2015-03-14 20:19:52 +01:00
Alexander Neumann
23ac4f84dd Set file-mode in repository read-only (Closes #48) 2015-03-14 14:30:26 +01:00
Alexander Neumann
f69a39cff5 Add ID to repository 
This allows identifying a repository regardless if it's accessed over
SFTP or locally. Introduced for having a per-repository cache.
 2015-03-14 12:06:27 +01:00
Alexander Neumann
b6f25aa690 Remove Each(), add basic stats 2015-02-17 23:11:56 +01:00
Alexander Neumann
c631b5c738 Add Len() functions for IDSet and Map 2015-02-17 22:39:16 +01:00
Alexander Neumann
06ed5c12b8 Remove Compress() and Uncompress() from backend 2015-02-16 00:30:51 +01:00
Alexander Neumann
fb2cd7b485 HashingWriter: Track size 2015-02-16 00:24:43 +01:00
Alexander Neumann
b816192d1c backend: Throw error when Close() is called twice 2015-02-16 00:24:02 +01:00
Alexander Neumann
907d401e32 backend: use HashingWriter 2015-02-15 23:46:21 +01:00
Alexander Neumann
5c724b92b8 Add HashingWriter and HashAppendWriter 2015-02-15 23:21:35 +01:00
Alexander Neumann
72c951a6ec Simply backend interface 
Rename CreateBlob() method to Create(), remove old Create* methods
 2015-02-15 17:26:20 +01:00
Alexander Neumann
35636a9d92 Introduce CreateBlob() method for backend 2015-02-15 17:09:49 +01:00
Alexander Neumann
92e2647505 Add tests for sftp backend, split out id tests 2015-02-11 20:17:55 +01:00
Alexander Neumann
6c68150e45 Add streaming functions to mockBackend 2015-02-11 20:01:01 +01:00
Alexander Neumann
89bf88df7a Add streaming methods to interface 2015-02-11 19:55:23 +01:00
Alexander Neumann
fdbe2f0c2d Implement streaming methods for sftp backend 2015-02-11 19:55:11 +01:00
Alexander Neumann
0a45ec1d3a Implement streaming functions for local backend 2015-02-11 19:50:29 +01:00
Alexander Neumann
64f7b4e5e9 Rename HashReader -> HashAppendReader, add HashingReader 2015-02-11 19:37:48 +01:00
Alexander Neumann
842dea173c Generalize hash in backend 2015-02-11 19:25:43 +01:00
Alexander Neumann
5f0cce8546 Move HashReader to backend 2015-02-11 19:20:45 +01:00
Alexander Neumann
203a911de9 Refactor repository structure 
Merge Map data type into Tree.
 2015-01-14 16:34:30 +01:00
Alexander Neumann
cc147c002e Introduce type Server 2014-12-21 17:37:29 +01:00
Alexander Neumann
661c1e9aa1 Restructure backend 2014-12-21 17:29:58 +01:00
Alexander Neumann
e2fea0d088 Rename khepri -> restic 2014-12-05 21:45:49 +01:00
Alexander Neumann
4b6c7f1584 Reduce dir depth for trees and data by one 2014-12-01 00:52:12 +01:00
Alexander Neumann
3325fa07ea Check timestamps for files 2014-11-30 22:34:21 +01:00
Alexander Neumann
6d56d7d4c6 Optimize repository layout 
Use subdirectoriers for Data and Tree objects, e.g. store blob
`aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f`
at
```
ac/c0/aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f
```

Issue #58
 2014-11-30 16:24:09 +01:00
Alexander Neumann
08690b99bf Merge branch 'list-snapshots' 2014-11-24 22:18:16 +01:00
Alexander Neumann
4ab3d51996 Add more checks befor decrypting data 2014-11-24 22:11:09 +01:00
Alexander Neumann
bb62218743 Add functions to compute unique prefix length 2014-11-24 22:09:22 +01:00
Alexander Neumann
2c5d07a571 List snapshots, accept snapshot id prefix 
Example:

    $ ./khepri snapshots
    ID        Date                 Source      Directory
    --------------------------------------------------------------------------------
    fa31d65b  2014-11-24 19:45:11  kasimir     /home/user/testdata
    20bdc140  2014-11-24 20:00:47  kasimir     /home/user/testdata
    326cb59d  2014-11-24 20:01:40  kasimir     /home/user/testdata
    20ff988b  2014-11-24 20:35:35  kasimir     /home/user
 2014-11-24 21:12:32 +01:00
Alexander Neumann
26cd6c5372 Add maps directory to sftp backend 2014-11-23 23:37:40 +01:00
Alexander Neumann
923a9fa49b Check maps directory when opening repository 2014-11-23 23:34:34 +01:00
Alexander Neumann
33ce4e33aa Store maps in new subdir "maps" 2014-11-23 22:26:01 +01:00
Alexander Neumann
f95788ed90 Limit memory usage, add several sync.Pool 2014-11-23 21:26:20 +01:00
Alexander Neumann
d1e4431514 Refactor StorageMap to BlobList 2014-11-22 21:05:18 +01:00
Alexander Neumann
804cebde67 Rename backend subdir 'blobs' to 'data' 2014-11-16 13:22:19 +01:00
Alexander Neumann
f37d0bf45f Add newlink to version file 2014-11-15 15:30:54 +01:00
Alexander Neumann
24b4d58bdf Remove double check 2014-11-15 15:25:30 +01:00
Alexander Neumann
9b75f2cab0 Better error handling and annotation 2014-10-07 23:23:39 +02:00
Alexander Neumann
f848afed27 Add SFTP backend 2014-10-04 19:20:15 +02:00
Alexander Neumann
bfd99a9be6 Add 'version' file to local backend 2014-10-04 16:49:39 +02:00
Alexander Neumann
30ab03b7b7 Add decrypt, refactor 2014-10-03 21:44:55 +02:00