octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-05-30 07:30:50 +00:00
Code Issues Releases Wiki Activity
1,246 Commits 7 Branches 33 Tags 65 MiB
9175f0b6af
Commit Graph

109 Commits

Author SHA1 Message Date
Alexander Neumann
4f1f03cdb9 Move testing for known blobs to Archiver 
This removes the list of in-flight blobs from the master index and
instead keeps a list of "known" blobs in the Archiver. "known" here
means: either already processed, or included in an index. This property
is tested atomically, when the blob is not in the list of "known" blobs,
it is added to the list and the caller is responsible to make this
happen (i.e. save the blob).
 2016-02-01 23:50:56 +01:00
Alexander Neumann
4484a3ea0d archiver: ignore dir nodes with errors 2015-11-07 11:42:28 +01:00
Alexander Neumann
1569176e48 pipe: propagate errors properly 2015-11-06 19:41:57 +01:00
Alexander Neumann
6fa4be5af2 Regularly save intermediate indexes 2015-10-12 23:59:17 +02:00
Alexander Neumann
86fcd170f6 Add and use MasterIndex 2015-10-12 22:34:12 +02:00
Alexander Neumann
f188cf81dc Add more panic() calls for invalid conditions 2015-10-11 20:45:42 +02:00
Alexander Neumann
5cdcc99eba Use array instead of hash for backend.ID 
Since backend.ID is always a slice of constant length, use an array
instead of a slice. Mostly, arrays behave as slices, except that an
array cannot be nil, so use `*backend.ID` insteaf of `backend.ID` in
places where the absence of an ID is possible (e.g. for the Subtree of a
Node, which may not present when the node is a file node).

This change allows to directly use backend.ID as the the key for a map,
so that arbitrary data structures (e.g. a Set implemented as a
map[backend.ID]struct{}) can easily be formed.
 2015-07-25 18:01:57 +02:00
Alexander Neumann
cc34401152 Record exclude patterns in snapshot 
This adds the exclude patterns used to create a backup in the snapshot,
so we can later compute statistics (like git does) on the data
structure, e.g. added/removed files etc. For that, we need the exclude
pattern.
 2015-07-22 22:43:32 +02:00
Alexander Neumann
7fd52f9f57 Add exclude filter to archiver and 'backup' command 2015-07-20 19:21:01 +02:00
Alexander Neumann
0762f7daba Merge pull request #219 from restic/extract_chunker 
Extract chunker
 2015-07-08 23:40:43 +02:00
Florian Weingarten
36a62cf77d Extract chunker 2015-07-08 16:58:23 -04:00
Florian Weingarten
82139912e8 Count non-regular files for progress as well 2015-07-08 08:56:47 -04:00
Alexander Neumann
5ae04b6834 Add last integration tests, remove testsuite 2015-06-21 17:40:22 +02:00
Alexander Neumann
eb6dfcf58c Remove Repository.Test() 2015-05-17 20:58:22 +02:00
Alexander Neumann
95536e8a21 Rename 'Repo' -> 'Repository' 2015-05-09 23:59:58 +02:00
Alexander Neumann
232c472836 Move package 'repo' to package 'repository' 2015-05-09 23:52:03 +02:00
Alexander Neumann
ae21938f3e Rename 'Repository' -> Repo 2015-05-09 17:41:28 +02:00
Alexander Neumann
5fc1583acc Fix comments 2015-05-09 13:47:23 +02:00
Alexander Neumann
d9b5832034 Rename variables 2015-05-09 13:47:21 +02:00
Alexander Neumann
87ebf12945 Rename 'Server' to 'Repository' 2015-05-09 13:25:52 +02:00
Alexander Neumann
8be9e95d20 Rename package 'server' to 'repo' 2015-05-09 13:24:15 +02:00
Alexander Neumann
ea22b2dfb1 chunker: move buffer pool to chunker package 2015-05-05 00:51:47 +02:00
Alexander Neumann
64a290c8db chunker: Add buffer to New() 2015-05-04 23:25:07 +02:00
Florian Weingarten
282c4379db Remove chunker.Reset() and chunker pool 2015-05-04 16:49:49 -04:00
Alexander Neumann
d4bf5bb279 server: Add config 2015-05-03 18:56:20 +02:00
Alexander Neumann
ba1125df10 Cleanup commets with feedback from @fw42 2015-05-02 17:01:31 +02:00
Alexander Neumann
c9664e9eac Remove unused function saveTree() 2015-05-02 16:55:29 +02:00
Alexander Neumann
d66780a45d Remove unused global variable 2015-05-02 16:39:24 +02:00
Alexander Neumann
71f3fb2dac Rename variables in copyJobs() 2015-05-02 16:39:11 +02:00
Alexander Neumann
ff4dd1f1fb Add more comments 2015-05-02 15:51:40 +02:00
Alexander Neumann
819b6da762 restorer: unexport archive pipe 2015-05-02 15:31:31 +02:00
Alexander Neumann
4a1cb65645 Merge branch 'replace_arrar' 2015-05-02 14:34:33 +02:00
Florian Weingarten
e19c87fa7d Break Archiver.SaveFile() method down into smaller ones 2015-05-02 01:05:49 -04:00
Alexander Neumann
97619e743c replace github.com/juju/arrar by /errors 2015-05-02 01:29:54 +02:00
Alexander Neumann
0662c14fee Move chunkerBufSize to more appropiate location 2015-05-01 17:40:50 +02:00
Florian Weingarten
0d9360a815 more cleanup 2015-04-29 22:33:34 -04:00
Alexander Neumann
60a0fe8349 Major restructure, bundle blobs 
This commit also breaks the repository format.
 2015-04-29 23:27:32 +02:00
Alexander Neumann
d19b23d4f1 Move Server and Key to new sub-package 2015-04-26 14:46:15 +02:00
Alexander Neumann
434c79354e Merge pull request #143 from fw42/count_errors 
Count errors separately and not as files that are "done"
 2015-04-26 12:05:09 +02:00
Florian Weingarten
61b1a5b1de Count errors separately and not as files that are "done" 2015-04-26 01:54:35 +00:00
Florian Weingarten
c947623707 Fix progress off-by-1 2015-04-25 15:40:42 -04:00
Florian Weingarten
262e5a8b4a O_NOATIME on Linux 2015-04-25 00:45:20 +00:00
Alexander Neumann
b2307cafa2 Seed chunker with random per-repository polynomial 2015-04-06 00:22:19 +02:00
Alexander Neumann
24b14e21cc Remove pools for nodes and IDs 
This removes the allocation pools for nodes and IDs. I feel they aren't
really needed. Benchmarks:

    benchmark                         old ns/op      new ns/op      delta
    BenchmarkChunkEncrypt             197890867      198616293      +0.37%
    BenchmarkChunkEncryptParallel     196127004      198819818      +1.37%
    BenchmarkArchiveDirectory         1098848419     1087237723     -1.06%
    BenchmarkPreload                  30464455       29910239       -1.82%
    BenchmarkLoadTree                 3265092        3088543        -5.41%
    BenchmarkEncryptWriter            37213511       37134683       -0.21%
    BenchmarkEncrypt                  36037879       36166546       +0.36%
    BenchmarkDecryptReader            38165659       38556734       +1.02%
    BenchmarkEncryptDecryptReader     77027044       77194987       +0.22%
    BenchmarkDecrypt                  36017602       35937888       -0.22%
    BenchmarkSaveJSON                 47906          50270          +4.93%
    BenchmarkSaveFrom                 49775973       50520969       +1.50%
    BenchmarkLoadJSONID               105290245      107281849      +1.89%
    BenchmarkChunkerWithSHA256        151501430      148264078      -2.14%
    BenchmarkChunkerWithMD5           93606346       94036392       +0.46%
    BenchmarkChunker                  74285431       75933882       +2.22%
    BenchmarkPipelineWalker           387689         346467         -10.63%

    benchmark                         old MB/s     new MB/s     speedup
    BenchmarkChunkEncrypt             52.99        52.79        1.00x
    BenchmarkChunkEncryptParallel     53.46        52.74        0.99x
    BenchmarkEncryptWriter            225.42       225.90       1.00x
    BenchmarkEncrypt                  232.77       231.94       1.00x
    BenchmarkDecryptReader            219.79       217.57       0.99x
    BenchmarkEncryptDecryptReader     108.90       108.67       1.00x
    BenchmarkDecrypt                  232.90       233.42       1.00x
    BenchmarkSaveFrom                 84.26        83.02        0.99x
    BenchmarkChunkerWithSHA256        69.21        70.72        1.02x
    BenchmarkChunkerWithMD5           112.02       111.51       1.00x
    BenchmarkChunker                  141.15       138.09       0.98x

    benchmark                         old allocs     new allocs     delta
    BenchmarkChunkEncrypt             110            110            +0.00%
    BenchmarkChunkEncryptParallel     100            100            +0.00%
    BenchmarkArchiveDirectory         475591         476635         +0.22%
    BenchmarkPreload                  28059          24182          -13.82%
    BenchmarkLoadTree                 3124           2889           -7.52%
    BenchmarkEncryptWriter            19             19             +0.00%
    BenchmarkEncrypt                  13             13             +0.00%
    BenchmarkDecryptReader            16             15             -6.25%
    BenchmarkEncryptDecryptReader     39             39             +0.00%
    BenchmarkDecrypt                  11             11             +0.00%
    BenchmarkSaveJSON                 74             74             +0.00%
    BenchmarkSaveFrom                 109            112            +2.75%
    BenchmarkLoadJSONID               103630         97849          -5.58%
    BenchmarkChunkerWithSHA256        13             13             +0.00%
    BenchmarkChunkerWithMD5           12             12             +0.00%
    BenchmarkChunker                  6              6              +0.00%
    BenchmarkPipelineWalker           212            165            -22.17%

    benchmark                         old bytes     new bytes     delta
    BenchmarkChunkEncrypt             64697         64697         +0.00%
    BenchmarkChunkEncryptParallel     64681         64681         +0.00%
    BenchmarkArchiveDirectory         193385504     193790864     +0.21%
    BenchmarkPreload                  4064701       3942000       -3.02%
    BenchmarkLoadTree                 344954        325396        -5.67%
    BenchmarkEncryptWriter            12793         12793         +0.00%
    BenchmarkEncrypt                  1950          1950          +0.00%
    BenchmarkDecryptReader            3120          2774          -11.09%
    BenchmarkEncryptDecryptReader     1528036       1528036       +0.00%
    BenchmarkDecrypt                  1919          1919          +0.00%
    BenchmarkSaveJSON                 5524          5524          +0.00%
    BenchmarkSaveFrom                 31353         40804         +30.14%
    BenchmarkLoadJSONID               12872020      16010968      +24.39%
    BenchmarkChunkerWithSHA256        26821         26821         +0.00%
    BenchmarkChunkerWithMD5           13554         13554         +0.00%
    BenchmarkChunker                  13458         13458         +0.00%
    BenchmarkPipelineWalker           58584         55560         -5.16%
 2015-03-28 16:51:37 +01:00
Alexander Neumann
f157f775da Add checks for Server.Load(), use Blob for load 2015-03-28 15:07:08 +01:00
Alexander Neumann
5e69788eac Refactor backends 2015-03-28 14:36:50 +01:00
Alexander Neumann
cfa2229bc0 Refactor cache refresh for blobs, add progress 2015-03-22 15:09:50 +01:00
Alexander Neumann
aca0692ee6 Add more error reporting 2015-03-21 14:43:33 +01:00
Alexander Neumann
71a57537ef Report errors, ignore files and continue. Closes #108 2015-03-15 15:48:05 +01:00
Alexander Neumann
3d768e39ea Add more error reporting 2015-03-15 12:20:30 +01:00