octoleo/restic
2
2
Fork 0
mirror of https://github.com/octoleo/restic.git synced 2024-11-02 11:46:36 +00:00
Code Issues Releases Wiki Activity
2,572 Commits 7 Branches 33 Tags 66 MiB
c4c731bd9a
Commit Graph

825 Commits

Author SHA1 Message Date
Alexander Neumann
05365706c0 backend/tests: Correct error message and delayed remove 2017-06-15 20:05:35 +02:00
Alexander Neumann
bbca31b661 test/s3: Retry connection to Minio server 2017-06-15 19:51:55 +02:00
Alexander Neumann
eb7fc12e01 backend tests: Delay listing for swift backend 2017-06-15 19:41:07 +02:00
Alexander Neumann
98ae7b1210 s3: Save config in backend 2017-06-15 16:41:09 +02:00
Alexander Neumann
51877cecf7 s3: Prevent closing of the reader for GCS 2017-06-15 16:39:42 +02:00
Alexander Neumann
9053b2000b s3: Delete ignores error if the object doesn't exist 2017-06-15 16:27:19 +02:00
Alexander Neumann
dd6ce5f9d8 Remove backend.Closer, use ioutil.NopCloser() instead 2017-06-15 15:58:23 +02:00
Alexander Neumann
9a8301fc74 prune: Fix status string for narrow terminals 
Closes #1030
 2017-06-15 15:41:07 +02:00
Alexander Neumann
c79fb6fcdd prune: Delete repacked files as the very last step 2017-06-15 14:46:50 +02:00
Alexander Neumann
af9ba3be91 backend: Add IsNotExist 2017-06-15 13:40:27 +02:00
Alexander Neumann
6f24d038f8 prune: Only remove data after index has been uploaded 
Closes #1032
 2017-06-15 13:12:46 +02:00
Alexander Neumann
cf65893c4b s3: Allow setting the number of retries for minio-go 
https://github.com/restic/restic/issues/1013#issuecomment-307883970
 2017-06-12 21:09:37 +02:00
Alexander Neumann
422c0dfb5e s3: Exit test loop for minio server on success 2017-06-11 20:49:56 +02:00
Alexander Neumann
73b296918b s3: Reorder debug messages 
This way the semaphore token acquisition can be observed in the debug
log.
 2017-06-11 20:49:53 +02:00
Alexander Neumann
907c201693 debug: Add version number to debug log 2017-06-11 20:48:46 +02:00
Alexander Neumann
58de8bf392 swift/rest: Reduce number of connections 2017-06-11 20:48:46 +02:00
Alexander Neumann
a89a7a783a s3: Correct comment on the connections option 2017-06-11 20:48:46 +02:00
Alexander Neumann
c422010597 s3: Fix test 2017-06-11 20:48:46 +02:00
Alexander Neumann
08e1d9ffad s3: Switch back to high-level API, limit connections 2017-06-11 20:48:42 +02:00
Alexander Neumann
a4e8dc3371 s3: Improve error message in debug log 2017-06-11 11:22:25 +02:00
Alexander Neumann
19da56a6ea debug: Add log before panic() 2017-06-11 11:22:25 +02:00
Alexander Neumann
d3c06c39f9 debug: Fix EOF detection in HTTP transport 2017-06-11 11:22:25 +02:00
Alexander Neumann
6301620428 s3: Add more debug logs 2017-06-11 11:22:25 +02:00
Alexander Neumann
8d4417ec92 Remove unused code/variables 2017-06-11 09:29:53 +02:00
Alexander Neumann
0b55be2581 prune: Fix debug log 2017-06-10 22:16:42 +02:00
Alexander Neumann
88a59fd0ca options: Handle uint 2017-06-10 21:07:10 +02:00
Alexander Neumann
9d1b9157d4 ls: Print names with percent correctly 
Closes #1017
 2017-06-10 12:17:21 +02:00
Alexander Neumann
3e4d236751 Merge pull request #1010 from restic/update-minio-go 
Update github.com/minio/minio-go
 2017-06-09 20:55:49 +02:00
Alexander Neumann
4fe6593fbe Merge pull request #1011 from restic/fix-1009 
pack: Handle small files
 2017-06-09 20:53:52 +02:00
Alexander Neumann
48fecd791d pack: Handle more invalid header cases 2017-06-08 21:04:07 +02:00
Alexander Neumann
a325a20fb4 s3: Increase wait time for minio server 2017-06-08 20:50:56 +02:00
Alexander Neumann
1f0916b01b Merge pull request #1004 from restic/add-migrate-s3 
Add 'migrate' command, change s3 layout
 2017-06-08 20:48:27 +02:00
Alexander Neumann
eb767ab15f pack: Handle small files 2017-06-08 20:40:12 +02:00
Alexander Neumann
ff3d2e42f4 migrate: Be a bit more verbose 2017-06-08 19:19:45 +02:00
Alexander Neumann
04ded881f6 s3: Change the default layout to "default" 2017-06-07 23:08:20 +02:00
Alexander Neumann
4f9bf5312b Add migrate 
This commits adds a 'migrate' command and a migration to move s3
repositories from the 's3legacy' to the 'default' layout.
 2017-06-07 23:08:02 +02:00
Alexander Neumann
7cf8f59987 layout: Add String() and Name() 2017-06-07 21:59:41 +02:00
Alexander Neumann
b8b5c8e8c9 s3: Rename struct to Backend 2017-06-07 21:59:01 +02:00
Alexander Neumann
a46baf7685 s3: Remove cache 2017-06-07 20:51:45 +02:00
Alexander Neumann
233eaf8ee9 fuse: Improve semantics of the blob size cache 
Wrap it in a struct and add a Lookup() function to make clear that it
is only queried, not changed, so we don't have any race conditions.
 2017-06-07 20:04:58 +02:00
Alexander Neumann
067be2c551 fuse: Add cache for blob sizes 
Closes: #820
 2017-06-07 20:04:15 +02:00
Alexander Neumann
550e1feaec Merge pull request #999 from restic/backend-use-semaphore 
backends: Use new semaphore
 2017-06-07 19:48:32 +02:00
Alexander Neumann
48c1e7b00d Fix location tests 2017-06-06 21:12:38 +02:00
Alexander Neumann
d4e74f20aa Add context to dump command 2017-06-06 00:37:25 +02:00
Alexander Neumann
aa5bc39311 swift: Use semaphore 2017-06-06 00:33:25 +02:00
Alexander Neumann
46049b4236 rest: Use semaphore 2017-06-06 00:26:29 +02:00
Alexander Neumann
683ebef6c6 s3: Use semaphore 2017-06-06 00:17:39 +02:00
Alexander Neumann
5010e95c23 Add error handling to semaphore 2017-06-06 00:17:21 +02:00
Alexander Neumann
46b7a270a6 Add context parameters to tests 2017-06-05 23:56:59 +02:00
Alexander Neumann
cf497c2728 Add context to restic packages 2017-06-04 14:35:14 +02:00
Alexander Neumann
16fcd07110 Add a Context to the backend 2017-06-04 14:02:44 +02:00
Alexander Neumann
a90e0c6595 find: Check trees only once 2017-06-04 11:42:40 +02:00
Alexander Neumann
7b5efaf7b0 find: Move functions to struct 2017-06-04 11:38:46 +02:00
Alexander Neumann
3b7ca4ac35 find: Improve debug log 2017-06-04 11:22:56 +02:00
Alexander Neumann
40a61b82ce Merge pull request #978 from restic/add-backblaze-backend 
Add Backblaze B2 backend
 2017-06-03 14:54:04 +02:00
Alexander Neumann
fec89f95fb Improve swift backend 2017-06-03 14:28:18 +02:00
Bartłomiej Święcki
5681d41f76 Implement OpenStack swift backend 
This commit implements support for OpenStack swift
storage server, tested on OVH public cloud storage.

Special thanks to jayme-github <tuxnet@gmail.com>
who helped with the implementation.
 2017-06-03 14:26:29 +02:00
Alexander Neumann
122462b9b1 Add Backblaze B2 backend 
This is based on prior work by Joe Turgeon <arithmetric@gmail.com>
@arithmetric.
 2017-06-03 14:24:59 +02:00
Alexander Neumann
aa0f874c8d s3: Simplify IsNotExist() 2017-05-31 21:23:01 +02:00
Alexander Neumann
5c59484d2b s3: Return only basename in List() 2017-05-31 21:22:55 +02:00
Alexander Neumann
04b262d8f1 Allow many idle connections per host 
Closes #985
 2017-05-31 19:39:19 +02:00
Alexander Neumann
f3f6924b61 backend/test: Loose requirement about early error 2017-05-28 13:06:27 +02:00
Alexander Neumann
c5244abad9 rest: Improve error messages 2017-05-28 12:33:47 +02:00
Alexander Neumann
1f5954e2c1 layout: Test DefaultLayout for empty path prefix 2017-05-28 12:33:47 +02:00
Alexander Neumann
e046a2a6da sftp: Use path instead of filepath 2017-05-28 12:33:47 +02:00
Alexander Neumann
8395b53400 backend/test: Reduce verbosity in logs 2017-05-28 12:33:47 +02:00
Alexander Neumann
24ec14738d backend/test: Skip offset == length test 2017-05-28 12:33:47 +02:00
Alexander Neumann
79477fdfe4 backend/test: Randomize test suite 2017-05-28 12:33:47 +02:00
Alexander Neumann
7ec0543af3 testing: Add id to error message in panic 2017-05-28 10:17:04 +02:00
Alexander Neumann
5247de552a Remove regular status printing for non terminals 2017-05-25 17:03:48 +02:00
Alexander Neumann
d1cc87ba28 Merge pull request #966 from restic/unify-repo-layout 
WIP: Unify repository layout
 2017-05-22 21:15:38 +02:00
Alexander Neumann
69e114c5c4 Merge pull request #968 from mcastilho/patch-2 
Fixing CPUProfile
 2017-05-17 21:13:42 +02:00
Marcio Castilho
84f1037b8f Update global_debug.go 
The variable was wrong and the CPU profile was never being created.
 2017-05-17 11:07:33 -04:00
Pauline Middelink
120af801cf Fix golint warnings 
(except the exported fields/functions without comments)
 2017-05-17 01:39:39 +02:00
Pauline Middelink
f3d09ce7c8 Fix vet warnings 2017-05-17 01:34:33 +02:00
Pauline Middelink
2b9323529f Fix gofmt -s warnings 2017-05-17 01:28:39 +02:00
Pauline Middelink
971ecee171 Fix ineffassign mistakes 2017-05-17 01:25:52 +02:00
Alexander Neumann
8b461a7456 Make TestFlags less verbose 2017-05-16 20:49:18 +02:00
Alexander Neumann
2fa1238b8a Fix filenames for layout tests 2017-05-16 20:45:17 +02:00
Alexander Neumann
fa41183a53 s3: Add s3.layout option and layout auto detection 2017-05-15 23:37:02 +02:00
Alexander Neumann
959aa0f595 Simplify layout detection 2017-05-15 23:36:23 +02:00
Alexander Neumann
069752cb42 Make layout default Dirname() consistent 
Always return a trailing slash now.
 2017-05-15 23:35:52 +02:00
Alexander Neumann
0c537837d9 Rename testdata files 2017-05-15 22:39:50 +02:00
Alexander Neumann
f19852a738 Rename CloudLayout -> RESTLayout 
The REST backend uses a special layout without subdirs below data/. This
layout is just used there and nowhere else, and our REST server
implementation uses the default layout for the on disk storage. So we
remove the REST layout from the auto detection code.
 2017-05-15 22:05:43 +02:00
Alexander Neumann
61cade6222 Rename layout s3 -> s3legacy 2017-05-15 21:59:28 +02:00
Alexander Neumann
13393c76dc Merge pull request #957 from middelink/fix-953 
Change backup policy to be inclusive
 2017-05-15 17:19:04 +02:00
Pauline Middelink
7fffd408af Do not use singleton counter. 
Revert change for running the numbered tests as subtests.
 2017-05-15 08:53:59 +02:00
Alexander Neumann
250a45ab15 s3: Move interfaces to function 2017-05-14 20:36:26 +02:00
Alexander Neumann
3b44b87137 s3: Remove file after usage in test 2017-05-14 20:34:22 +02:00
Alexander Neumann
26c16b9fd3 s3: Correctly return token in Load 2017-05-14 20:19:12 +02:00
Alexander Neumann
0bd40bae6e s3: Prevent net/http client from closing the reader 2017-05-14 20:19:12 +02:00
Alexander Neumann
246ccf09b9 s3: add getRemainingSize 2017-05-14 20:19:12 +02:00
Alexander Neumann
9452f416bf s3: Use low level API for saving files 
benchmark                        old ns/op       new ns/op       delta
    BenchmarkBackendMinio/Save-4     184482294       40663344        -77.96%
    BenchmarkBackendS3/Save-4        35030825568     54475455819     +55.51%

    benchmark                        old MB/s     new MB/s     speedup
    BenchmarkBackendMinio/Save-4     90.95        412.64       4.54x
    BenchmarkBackendS3/Save-4        0.48         0.31         0.65x

    benchmark                        old allocs     new allocs     delta
    BenchmarkBackendMinio/Save-4     631            560            -11.25%
    BenchmarkBackendS3/Save-4        646            584            -9.60%

    benchmark                        old bytes     new bytes     delta
    BenchmarkBackendMinio/Save-4     66818060      50735         -99.92%
    BenchmarkBackendS3/Save-4        66834000      73024         -99.89%
 2017-05-14 20:19:12 +02:00
Alexander Neumann
1e0e6ee573 s3: Use low-level API with a Range header for Load 
benchmark                                         old ns/op      new ns/op      delta
    BenchmarkBackendMinio/LoadFile-4                  9213315        11001787       +19.41%
    BenchmarkBackendMinio/LoadPartialFile-4           4176619        3479707        -16.69%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     4391521        3139214        -28.52%
    BenchmarkBackendS3/LoadFile-4                     2886070905     2505907501     -13.17%
    BenchmarkBackendS3/LoadPartialFile-4              762702722      735694398      -3.54%
    BenchmarkBackendS3/LoadPartialFileOffset-4        789724328      1108989142     +40.43%

    benchmark                                         old MB/s     new MB/s     speedup
    BenchmarkBackendMinio/LoadFile-4                  1821.21      1525.15      0.84x
    BenchmarkBackendMinio/LoadPartialFile-4           1004.49      1205.67      1.20x
    BenchmarkBackendMinio/LoadPartialFileOffset-4     955.34       1336.45      1.40x
    BenchmarkBackendS3/LoadFile-4                     5.81         6.70         1.15x
    BenchmarkBackendS3/LoadPartialFile-4              5.50         5.70         1.04x
    BenchmarkBackendS3/LoadPartialFileOffset-4        5.31         3.78         0.71x

    benchmark                                         old allocs     new allocs     delta
    BenchmarkBackendMinio/LoadFile-4                  406            204            -49.75%
    BenchmarkBackendMinio/LoadPartialFile-4           225            206            -8.44%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     227            207            -8.81%
    BenchmarkBackendS3/LoadFile-4                     600            388            -35.33%
    BenchmarkBackendS3/LoadPartialFile-4              416            302            -27.40%
    BenchmarkBackendS3/LoadPartialFileOffset-4        417            303            -27.34%

    benchmark                                         old bytes     new bytes     delta
    BenchmarkBackendMinio/LoadFile-4                  29475         13904         -52.83%
    BenchmarkBackendMinio/LoadPartialFile-4           4218838       13958         -99.67%
    BenchmarkBackendMinio/LoadPartialFileOffset-4     4219175       14332         -99.66%
    BenchmarkBackendS3/LoadFile-4                     114152        97424         -14.65%
    BenchmarkBackendS3/LoadPartialFile-4              4265416       56212         -98.68%
    BenchmarkBackendS3/LoadPartialFileOffset-4        4266520       56308         -98.68%
 2017-05-14 20:19:12 +02:00
Alexander Neumann
be0e53c07b tests: Add test for backend Save() from file 2017-05-14 19:55:32 +02:00
Alexander Neumann
66b4999765 tests: Remove code generation, use reflection 
This simplifies the code.
 2017-05-14 12:59:07 +02:00
Alexander Neumann
403e201e1a tests: Improve robustness of config tests 2017-05-14 12:50:20 +02:00
Alexander Neumann
77a55fbe5c tests: Add documentation 2017-05-14 11:48:30 +02:00
Alexander Neumann
90c1608d88 sftp: Add Benchmarks 2017-05-14 11:48:30 +02:00
Alexander Neumann
7a51640262 rest: Add benchmarks 2017-05-14 11:48:30 +02:00