Commit Graph

373 Commits

Author SHA1 Message Date
Takeshi Nakatani
1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00
Takeshi Nakatani
bedd648d47 Fixed build error by cppcheck 1.89 2019-09-23 10:49:49 +00:00
Takeshi Nakatani
58b3cce320
Merge pull request #1066 from gaul/sign-requests
Sign request immediately before sending
2019-09-23 15:58:21 +09:00
Takeshi Nakatani
81102a5963
Merge pull request #1149 from ggtakec/fix_retry
Fixed a bug that type and op are not inherited
2019-09-08 20:32:43 +09:00
Takeshi Nakatani
42fb30852b
Merge pull request #1145 from gaul/bool
Fix a few bool types
2019-09-08 20:04:30 +09:00
Takeshi Nakatani
e2129001eb Fixed a bug that type and op are not inherited 2019-09-08 09:59:04 +00:00
Andrew Gaul
805cc064af Fix a few bool types
Found via clang-tidy.
2019-09-05 10:42:42 -07:00
Andrew Gaul
412876ca33 Expose base parameter in s3fs_strtoofft
This fixes a regression from ccf3e7bfa2
which caused the misparsing of octal inputs for the mp_umask and umask
flags.  It also allows some callers to be more precise about their
decimal inputs.
2019-09-04 20:41:47 -07:00
Andrew Gaul
461a346bf4 Sign request immediately before sending
Previously s3fs could create a long list of pre-signed requests which
could take longer than the default S3 clock skew limit of 15 minutes.
This also changes SHA-256 computation from single- to multi-threaded
since this is now computed in the worker threads.  Regression from
88cd8feb05.  Fixes #1019.
2019-08-29 15:35:27 -07:00
liuyongqing
ae4bcd405c use correct content-type when overwriting and copying files 2019-08-28 22:25:09 -07:00
Takeshi Nakatani
0536dc1112 Refixed strict processing of MultiRead method 2019-08-25 20:46:01 -07:00
Andrew Gaul
cc2eed84a5 Promote default ACL to enum
This sanity checks ACLs during initialization and also omits sending
the header when set to PRIVATE which is the default.
2019-08-20 15:05:58 -07:00
Takeshi Nakatani
873e376098 Strict processing of MultiRead method 2019-08-13 15:23:13 +00:00
Takeshi Nakatani
4a0c23258e Fixed MultiRead method to exit after waiting other parts at error 2019-08-12 14:45:06 -07:00
Takeshi Nakatani
5c4a0a862a Fixed multipart copy and its retry handler 2019-08-11 16:21:26 +00:00
Takeshi Nakatani
f74c7407db
Merge pull request #1118 from gaul/atoi
Prefer s3fs_strtoofft over atoi and strtol
2019-08-06 21:23:37 +09:00
Takeshi Nakatani
433c04af26
Merge pull request #1116 from gaul/bodydata
Do not heap allocate BodyData
2019-08-06 21:08:02 +09:00
Andrew Gaul
ccf3e7bfa2 Prefer s3fs_strtoofft over atoi and strtol
The former propagates errors consistently.
2019-08-03 16:13:48 -07:00
Andrew Gaul
7ecfba811e Do not heap allocate BodyData
Callers already manage lifetimes via Clear().
2019-08-03 15:50:46 -07:00
Andrew Gaul
a5c20175a1 Use length from s3fs_HMAC256 for consistency
EVP_MAX_MD_SIZE can range up to 64.  Found by Coverity.
2019-08-02 10:40:55 -07:00
Andrew Gaul
bacd15714a Use consistent default for readwrite_timeout
Previously curl threads mutated this shared state without a lock.
Found via ThreadSanitizer.
2019-07-16 22:03:56 -07:00
Takeshi Nakatani
543aed2a32
Merge pull request #1096 from gaul/pthread/lock-checking
Expand error checking to all pthread_mutex_t
2019-07-15 11:43:56 +09:00
Andrew Gaul
43ec064fb9 Expand error checking to all pthread_mutex_t 2019-07-14 18:21:23 -07:00
Andrew Gaul
ffac4c8417 Avoid narrowing time_t and off_t args in logging
This displays correct results on 32-bit platforms.
2019-07-14 17:02:36 -07:00
Takeshi Nakatani
e936854493
Merge pull request #1094 from larsch/master
Fix multipart upload not setting curl options
2019-07-15 05:13:37 +09:00
Lars Christensen
58037da061 Fix multipart upload not setting curl options 2019-07-14 14:09:37 +02:00
Andrew Gaul
6fd42d9fe4 Prefer modern C headers
Found and fixed via clang-tidy.
2019-07-12 03:50:59 -07:00
Takeshi Nakatani
e5e63d6ac3
Merge pull request #1067 from gaul/mismatched-free
Fix mismatched new[]/free
2019-07-07 16:55:07 +09:00
Takeshi Nakatani
7a65a414c3
Merge pull request #1080 from gaul/xml/simplification
Add simple XML parsing wrapper
2019-07-07 16:44:18 +09:00
Andrew Gaul
944d21cabb Fix mismatched new[]/free
Found via Valgrind.
2019-07-07 00:36:48 -07:00
Andrew Gaul
b8ff6a647e Add simple XML parsing wrapper
Also simplify check_region_error.
2019-07-05 17:39:09 -07:00
Andrew Gaul
bbfa91141a Configure clang-tidy target
Also fix nits.
2019-07-03 14:04:11 -07:00
Takeshi Nakatani
fbebc6fa57
Merge pull request #1048 from gaul/log/curl
Emit older curl warnings only once
2019-06-24 00:42:44 +09:00
Andrew Gaul
738eaadcbf Emit older curl warnings only once
This makes Travis logs actually readable.
2019-06-22 10:01:20 -07:00
Andrew Gaul
15b7450713 Convert file offsets from size_t to off_t
The latter is 64-bits on 32-bit platforms when specifying
-D_FILE_OFFSET_BITS=64.  This allows early Raspberry Pis to use files
larger than 2 GB.  It also cleans up some ugly casting.  Fixes #620.
Fixes #656.
2019-06-15 17:05:37 -07:00
Michal Lula
0d4847596e fix issue with aws session token provided inside ~/.aws/credentials file 2019-04-25 14:29:35 +02:00
Andrew Gaul
0677a096a9 Prefer simple over compound statements in macros
This prohibits missing semicolons.
2019-04-17 21:46:24 +09:00
Michal Lula
f9cd43b684 add session token support 2019-04-16 16:53:05 +02:00
Andrew Gaul
6f8ecb28c5 Correctly calculate MD5 hash for SSE-C keys
Previously s3fs calculated the strlen of a binary input instead of
using the stored length.  This yielded IO errors when interacting with
SSE-encrypted objects.  Fixes #696.
2019-04-08 21:49:42 +09:00
Andrew Gaul
6f6a67807b Prefer new over malloc
The former cannot return NULL and will allow use of scoped pointers.
2019-04-07 16:54:24 +09:00
Takeshi Nakatani
71766039ff Support undefined CURLoption in libcurl library used in build 2019-03-22 10:47:42 +00:00
Andrew Gaul
1e86cc643d Fix lazy typo 2019-03-09 17:46:02 +09:00
Takeshi Nakatani
10d9f75366 Improvement of curl session pool for multipart
Improvement of curl session pool for multipart(2)

Improvement of curl session pool for multipart(3)
2019-02-25 14:46:24 +00:00
Takeshi Nakatani
eeb839242b Fixed code for latest cppcheck error on OSX 2019-02-17 13:59:11 +00:00
Takeshi Nakatani
e07cb020cc
Merge pull request #954 from gaul/clear-iter
Clear containers instead of individual erases
2019-02-06 21:12:36 +09:00
Andrew Gaul
a1d3ff9766 Automatically abort failed multipart requests
This can avoid dangling parts.  However, many transfers fail due to
network errors so we still need other mechanisms to handle these
parts.
2019-02-03 10:29:20 -08:00
Andrew Gaul
4d0bef1e90 Clear containers instead of individual erases
This has O(n) runtime instead of O(n log n).
2019-02-02 23:58:43 -08:00
Takeshi Nakatani
c04e8e7a9d
Merge pull request #949 from gaul/503-slow-down
Implement exponential backoff for 503
2019-02-03 15:19:28 +09:00
Takeshi Nakatani
d8185a25aa
Merge pull request #946 from gaul/async/completed-tids
Simplify async request completion code
2019-02-03 14:08:49 +09:00
Takeshi Nakatani
53337a0a28
Merge pull request #944 from gaul/utility-mode
Repair utility mode
2019-02-03 13:59:56 +09:00
Takeshi Nakatani
055ecf6ea7
Merge pull request #940 from gaul/parallel-multipart-copy
Copy parts in parallel
2019-02-03 11:44:15 +09:00
Takeshi Nakatani
c603680e02
Merge pull request #939 from gaul/stringstream-specific
Prefer specific [io]stringstream where possible
2019-02-03 11:06:57 +09:00
Andrew Gaul
8ff05d8e38 Implement exponential backoff for 503
Amazon returns SlowDown when overloaded.  Also return ENOTSUP for 501
and immediately return EIO for 500 instead of retrying.  Fixes #603.
2019-02-02 00:03:50 -08:00
Andrew Gaul
8c527c3616 Simplify async request completion code
Workers now notify the master thread when they complete, unifying the
Linux and macOS code paths.  This also avoids excessive
pthread_tryjoin_np calls.  Follows on to
88cd8feb05.
2019-01-31 22:55:23 -08:00
Andrew Gaul
54a074647e Repair utility mode
This deinitialized S3fsCurl twice and incorrectly calculated V4
signatures.
2019-01-31 18:45:39 -08:00
Andrew Gaul
c5ebf5d328 Copy parts in parallel
S3 can copy multipart much faster than single part due to IO
parallelization.  Renaming a 4 GB file reduces from 72 to 20 seconds
with bigger gains with larger files.
2019-01-31 10:21:39 -08:00
Andrew Gaul
07636c8a8d Prefer specific [io]stringstream where possible
These better communicate intent and are slightly more efficient.
2019-01-29 10:44:33 -08:00
Andrew Gaul
35d55ee513 Remove unneeded void parameter
This is implicit in C++.  Found and fixed via clang-tidy.
2019-01-28 23:22:27 -08:00
Takeshi Nakatani
a0c1f30ae7
Merge pull request #932 from gaul/autolock
Prefer AutoLock for synchronization
2019-01-27 15:59:18 +09:00
Andrew Gaul
92d3114584 Prefer AutoLock for synchronization
This simplifies the code and fixes an issue with unlocked access.
Also use a recursive lock for StatCache to avoid races between
lock..unlock..lock sequences.
2019-01-25 15:28:41 -08:00
Andrew Gaul
84b421d6ef Prefer empty over size checks
Found and fixed via clang-tidy.
2019-01-23 11:30:28 -08:00
Takeshi Nakatani
010276ceab
Merge pull request #921 from gaul/clang-tidy/redundant-string-init
Remove redundant string initializations
2019-01-23 19:44:59 +09:00
Andrew Gaul
1fc25e8c3f Remove redundant string initializations
Found and fixed via clang-tidy.
2019-01-22 23:16:37 -08:00
Andrew Gaul
61ecafd426 Remove unnecessary string copies
Found via clang-tidy.
2019-01-22 23:09:37 -08:00
Takeshi Nakatani
fada95f58e
Merge pull request #914 from gaul/readdir/head-of-line
Issue readdir HEAD requests without batching
2019-01-21 22:10:50 +09:00
Takeshi Nakatani
014b8c5982
Merge pull request #913 from gaul/assert
Prefer abort over assert(false)
2019-01-21 21:56:41 +09:00
Andrew Gaul
46d79c5bc2 Issue readdir HEAD requests without batching
Previously s3fs would issue a batch of HEAD requests and wait for all
to succeed before issuing the next batch.  Now it issues the first
batch and only waits for a single call to succeed before issuing the
next call.  This can improve performance when one call lags due to
network errors.  I measured 25% improvement with the same level of
parallelism.  This commit also reparents parallelism knobs for
consistency.  Follows on to 88cd8feb05.
Fixes #223.
2019-01-20 18:07:22 -08:00
Andrew Gaul
40ba3b44a1 Prefer abort over assert(false)
The compiler can remove the latter when compiled with NDEBUG which may
cause unintended control flow.
2019-01-20 12:30:27 -08:00
Takeshi Nakatani
ac72431195 Added detail error message when HTTP 301/307 status 2019-01-20 10:07:58 +00:00
Takeshi Nakatani
36a4903843 Added an error message when HTTP 301 status 2019-01-20 07:17:40 +00:00
Takeshi Nakatani
05014c49c8
Merge pull request #906 from gaul/doc/https
Prefer HTTPS links where possible
2019-01-20 15:10:51 +09:00
Takeshi Nakatani
d373b0eca3
Merge pull request #904 from gaul/clang-tidy/c-str
Remove unnecessary calls to std::string::c_str
2019-01-20 14:13:46 +09:00
Takeshi Nakatani
61dc7f0a70
Merge pull request #900 from gaul/leak
Plug memory leak
2019-01-20 12:30:31 +09:00
Andrew Gaul
aa9bd1fa3c Prefer HTTPS links where possible
Fix a few stale links as well.
2019-01-18 11:09:08 -08:00
Andrew Gaul
508fafbe62 Remove unnecessary calls to std::string::c_str
Found via clang-tidy.
2019-01-17 22:05:16 -08:00
Andrew Gaul
58ce544e83 Plug memory leak
Previously this appended to an empty list, zeroed the list, then
appended to the list.  Instead zero the list first and then append.
This also enables sending Content-MD5 which can eagerly detect
transmission errors.  Found via Valgrind.
2019-01-17 16:24:34 -08:00
Andrew Gaul
e98ce36301 Tighten up HTTP response code check
Previously s3fs considered 3xx codes as successful.  When writing an
object to a misconfigured region, s3fs did not propagate the 301 as a
write error to the client.  References #693.
2019-01-17 15:04:59 -08:00
Takeshi Nakatani
4ed0e5f35a
Merge pull request #882 from earlchew/issue-817
[curl] Assume long encryption keys are base64 encoded
2019-01-06 17:29:04 +09:00
Earl Chew
ed5795eead [curl] Assume long encryption keys are base64 encoded
Correct tabs and whitespace.

Signed-off-by: Earl Chew <earl_chew@yahoo.com>
2019-01-05 21:08:41 +00:00
Andrew Gaul
a2f8ac535e Address cppcheck 1.86 errors
Lifetime, shadowing, and unused variables.  Found via the Travis macOS
builder.
2018-12-20 14:56:31 -08:00
Or Ozeri
b91fc5409e fix multihead warning check 2018-11-28 09:54:02 +02:00
Andrew Gaul
88cd8feb05 Upload S3 parts without batching
Previously s3fs would issue a batch of uploads and wait for all to
succeed before issuing the next batch.  Now it issues the first batch
and only waits for a single part to succeed before uploading the next
part.  This can improve performance when one part lags due to network
errors.  Fixes #183.
2018-11-16 18:32:38 -08:00
Jonatan Nilsson
d9c106cfde Add support for storage class ONEZONE_IA. 2018-09-30 14:27:31 +02:00
Earl Chew
41c23adb0e [curl] Assume long encryption keys are base64 encoded
Amazon SSE-C https://tinyurl.com/ychug4cg writes:

> Use this header to provide the 256-bit, base64-encoded encryption key
> for Amazon S3 to use to encrypt or decrypt your data.

It seems likely that future keys would be as long, or longer, so
this change assumes that text longer than 32 characters are keys
that are base64 encoded.

Signed-off-by: Earl Chew <earl_chew@yahoo.com>
2018-09-16 21:23:22 +00:00
Takeshi Nakatani
7e0c53dfe9 Added list_object_max_keys option based on #783 PR 2018-07-08 03:49:10 +00:00
root
35d3fce7a0 Review comment: Include the error code being returned 2018-07-06 05:14:32 -04:00
root
4177d8bd3b Review comment: Include the error code being returned 2018-07-06 03:03:57 -04:00
root
ad5349a488 Changes as per review comments 2018-07-05 05:02:04 -04:00
root
92a4034c5e Log messages for 5xx and 4xx HTTP response code 2018-07-04 03:50:45 -04:00
Takeshi Nakatani
0d3fb0658a Fixed a error by cppcheck on OSX 2018-06-24 02:38:59 +00:00
Takeshi Nakatani
ae9d8eb734 Fixed memory leak 2018-05-27 10:48:03 +00:00
Takeshi Nakatani
57b2a60172
Merge pull request #764 from orozery/remove_false_multihead_warnings
Remove false multihead warnings
2018-05-23 22:38:35 +09:00
Or Ozeri
0bd875eb9e remove false readdir_multi_head warnings 2018-05-22 17:10:50 +03:00
Or Ozeri
af63a42773 cleanup curl handles before curl share 2018-05-21 13:20:09 +03:00
Takeshi Nakatani
e3de6ea458 Added reset curl handle when returning to handle pool 2018-05-06 12:11:53 +00:00
Or Ozeri
adcf5754ae cleanup failing curl handles on retries 2018-03-29 13:56:08 +03:00
Takeshi Nakatani
987a166bf4
Merge pull request #726 from orozery/instance_name_logging
add an instance_name option for logging
2018-03-04 15:41:12 +09:00
Takeshi Nakatani
57b6f0eeaf
Merge pull request #724 from orozery/dont_fail_multirequest
don't fail multirequest on single thread error
2018-03-04 15:35:29 +09:00
Takeshi Nakatani
be9d407fa0 Fixed cppcheck error on osx 2018-02-28 12:29:58 +00:00
Takeshi Nakatani
c494e54320 Fixed cppcheck error on osx 2018-02-28 12:06:06 +00:00
Or Ozeri
b52b6f3fc5 add an instance_name option for logging 2018-02-28 09:51:35 +02:00
Or Ozeri
82c9733101 don't fail multirequest on single thread error 2018-02-26 12:06:08 +02:00
Takeshi Nakatani
a45ff6cdaa Fixed cppcheck error and clean ^M code 2018-02-25 13:08:41 +00:00
Takeshi Nakatani
960d45c853 Fixed cppcheck error on osx 2018-02-25 08:51:19 +00:00
Or Ozeri
bd4bc0e7f1 add support for IBM IAM authentication 2017-11-23 12:01:52 +02:00
Takeshi Nakatani
99fe93b7f1
Merge pull request #684 from gaul/signedness
Correct signedness warning
2017-11-23 17:33:46 +09:00
Or Ozeri
11bd7128d2 remove use of jsoncpp 2017-11-22 13:36:27 +02:00
Andrew Gaul
7cda32664b Correct signedness warning
Fixes regression from 0418e53b3c.
2017-11-19 11:14:37 -08:00
Takeshi Nakatani
e1928288fe
Merge pull request #676 from gaul/sstream
Move str definition from header to implementation
2017-11-19 16:34:15 +09:00
Andrew Gaul
0418e53b3c Reduce use of preprocessor
This provides type-safety and avoids token expansion side effects.
2017-11-18 22:40:06 -08:00
Takeshi Nakatani
bad48ab59a
Merge pull request #671 from psyvision/master
Add support for ECS metadata endpoint
2017-11-19 14:43:06 +09:00
Andrew Gaul
bbad76bb71 Move str definition from header to implementation
Also clean up some char * to std::string conversions.
2017-11-18 11:34:34 -08:00
Richard Caunt
b95e4acaeb
Remove debug logging statements 2017-11-08 15:52:35 +00:00
Richard Caunt
c238701d09
Corrected ECS headers 2017-11-08 15:21:49 +00:00
Richard Caunt
60d2ac3c7a
Adding x-amz-security-token header 2017-11-08 15:09:59 +00:00
Richard Caunt
967ef4d56b
Corrected fat finger mistakes 2017-11-08 13:14:49 +00:00
Richard Caunt
ad57bdda6c
Corrected keycount check 2017-11-08 13:06:22 +00:00
Richard Caunt
a0b69d1d3d
Corrected keyval[].c_str() 2017-11-08 13:01:52 +00:00
Richard Caunt
5df94d7e33
Add debug messages 2017-11-08 09:50:39 +00:00
Richard Caunt
1cbe9fb7a3 Gotta pass that cppcheckgit add . 2017-11-07 21:41:51 +00:00
Richard Caunt
8660abaea2 Use jsoncpp to parse AWS JSON 2017-11-07 21:20:02 +00:00
Richard Caunt
366f0705a0 ECS credentials bug fixes 2017-11-06 21:45:58 +00:00
Richard Caunt
5d54883e2f Remove commented out code 2017-11-05 19:25:34 +00:00
Richard Caunt
662f65c3c8 Add support for ECS metadata endpoint 2017-11-05 19:24:02 +00:00
Takeshi Nakatani
5db550a298 Fixed a bug in S3fsCurl::LocateBundle 2017-11-05 11:26:05 +00:00
Or Ozeri
384b4cbafa auth headers insertion refactoring 2017-10-30 11:52:58 +02:00
Takeshi Nakatani
5957d9ead0 Fixed with unnecessary equal in POST uploads url argment - #643 2017-09-17 10:52:28 +00:00
Takeshi Nakatani
00bc9142c4 Fixed potential atomic violation in S3fsCurl::AddUserAgent - #633 2017-09-17 09:16:05 +00:00
Takeshi Nakatani
a22675bafd Merge pull request #567 from andrewgaul/default-acl
Do not send ACL unless overridden
2017-05-09 23:03:27 +09:00
Andrew Gaul
7b30d5d15b Do not send canned ACL header when empty string
Some providers such as StorageGRID do not support canned ACLs.
Setting to empty allows callers to omit the header.  References #125.
2017-05-07 10:52:31 -07:00
Takeshi Nakatani
9d10a5aa70 Changed copyright year format for debian pkg 2017-05-07 11:24:17 +00:00
Takeshi Nakatani
a12e0d5ec4 Fixed failure to upload/copy with SSE_C and SSE_KMS 2017-05-07 09:29:08 +00:00
Takeshi Nakatani
d07c3f38b7 Check errors returned in 200 OK responses for put header request 2017-05-06 02:15:53 +00:00
Takeshi Nakatani
df0ff3a2fd Merge pull request #556 from orozery/fix_nocache_multipart_upload
fix multipart upload handling without cache
2017-04-16 19:22:15 +09:00
Takeshi Nakatani
edcf4c6218 Merge pull request #555 from orozery/dont_sign_empty_headers
don't sign empty headers (as they are discarded by libcurl)
2017-04-16 19:16:47 +09:00
Takeshi Nakatani
efba9bcbc1 Merge pull request #553 from orozery/custom_cipher_suite
add TLS cipher suites customization
2017-04-16 19:09:27 +09:00
Takeshi Nakatani
6bd179c92b Merge pull request #552 from orozery/foreground_threads
switch S3fsMultiCurl to use foreground threads
2017-04-16 19:05:16 +09:00
Or Ozeri
96764b7410 switch S3fsMultiCurl to use foreground threads 2017-04-09 16:56:49 +03:00
Takeshi Nakatani
b4c90d6957 Fixed a bug about multipart uploading at no disk free space related to #509 2017-04-09 04:37:20 +00:00
Or Ozeri
75b59a7c16 switch S3fsMultiCurl to use foreground threads 2017-04-04 15:32:53 +03:00
Or Ozeri
3bcca75a88 don't sign empty headers (as they are discarded by libcurl) 2017-04-04 15:24:20 +03:00
Or Ozeri
8ee95ff7ab fix multipart upload handling without cache 2017-04-02 10:27:43 +03:00
Andrew Gaul
03217baa99 Address cppcheck 1.77 warnings 2017-03-06 12:41:08 -08:00
Andrew Gaul
915a1321c7 Use server-provided ETag during complete upload
This avoids calculating the MD5 locally and enables use with object
stores which do not use MD5 as ETag.
2016-11-23 18:48:57 -08:00
Andrew Gaul
d375bca0d0 Correct typos 2016-11-19 15:57:41 -08:00
Takeshi Nakatani
edd0a11fb5 Merge pull request #494 from mapreri/typo
Fix typo s/destroied/destroyed/
2016-11-20 05:53:23 +09:00
Takeshi Nakatani
67a836223a Merge pull request #495 from driskell/fix_sse_copy
Fix invalid V4 signature on multipart copy requests
2016-11-20 05:43:10 +09:00
Jason Woods
6f688770fd Fix invalid V4 signature on multipart copy requests 2016-11-13 13:22:00 +00:00
Mattia Rizzolo
8c0b1d9c5b
Fix typo s/destroied/destroyed/ 2016-11-11 23:27:17 +00:00
Takeshi Nakatani
cca217f613 Merge pull request #487 from driskell/debugging
Split header debugging onto multiple lines for easier reading
2016-10-23 21:51:38 +09:00