octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-06-07 11:30:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,986 Commits 1 Branch 22 Tags 12 MiB
526700f2de
Commit Graph

401 Commits

Author SHA1 Message Date
Andrew Gaul
096a230b70
Allow arbitrary size AWS secret keys (#1648) 
Previously s3fs limited these to 123 characters.  Fixes #1626.
 2021-05-06 22:24:38 +09:00
Andrew Gaul
8ef01d37a9
Fix a few nits (#1645) 
Make some strings more const, initialize members, and abort if lock
initialization fails.  Partially found via clang-tidy.
 2021-05-06 19:40:35 +09:00
Andrew Gaul
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses (#1635) 
S3 can emit these in unusual situations.  Fixes #1317.
 2021-04-25 13:18:11 +09:00
Takeshi Nakatani
2effffd8e2 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Takeshi Nakatani
2908878988 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Andrew Gaul
9645d57c05
Translate KeyTooLongError to ENAMETOOLONG (#1602) 
AWS does not support keys longer than 1024 characters.  Add special
handling for HeadObject which does not have a response body.
Found via pjdfstests.  References #1589.
 2021-03-28 13:17:41 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583) 
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
 2021-02-23 09:45:13 +09:00
Takeshi Nakatani
4da56acdcc Improved processing when HEAD response is 400 2021-02-13 17:00:13 +09:00
Andrew Gaul
a4d916af13
Allow configuring multipart copy size (#1555) 
Also align configuration with mixupload.
 2021-02-08 20:32:12 +09:00
Andrew Gaul
c692093921
Copy at most 5 GB per multipart copy part request (#1553) 
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes.  This symptom was seen more frequently due
to periodic flushing of dirty data.  Fixes #1547.
 2021-02-07 23:10:07 +09:00
Andrew Gaul
ff9d6a75c7
Parse more specific AWS error codes (#1549) 
Fixes #1545.
 2021-02-07 11:29:08 +09:00
Andrew Gaul
4b53d4bf6b
Suppress spurious SHA256 error message (#1544) 2021-02-04 07:28:51 +09:00
Andrew Gaul
17fda89ae9
Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors (#1534) 
Follows on to 4d833a4fb9.
 2021-01-25 19:08:14 +09:00
Andrew Gaul
d019dda4f7
Simplify substr manipulations with erase (#1532) 
This avoids creating a new std::string.
 2021-01-25 18:02:32 +09:00
Andrew Gaul
dc9255bc5f
Prefer static_cast where possible (#1531) 2021-01-25 08:15:17 +09:00
Andrew Gaul
b0e8758b63
Use result instead of res for consistency (#1530) 2021-01-25 07:56:10 +09:00
Andrew Gaul
4d833a4fb9
Return more specific errno when available (#1520) 
Previously s3fs threw away some function return values and returned
EIO instead.  This was due to not trusting the mix of -1 and errno
return codes.  Correct the obviously incorrect ones via visual
inspection.  Stronger typing may find more occurrences.  Fixes #1519.
 2021-01-18 18:50:49 +09:00
fly3366
168e588ac7
fix: Add reset offset (#1503) 2021-01-04 22:57:56 +09:00
Andrew Gaul
bd0fadbe5f
Remove authorization header when remaking handle (#1505) 
This avoids including Authorization in SignedHeaders.  s3fs will
recreate the Authorization header before sending the request.
 2021-01-04 21:37:34 +09:00
fly3366
51f65d7b14 fix: miss header when retry 2020-12-23 22:16:07 +09:00
Andrew Gaul
f8d5b76edb
Simplify handling of returned ETag (#1479) 
This works around lifetime warnings uncovered by cppcheck.
References #1478.
 2020-11-21 06:56:05 +09:00
Andrew Gaul
543231c9f2
Address warnings from clang-tidy 11 (#1470) 2020-11-09 21:15:20 +09:00
Noah Meyerhans
d96a08d4ad fixup! Implement AWS IMDSv2 support 2020-11-07 14:48:52 +09:00
Noah Meyerhans
f2f930300a Implement AWS IMDSv2 support 
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2.  It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.

With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials.  In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode.  It can also be selected explicitly via
the imdsv1only mount option.

More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
 2020-11-07 14:48:52 +09:00
Takeshi Nakatani
38e1eaa8a3 Added logfile option for non-syslog logging 2020-10-19 20:10:19 +09:00
Takeshi Nakatani
0492f75197
Merge pull request #1439 from gaul/run-once 
Protect curl_warnings_once with a mutex
 2020-10-04 22:43:56 +09:00
Takeshi Nakatani
493cf20f95
Merge pull request #1435 from gaul/no-exceptions 
Remove exceptions from s3fs_strtoofft
 2020-10-04 22:19:25 +09:00
Andrew Gaul
4df4ffe06f Protect curl_warnings_once with a mutex 
This allows ThreadSanitizer to run without warnings.
 2020-10-03 21:09:35 +09:00
Takeshi Nakatani
133feb67c3
Merge pull request #1434 from gaul/const-string 
Use const std::string& where possible
 2020-10-03 10:02:03 +09:00
Takeshi Nakatani
6e1e0d1d31
Merge pull request #1432 from gaul/v2-or-v4 
Add sigv4 flag and fix v4 regression
 2020-10-03 00:09:55 +09:00
Takeshi Nakatani
66419e7292
Merge pull request #1431 from gaul/string/find_first_of 
Prefer string::find_first_of(char) where possible
 2020-10-02 23:42:17 +09:00
Takeshi Nakatani
9e998877e9
Merge pull request #1430 from gaul/ostringstream 
Simplify use of ostringstream
 2020-10-02 23:26:41 +09:00
Andrew Gaul
3628b9d1e2 Remove exceptions from s3fs_strtoofft 
Explicit return codes are simpler and safer.
 2020-10-02 22:53:12 +09:00
Andrew Gaul
8d2bd874d7 Use const std::string& where possible 
This hints to callers on usage.
 2020-10-02 08:48:43 +09:00
Andrew Gaul
b8ff4ede49 Sign GET requests with V4 signature correctly 
This bug was previously masked by v2 fallback.  Fixes regression from
81805715bd.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
081d6c1245 Allow setting signature V4-only 
Default to allowing V2 or V4.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
eb8004c355 Prefer string::find_first_of(char) where possible 2020-10-01 18:24:45 +09:00
Andrew Gaul
b3bf9f8f54 Simplify use of ostringstream 
Resetting this does not re-use the underlying buffer so creating a new
ostringstream has similar efficiency.
 2020-10-01 17:55:34 +09:00
Andrew Gaul
503c86bb8a Call is_prefix instead of compare and substr 2020-09-27 22:19:54 +09:00
Takeshi Nakatani
757f4caee8
Merge pull request #1425 from gaul/create-bucket/sse 
Do not send SSE headers during bucket creation
 2020-09-27 11:34:15 +09:00
Takeshi Nakatani
0b42e08636
Merge pull request #1422 from gaul/c++/append-assign-at 
Remove calls to append, assign, and at
 2020-09-27 10:59:48 +09:00
Andrew Gaul
f35fe850c0 Do not send SSE headers during bucket creation 
This allows integration tests to pass with use_sse.
 2020-09-26 13:25:18 +09:00
Neeraj Kumar Kashyap
041b4ec05c enforce exact key match in IBM IAM response 2020-09-25 22:41:19 +09:00
Andrew Gaul
2438066d52 Remove calls to append, assign, and at 
operator+, operator=, and operator[] are more idiomatic and consistent
with the code base.
 2020-09-25 12:19:51 +09:00
Takeshi Nakatani
075d161bb1
Merge pull request #1408 from gaul/env 
Ensure environment variable is set when using ECS
 2020-09-20 13:01:17 +09:00
Andrew Gaul
44d2cc15f7 Ensure environment variable is set when using ECS 
Previously s3fs crashed when AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
was not set.  References #1162.
 2020-09-18 08:14:58 +09:00
Andrew Gaul
81805715bd Remove unnecessary calls to strcmp(str.c_str()) 2020-09-17 20:00:45 +09:00
Andrew Gaul
b7187352e1 Use more descriptive names for digest functions 
Previously they used hex to refer to binary data which is confusing
since other std::string return types are actually hex.  Also remove
unused s3fs_md5sum.
 2020-09-15 21:18:58 +09:00
Takeshi Nakatani
1520ca6220
Merge pull request #1397 from gaul/hex 
Centralize hex conversion
 2020-09-15 21:14:23 +09:00
Andrew Gaul
44eaac8471 Centralize hex conversion 2020-09-14 19:28:59 +09:00
Andrew Gaul
77501c3600 Use off_t for read sizes 
This fixes an issue with large multipart_size on 32-bit platforms like
Raspberry Pi.
 2020-09-14 18:12:23 +09:00
Andrew Gaul
1043e08dfa Remove uses of implicit namespace std 
Fixed via:

sed -i '/using namespace std/{N;d}' src/*.cpp
sed -i 's/ string/ std::string/g' src/*.cpp
sed -i 's/(string/(std::string/g' src/*.cpp
sed -i 's/\[string/\[std::string/g' src/*.cpp
sed -i 's/^string/std::string/g' src/*.cpp
sed -i 's/ ifstream/ std::ifstream/g' src/*.cpp
sed -i 's/ istringstream/ std::istringstream/g' src/*.cpp
sed -i 's/ ostringstream/ std::ostringstream/g' src/*.cpp
sed -i 's/ max(/ std::max(/g' src/*.cpp
sed -i 's/ min(/ std::min(/g' src/*.cpp
sed -i 's/ endl/ std::endl/g' src/*.cpp
 2020-09-13 11:57:20 +09:00
Andrew Gaul
6d1e704e34 Fix casting warnings on 32-bit platforms 2020-09-12 18:22:32 +09:00
Takeshi Nakatani
b5ffd419d8 Source file division and set 4 spaces and cleanup 2020-08-26 17:43:50 +09:00
Andrew Gaul
60d456a993 Use scoped enums for acl_t, sse_type_t, and storage_class_t 
This prevents some kinds of implicit conversions.  Also deduplicate
str/from_str logic.  References #1371.
 2020-08-20 18:46:18 +09:00
Takeshi Nakatani
bcf6838e86
Merge pull request #1370 from gaul/vim/modeline 
Correct vim modeline to 2-space indentation
 2020-08-19 22:44:40 +09:00
Andrew Gaul
958ad83a4b Correct vim modeline to 2-space indentation 2020-08-19 21:03:46 +09:00
Andrew Gaul
ba61470bae Prefer named locks in curl 
This removes the confusing and unsafe pthread_mutex_t array.
 2020-08-18 22:20:41 +09:00
Andrew Gaul
8210a1b2f2 Consistently lock curl_times and curl_progress 
References #1362.
 2020-08-17 09:47:38 +09:00
Andrew Gaul
132a1bebbb Prefer %zd and %zu for ssize_t and size_t 
This removes unnecessary casting.
 2020-08-15 17:30:23 +09:00
Takeshi Nakatani
dc365b65a0 Fix NULL pointer deference 
Found via cppcheck 2.1.
 2020-07-09 21:40:23 +09:00
Takeshi Nakatani
35006e318f Fixed about ParallelMixMultipartUpload 2020-06-24 12:48:55 +09:00
Eric Vantillard
7d0c66e08a Add support for glacier storage class. 
Just a copy of what have been done in PR #271.
 2020-06-23 11:23:21 +09:00
Takeshi Nakatani
f16ee96d7e
Merge pull request #1306 from gaul/http/500 
Retry with exponential backoff during 500 error
 2020-06-06 15:30:22 +09:00
Andrew Gaul
d8766b2051 Retry with exponential backoff during 500 error 
Amazon suggests retrying on both 500 and 503:

https://aws.amazon.com/premiumsupport/knowledge-center/http-5xx-errors-s3/

Fixes #1251.
 2020-06-05 21:01:30 +09:00
Andrew Gaul
9db70bab63 Ignore case when comparing ETags 
This allows multipart upload to work with Alibaba OSS.
References #1297.
 2020-06-05 18:17:52 +09:00
Andrew Gaul
40f7007263 Check results from pthread mutex calls 
Also remove some unnecessary exception handling.
 2020-05-30 16:37:55 +09:00
Takeshi Nakatani
75e72385cc Added a parameter to output body to curldbg option 2020-05-25 08:49:01 +09:00
Takeshi Nakatani
97c249d5b9 Not abort process by exception threw from s3fs_strtoofft 2020-05-03 08:08:28 +00:00
Andrew Gaul
2cc88b933f Warn about missing MIME types instead of exiting 
s3fs uses the MIME types file to set Content-Type for uploaded
objects.  Most distribution packages should install this via
recommended (not required) dependencies.  Users compiling from source
may not have this installed and s3fs should not prevent launching
since most users do not care about Content-Type.  Instead warn about
MIME types absence.  Fixes #1270.
 2020-04-29 20:03:50 +09:00
Andrew Gaul
cf529e0af7 Add handler for HTTP 416 
This prevents retries when the server indicates an unsatisfiable MPU
copy part request.  References #1220.
 2020-04-21 19:45:10 +09:00
Takeshi Nakatani
7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
yongqingliu
b72f4b43a4 use correct content-type when complete multipart upload 2020-02-10 16:58:28 +09:00
Andrew Gaul
bc9126d774 Set directory MIME type to application/x-directory 
Previously s3fs auto-detected the MIME type of directories like
"TOYOTA TRUCK 8.2.2" as application/x-troff-man.  This caused get_mode
to not set S_IFDIR which failed directory creation.  Instead force all
object names ending in / to application/x-directory.  Fixes #1183.
 2020-02-04 20:03:21 +09:00
rallister
4e26728cbf
break recursion when calling GetIAMCredentials (#1233) 
break recursion when calling GetIAMCredentials
 2020-01-31 16:48:37 +09:00
Andrei Precup
39102608aa Add intelligent tiering as option for storage class 
Resolves Issue #1219
 2020-01-13 20:24:35 +09:00
Alvaro Huarte
a24f78f5a4 Adds requester_pays option to enable requests involving Requester Pays buckets 2019-11-18 12:38:16 +01:00
Bryce Larson
0c42a74a8a
handle edge case when bucket name and first part of endpoint match 2019-10-26 00:51:19 -06:00
Takeshi Nakatani
1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00
Takeshi Nakatani
bedd648d47 Fixed build error by cppcheck 1.89 2019-09-23 10:49:49 +00:00
Takeshi Nakatani
58b3cce320
Merge pull request #1066 from gaul/sign-requests 
Sign request immediately before sending
 2019-09-23 15:58:21 +09:00
Takeshi Nakatani
81102a5963
Merge pull request #1149 from ggtakec/fix_retry 
Fixed a bug that type and op are not inherited
 2019-09-08 20:32:43 +09:00
Takeshi Nakatani
42fb30852b
Merge pull request #1145 from gaul/bool 
Fix a few bool types
 2019-09-08 20:04:30 +09:00
Takeshi Nakatani
e2129001eb Fixed a bug that type and op are not inherited 2019-09-08 09:59:04 +00:00
Andrew Gaul
805cc064af Fix a few bool types 
Found via clang-tidy.
 2019-09-05 10:42:42 -07:00
Andrew Gaul
412876ca33 Expose base parameter in s3fs_strtoofft 
This fixes a regression from ccf3e7bfa2
which caused the misparsing of octal inputs for the mp_umask and umask
flags.  It also allows some callers to be more precise about their
decimal inputs.
 2019-09-04 20:41:47 -07:00
Andrew Gaul
461a346bf4 Sign request immediately before sending 
Previously s3fs could create a long list of pre-signed requests which
could take longer than the default S3 clock skew limit of 15 minutes.
This also changes SHA-256 computation from single- to multi-threaded
since this is now computed in the worker threads.  Regression from
88cd8feb05.  Fixes #1019.
 2019-08-29 15:35:27 -07:00
liuyongqing
ae4bcd405c use correct content-type when overwriting and copying files 2019-08-28 22:25:09 -07:00
Takeshi Nakatani
0536dc1112 Refixed strict processing of MultiRead method 2019-08-25 20:46:01 -07:00
Andrew Gaul
cc2eed84a5 Promote default ACL to enum 
This sanity checks ACLs during initialization and also omits sending
the header when set to PRIVATE which is the default.
 2019-08-20 15:05:58 -07:00
Takeshi Nakatani
873e376098 Strict processing of MultiRead method 2019-08-13 15:23:13 +00:00
Takeshi Nakatani
4a0c23258e Fixed MultiRead method to exit after waiting other parts at error 2019-08-12 14:45:06 -07:00
Takeshi Nakatani
5c4a0a862a Fixed multipart copy and its retry handler 2019-08-11 16:21:26 +00:00
Takeshi Nakatani
f74c7407db
Merge pull request #1118 from gaul/atoi 
Prefer s3fs_strtoofft over atoi and strtol
 2019-08-06 21:23:37 +09:00
Takeshi Nakatani
433c04af26
Merge pull request #1116 from gaul/bodydata 
Do not heap allocate BodyData
 2019-08-06 21:08:02 +09:00
Andrew Gaul
ccf3e7bfa2 Prefer s3fs_strtoofft over atoi and strtol 
The former propagates errors consistently.
 2019-08-03 16:13:48 -07:00
Andrew Gaul
7ecfba811e Do not heap allocate BodyData 
Callers already manage lifetimes via Clear().
 2019-08-03 15:50:46 -07:00
Andrew Gaul
a5c20175a1 Use length from s3fs_HMAC256 for consistency 
EVP_MAX_MD_SIZE can range up to 64.  Found by Coverity.
 2019-08-02 10:40:55 -07:00
Andrew Gaul
bacd15714a Use consistent default for readwrite_timeout 
Previously curl threads mutated this shared state without a lock.
Found via ThreadSanitizer.
 2019-07-16 22:03:56 -07:00
Takeshi Nakatani
543aed2a32
Merge pull request #1096 from gaul/pthread/lock-checking 
Expand error checking to all pthread_mutex_t
 2019-07-15 11:43:56 +09:00