octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-09-27 16:59:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,728 Commits 1 Branch 22 Tags 12 MiB
8ef01d37a9
Commit Graph

350 Commits

Author SHA1 Message Date
Andrew Gaul
8ef01d37a9
Fix a few nits (#1645) 
Make some strings more const, initialize members, and abort if lock
initialization fails.  Partially found via clang-tidy.
 2021-05-06 19:40:35 +09:00
Andrew Gaul
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses (#1635) 
S3 can emit these in unusual situations.  Fixes #1317.
 2021-04-25 13:18:11 +09:00
Takeshi Nakatani
2effffd8e2 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Takeshi Nakatani
2908878988 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Andrew Gaul
9645d57c05
Translate KeyTooLongError to ENAMETOOLONG (#1602) 
AWS does not support keys longer than 1024 characters.  Add special
handling for HeadObject which does not have a response body.
Found via pjdfstests.  References #1589.
 2021-03-28 13:17:41 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583) 
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
 2021-02-23 09:45:13 +09:00
Takeshi Nakatani
4da56acdcc Improved processing when HEAD response is 400 2021-02-13 17:00:13 +09:00
Andrew Gaul
a4d916af13
Allow configuring multipart copy size (#1555) 
Also align configuration with mixupload.
 2021-02-08 20:32:12 +09:00
Andrew Gaul
c692093921
Copy at most 5 GB per multipart copy part request (#1553) 
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes.  This symptom was seen more frequently due
to periodic flushing of dirty data.  Fixes #1547.
 2021-02-07 23:10:07 +09:00
Andrew Gaul
ff9d6a75c7
Parse more specific AWS error codes (#1549) 
Fixes #1545.
 2021-02-07 11:29:08 +09:00
Andrew Gaul
4b53d4bf6b
Suppress spurious SHA256 error message (#1544) 2021-02-04 07:28:51 +09:00
Andrew Gaul
17fda89ae9
Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors (#1534) 
Follows on to 4d833a4fb9.
 2021-01-25 19:08:14 +09:00
Andrew Gaul
d019dda4f7
Simplify substr manipulations with erase (#1532) 
This avoids creating a new std::string.
 2021-01-25 18:02:32 +09:00
Andrew Gaul
dc9255bc5f
Prefer static_cast where possible (#1531) 2021-01-25 08:15:17 +09:00
Andrew Gaul
b0e8758b63
Use result instead of res for consistency (#1530) 2021-01-25 07:56:10 +09:00
Andrew Gaul
4d833a4fb9
Return more specific errno when available (#1520) 
Previously s3fs threw away some function return values and returned
EIO instead.  This was due to not trusting the mix of -1 and errno
return codes.  Correct the obviously incorrect ones via visual
inspection.  Stronger typing may find more occurrences.  Fixes #1519.
 2021-01-18 18:50:49 +09:00
fly3366
168e588ac7
fix: Add reset offset (#1503) 2021-01-04 22:57:56 +09:00
Andrew Gaul
bd0fadbe5f
Remove authorization header when remaking handle (#1505) 
This avoids including Authorization in SignedHeaders.  s3fs will
recreate the Authorization header before sending the request.
 2021-01-04 21:37:34 +09:00
fly3366
51f65d7b14 fix: miss header when retry 2020-12-23 22:16:07 +09:00
Andrew Gaul
f8d5b76edb
Simplify handling of returned ETag (#1479) 
This works around lifetime warnings uncovered by cppcheck.
References #1478.
 2020-11-21 06:56:05 +09:00
Andrew Gaul
543231c9f2
Address warnings from clang-tidy 11 (#1470) 2020-11-09 21:15:20 +09:00
Noah Meyerhans
d96a08d4ad fixup! Implement AWS IMDSv2 support 2020-11-07 14:48:52 +09:00
Noah Meyerhans
f2f930300a Implement AWS IMDSv2 support 
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2.  It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.

With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials.  In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode.  It can also be selected explicitly via
the imdsv1only mount option.

More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
 2020-11-07 14:48:52 +09:00
Takeshi Nakatani
38e1eaa8a3 Added logfile option for non-syslog logging 2020-10-19 20:10:19 +09:00
Takeshi Nakatani
0492f75197
Merge pull request #1439 from gaul/run-once 
Protect curl_warnings_once with a mutex
 2020-10-04 22:43:56 +09:00
Takeshi Nakatani
493cf20f95
Merge pull request #1435 from gaul/no-exceptions 
Remove exceptions from s3fs_strtoofft
 2020-10-04 22:19:25 +09:00
Andrew Gaul
4df4ffe06f Protect curl_warnings_once with a mutex 
This allows ThreadSanitizer to run without warnings.
 2020-10-03 21:09:35 +09:00
Takeshi Nakatani
133feb67c3
Merge pull request #1434 from gaul/const-string 
Use const std::string& where possible
 2020-10-03 10:02:03 +09:00
Takeshi Nakatani
6e1e0d1d31
Merge pull request #1432 from gaul/v2-or-v4 
Add sigv4 flag and fix v4 regression
 2020-10-03 00:09:55 +09:00
Takeshi Nakatani
66419e7292
Merge pull request #1431 from gaul/string/find_first_of 
Prefer string::find_first_of(char) where possible
 2020-10-02 23:42:17 +09:00
Takeshi Nakatani
9e998877e9
Merge pull request #1430 from gaul/ostringstream 
Simplify use of ostringstream
 2020-10-02 23:26:41 +09:00
Andrew Gaul
3628b9d1e2 Remove exceptions from s3fs_strtoofft 
Explicit return codes are simpler and safer.
 2020-10-02 22:53:12 +09:00
Andrew Gaul
8d2bd874d7 Use const std::string& where possible 
This hints to callers on usage.
 2020-10-02 08:48:43 +09:00
Andrew Gaul
b8ff4ede49 Sign GET requests with V4 signature correctly 
This bug was previously masked by v2 fallback.  Fixes regression from
81805715bd.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
081d6c1245 Allow setting signature V4-only 
Default to allowing V2 or V4.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
eb8004c355 Prefer string::find_first_of(char) where possible 2020-10-01 18:24:45 +09:00
Andrew Gaul
b3bf9f8f54 Simplify use of ostringstream 
Resetting this does not re-use the underlying buffer so creating a new
ostringstream has similar efficiency.
 2020-10-01 17:55:34 +09:00
Andrew Gaul
503c86bb8a Call is_prefix instead of compare and substr 2020-09-27 22:19:54 +09:00
Takeshi Nakatani
757f4caee8
Merge pull request #1425 from gaul/create-bucket/sse 
Do not send SSE headers during bucket creation
 2020-09-27 11:34:15 +09:00
Takeshi Nakatani
0b42e08636
Merge pull request #1422 from gaul/c++/append-assign-at 
Remove calls to append, assign, and at
 2020-09-27 10:59:48 +09:00
Andrew Gaul
f35fe850c0 Do not send SSE headers during bucket creation 
This allows integration tests to pass with use_sse.
 2020-09-26 13:25:18 +09:00
Neeraj Kumar Kashyap
041b4ec05c enforce exact key match in IBM IAM response 2020-09-25 22:41:19 +09:00
Andrew Gaul
2438066d52 Remove calls to append, assign, and at 
operator+, operator=, and operator[] are more idiomatic and consistent
with the code base.
 2020-09-25 12:19:51 +09:00
Takeshi Nakatani
075d161bb1
Merge pull request #1408 from gaul/env 
Ensure environment variable is set when using ECS
 2020-09-20 13:01:17 +09:00
Andrew Gaul
44d2cc15f7 Ensure environment variable is set when using ECS 
Previously s3fs crashed when AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
was not set.  References #1162.
 2020-09-18 08:14:58 +09:00
Andrew Gaul
81805715bd Remove unnecessary calls to strcmp(str.c_str()) 2020-09-17 20:00:45 +09:00
Andrew Gaul
b7187352e1 Use more descriptive names for digest functions 
Previously they used hex to refer to binary data which is confusing
since other std::string return types are actually hex.  Also remove
unused s3fs_md5sum.
 2020-09-15 21:18:58 +09:00
Takeshi Nakatani
1520ca6220
Merge pull request #1397 from gaul/hex 
Centralize hex conversion
 2020-09-15 21:14:23 +09:00
Andrew Gaul
44eaac8471 Centralize hex conversion 2020-09-14 19:28:59 +09:00
Andrew Gaul
77501c3600 Use off_t for read sizes 
This fixes an issue with large multipart_size on 32-bit platforms like
Raspberry Pi.
 2020-09-14 18:12:23 +09:00