octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-15 00:44:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,973 Commits 1 Branch 23 Tags 13 MiB
14eb1a7fd8
Commit Graph

132 Commits

Author SHA1 Message Date
Takeshi Nakatani
c491fbeabc Replace uses of lock_already_held flag with AutoLock::Type 2022-07-30 16:20:43 +09:00
Andrew Gaul
48e9e51f4f
Remove more unneeded headers identified by IWYU (#2011) 2022-07-30 12:06:47 +09:00
Andrew Gaul
01a92476e6
Remove unneeded headers identified by IWYU (#2007) 2022-07-28 23:38:38 +09:00
Takeshi Nakatani
e30a5939d0 Fixed deadlock in S3fsCurl::DestroyCurlHandle 2022-07-28 13:47:27 +09:00
Takeshi Nakatani
ccfc119e45 Improved to avoid unnecessary head request 2022-07-27 21:51:28 +09:00
Takeshi Nakatani
d22e1dc018 Add the stream upload which starts uploading parts before Flush 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
684ced5a41 Changed handling the credential in S3fsCred more robust 2022-03-02 22:39:15 +09:00
Takeshi Nakatani
1678803566 Added S3fsCred class and moved Credential related processing in it 2022-02-13 21:38:30 +09:00
Andrew Gaul
3cf00626a2
Add option to allow unsigned payloads (#1801) 
This reduces CPU usage of sigv4.  This reduces test run-time by 7
seconds per flag.
 2021-11-01 23:33:55 +09:00
Takeshi Nakatani
2f412804e2 Fixed forgetting to clear the dirty flag for meta information 
Addressed an error in macos cpp check
 2021-10-15 22:54:55 +09:00
Takeshi Nakatani
a868c0656e Changed etaglist_t from string list to new structure etagpairs list 2021-08-16 09:27:12 +09:00
Andrew Gaul
15e89b78de
Add a partial page_list unit test (#1735) 2021-08-04 07:36:32 +09:00
Andrew Gaul
2f59cb5a0a
Remove unneeded volatile qualifiers (#1702) 
We should use proper locking instead.  Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
 2021-06-30 08:20:44 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Andrew Gaul
a100be9dce
Prefer std::string::empty over length == 0 (#1679) 
This is more concise.
 2021-06-13 20:03:10 +09:00
Andrew Gaul
600cee118d
Fix implicit narrowing conversions (#1672) 
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
 2021-06-13 12:50:07 +09:00
Takeshi Nakatani
c2c56d0263 Added info object about multipart uploading for each pseudo fd 
(and fixed typo about method name)
 2021-06-04 22:42:58 +09:00
Takeshi Nakatani
ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
Andrew Gaul
bbcccd6e98
Make storage class a string (#1663) 
This allows non-standard storage classes like Google Cloud Storage
Nearline.  Fixes #1613.
 2021-05-21 23:34:31 +09:00
Andrew Gaul
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses (#1635) 
S3 can emit these in unusual situations.  Fixes #1317.
 2021-04-25 13:18:11 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583) 
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
 2021-02-23 09:45:13 +09:00
Andrew Gaul
a4d916af13
Allow configuring multipart copy size (#1555) 
Also align configuration with mixupload.
 2021-02-08 20:32:12 +09:00
fly3366
168e588ac7
fix: Add reset offset (#1503) 2021-01-04 22:57:56 +09:00
Noah Meyerhans
d96a08d4ad fixup! Implement AWS IMDSv2 support 2020-11-07 14:48:52 +09:00
Noah Meyerhans
f2f930300a Implement AWS IMDSv2 support 
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2.  It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.

With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials.  In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode.  It can also be selected explicitly via
the imdsv1only mount option.

More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
 2020-11-07 14:48:52 +09:00
Andrew Gaul
4df4ffe06f Protect curl_warnings_once with a mutex 
This allows ThreadSanitizer to run without warnings.
 2020-10-03 21:09:35 +09:00
Takeshi Nakatani
133feb67c3
Merge pull request #1434 from gaul/const-string 
Use const std::string& where possible
 2020-10-03 10:02:03 +09:00
Andrew Gaul
8d2bd874d7 Use const std::string& where possible 
This hints to callers on usage.
 2020-10-02 08:48:43 +09:00
Andrew Gaul
081d6c1245 Allow setting signature V4-only 
Default to allowing V2 or V4.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
05863a3178 Fix clang-tidy warnings 2020-09-21 07:03:42 +09:00
Andrew Gaul
77501c3600 Use off_t for read sizes 
This fixes an issue with large multipart_size on 32-bit platforms like
Raspberry Pi.
 2020-09-14 18:12:23 +09:00
Takeshi Nakatani
b5ffd419d8 Source file division and set 4 spaces and cleanup 2020-08-26 17:43:50 +09:00
Andrew Gaul
60d456a993 Use scoped enums for acl_t, sse_type_t, and storage_class_t 
This prevents some kinds of implicit conversions.  Also deduplicate
str/from_str logic.  References #1371.
 2020-08-20 18:46:18 +09:00
Takeshi Nakatani
bcf6838e86
Merge pull request #1370 from gaul/vim/modeline 
Correct vim modeline to 2-space indentation
 2020-08-19 22:44:40 +09:00
Andrew Gaul
958ad83a4b Correct vim modeline to 2-space indentation 2020-08-19 21:03:46 +09:00
Andrew Gaul
ba61470bae Prefer named locks in curl 
This removes the confusing and unsafe pthread_mutex_t array.
 2020-08-18 22:20:41 +09:00
Andrew Gaul
8210a1b2f2 Consistently lock curl_times and curl_progress 
References #1362.
 2020-08-17 09:47:38 +09:00
Takeshi Nakatani
35006e318f Fixed about ParallelMixMultipartUpload 2020-06-24 12:48:55 +09:00
Eric Vantillard
7d0c66e08a Add support for glacier storage class. 
Just a copy of what have been done in PR #271.
 2020-06-23 11:23:21 +09:00
Takeshi Nakatani
75e72385cc Added a parameter to output body to curldbg option 2020-05-25 08:49:01 +09:00
Takeshi Nakatani
7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
rallister
4e26728cbf
break recursion when calling GetIAMCredentials (#1233) 
break recursion when calling GetIAMCredentials
 2020-01-31 16:48:37 +09:00
Andrei Precup
39102608aa Add intelligent tiering as option for storage class 
Resolves Issue #1219
 2020-01-13 20:24:35 +09:00
Alvaro Huarte
a24f78f5a4 Adds requester_pays option to enable requests involving Requester Pays buckets 2019-11-18 12:38:16 +01:00
Takeshi Nakatani
1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00
Takeshi Nakatani
bedd648d47 Fixed build error by cppcheck 1.89 2019-09-23 10:49:49 +00:00
Takeshi Nakatani
0536dc1112 Refixed strict processing of MultiRead method 2019-08-25 20:46:01 -07:00
Andrew Gaul
cc2eed84a5 Promote default ACL to enum 
This sanity checks ACLs during initialization and also omits sending
the header when set to PRIVATE which is the default.
 2019-08-20 15:05:58 -07:00
Takeshi Nakatani
5c4a0a862a Fixed multipart copy and its retry handler 2019-08-11 16:21:26 +00:00
Andrew Gaul
7ecfba811e Do not heap allocate BodyData 
Callers already manage lifetimes via Clear().
 2019-08-03 15:50:46 -07:00