octoleo
/
s3fs-fuse
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,998 Commits 1 Branch 22 Tags 11 MiB
Commit Graph

137 Commits

Author SHA1 Message Date
Takeshi Nakatani ca9a257eec
Allow mount points without directory objects by compat_dir (#2153) 2023-05-07 09:15:27 +09:00
Takeshi Nakatani d1388ff446 Added proxy and proxy_cred_file option 2023-02-08 21:53:34 +09:00
Takeshi Nakatani 16bc44948e Retry BucketCheck containing directory paths 2023-01-09 21:18:47 +09:00
Takeshi Nakatani e4f85c1e08 Fixed renaming bug with SSE 2023-01-09 21:17:48 +09:00
Andrew Gaul 0ba49518e9
Make some methods and parameters const (#2078) 
This requires making some locks mutable.
 2023-01-04 20:23:39 +09:00
Takeshi Nakatani c491fbeabc Replace uses of lock_already_held flag with AutoLock::Type 2022-07-30 16:20:43 +09:00
Andrew Gaul 48e9e51f4f
Remove more unneeded headers identified by IWYU (#2011) 2022-07-30 12:06:47 +09:00
Andrew Gaul 01a92476e6
Remove unneeded headers identified by IWYU (#2007) 2022-07-28 23:38:38 +09:00
Takeshi Nakatani e30a5939d0 Fixed deadlock in S3fsCurl::DestroyCurlHandle 2022-07-28 13:47:27 +09:00
Takeshi Nakatani ccfc119e45 Improved to avoid unnecessary head request 2022-07-27 21:51:28 +09:00
Takeshi Nakatani d22e1dc018 Add the stream upload which starts uploading parts before Flush 2022-07-17 22:20:45 +09:00
Takeshi Nakatani 684ced5a41 Changed handling the credential in S3fsCred more robust 2022-03-02 22:39:15 +09:00
Takeshi Nakatani 1678803566 Added S3fsCred class and moved Credential related processing in it 2022-02-13 21:38:30 +09:00
Andrew Gaul 3cf00626a2
Add option to allow unsigned payloads (#1801) 
This reduces CPU usage of sigv4.  This reduces test run-time by 7
seconds per flag.
 2021-11-01 23:33:55 +09:00
Takeshi Nakatani 2f412804e2 Fixed forgetting to clear the dirty flag for meta information 
Addressed an error in macos cpp check
 2021-10-15 22:54:55 +09:00
Takeshi Nakatani a868c0656e Changed etaglist_t from string list to new structure etagpairs list 2021-08-16 09:27:12 +09:00
Andrew Gaul 15e89b78de
Add a partial page_list unit test (#1735) 2021-08-04 07:36:32 +09:00
Andrew Gaul 2f59cb5a0a
Remove unneeded volatile qualifiers (#1702) 
We should use proper locking instead.  Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
 2021-06-30 08:20:44 +09:00
Andrew Gaul 2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Andrew Gaul a100be9dce
Prefer std::string::empty over length == 0 (#1679) 
This is more concise.
 2021-06-13 20:03:10 +09:00
Andrew Gaul 600cee118d
Fix implicit narrowing conversions (#1672) 
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
 2021-06-13 12:50:07 +09:00
Takeshi Nakatani c2c56d0263 Added info object about multipart uploading for each pseudo fd 
(and fixed typo about method name)
 2021-06-04 22:42:58 +09:00
Takeshi Nakatani ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
Andrew Gaul bbcccd6e98
Make storage class a string (#1663) 
This allows non-standard storage classes like Google Cloud Storage
Nearline.  Fixes #1613.
 2021-05-21 23:34:31 +09:00
Andrew Gaul fbf3c83019
Consider S3 errors in HTTP PUT 200 responses (#1635) 
S3 can emit these in unusual situations.  Fixes #1317.
 2021-04-25 13:18:11 +09:00
Andrew Gaul 032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583) 
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
 2021-02-23 09:45:13 +09:00
Andrew Gaul a4d916af13
Allow configuring multipart copy size (#1555) 
Also align configuration with mixupload.
 2021-02-08 20:32:12 +09:00
fly3366 168e588ac7
fix: Add reset offset (#1503) 2021-01-04 22:57:56 +09:00
Noah Meyerhans d96a08d4ad fixup! Implement AWS IMDSv2 support 2020-11-07 14:48:52 +09:00
Noah Meyerhans f2f930300a Implement AWS IMDSv2 support 
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2.  It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.

With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials.  In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode.  It can also be selected explicitly via
the imdsv1only mount option.

More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
 2020-11-07 14:48:52 +09:00
Andrew Gaul 4df4ffe06f Protect curl_warnings_once with a mutex 
This allows ThreadSanitizer to run without warnings.
 2020-10-03 21:09:35 +09:00
Takeshi Nakatani 133feb67c3
Merge pull request #1434 from gaul/const-string 
Use const std::string& where possible
 2020-10-03 10:02:03 +09:00
Andrew Gaul 8d2bd874d7 Use const std::string& where possible 
This hints to callers on usage.
 2020-10-02 08:48:43 +09:00
Andrew Gaul 081d6c1245 Allow setting signature V4-only 
Default to allowing V2 or V4.
 2020-10-01 20:03:14 +09:00
Andrew Gaul 05863a3178 Fix clang-tidy warnings 2020-09-21 07:03:42 +09:00
Andrew Gaul 77501c3600 Use off_t for read sizes 
This fixes an issue with large multipart_size on 32-bit platforms like
Raspberry Pi.
 2020-09-14 18:12:23 +09:00
Takeshi Nakatani b5ffd419d8 Source file division and set 4 spaces and cleanup 2020-08-26 17:43:50 +09:00
Andrew Gaul 60d456a993 Use scoped enums for acl_t, sse_type_t, and storage_class_t 
This prevents some kinds of implicit conversions.  Also deduplicate
str/from_str logic.  References #1371.
 2020-08-20 18:46:18 +09:00
Takeshi Nakatani bcf6838e86
Merge pull request #1370 from gaul/vim/modeline 
Correct vim modeline to 2-space indentation
 2020-08-19 22:44:40 +09:00
Andrew Gaul 958ad83a4b Correct vim modeline to 2-space indentation 2020-08-19 21:03:46 +09:00
Andrew Gaul ba61470bae Prefer named locks in curl 
This removes the confusing and unsafe pthread_mutex_t array.
 2020-08-18 22:20:41 +09:00
Andrew Gaul 8210a1b2f2 Consistently lock curl_times and curl_progress 
References #1362.
 2020-08-17 09:47:38 +09:00
Takeshi Nakatani 35006e318f Fixed about ParallelMixMultipartUpload 2020-06-24 12:48:55 +09:00
Eric Vantillard 7d0c66e08a Add support for glacier storage class. 
Just a copy of what have been done in PR #271.
 2020-06-23 11:23:21 +09:00
Takeshi Nakatani 75e72385cc Added a parameter to output body to curldbg option 2020-05-25 08:49:01 +09:00
Takeshi Nakatani 7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
rallister 4e26728cbf
break recursion when calling GetIAMCredentials (#1233) 
break recursion when calling GetIAMCredentials
 2020-01-31 16:48:37 +09:00
Andrei Precup 39102608aa Add intelligent tiering as option for storage class 
Resolves Issue #1219
 2020-01-13 20:24:35 +09:00
Alvaro Huarte a24f78f5a4 Adds requester_pays option to enable requests involving Requester Pays buckets 2019-11-18 12:38:16 +01:00
Takeshi Nakatani 1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00