octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-16 17:25:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,978 Commits 1 Branch 23 Tags 13 MiB
16bc44948e
Commit Graph

397 Commits

Author SHA1 Message Date
Takeshi Nakatani
16bc44948e Retry BucketCheck containing directory paths 2023-01-09 21:18:47 +09:00
Takeshi Nakatani
e4f85c1e08 Fixed renaming bug with SSE 2023-01-09 21:17:48 +09:00
Takeshi Nakatani
222110e153 Fixed a bug about multipart upload with SSE 2023-01-07 09:12:55 +05:30
Andrew Gaul
0ba49518e9
Make some methods and parameters const (#2078) 
This requires making some locks mutable.
 2023-01-04 20:23:39 +09:00
Takeshi Nakatani
aec7efc3af
Updated comments in curl.cpp (related to #1827) (#2052) 2022-11-05 11:45:28 +09:00
Andrew Gaul
30353f1a83
Remove unnecessary calls to dup (#1827) 
These functions already use pread and keep track of an explicit offset
for the SHA-256 calculations.
 2022-11-05 10:24:41 +09:00
Takeshi Nakatani
c491fbeabc Replace uses of lock_already_held flag with AutoLock::Type 2022-07-30 16:20:43 +09:00
Andrew Gaul
48e9e51f4f
Remove more unneeded headers identified by IWYU (#2011) 2022-07-30 12:06:47 +09:00
Andrew Gaul
01a92476e6
Remove unneeded headers identified by IWYU (#2007) 2022-07-28 23:38:38 +09:00
Andrew Gaul
3928a7e359
Remove more uses of const_cast (#2006) 
Follows on to #2004.
 2022-07-28 23:37:15 +09:00
Takeshi Nakatani
e30a5939d0 Fixed deadlock in S3fsCurl::DestroyCurlHandle 2022-07-28 13:47:27 +09:00
Takeshi Nakatani
ccfc119e45 Improved to avoid unnecessary head request 2022-07-27 21:51:28 +09:00
Takeshi Nakatani
d22e1dc018 Add the stream upload which starts uploading parts before Flush 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
92fd5bc3e1
Fixed errors reported by cppcheck 2.8 (#1949) 2022-05-26 01:48:53 +09:00
Takeshi Nakatani
684ced5a41 Changed handling the credential in S3fsCred more robust 2022-03-02 22:39:15 +09:00
Takeshi Nakatani
1678803566 Added S3fsCred class and moved Credential related processing in it 2022-02-13 21:38:30 +09:00
Naoki Ikeguchi
6edb6067f3 Remove strcasestr polyfill 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
b14e39815b Use polyfills in MSYS2 environment 2022-01-23 21:49:51 +09:00
Andrew Gaul
93d1c30d4d
Use XML parsing with PUT HTTP 200 responses (#1858) 
This works around the missing strcasestr on win32.  References #728.
 2022-01-14 16:10:22 +09:00
LiuBingrun
85ca2a3e45
fix mixupload return EntityTooSmall while a copypart is less than 5MB after split (#1809) 
* fix  mixupload return EntityTooSmall while a copypart is less than 5MB after split
* fix possible part exceeds 5GB when multipart_copy_size is set to 5120MB
* Update curl.cpp
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
 2021-11-27 16:53:26 +09:00
Andrew Gaul
3cf00626a2
Add option to allow unsigned payloads (#1801) 
This reduces CPU usage of sigv4.  This reduces test run-time by 7
seconds per flag.
 2021-11-01 23:33:55 +09:00
Takeshi Nakatani
2f412804e2 Fixed forgetting to clear the dirty flag for meta information 
Addressed an error in macos cpp check
 2021-10-15 22:54:55 +09:00
Kamil Jakrzewski
b4edad86d6 remove Expect: 100-continue header when requesting an IMDSv2 access token 2021-09-09 08:12:36 +09:00
Noah Meyerhans
9d1552a54e fix IAM role retrieval from IMDSv2 
AWS IMDSv2 support was added in #1462, but the implementation did not
cover the addional IMDS access that occurs with the iam_role=auto
configuration.  This change implements IMDSv2 support for the IMDS
call to determine the instance's role name.

See also
https://stackoverflow.com/questions/69031023/how-to-make-s3fs-use-imds-v2-when-mounting-s3-buckets-from-ec2-instance
 2021-09-03 20:36:34 +09:00
Andrew Gaul
47ebfcc60a
Consume return value from curl_easy_setopt (#1759) 
Found via Coverity.
 2021-09-02 08:07:06 +09:00
Andrew Gaul
57b2e4a4f1
Fix 32-bit compilation issues (#1757) 2021-08-31 19:36:02 +09:00
Andrew Gaul
48817d849f
Require explicit length in s3fs_decode64 (#1755) 
This is available from std::string::size in callers.
 2021-08-31 09:22:10 +09:00
Takeshi Nakatani
a868c0656e Changed etaglist_t from string list to new structure etagpairs list 2021-08-16 09:27:12 +09:00
Andrew Gaul
66006ba48d
Add dedicated upper- and lower-case hex functions (#1734) 
This makes the call sites more readable than a boolean parameter.
 2021-08-04 07:28:51 +09:00
Andrew Gaul
18e9c62087
Make string constants read-only const (#1733) 
This removes some global constructors.  Also use a consistent ALL_CAPS
style.
 2021-08-03 00:10:27 +09:00
Andrew Gaul
d3278f4886
Loosen CheckBucket to check only the bucket (#1728) 
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460.  Fixes #1687.
 2021-07-25 12:29:00 +09:00
Andrew Gaul
77f0b75d2f
Add jitter to avoid thundering herd (#1723) 2021-07-25 10:22:19 +09:00
Andrew Gaul
7890989cbb
Address cppcheck 2.5 warnings (#1720) 
This slightly simplifies memory management.
 2021-07-14 22:18:09 +09:00
Andrew Gaul
487df27008
Return EREMOTE when reading Glacier objects (#1705) 
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466.
 2021-06-30 09:25:36 +09:00
Andrew Gaul
1965916f7a
Convert storage class values to uppercase (#1704) 
AWS requires uppercase values.  Fixes a regression from
bbcccd6e98.  References #1613.
 2021-06-30 09:03:31 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Jan Stastny
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating (#1681) 
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/.

Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-06-25 20:26:12 +09:00
Andrew Gaul
20281737b1
Set explicit Content-Length: 0 when initiating MPU (#1696) 
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661.
 2021-06-25 18:36:40 +09:00
Dmitrii Vasilev
b9ce0faee2 Set Content-MD5 header even for empty objects 
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
 2021-06-19 16:10:58 +09:00
Andrew Gaul
a100be9dce
Prefer std::string::empty over length == 0 (#1679) 
This is more concise.
 2021-06-13 20:03:10 +09:00
Andrew Gaul
600cee118d
Fix implicit narrowing conversions (#1672) 
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
 2021-06-13 12:50:07 +09:00
Takeshi Nakatani
c2c56d0263 Added info object about multipart uploading for each pseudo fd 
(and fixed typo about method name)
 2021-06-04 22:42:58 +09:00
Takeshi Nakatani
ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
Andrew Gaul
53dfd48f59
Correct usage of istringstream (#1670) 
Previously this looped one more time than necessary due to the eof
check:

https://isocpp.org/wiki/faq/input-output#istream-and-eof

Remove now redundant empty check.
 2021-06-02 07:14:32 +09:00
Andrew Gaul
bbcccd6e98
Make storage class a string (#1663) 
This allows non-standard storage classes like Google Cloud Storage
Nearline.  Fixes #1613.
 2021-05-21 23:34:31 +09:00
Andrew Gaul
bb6d2b1b74
Replace snprintf with string and ostringstream (#1649) 
These uses are probably safe from a buffer overflow perspective but
can cause data race issues in logging due to static buffers.
 2021-05-08 02:48:47 +09:00
Andrew Gaul
096a230b70
Allow arbitrary size AWS secret keys (#1648) 
Previously s3fs limited these to 123 characters.  Fixes #1626.
 2021-05-06 22:24:38 +09:00
Andrew Gaul
8ef01d37a9
Fix a few nits (#1645) 
Make some strings more const, initialize members, and abort if lock
initialization fails.  Partially found via clang-tidy.
 2021-05-06 19:40:35 +09:00
Andrew Gaul
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses (#1635) 
S3 can emit these in unusual situations.  Fixes #1317.
 2021-04-25 13:18:11 +09:00
Takeshi Nakatani
2effffd8e2 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00