Commit Graph

397 Commits

Author SHA1 Message Date
Takeshi Nakatani
16bc44948e Retry BucketCheck containing directory paths 2023-01-09 21:18:47 +09:00
Takeshi Nakatani
e4f85c1e08 Fixed renaming bug with SSE 2023-01-09 21:17:48 +09:00
Takeshi Nakatani
222110e153 Fixed a bug about multipart upload with SSE 2023-01-07 09:12:55 +05:30
Andrew Gaul
0ba49518e9
Make some methods and parameters const (#2078)
This requires making some locks mutable.
2023-01-04 20:23:39 +09:00
Takeshi Nakatani
aec7efc3af
Updated comments in curl.cpp (related to #1827) (#2052) 2022-11-05 11:45:28 +09:00
Andrew Gaul
30353f1a83
Remove unnecessary calls to dup (#1827)
These functions already use pread and keep track of an explicit offset
for the SHA-256 calculations.
2022-11-05 10:24:41 +09:00
Takeshi Nakatani
c491fbeabc Replace uses of lock_already_held flag with AutoLock::Type 2022-07-30 16:20:43 +09:00
Andrew Gaul
48e9e51f4f
Remove more unneeded headers identified by IWYU (#2011) 2022-07-30 12:06:47 +09:00
Andrew Gaul
01a92476e6
Remove unneeded headers identified by IWYU (#2007) 2022-07-28 23:38:38 +09:00
Andrew Gaul
3928a7e359
Remove more uses of const_cast (#2006)
Follows on to #2004.
2022-07-28 23:37:15 +09:00
Takeshi Nakatani
e30a5939d0 Fixed deadlock in S3fsCurl::DestroyCurlHandle 2022-07-28 13:47:27 +09:00
Takeshi Nakatani
ccfc119e45 Improved to avoid unnecessary head request 2022-07-27 21:51:28 +09:00
Takeshi Nakatani
d22e1dc018 Add the stream upload which starts uploading parts before Flush 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
92fd5bc3e1
Fixed errors reported by cppcheck 2.8 (#1949) 2022-05-26 01:48:53 +09:00
Takeshi Nakatani
684ced5a41 Changed handling the credential in S3fsCred more robust 2022-03-02 22:39:15 +09:00
Takeshi Nakatani
1678803566 Added S3fsCred class and moved Credential related processing in it 2022-02-13 21:38:30 +09:00
Naoki Ikeguchi
6edb6067f3 Remove strcasestr polyfill 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
b14e39815b Use polyfills in MSYS2 environment 2022-01-23 21:49:51 +09:00
Andrew Gaul
93d1c30d4d
Use XML parsing with PUT HTTP 200 responses (#1858)
This works around the missing strcasestr on win32.  References #728.
2022-01-14 16:10:22 +09:00
LiuBingrun
85ca2a3e45
fix mixupload return EntityTooSmall while a copypart is less than 5MB after split (#1809)
* fix  mixupload return EntityTooSmall while a copypart is less than 5MB after split
* fix possible part exceeds 5GB when multipart_copy_size is set to 5120MB
* Update curl.cpp
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
2021-11-27 16:53:26 +09:00
Andrew Gaul
3cf00626a2
Add option to allow unsigned payloads (#1801)
This reduces CPU usage of sigv4.  This reduces test run-time by 7
seconds per flag.
2021-11-01 23:33:55 +09:00
Takeshi Nakatani
2f412804e2 Fixed forgetting to clear the dirty flag for meta information
Addressed an error in macos cpp check
2021-10-15 22:54:55 +09:00
Kamil Jakrzewski
b4edad86d6 remove Expect: 100-continue header when requesting an IMDSv2 access token 2021-09-09 08:12:36 +09:00
Noah Meyerhans
9d1552a54e fix IAM role retrieval from IMDSv2
AWS IMDSv2 support was added in #1462, but the implementation did not
cover the addional IMDS access that occurs with the iam_role=auto
configuration.  This change implements IMDSv2 support for the IMDS
call to determine the instance's role name.

See also
https://stackoverflow.com/questions/69031023/how-to-make-s3fs-use-imds-v2-when-mounting-s3-buckets-from-ec2-instance
2021-09-03 20:36:34 +09:00
Andrew Gaul
47ebfcc60a
Consume return value from curl_easy_setopt (#1759)
Found via Coverity.
2021-09-02 08:07:06 +09:00
Andrew Gaul
57b2e4a4f1
Fix 32-bit compilation issues (#1757) 2021-08-31 19:36:02 +09:00
Andrew Gaul
48817d849f
Require explicit length in s3fs_decode64 (#1755)
This is available from std::string::size in callers.
2021-08-31 09:22:10 +09:00
Takeshi Nakatani
a868c0656e Changed etaglist_t from string list to new structure etagpairs list 2021-08-16 09:27:12 +09:00
Andrew Gaul
66006ba48d
Add dedicated upper- and lower-case hex functions (#1734)
This makes the call sites more readable than a boolean parameter.
2021-08-04 07:28:51 +09:00
Andrew Gaul
18e9c62087
Make string constants read-only const (#1733)
This removes some global constructors.  Also use a consistent ALL_CAPS
style.
2021-08-03 00:10:27 +09:00
Andrew Gaul
d3278f4886
Loosen CheckBucket to check only the bucket (#1728)
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460.  Fixes #1687.
2021-07-25 12:29:00 +09:00
Andrew Gaul
77f0b75d2f
Add jitter to avoid thundering herd (#1723) 2021-07-25 10:22:19 +09:00
Andrew Gaul
7890989cbb
Address cppcheck 2.5 warnings (#1720)
This slightly simplifies memory management.
2021-07-14 22:18:09 +09:00
Andrew Gaul
487df27008
Return EREMOTE when reading Glacier objects (#1705)
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466.
2021-06-30 09:25:36 +09:00
Andrew Gaul
1965916f7a
Convert storage class values to uppercase (#1704)
AWS requires uppercase values.  Fixes a regression from
bbcccd6e98.  References #1613.
2021-06-30 09:03:31 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Jan Stastny
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating (#1681)
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/.

Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-06-25 20:26:12 +09:00
Andrew Gaul
20281737b1
Set explicit Content-Length: 0 when initiating MPU (#1696)
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661.
2021-06-25 18:36:40 +09:00
Dmitrii Vasilev
b9ce0faee2 Set Content-MD5 header even for empty objects
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
2021-06-19 16:10:58 +09:00
Andrew Gaul
a100be9dce
Prefer std::string::empty over length == 0 (#1679)
This is more concise.
2021-06-13 20:03:10 +09:00
Andrew Gaul
600cee118d
Fix implicit narrowing conversions (#1672)
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
2021-06-13 12:50:07 +09:00
Takeshi Nakatani
c2c56d0263 Added info object about multipart uploading for each pseudo fd
(and fixed typo about method name)
2021-06-04 22:42:58 +09:00
Takeshi Nakatani
ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
Andrew Gaul
53dfd48f59
Correct usage of istringstream (#1670)
Previously this looped one more time than necessary due to the eof
check:

https://isocpp.org/wiki/faq/input-output#istream-and-eof

Remove now redundant empty check.
2021-06-02 07:14:32 +09:00
Andrew Gaul
bbcccd6e98
Make storage class a string (#1663)
This allows non-standard storage classes like Google Cloud Storage
Nearline.  Fixes #1613.
2021-05-21 23:34:31 +09:00
Andrew Gaul
bb6d2b1b74
Replace snprintf with string and ostringstream (#1649)
These uses are probably safe from a buffer overflow perspective but
can cause data race issues in logging due to static buffers.
2021-05-08 02:48:47 +09:00
Andrew Gaul
096a230b70
Allow arbitrary size AWS secret keys (#1648)
Previously s3fs limited these to 123 characters.  Fixes #1626.
2021-05-06 22:24:38 +09:00
Andrew Gaul
8ef01d37a9
Fix a few nits (#1645)
Make some strings more const, initialize members, and abort if lock
initialization fails.  Partially found via clang-tidy.
2021-05-06 19:40:35 +09:00
Andrew Gaul
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses (#1635)
S3 can emit these in unusual situations.  Fixes #1317.
2021-04-25 13:18:11 +09:00
Takeshi Nakatani
2effffd8e2 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00