Commit Graph

384 Commits

Author SHA1 Message Date
Takeshi Nakatani
92fd5bc3e1
Fixed errors reported by cppcheck 2.8 (#1949) 2022-05-26 01:48:53 +09:00
Takeshi Nakatani
684ced5a41 Changed handling the credential in S3fsCred more robust 2022-03-02 22:39:15 +09:00
Takeshi Nakatani
1678803566 Added S3fsCred class and moved Credential related processing in it 2022-02-13 21:38:30 +09:00
Naoki Ikeguchi
6edb6067f3 Remove strcasestr polyfill 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
b14e39815b Use polyfills in MSYS2 environment 2022-01-23 21:49:51 +09:00
Andrew Gaul
93d1c30d4d
Use XML parsing with PUT HTTP 200 responses (#1858)
This works around the missing strcasestr on win32.  References #728.
2022-01-14 16:10:22 +09:00
LiuBingrun
85ca2a3e45
fix mixupload return EntityTooSmall while a copypart is less than 5MB after split (#1809)
* fix  mixupload return EntityTooSmall while a copypart is less than 5MB after split
* fix possible part exceeds 5GB when multipart_copy_size is set to 5120MB
* Update curl.cpp
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
2021-11-27 16:53:26 +09:00
Andrew Gaul
3cf00626a2
Add option to allow unsigned payloads (#1801)
This reduces CPU usage of sigv4.  This reduces test run-time by 7
seconds per flag.
2021-11-01 23:33:55 +09:00
Takeshi Nakatani
2f412804e2 Fixed forgetting to clear the dirty flag for meta information
Addressed an error in macos cpp check
2021-10-15 22:54:55 +09:00
Kamil Jakrzewski
b4edad86d6 remove Expect: 100-continue header when requesting an IMDSv2 access token 2021-09-09 08:12:36 +09:00
Noah Meyerhans
9d1552a54e fix IAM role retrieval from IMDSv2
AWS IMDSv2 support was added in #1462, but the implementation did not
cover the addional IMDS access that occurs with the iam_role=auto
configuration.  This change implements IMDSv2 support for the IMDS
call to determine the instance's role name.

See also
https://stackoverflow.com/questions/69031023/how-to-make-s3fs-use-imds-v2-when-mounting-s3-buckets-from-ec2-instance
2021-09-03 20:36:34 +09:00
Andrew Gaul
47ebfcc60a
Consume return value from curl_easy_setopt (#1759)
Found via Coverity.
2021-09-02 08:07:06 +09:00
Andrew Gaul
57b2e4a4f1
Fix 32-bit compilation issues (#1757) 2021-08-31 19:36:02 +09:00
Andrew Gaul
48817d849f
Require explicit length in s3fs_decode64 (#1755)
This is available from std::string::size in callers.
2021-08-31 09:22:10 +09:00
Takeshi Nakatani
a868c0656e Changed etaglist_t from string list to new structure etagpairs list 2021-08-16 09:27:12 +09:00
Andrew Gaul
66006ba48d
Add dedicated upper- and lower-case hex functions (#1734)
This makes the call sites more readable than a boolean parameter.
2021-08-04 07:28:51 +09:00
Andrew Gaul
18e9c62087
Make string constants read-only const (#1733)
This removes some global constructors.  Also use a consistent ALL_CAPS
style.
2021-08-03 00:10:27 +09:00
Andrew Gaul
d3278f4886
Loosen CheckBucket to check only the bucket (#1728)
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460.  Fixes #1687.
2021-07-25 12:29:00 +09:00
Andrew Gaul
77f0b75d2f
Add jitter to avoid thundering herd (#1723) 2021-07-25 10:22:19 +09:00
Andrew Gaul
7890989cbb
Address cppcheck 2.5 warnings (#1720)
This slightly simplifies memory management.
2021-07-14 22:18:09 +09:00
Andrew Gaul
487df27008
Return EREMOTE when reading Glacier objects (#1705)
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466.
2021-06-30 09:25:36 +09:00
Andrew Gaul
1965916f7a
Convert storage class values to uppercase (#1704)
AWS requires uppercase values.  Fixes a regression from
bbcccd6e98.  References #1613.
2021-06-30 09:03:31 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Jan Stastny
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating (#1681)
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/.

Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-06-25 20:26:12 +09:00
Andrew Gaul
20281737b1
Set explicit Content-Length: 0 when initiating MPU (#1696)
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661.
2021-06-25 18:36:40 +09:00
Dmitrii Vasilev
b9ce0faee2 Set Content-MD5 header even for empty objects
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
2021-06-19 16:10:58 +09:00
Andrew Gaul
a100be9dce
Prefer std::string::empty over length == 0 (#1679)
This is more concise.
2021-06-13 20:03:10 +09:00
Andrew Gaul
600cee118d
Fix implicit narrowing conversions (#1672)
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
2021-06-13 12:50:07 +09:00
Takeshi Nakatani
c2c56d0263 Added info object about multipart uploading for each pseudo fd
(and fixed typo about method name)
2021-06-04 22:42:58 +09:00
Takeshi Nakatani
ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
Andrew Gaul
53dfd48f59
Correct usage of istringstream (#1670)
Previously this looped one more time than necessary due to the eof
check:

https://isocpp.org/wiki/faq/input-output#istream-and-eof

Remove now redundant empty check.
2021-06-02 07:14:32 +09:00
Andrew Gaul
bbcccd6e98
Make storage class a string (#1663)
This allows non-standard storage classes like Google Cloud Storage
Nearline.  Fixes #1613.
2021-05-21 23:34:31 +09:00
Andrew Gaul
bb6d2b1b74
Replace snprintf with string and ostringstream (#1649)
These uses are probably safe from a buffer overflow perspective but
can cause data race issues in logging due to static buffers.
2021-05-08 02:48:47 +09:00
Andrew Gaul
096a230b70
Allow arbitrary size AWS secret keys (#1648)
Previously s3fs limited these to 123 characters.  Fixes #1626.
2021-05-06 22:24:38 +09:00
Andrew Gaul
8ef01d37a9
Fix a few nits (#1645)
Make some strings more const, initialize members, and abort if lock
initialization fails.  Partially found via clang-tidy.
2021-05-06 19:40:35 +09:00
Andrew Gaul
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses (#1635)
S3 can emit these in unusual situations.  Fixes #1317.
2021-04-25 13:18:11 +09:00
Takeshi Nakatani
2effffd8e2 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Takeshi Nakatani
2908878988 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Andrew Gaul
9645d57c05
Translate KeyTooLongError to ENAMETOOLONG (#1602)
AWS does not support keys longer than 1024 characters.  Add special
handling for HeadObject which does not have a response body.
Found via pjdfstests.  References #1589.
2021-03-28 13:17:41 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583)
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
2021-02-23 09:45:13 +09:00
Takeshi Nakatani
4da56acdcc Improved processing when HEAD response is 400 2021-02-13 17:00:13 +09:00
Andrew Gaul
a4d916af13
Allow configuring multipart copy size (#1555)
Also align configuration with mixupload.
2021-02-08 20:32:12 +09:00
Andrew Gaul
c692093921
Copy at most 5 GB per multipart copy part request (#1553)
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes.  This symptom was seen more frequently due
to periodic flushing of dirty data.  Fixes #1547.
2021-02-07 23:10:07 +09:00
Andrew Gaul
ff9d6a75c7
Parse more specific AWS error codes (#1549)
Fixes #1545.
2021-02-07 11:29:08 +09:00
Andrew Gaul
4b53d4bf6b
Suppress spurious SHA256 error message (#1544) 2021-02-04 07:28:51 +09:00
Andrew Gaul
17fda89ae9
Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors (#1534)
Follows on to 4d833a4fb9.
2021-01-25 19:08:14 +09:00
Andrew Gaul
d019dda4f7
Simplify substr manipulations with erase (#1532)
This avoids creating a new std::string.
2021-01-25 18:02:32 +09:00
Andrew Gaul
dc9255bc5f
Prefer static_cast where possible (#1531) 2021-01-25 08:15:17 +09:00
Andrew Gaul
b0e8758b63
Use result instead of res for consistency (#1530) 2021-01-25 07:56:10 +09:00
Andrew Gaul
4d833a4fb9
Return more specific errno when available (#1520)
Previously s3fs threw away some function return values and returned
EIO instead.  This was due to not trusting the mix of -1 and errno
return codes.  Correct the obviously incorrect ones via visual
inspection.  Stronger typing may find more occurrences.  Fixes #1519.
2021-01-18 18:50:49 +09:00