Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes#1460. Fixes#1687.
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466.
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/.
Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
AWS does not support keys longer than 1024 characters. Add special
handling for HeadObject which does not have a response body.
Found via pjdfstests. References #1589.
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes. This symptom was seen more frequently due
to periodic flushing of dirty data. Fixes#1547.
Previously s3fs threw away some function return values and returned
EIO instead. This was due to not trusting the mix of -1 and errno
return codes. Correct the obviously incorrect ones via visual
inspection. Stronger typing may find more occurrences. Fixes#1519.
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2. It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.
With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials. In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode. It can also be selected explicitly via
the imdsv1only mount option.
More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/
Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
