AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2. It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.
With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials. In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode. It can also be selected explicitly via
the imdsv1only mount option.
More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/
Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
The latter is 64-bits on 32-bit platforms when specifying
-D_FILE_OFFSET_BITS=64. This allows early Raspberry Pis to use files
larger than 2 GB. It also cleans up some ugly casting. Fixes#620.
Fixes#656.
Workers now notify the master thread when they complete, unifying the
Linux and macOS code paths. This also avoids excessive
pthread_tryjoin_np calls. Follows on to
88cd8feb05.
S3 can copy multipart much faster than single part due to IO
parallelization. Renaming a 4 GB file reduces from 72 to 20 seconds
with bigger gains with larger files.
Previously s3fs would issue a batch of HEAD requests and wait for all
to succeed before issuing the next batch. Now it issues the first
batch and only waits for a single call to succeed before issuing the
next call. This can improve performance when one call lags due to
network errors. I measured 25% improvement with the same level of
parallelism. This commit also reparents parallelism knobs for
consistency. Follows on to 88cd8feb05.
Fixes#223.
Previously s3fs would issue a batch of uploads and wait for all to
succeed before issuing the next batch. Now it issues the first batch
and only waits for a single part to succeed before uploading the next
part. This can improve performance when one part lags due to network
errors. Fixes#183.
