Few applications create the dir_$folder$ objects and users can enable
compat_dir if required. This commit reduces readdir latency by 33%.
Also remove notsup_compat_dir from tests since these directories are
never created. Fixes#927. References #1643.
This should improve performance in many situations. s3fs already
enables the stat cache by default so memorizing noobj makes this more
consistent. Fixes#1901.
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2. It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.
With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials. In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode. It can also be selected explicitly via
the imdsv1only mount option.
More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/
Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
This allows s3fs to write large files without consuming a large amount
of temporary local storage but can slow uploads due to server-side
copies. References #617. Fixed#1056. Fixes#1257.
