s3fs periodically flushes data to S3 and hole-punches the written
data, marking it as clean. Without multipart copy, s3fs would write
the now-zeroed data along with any new data. s3fs should not attempt
to periodically flush dirty data at all without multipart copy
support. Fixes#1542.
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes. This symptom was seen more frequently due
to periodic flushing of dirty data. Fixes#1547.
Previously these failed due to dd only copying 32 MB from /dev/urandom
and exhausting the limited space in /tmp. References #1543.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
This logic has always been broken but happened to try one time due to
&& operator precedence. 7158e50ee2
broke this further when quoting && since the command was not
evaluated.
Previously s3fs threw away some function return values and returned
EIO instead. This was due to not trusting the mix of -1 and errno
return codes. Correct the obviously incorrect ones via visual
inspection. Stronger typing may find more occurrences. Fixes#1519.
This can find errors in retry logic. Chaos HTTP Proxy does not
support SSL bouncestorage/chaos-http-proxy#1 so users must set
s3proxy.endpoint and run via:
CHAOS_HTTP_PROXY=1 S3_URL=http://127.0.0.1:8080 make check -C test
It can also be helpful to increase retries and reduce sleep times.
References #1504.
* Homebrew changed the arguments required to install casks. Previously,
`brew cask install <package>` was the correct syntax, but now it's
`brew install --cask <package>`. This small commit updates the
instructions for installing `osxfuse` to reflect this change
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2. It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.
With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials. In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode. It can also be selected explicitly via
the imdsv1only mount option.
More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/
Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
This allows s3fs to write large files without consuming a large amount
of temporary local storage but can slow uploads due to server-side
copies. References #617. Fixed#1056. Fixes#1257.
