Commit Graph

27 Commits

Author SHA1 Message Date
Takeshi Nakatani
d1388ff446 Added proxy and proxy_cred_file option 2023-02-08 21:53:34 +09:00
Takeshi Nakatani
7605c2e8fb Updated help contents and man pages for streamupload and etc 2023-01-31 20:53:20 +09:00
Takeshi Nakatani
6e89e69bba
Enabled to load shared library for Credential and Token (#1927) 2022-10-22 10:42:07 +09:00
Andrew Gaul
48e9e51f4f
Remove more unneeded headers identified by IWYU (#2011) 2022-07-30 12:06:47 +09:00
Andrew Gaul
01a92476e6
Remove unneeded headers identified by IWYU (#2007) 2022-07-28 23:38:38 +09:00
Andrew Gaul
8b90cd6ba1
Enable notsup_compat_dir by default (#1970)
Few applications create the dir_$folder$ objects and users can enable
compat_dir if required.  This commit reduces readdir latency by 33%.
Also remove notsup_compat_dir from tests since these directories are
never created.  Fixes #927.  References #1643.
2022-06-28 07:56:06 +09:00
Andrew Gaul
5c57e17b77
Enable noobj_cache by default (#1922)
This should improve performance in many situations.  s3fs already
enables the stat cache by default so memorizing noobj makes this more
consistent.  Fixes #1901.
2022-03-12 16:57:31 +09:00
Andrew Gaul
deaa85c40e
Document glacier_ir storage class (#1921)
Announcement:
https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-s3-glacier-instant-retrieval-storage-class/

This does not require code changes due to
bbcccd6e98.
2022-03-12 16:20:38 +09:00
Carsten Grohmann
afb0897553 Typos 2022-02-24 19:15:00 +09:00
Andrew Gaul
08adffd2fe
Fix typos (#1916) 2022-02-23 23:31:52 +09:00
Carsten Grohmann
a716c72d37 Update notsup_compat_dir in --help 2022-02-21 19:29:15 +09:00
Andrew Gaul
25012f3839 Fix typo in -o enable_unsigned_payload 2022-01-12 22:50:49 +09:00
Andrew Gaul
3cf00626a2
Add option to allow unsigned payloads (#1801)
This reduces CPU usage of sigv4.  This reduces test run-time by 7
seconds per flag.
2021-11-01 23:33:55 +09:00
Carsten Grohmann
d67b83e671 Allow configuration for temporary files directory 2021-06-16 21:29:58 +09:00
Ambika Nair
77581eda59 Change in ibm iam endpoint 2021-04-21 21:42:55 +09:00
Takeshi Nakatani
b589ebec23 Output version and command parameters at startup 2021-03-06 19:11:57 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583)
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
2021-02-23 09:45:13 +09:00
Takeshi Nakatani
a0f347b10f Added no_time_stamp_ms option mainly for testing 2021-02-20 20:24:26 +09:00
Andrew Gaul
134a54b32f
Allow configuring the multipart threshold (#1562)
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-11 23:35:46 +09:00
Andrew Gaul
a4d916af13
Allow configuring multipart copy size (#1555)
Also align configuration with mixupload.
2021-02-08 20:32:12 +09:00
Noah Meyerhans
d96a08d4ad fixup! Implement AWS IMDSv2 support 2020-11-07 14:48:52 +09:00
Noah Meyerhans
f2f930300a Implement AWS IMDSv2 support
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2.  It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.

With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials.  In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode.  It can also be selected explicitly via
the imdsv1only mount option.

More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
2020-11-07 14:48:52 +09:00
Takeshi Nakatani
38e1eaa8a3 Added logfile option for non-syslog logging 2020-10-19 20:10:19 +09:00
Andrew Gaul
6aa786b886
Flush dirty data after a writing a number of bytes (#1448)
This allows s3fs to write large files without consuming a large amount
of temporary local storage but can slow uploads due to server-side
copies.  References #617.  Fixed #1056.  Fixes #1257.
2020-10-13 22:30:42 +09:00
Andrew Gaul
081d6c1245 Allow setting signature V4-only
Default to allowing V2 or V4.
2020-10-01 20:03:14 +09:00
Andrew Gaul
1043e08dfa Remove uses of implicit namespace std
Fixed via:

sed -i '/using namespace std/{N;d}' src/*.cpp
sed -i 's/ string/ std::string/g' src/*.cpp
sed -i 's/(string/(std::string/g' src/*.cpp
sed -i 's/\[string/\[std::string/g' src/*.cpp
sed -i 's/^string/std::string/g' src/*.cpp
sed -i 's/ ifstream/ std::ifstream/g' src/*.cpp
sed -i 's/ istringstream/ std::istringstream/g' src/*.cpp
sed -i 's/ ostringstream/ std::ostringstream/g' src/*.cpp
sed -i 's/ max(/ std::max(/g' src/*.cpp
sed -i 's/ min(/ std::min(/g' src/*.cpp
sed -i 's/ endl/ std::endl/g' src/*.cpp
2020-09-13 11:57:20 +09:00
Takeshi Nakatani
b5ffd419d8 Source file division and set 4 spaces and cleanup 2020-08-26 17:43:50 +09:00