Commit Graph

139 Commits

Author SHA1 Message Date
Takeshi Nakatani cf6102f91b
Changed due to s3fs-fuse logo change (#2448) 2024-04-28 14:28:24 +09:00
Takeshi Nakatani 95026804e9 Support SSL client cert and added ssl_client_cert option 2024-04-14 10:21:48 +09:00
Takeshi Nakatani a5cdd05c25 Added ipresolve option 2024-03-13 22:29:17 +09:00
Takeshi Nakatani b97fd470a5 Abort for SSE-KMS encryption type and not SSL/TLS specified 2024-02-23 13:11:56 +09:00
AdamQQQ 3856637cd2
s3fs: add option free_space_ratio to control cache size (#2351)
* Try to cleanup cache directory when initing without enough disk space

Also optimize log messages to print detailed errors to the user.

Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>

* s3fs: add option free_space_ratio to control cache size

Since the ensure_diskfree option is not convenient enough, we have added
a new option "-o free_space_ratio" to control the space used by the s3fs
cache based on the current disk size.

The value of this option can be between 0 and 100. It will control the
size of the cache according to this ratio to ensure that the idle ratio
of the disk is greater than this value.

For example, when the value is 10 and the disk space is 50GB, it will
ensure that the disk will reserve at least 50GB * 10% = 5GB of remaining
space.

Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>

---------

Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>
2023-10-20 18:11:47 +09:00
Takeshi Nakatani ca9a257eec
Allow mount points without directory objects by compat_dir (#2153) 2023-05-07 09:15:27 +09:00
Ottavia Balducci 9b75abfbe6
New option: bucket_size (#2148)
* Added bucket_size option
2023-04-23 14:04:38 +09:00
Takeshi Nakatani e715b77307
Added the function to update mtime/ctime of the parent directory (#2016) 2023-02-12 17:59:40 +09:00
Takeshi Nakatani d1388ff446 Added proxy and proxy_cred_file option 2023-02-08 21:53:34 +09:00
Takeshi Nakatani 7605c2e8fb Updated help contents and man pages for streamupload and etc 2023-01-31 20:53:20 +09:00
Pavel Knoblokh d98fdf4b2e Fix man page inaccuracies 2022-11-22 22:47:11 +09:00
Takeshi Nakatani 6e89e69bba
Enabled to load shared library for Credential and Token (#1927) 2022-10-22 10:42:07 +09:00
Andrew Gaul 2723e1049e
Add missing paragraph to help (#1974) 2022-07-03 12:35:02 +09:00
Andrew Gaul 8b90cd6ba1
Enable notsup_compat_dir by default (#1970)
Few applications create the dir_$folder$ objects and users can enable
compat_dir if required.  This commit reduces readdir latency by 33%.
Also remove notsup_compat_dir from tests since these directories are
never created.  Fixes #927.  References #1643.
2022-06-28 07:56:06 +09:00
Andrew Gaul 5c57e17b77
Enable noobj_cache by default (#1922)
This should improve performance in many situations.  s3fs already
enables the stat cache by default so memorizing noobj makes this more
consistent.  Fixes #1901.
2022-03-12 16:57:31 +09:00
Andrew Gaul deaa85c40e
Document glacier_ir storage class (#1921)
Announcement:
https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-s3-glacier-instant-retrieval-storage-class/

This does not require code changes due to
bbcccd6e98.
2022-03-12 16:20:38 +09:00
Carsten Grohmann afb0897553 Typos 2022-02-24 19:15:00 +09:00
Takeshi Nakatani 5452e9cb10 Dynamically generate dates for man page file 2022-02-23 21:51:47 +09:00
Carsten Grohmann c2a49b7b1a Rephrase the description of notsup_compat_dir 2022-02-20 16:06:57 +09:00
Carsten Grohmann 265fa9e47a Add performance considerations section to man page 2022-02-20 16:06:57 +09:00
Andrew Gaul 25012f3839 Fix typo in -o enable_unsigned_payload 2022-01-12 22:50:49 +09:00
Andrew Gaul 3cf00626a2
Add option to allow unsigned payloads (#1801)
This reduces CPU usage of sigv4.  This reduces test run-time by 7
seconds per flag.
2021-11-01 23:33:55 +09:00
Carsten Grohmann e1f3b9d8c1
Add support for AWS-style environment variables (#1729)
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.

The old environment variables are still supported, but they are
deprecated and no longer documented.

Close #1708
2021-07-26 23:29:45 +09:00
Carsten Grohmann d67b83e671 Allow configuration for temporary files directory 2021-06-16 21:29:58 +09:00
Carsten Grohmann c4fa53ec8b
Add section about storage consumption to manpage (#1639)
References s3fs-fuse#1591. References s3fs-fuse#1595.
2021-05-01 15:09:29 +09:00
Ambika Nair 77581eda59 Change in ibm iam endpoint 2021-04-21 21:42:55 +09:00
Andrew Gaul 032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583)
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
2021-02-23 09:45:13 +09:00
Takeshi Nakatani a0f347b10f Added no_time_stamp_ms option mainly for testing 2021-02-20 20:24:26 +09:00
Andrew Gaul 134a54b32f
Allow configuring the multipart threshold (#1562)
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-11 23:35:46 +09:00
Andrew Gaul a4d916af13
Allow configuring multipart copy size (#1555)
Also align configuration with mixupload.
2021-02-08 20:32:12 +09:00
Noah Meyerhans d96a08d4ad fixup! Implement AWS IMDSv2 support 2020-11-07 14:48:52 +09:00
Takeshi Nakatani 38e1eaa8a3 Added logfile option for non-syslog logging 2020-10-19 20:10:19 +09:00
Andrew Gaul 6aa786b886
Flush dirty data after a writing a number of bytes (#1448)
This allows s3fs to write large files without consuming a large amount
of temporary local storage but can slow uploads due to server-side
copies.  References #617.  Fixed #1056.  Fixes #1257.
2020-10-13 22:30:42 +09:00
Andrew Gaul 081d6c1245 Allow setting signature V4-only
Default to allowing V2 or V4.
2020-10-01 20:03:14 +09:00
Andrew Gaul 892e7129c5 Add support for deep archive storage class 2020-08-20 23:42:45 +09:00
Takeshi Nakatani 63bbb47378
Merge pull request #1341 from gaul/stat-cache/default
Change default stat_cache_expire
2020-07-26 23:54:14 +09:00
Andrew Gaul 0fbd0eac80 Change default stat_cache_expire
Previously s3fs cached files forever which confused users with
creating objects using another client.
2020-07-26 23:04:43 +09:00
Takeshi Nakatani ad1961417d Added SIGUSR1 option for cache file integrity test 2020-07-26 21:04:11 +09:00
Eric Vantillard 7d0c66e08a Add support for glacier storage class.
Just a copy of what have been done in PR #271.
2020-06-23 11:23:21 +09:00
Takeshi Nakatani 75e72385cc Added a parameter to output body to curldbg option 2020-05-25 08:49:01 +09:00
Andrew Gaul 005a684600 Fix typos 2020-04-22 21:49:11 +09:00
Takeshi Nakatani 7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
Andrew Gaul 924eeb3587 Document host and servicepath
The various bits of host, url, and servicepath seem to overlap.
References #1203.
2020-02-04 21:13:29 +09:00
Andrew Gaul 913b72fdaf Add intelligent_tiering to man page
Follows on to 39102608aa.
References #1219.
2020-01-13 20:26:01 +09:00
Takeshi Nakatani 23945a0130 In memory cache for softlinks with cache out 2020-01-13 20:23:10 +09:00
Alvaro Huarte bdfb9ee815 Document requester_pays option 2019-11-25 22:53:50 +09:00
Takeshi Nakatani 1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00
Andrew Gaul 3c2279db39 Document umask flag 2019-09-04 20:53:58 -07:00
Andrew Gaul cc2eed84a5 Promote default ACL to enum
This sanity checks ACLs during initialization and also omits sending
the header when set to PRIVATE which is the default.
2019-08-20 15:05:58 -07:00
Andrew Gaul bacd15714a Use consistent default for readwrite_timeout
Previously curl threads mutated this shared state without a lock.
Found via ThreadSanitizer.
2019-07-16 22:03:56 -07:00