AdamQQQ
3856637cd2
s3fs: add option free_space_ratio to control cache size ( #2351 )
...
* Try to cleanup cache directory when initing without enough disk space
Also optimize log messages to print detailed errors to the user.
Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>
* s3fs: add option free_space_ratio to control cache size
Since the ensure_diskfree option is not convenient enough, we have added
a new option "-o free_space_ratio" to control the space used by the s3fs
cache based on the current disk size.
The value of this option can be between 0 and 100. It will control the
size of the cache according to this ratio to ensure that the idle ratio
of the disk is greater than this value.
For example, when the value is 10 and the disk space is 50GB, it will
ensure that the disk will reserve at least 50GB * 10% = 5GB of remaining
space.
Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>
---------
Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>
2023-10-20 18:11:47 +09:00
Takeshi Nakatani
ca9a257eec
Allow mount points without directory objects by compat_dir ( #2153 )
2023-05-07 09:15:27 +09:00
Ottavia Balducci
9b75abfbe6
New option: bucket_size ( #2148 )
...
* Added bucket_size option
2023-04-23 14:04:38 +09:00
Takeshi Nakatani
e715b77307
Added the function to update mtime/ctime of the parent directory ( #2016 )
2023-02-12 17:59:40 +09:00
Takeshi Nakatani
d1388ff446
Added proxy and proxy_cred_file option
2023-02-08 21:53:34 +09:00
Takeshi Nakatani
7605c2e8fb
Updated help contents and man pages for streamupload and etc
2023-01-31 20:53:20 +09:00
Pavel Knoblokh
d98fdf4b2e
Fix man page inaccuracies
2022-11-22 22:47:11 +09:00
Takeshi Nakatani
6e89e69bba
Enabled to load shared library for Credential and Token ( #1927 )
2022-10-22 10:42:07 +09:00
Andrew Gaul
2723e1049e
Add missing paragraph to help ( #1974 )
2022-07-03 12:35:02 +09:00
Andrew Gaul
8b90cd6ba1
Enable notsup_compat_dir by default ( #1970 )
...
Few applications create the dir_$folder$ objects and users can enable
compat_dir if required. This commit reduces readdir latency by 33%.
Also remove notsup_compat_dir from tests since these directories are
never created. Fixes #927 . References #1643 .
2022-06-28 07:56:06 +09:00
Andrew Gaul
5c57e17b77
Enable noobj_cache by default ( #1922 )
...
This should improve performance in many situations. s3fs already
enables the stat cache by default so memorizing noobj makes this more
consistent. Fixes #1901 .
2022-03-12 16:57:31 +09:00
Andrew Gaul
deaa85c40e
Document glacier_ir storage class ( #1921 )
...
Announcement:
https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-s3-glacier-instant-retrieval-storage-class/
This does not require code changes due to
bbcccd6e98
.
2022-03-12 16:20:38 +09:00
Carsten Grohmann
afb0897553
Typos
2022-02-24 19:15:00 +09:00
Takeshi Nakatani
5452e9cb10
Dynamically generate dates for man page file
2022-02-23 21:51:47 +09:00
Carsten Grohmann
c2a49b7b1a
Rephrase the description of notsup_compat_dir
2022-02-20 16:06:57 +09:00
Carsten Grohmann
265fa9e47a
Add performance considerations section to man page
2022-02-20 16:06:57 +09:00
Andrew Gaul
25012f3839
Fix typo in -o enable_unsigned_payload
2022-01-12 22:50:49 +09:00
Andrew Gaul
3cf00626a2
Add option to allow unsigned payloads ( #1801 )
...
This reduces CPU usage of sigv4. This reduces test run-time by 7
seconds per flag.
2021-11-01 23:33:55 +09:00
Carsten Grohmann
e1f3b9d8c1
Add support for AWS-style environment variables ( #1729 )
...
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.
The old environment variables are still supported, but they are
deprecated and no longer documented.
Close #1708
2021-07-26 23:29:45 +09:00
Carsten Grohmann
d67b83e671
Allow configuration for temporary files directory
2021-06-16 21:29:58 +09:00
Carsten Grohmann
c4fa53ec8b
Add section about storage consumption to manpage ( #1639 )
...
References s3fs-fuse#1591. References s3fs-fuse#1595.
2021-05-01 15:09:29 +09:00
Ambika Nair
77581eda59
Change in ibm iam endpoint
2021-04-21 21:42:55 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 ( #1583 )
...
This allows use of s3fs on object stores that do not implement
the V1 API. Fixes #1573 .
2021-02-23 09:45:13 +09:00
Takeshi Nakatani
a0f347b10f
Added no_time_stamp_ms option mainly for testing
2021-02-20 20:24:26 +09:00
Andrew Gaul
134a54b32f
Allow configuring the multipart threshold ( #1562 )
...
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-11 23:35:46 +09:00
Andrew Gaul
a4d916af13
Allow configuring multipart copy size ( #1555 )
...
Also align configuration with mixupload.
2021-02-08 20:32:12 +09:00
Noah Meyerhans
d96a08d4ad
fixup! Implement AWS IMDSv2 support
2020-11-07 14:48:52 +09:00
Takeshi Nakatani
38e1eaa8a3
Added logfile option for non-syslog logging
2020-10-19 20:10:19 +09:00
Andrew Gaul
6aa786b886
Flush dirty data after a writing a number of bytes ( #1448 )
...
This allows s3fs to write large files without consuming a large amount
of temporary local storage but can slow uploads due to server-side
copies. References #617 . Fixed #1056 . Fixes #1257 .
2020-10-13 22:30:42 +09:00
Andrew Gaul
081d6c1245
Allow setting signature V4-only
...
Default to allowing V2 or V4.
2020-10-01 20:03:14 +09:00
Andrew Gaul
892e7129c5
Add support for deep archive storage class
2020-08-20 23:42:45 +09:00
Takeshi Nakatani
63bbb47378
Merge pull request #1341 from gaul/stat-cache/default
...
Change default stat_cache_expire
2020-07-26 23:54:14 +09:00
Andrew Gaul
0fbd0eac80
Change default stat_cache_expire
...
Previously s3fs cached files forever which confused users with
creating objects using another client.
2020-07-26 23:04:43 +09:00
Takeshi Nakatani
ad1961417d
Added SIGUSR1 option for cache file integrity test
2020-07-26 21:04:11 +09:00
Eric Vantillard
7d0c66e08a
Add support for glacier storage class.
...
Just a copy of what have been done in PR #271 .
2020-06-23 11:23:21 +09:00
Takeshi Nakatani
75e72385cc
Added a parameter to output body to curldbg option
2020-05-25 08:49:01 +09:00
Andrew Gaul
005a684600
Fix typos
2020-04-22 21:49:11 +09:00
Takeshi Nakatani
7fbda230f5
Added mime option for strict checking of mime types file
2020-03-30 14:41:18 +00:00
Andrew Gaul
924eeb3587
Document host and servicepath
...
The various bits of host, url, and servicepath seem to overlap.
References #1203 .
2020-02-04 21:13:29 +09:00
Andrew Gaul
913b72fdaf
Add intelligent_tiering to man page
...
Follows on to 39102608aa
.
References #1219 .
2020-01-13 20:26:01 +09:00
Takeshi Nakatani
23945a0130
In memory cache for softlinks with cache out
2020-01-13 20:23:10 +09:00
Alvaro Huarte
bdfb9ee815
Document requester_pays option
2019-11-25 22:53:50 +09:00
Takeshi Nakatani
1db94a0b30
Fixed to use copy api in multipart upload
2019-09-25 19:30:58 -07:00
Andrew Gaul
3c2279db39
Document umask flag
2019-09-04 20:53:58 -07:00
Andrew Gaul
cc2eed84a5
Promote default ACL to enum
...
This sanity checks ACLs during initialization and also omits sending
the header when set to PRIVATE which is the default.
2019-08-20 15:05:58 -07:00
Andrew Gaul
bacd15714a
Use consistent default for readwrite_timeout
...
Previously curl threads mutated this shared state without a lock.
Found via ThreadSanitizer.
2019-07-16 22:03:56 -07:00
Andrew Gaul
8aa06d621a
Add documentation for use_session_token and use_rrs
...
Fixes #929 .
2019-07-06 10:04:43 -07:00
Andrew Gaul
f02105c346
Document ssl_verify_hostname
...
References #1064 .
2019-07-02 10:26:29 -07:00
Takeshi Nakatani
245f14c8c1
Merge pull request #1041 from gaul/doc/typo
...
Fix singlepart_copy_limit default
2019-06-23 22:57:43 +09:00
Andrew Gaul
07cfdcf205
Fix singlepart_copy_limit default
...
Follows on to c5ebf5d328
.
2019-06-15 17:38:14 -07:00