AdamQQQ
3856637cd2
s3fs: add option free_space_ratio to control cache size ( #2351 )
...
* Try to cleanup cache directory when initing without enough disk space
Also optimize log messages to print detailed errors to the user.
Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>
* s3fs: add option free_space_ratio to control cache size
Since the ensure_diskfree option is not convenient enough, we have added
a new option "-o free_space_ratio" to control the space used by the s3fs
cache based on the current disk size.
The value of this option can be between 0 and 100. It will control the
size of the cache according to this ratio to ensure that the idle ratio
of the disk is greater than this value.
For example, when the value is 10 and the disk space is 50GB, it will
ensure that the disk will reserve at least 50GB * 10% = 5GB of remaining
space.
Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>
---------
Signed-off-by: Qinqi Qu <quqinqi@linux.alibaba.com>
2023-10-20 18:11:47 +09:00
Andrew Gaul
a4a2841c05
Use C++11 nullptr instead of 0 or NULL ( #2234 )
...
This improves type-safety.
2023-07-27 21:56:58 +09:00
Takeshi Nakatani
faec0d9d15
Refixed for cppcheck 2.1x
2023-07-26 07:55:33 +09:00
Takeshi Nakatani
c491fbeabc
Replace uses of lock_already_held flag with AutoLock::Type
2022-07-30 16:20:43 +09:00
Takeshi Nakatani
4b2f3fecb5
Set mtime/ctime/atime of all objects as nanosecond
2022-07-28 13:47:03 +09:00
Takeshi Nakatani
d842d45b2b
Fixed a bug about truncation for shrinking file
2022-03-02 22:41:10 +09:00
Takeshi Nakatani
34f89e5936
Added fake_diskfree option to deceive free disk space for test
2021-08-02 11:04:42 +09:00
Andrew Gaul
f505c8224e
Explicitly specify lock type ( #1701 )
...
This makes it more clear and type-safe if the caller already has the
lock. Follows on to 84174c560d
.
2021-06-27 15:15:48 +09:00
Andrew Gaul
2154e898bc
Fix typos ( #1700 )
2021-06-27 11:22:33 +09:00
Carsten Grohmann
d67b83e671
Allow configuration for temporary files directory
2021-06-16 21:29:58 +09:00
Takeshi Nakatani
c2c56d0263
Added info object about multipart uploading for each pseudo fd
...
(and fixed typo about method name)
2021-06-04 22:42:58 +09:00
Takeshi Nakatani
ac578d188e
Introduced pseudo fd and separated fd for each file opening
2021-06-04 22:42:58 +09:00
Andrew Gaul
05863a3178
Fix clang-tidy warnings
2020-09-21 07:03:42 +09:00
Takeshi Nakatani
3958450c05
Added a class for automating fdentity reference counts
2020-09-14 22:02:28 +09:00
Takeshi Nakatani
b5ffd419d8
Source file division and set 4 spaces and cleanup
2020-08-26 17:43:50 +09:00
Takeshi Nakatani
a5186c73c2
Merge pull request #1365 from ggtakec/check_lseek
...
Dynamically determine whether lseek extended options are supported
2020-08-20 22:36:56 +09:00
Andrew Gaul
958ad83a4b
Correct vim modeline to 2-space indentation
2020-08-19 21:03:46 +09:00
Takeshi Nakatani
55cb8920d5
Dynamically determine whether lseek extended options are supported
2020-08-16 12:37:11 +00:00
Andrew Gaul
8de6cb3504
Lock fdent_data_lock before accessing pagelist
...
Found via ThreadSanitizer. References #1353 . References #1362 .
2020-08-16 17:44:03 +09:00
Takeshi Nakatani
2bb745cdd7
Fixed a bug about move file over limit of ensure space
2020-08-16 17:42:20 +09:00
Takeshi Nakatani
ad1961417d
Added SIGUSR1 option for cache file integrity test
2020-07-26 21:04:11 +09:00
Takeshi Nakatani
700e288718
Put similar processing together into method GetCacheFileStatTopDir
2020-07-12 08:12:02 +00:00
Takeshi Nakatani
35006e318f
Fixed about ParallelMixMultipartUpload
2020-06-24 12:48:55 +09:00
Takeshi Nakatani
f26a0aa71d
Fixed insufficient upload size for mix multipart upload
2020-04-22 09:31:22 +09:00
Takeshi Nakatani
4da02d023b
Improved strictness of cache file stats(file)
2020-04-21 19:45:03 +09:00
yongqingliu
b762a0a85b
fix deadlock due to fdmanager_lock and fdent_data_lock
2020-01-29 12:03:53 +08:00
Andrew Gaul
e0712f444d
Update source for clang-tidy 9.0.0
2020-01-13 20:56:45 +09:00
Takeshi Nakatani
cf3e82d10a
Fixed a rename bug when enable_noobj_cache
2019-10-23 12:39:42 +00:00
Takeshi Nakatani
1db94a0b30
Fixed to use copy api in multipart upload
2019-09-25 19:30:58 -07:00
Takeshi Nakatani
f5bf41cf11
Fixed simple copy with nocpoyapi and modify flag in fdcache
2019-08-11 07:42:48 +00:00
Andrew Gaul
5b15c7c4e9
Always hold lock when accessing free_disk_space
...
Slightly reorder locks to avoid double locking. Found via
ThreadSanitizer.
2019-07-30 08:36:54 -07:00
Andrew Gaul
a83d5baa90
Remove recursive locking
...
Recursive locking is frowned upon and is incompatible with
PTHREAD_MUTEX_ERRORCHECK. Also clean up pthread_mutex_lock error
checking.
2019-07-10 12:39:00 -07:00
Andrew Gaul
ecf13a8cb9
Break FdEntity locks into data and metadata
...
Previously long-running data operations like RowFlush would block
metadata operations like GetStats and thus user readdir. Fixes #928 .
2019-07-05 23:12:24 -07:00
Andrew Gaul
455e29cbea
Make fdpage a value type in fdpage_list_t
...
This simplifies memory management.
2019-07-01 09:30:48 -07:00
Andrew Gaul
15b7450713
Convert file offsets from size_t to off_t
...
The latter is 64-bits on 32-bit platforms when specifying
-D_FILE_OFFSET_BITS=64. This allows early Raspberry Pis to use files
larger than 2 GB. It also cleans up some ugly casting. Fixes #620 .
Fixes #656 .
2019-06-15 17:05:37 -07:00
Andrew Gaul
059ab1f0f4
Update ctime during nocopyapi operations
...
Follows on to 2c43b1e12b
. Fixes #971 .
2019-04-30 18:59:51 +09:00
Andrew Gaul
f53503438c
Increase FdEntity reference count when returning
...
Previously s3fs had a race condition where one client could delete
FdEntity that another client was using. Add a simple concurrent test
which previously failed but now succeeds. Fixes #964 .
2019-03-09 15:55:04 +09:00
Andrew Gaul
2c43b1e12b
Store and retrieve file change time
...
This introduces a new header with the change time; existing objects
will report modification time. Fixes #771 .
2019-01-14 10:05:11 -08:00
Takeshi Nakatani
f71a28f9b9
Merge pull request #714 from orozery/reduce_lock_contention
...
reduce lock contention on file open
2018-03-04 13:36:08 +09:00
Or Ozeri
8b657eee41
add disk space reservation
2018-02-28 19:20:23 +02:00
Or Ozeri
0edf056e95
reduce lock contention on file open
2018-02-04 17:13:58 +02:00
Scott Talbert
20da0e4dd3
Fix intermittent upload failures on macOS
...
There were multiple problems with the FdManager::GetFreeDiskSpace() function
on macOS:
1) When calling statvfs(), f_frsize should be used instead of f_bsize when
converting available blocks to bytes. This was causing the free space
calculation to be incorrect.
2) On macOS, fsblkcnt_t is a 32-bit integer. Thus, when calculating available
disk space, there were frequently overflows. This caused s3fs to incorrectly
determine that the cache location was out of space in the middle of a transfer
which caused uploads to fail. Changing this to a uint64_t resolves the
problem.
2017-09-08 15:23:10 -04:00
Takeshi Nakatani
9d10a5aa70
Changed copyright year format for debian pkg
2017-05-07 11:24:17 +00:00
Takeshi Nakatani
28efff5986
Merge pull request #554 from orozery/cache_cleanup
...
cleanup cache directory when running out of disk space
2017-04-16 19:13:11 +09:00
Takeshi Nakatani
fef3fbc225
Added check_cache_dir_exist option(refixed #347 ) - #538
2017-04-02 08:10:16 +00:00
Or Ozeri
95578cad43
cleanup cache directory when running out of disk space
2017-04-02 10:22:12 +03:00
Takeshi Nakatani
e8a8019a71
Add mirror file logic for removing cache file
2016-07-03 03:37:08 +00:00
Takeshi Nakatani
150b83f61e
Remove stat file cache dir if specified del_cache - #337
2016-02-06 18:59:13 +00:00
Takeshi Nakatani
7b62de80f6
Fixed a bug about mtime - #299
2015-11-29 15:53:53 +00:00
Takeshi Nakatani
8dd234dd8f
Fixed bugs about cppcheck error
2015-10-20 15:47:07 +00:00