7978395083
Use smart pointer to manage pcfstat object
...
Previously pcfstat points to a raw pointer, and it may be leaked if
function returned before deleting it.
So use smart pointer to automatically release the object.
Note that currently s3fs only uses c++03, so we use auto_ptr here, not
unique_ptr, which requires c++11.
Fixes: 6ca5a24a7f#2152 )")
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
2023-07-19 20:15:55 +09:00
d0a944fcaa
Fixed data race about fuse_fill_dir_t function and data pointer
2023-07-14 22:32:12 +09:00
537384e9b5
Guard filler calls with filled check ( #2215 )
...
Follows on to e650d8c55c
2023-07-13 22:46:19 +09:00
e650d8c55c
Explicitly handle CommonPrefixes with nocompat_dir ( #2212 )
...
Previously the test missed listing implicit directories and another
test was incorrect. This fixes a regression from 1.91.
2023-07-13 21:15:34 +09:00
9663215bb4
Fixed data race at OPENSSL_sk_dup/free in libcurl
2023-07-12 22:51:57 +09:00
b2537052ef
Fixed data race in threads found thread sanitizer
2023-07-09 20:53:27 +09:00
2e51908bec
Check FdEntity::Open() status correctly
...
FdEntity::Open() returns -errno on error, but FdManager::Open() only
checks if its ret is -1. This may lead to use '-errno' as pseudo fd
in next read or write, which would fail due to '-errno' is not in
fent map.
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
2023-07-05 08:50:12 +09:00
a8edbd8622
fix streamuplod not working ( #2204 )
...
fuse_main will do fork to daemonize. if ThreadPool is created before
this, worker thread will not be placed in child process.
Move ThreadPool Init to s3fs_init and destory to s3fs_destory.
Signed-off-by: liubingrun <liubr1@chinatelecom.cn>
2023-07-02 10:56:59 +09:00
03066ea29a
Fix extended attribute support when using FUSE-T ( #2201 )
...
* Add support for FUSE-T on macos
Signed-off-by: Alex Fishman <alex@fuse-t.org>
Signed-off-by: alex <alex@alex-NUC10.lan>
* Ignore value pointer when size is zero on setattrx
Signed-off-by: Alex Fishman <alex@fuse-t.org>
---------
Signed-off-by: Alex Fishman <alex@fuse-t.org>
Signed-off-by: alex <alex@alex-NUC10.lan>
2023-06-26 22:05:37 +09:00
6448c8f1a8
Protect FdEntity::physical_fd with fdent_lock ( #2194 )
...
* Protect FdEntity::physical_fd with fdent_lock
Found via ThreadSanitizer.
2023-06-25 16:43:15 +09:00
3b6688253f
Address cppcheck 2.10 warnings ( #2163 )
...
Disable newer cppcheck until we can diagnose this further.
References #2162 .
2023-06-25 16:04:16 +09:00
580775b47c
Removed unnecessary debug output
2023-06-07 20:48:50 -07:00
6ca5a24a7f
Fix two inconsistency issues between stat cache and cache file ( #2152 )
...
* Fix inconsistency between stat cache file and cache file
We unlock stat cache file too early in FdEntity::Open(), and would
truncate cache file and update stat cache file, so there's a window that
stat cache doesn't reflect cache file status.
Suggested-by: Takeshi Nakatani <ggtakec@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
* Mark pagelist as unloaded if cache file has been truncated
If cache file size doesn't match object size, the cache file might be
corrupted, so invalidate it and save new cache stat file.
Suggested-by: Takeshi Nakatani <ggtakec@gmail.com>
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
---------
Signed-off-by: Eryu Guan <eguan@linux.alibaba.com>
2023-05-30 18:39:50 +09:00
0d6b02090e
Revert "Update curl.cpp: reduce memory cache use ( #2157 )" ( #2170 )
...
This reverts commit 5b487f651a
2023-05-27 20:33:43 +09:00
5b487f651a
Update curl.cpp: reduce memory cache use ( #2157 )
...
flush the file and clean the page cache when filepart download done
2023-05-27 19:15:47 +09:00
1fc0e52dc3
Fixed mount point stat flag to be thread safe
2023-05-21 13:46:06 +09:00
59c3b26655
Fix clang-tidy 16 nits ( #2158 )
2023-05-14 01:59:26 +09:00
8296fe32cb
Directly and simplify requests in mount point checks ( #2155 )
2023-05-14 01:29:24 +09:00
ca9a257eec
Allow mount points without directory objects by compat_dir ( #2153 )
2023-05-07 09:15:27 +09:00
6d4bb59865
Corresponded to upload in case of calling release without flush ( #2150 )
2023-04-23 22:59:04 +09:00
9b75abfbe6
New option: bucket_size ( #2148 )
...
* Added bucket_size option
2023-04-23 14:04:38 +09:00
c4f95f14cb
Fixed a bug about attributes of mount point ( #2147 )
2023-04-15 14:34:39 +09:00
9c74014443
Fixed a bug in handling file names containing CR(0x1D) ( #2136 )
2023-03-26 13:19:16 +09:00
a25cb9e07a
Fixed a bug when reading a reduced file without flushing ( #2133 )
2023-03-26 11:45:21 +09:00
9648eba5bb
Changed to rename cache files when renaming large files ( #2135 )
2023-03-21 14:15:34 +09:00
18495c44aa
Added cache clearing on upload error ( #2127 )
2023-03-13 21:22:13 +09:00
4f22354aae
Fix an error to calculate no data page bytes ( #2122 )
...
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2023-03-11 17:23:56 +09:00
9d00b8d4a8
Fixed segmentation fault caused by file write failure ( #2123 )
2023-03-11 16:45:56 +09:00
526700f2de
Updated files for support ALPINE v3.17 ( #2116 )
2023-03-05 13:30:59 +09:00
a25a9450a6
Fixed to fail when the mount point directory is not existed ( #2114 )
2023-03-05 13:03:48 +09:00
f8a825e9b9
multipart upload id is converted by url encode ( #2097 )
2023-02-18 11:40:25 +09:00
dbddd762a9
Fixed token update process using ext-cred library ( #2101 )
2023-02-18 09:49:05 +09:00
e715b77307
Added the function to update mtime/ctime of the parent directory ( #2016 )
2023-02-12 17:59:40 +09:00
d1388ff446
Added proxy and proxy_cred_file option
2023-02-08 21:53:34 +09:00
38e8a830c9
Fixed a deadlock bug when specifying iam role(IAMv2)
2023-01-31 20:53:39 +09:00
7605c2e8fb
Updated help contents and man pages for streamupload and etc
2023-01-31 20:53:20 +09:00
16bc44948e
Retry BucketCheck containing directory paths
2023-01-09 21:18:47 +09:00
e4f85c1e08
Fixed renaming bug with SSE
2023-01-09 21:17:48 +09:00
222110e153
Fixed a bug about multipart upload with SSE
2023-01-07 09:12:55 +05:30
0ba49518e9
Make some methods and parameters const ( #2078 )
...
This requires making some locks mutable.
2023-01-04 20:23:39 +09:00
23a8124c51
Compatible with OpenSSL 3.0
2022-11-27 21:46:08 +09:00
d8e4e34b74
Fixed a warning by compiling on fedora 37
2022-11-23 21:57:31 +09:00
956eb77369
Updates for clang-tidy 15 ( #2058 )
2022-11-21 23:29:35 +09:00
465986e397
Fixed a bug when end of path for mount point is multi slash ( #2057 )
2022-11-19 15:07:27 +09:00
aec7efc3af
Updated comments in curl.cpp (related to #1827 ) ( #2052 )
2022-11-05 11:45:28 +09:00
30353f1a83
Remove unnecessary calls to dup ( #1827 )
...
These functions already use pread and keep track of an explicit offset
for the SHA-256 calculations.
2022-11-05 10:24:41 +09:00
2e77920943
Added support for xattr as POSIX ACL ( #2039 )
2022-10-22 21:48:02 +09:00
4a813aec42
Support the object under no directory object path by compat_dir ( #2023 )
2022-10-22 15:12:00 +09:00
4304ec63bb
Added stat information to the mount point ( #1964 )
2022-10-22 11:46:13 +09:00
6e89e69bba
Enabled to load shared library for Credential and Token ( #1927 )
2022-10-22 10:42:07 +09:00
9a4282208e
Improved fdcache_page performance
2022-09-25 17:14:35 +09:00
238fc0799e
Fixed statfs(free inodes) for macos
2022-09-25 17:10:37 +09:00
5b95a0fcb6
Fixed errors reported by cppcheck 2.9 ( #2038 )
2022-09-24 15:45:13 +09:00
2186317676
Include missing ctime header
...
This caused compilation errors on Fedora 36.
2022-09-01 22:40:16 +09:00
5ab1037094
Strictly reviewed the exclusive control of PseudoFdInfo class(additional change)
2022-08-08 07:51:49 +09:00
f6d7ff1084
Strictly reviewed the exclusive control of PseudoFdInfo class
2022-08-07 20:41:35 +09:00
3e242d0bad
Avoid C-style casts ( #2015 )
...
Prefer more precise C++-style casts.
2022-07-30 22:35:27 +09:00
c491fbeabc
Replace uses of lock_already_held flag with AutoLock::Type
2022-07-30 16:20:43 +09:00
e654e8ec8a
Removed const_cast in set_bucket
2022-07-30 16:18:52 +09:00
48e9e51f4f
Remove more unneeded headers identified by IWYU ( #2011 )
2022-07-30 12:06:47 +09:00
5a2172dc56
Fixed data race and memory leaks in PseudoFdInfo
2022-07-30 06:51:16 +09:00
07535ec3ec
Address clang-tidy warnings ( #2010 )
2022-07-29 22:00:07 +09:00
b8dd466988
Fixed data race in ThreadPoolMan
2022-07-29 13:24:09 +09:00
01a92476e6
Remove unneeded headers identified by IWYU ( #2007 )
2022-07-28 23:38:38 +09:00
3928a7e359
Remove more uses of const_cast ( #2006 )
...
Follows on to #2004 .
2022-07-28 23:37:15 +09:00
f9f614a474
Avoid mutating mybasename and mydirname parameters ( #2004 )
...
basename and dirname mutate their inputs but const_cast hid this
behavior. Also shuffle helpers to avoid unnecessary std::string.
Follows on to 404c284440
2022-07-28 23:34:01 +09:00
e30a5939d0
Fixed deadlock in S3fsCurl::DestroyCurlHandle
2022-07-28 13:47:27 +09:00
4b2f3fecb5
Set mtime/ctime/atime of all objects as nanosecond
2022-07-28 13:47:03 +09:00
ccfc119e45
Improved to avoid unnecessary head request
2022-07-27 21:51:28 +09:00
38b5018bab
Remove unneedd binary_function inheritance ( #1998 )
...
GCC 12 warns that C++17 removes this.
2022-07-24 22:41:58 +09:00
404c284440
Fixed race condition in dirname and basename call
2022-07-24 18:14:58 +09:00
e0655008b3
Protect pending_status in UploadPending ( #1992 )
...
This requires avoiding double-locking in RowFlush. References #1991 .
2022-07-22 23:30:04 +09:00
22f2392fca
Fixed bugs about stream upload
2022-07-19 21:29:56 +09:00
faddb4900f
Merged the code corresponding to the mknod fix( f11eb7d)
2022-07-17 22:20:45 +09:00
6ca7d5ec27
Fixed an error which reported by cppcheck 2.8
2022-07-17 22:20:45 +09:00
6f679a9e78
Reflect the result of the review in the code again
2022-07-17 22:20:45 +09:00
b0eeaa6679
Reflected the result of the review in the code
2022-07-17 22:20:45 +09:00
d22e1dc018
Add the stream upload which starts uploading parts before Flush
2022-07-17 22:20:45 +09:00
4e163b2888
Allow listing implicit directories ( #1986 )
...
This fixes an issue when using -o notsup_compat_dir flag, dating back
to its introduction 3ac39d61f8#927 .
References #1984 .
2022-07-17 01:54:29 +09:00
e7ed01b35f
Consume return code from get_object_attribute ( #1976 )
...
Found via clang-analyzer.
2022-07-09 16:45:23 +09:00
3f55c98a3f
Fixed a bug when the pool of curl handler exceeds the upper limit
2022-07-08 21:14:29 +09:00
f11eb7d69b
Fixed a bug that regular files could not be created by mknod
2022-06-29 16:56:19 +09:00
73b49c1038
Fixed a bug that regular files could not be created by mknod
2022-06-29 16:56:19 +09:00
8b90cd6ba1
Enable notsup_compat_dir by default ( #1970 )
...
Few applications create the dir_$folder$ objects and users can enable
compat_dir if required. This commit reduces readdir latency by 33%.
Also remove notsup_compat_dir from tests since these directories are
never created. Fixes #927 . References #1643 .
2022-06-28 07:56:06 +09:00
b52f916af6
Avoid extended initializer lists in C++03 ( #1960 )
...
C++11 introduces these.
2022-06-12 23:52:16 +09:00
ec7810f08e
Fixed a bug could not change the mode while the file was opened
2022-06-12 13:48:42 +09:00
904682b856
Fix double-iteration in curl_slist_remove ( #1951 )
...
Also backfill tests. Fixes #1948 .
2022-05-27 22:56:20 +09:00
92fd5bc3e1
Fixed errors reported by cppcheck 2.8 ( #1949 )
2022-05-26 01:48:53 +09:00
d75c6d6538
Changed the message level(ERR to INFO) of skip cleanup cache ( #1946 )
2022-05-22 14:36:56 +09:00
5c57e17b77
Enable noobj_cache by default ( #1922 )
...
This should improve performance in many situations. s3fs already
enables the stat cache by default so memorizing noobj makes this more
consistent. Fixes #1901 .
2022-03-12 16:57:31 +09:00
deaa85c40e
Document glacier_ir storage class ( #1921 )
...
Announcement:
https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-s3-glacier-instant-retrieval-storage-class/
This does not require code changes due to
bbcccd6e98
2022-03-12 16:20:38 +09:00
d842d45b2b
Fixed a bug about truncation for shrinking file
2022-03-02 22:41:10 +09:00
684ced5a41
Changed handling the credential in S3fsCred more robust
2022-03-02 22:39:15 +09:00
afb0897553
Typos
2022-02-24 19:15:00 +09:00
8a5c4306f5
Preserve sub-second precision where possible ( #1915 )
2022-02-23 23:58:51 +09:00
08adffd2fe
Fix typos ( #1916 )
2022-02-23 23:31:52 +09:00
81ed2bd91e
Propagate deferred exit status from main ( #1912 )
...
Previously s3fs always returned zero when the bucket did not mount.
Fixes #1911 .
2022-02-23 10:09:12 +09:00
a716c72d37
Update notsup_compat_dir in --help
2022-02-21 19:29:15 +09:00
d31cbda7b6
Fixed a bug about checking credential
2022-02-19 23:22:15 +09:00
b64dc7749c
Moved parameter analysis processing to S3fsCred class
2022-02-19 17:23:40 +09:00
839a33de49
Fixed not to call Flush even if the file size is increased ( #1887 )
...
Changed s3fs_truncate function.
This change reduces the number of file uploads if the file size is changed.
On macOS, I have found that the truncate call when "size=0" cannot reflect the file size.(This reason is not understood...)
To avoid this, only when "size=0", the flush method is called as before.
Other than that, I found a bug in FdEntity::Open() and fixed it.
Fixes #1875 .
2022-02-15 21:29:07 +09:00
4dfe2bfdd7
Include climits to support musl libc
...
PATH_MAX constant is not visible from any of currently included header
files in system with musl libc, where compilation fails with an error
below. The constant is defined in limits.h which is directly include via
climits header file.
fdcache.cpp: In static member function 'static FILE* FdManager::MakeTempFile()':
fdcache.cpp:381:14: error: 'PATH_MAX' was not declared in this scope
381 | char cfn[PATH_MAX];
| ^~~~~~~~
Fixes: d67b83e671
2022-02-14 09:19:30 +09:00
1678803566
Added S3fsCred class and moved Credential related processing in it
2022-02-13 21:38:30 +09:00
d7e929e0a8
Fixed some Github Actions errors. ( #1886 )
...
- Fix knownConditionTrueFalse cppcheck(2.7) error on MacOS
- Fixed package installing failure of appstream download on centos8
2022-02-13 14:23:35 +09:00
9224f792f0
Use CLOCK_REALTIME for UTIME_NOW ( #1881 )
...
Previously s3fs_utimens used CLOCK_MONOTONIC_COARSE which was not
1970-based. Found via pjdfstest. References #1589 .
2022-01-30 22:19:15 +09:00
f6ed972926
Always flush open files with O_CREAT flag ( #1879 )
...
Previously s3fs only created files that had dirty data and not those
with zero-bytes. Regression from
771bbfeac5#1013 . Found
via pjdfstest. References #1589 .
2022-01-30 22:02:37 +09:00
0c75a63184
Preserve sub-second precision with utimens ( #1880 )
...
Found via pjdfstest. References #1589 .
2022-01-30 21:45:51 +09:00
e452ef3940
Fixed the fault tolerance when time stamp getting fails
2022-01-30 18:31:36 +09:00
cd5a69b9eb
Handle UTIME_NOW and UTIME_OMIT special values ( #1868 )
...
FUSE 3 will require this behavior. References #1159 .
2022-01-29 11:35:37 +09:00
662882d2f0
Always call clock_gettime(2) ( #1871 )
...
e01ded9e27https://stackoverflow.com/a/39801564 . Also remove fallback to
time(3) which loses precision.
2022-01-25 08:36:27 +09:00
de0c87c801
Convert S3FS_LOW_LOGPRN from a macro to a function ( #1869 )
...
This shrinks the binary size from 770 to 540 KB and reduces compile
times.
2022-01-23 23:10:09 +09:00
41aaa4184f
Avoid double setting values in statfs
2022-01-23 21:49:51 +09:00
581f5c0356
Move strptime polyfill to string_util
2022-01-23 21:49:51 +09:00
e5f6f112db
Fix typo
2022-01-23 21:49:51 +09:00
b3cef944b2
Fix test_page_list_SOURCES has no if MSYS clause
2022-01-23 21:49:51 +09:00
6edb6067f3
Remove strcasestr polyfill
2022-01-23 21:49:51 +09:00
b2c659c0a6
Disable compiling polyfills in not MSYS2 env
2022-01-23 21:49:51 +09:00
3ac9f571f5
Use std::get_time instead in strptime polyfill
2022-01-23 21:49:51 +09:00
19303a546e
Fix the statfs issue, using f_frsize instead
2022-01-23 21:49:51 +09:00
6370e150dd
Disable features that causes problems on Windows
2022-01-23 21:49:51 +09:00
b14e39815b
Use polyfills in MSYS2 environment
2022-01-23 21:49:51 +09:00
6aaf9433a5
Add polyfills for MSYS2 environment
2022-01-23 21:49:51 +09:00
93d1c30d4d
Use XML parsing with PUT HTTP 200 responses ( #1858 )
...
This works around the missing strcasestr on win32. References #728 .
2022-01-14 16:10:22 +09:00
25012f3839
Fix typo in -o enable_unsigned_payload
2022-01-12 22:50:49 +09:00
e734763002
Remove createbucket option ( #1841 )
...
AWS CLI can do this. Fixes #1840 .
2022-01-05 01:59:31 +09:00
85ca2a3e45
fix mixupload return EntityTooSmall while a copypart is less than 5MB after split ( #1809 )
...
* fix mixupload return EntityTooSmall while a copypart is less than 5MB after split
* fix possible part exceeds 5GB when multipart_copy_size is set to 5120MB
* Update curl.cpp
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
2021-11-27 16:53:26 +09:00
3cf00626a2
Add option to allow unsigned payloads ( #1801 )
...
This reduces CPU usage of sigv4. This reduces test run-time by 7
seconds per flag.
2021-11-01 23:33:55 +09:00
7273d561f5
Added exclusive control of statc variables in s3fs xml parser
2021-10-27 08:18:19 +09:00
78126aea0b
Added exclusive control of statc variables in s3fs xml parser
2021-10-27 08:18:19 +09:00
7892eee207
Fixed a bug that copied without considering the length of xmlChar
2021-10-27 08:18:19 +09:00
023aaf7dff
Fixed wrong stat of cache after new creation file
...
And added a test for stat of cache after new creation file
2021-10-17 16:10:14 +09:00
2f412804e2
Fixed forgetting to clear the dirty flag for meta information
...
Addressed an error in macos cpp check
2021-10-15 22:54:55 +09:00
be0b17329a
Fix wrong function name in log message ( #1774 )
2021-10-10 11:08:32 +09:00
b4edad86d6
remove Expect: 100-continue header when requesting an IMDSv2 access token
2021-09-09 08:12:36 +09:00
9d1552a54e
fix IAM role retrieval from IMDSv2
...
AWS IMDSv2 support was added in #1462 , but the implementation did not
cover the addional IMDS access that occurs with the iam_role=auto
configuration. This change implements IMDSv2 support for the IMDS
call to determine the instance's role name.
See also
https://stackoverflow.com/questions/69031023/how-to-make-s3fs-use-imds-v2-when-mounting-s3-buckets-from-ec2-instance
2021-09-03 20:36:34 +09:00
47ebfcc60a
Consume return value from curl_easy_setopt ( #1759 )
...
Found via Coverity.
2021-09-02 08:07:06 +09:00
beecf32dff
fclose(FILE*) instead of close(fileno(FILE*)) ( #1758 )
...
This is the same thing but confuses Coverity.
2021-09-01 19:41:55 +09:00
57b2e4a4f1
Fix 32-bit compilation issues ( #1757 )
2021-08-31 19:36:02 +09:00
48817d849f
Require explicit length in s3fs_decode64 ( #1755 )
...
This is available from std::string::size in callers.
2021-08-31 09:22:10 +09:00
d9f2d17040
1. fix RowFlush can not upload last part smaller than 5MB using NoCacheMultipartPost; ( #1753 )
...
2. fix deadlock in UploadPendingMeta
2021-08-31 00:41:47 +09:00
cd98afdd7b
Do not NUL terminate base64 decoded output ( #1752 )
...
This is binary data and must use the explicit length.
2021-08-31 00:15:47 +09:00
dac6885fb0
Don't over-allocate in base64 encoding and decoding ( #1751 )
2021-08-30 00:03:10 +09:00
fcd180891b
fix misuse of IsUploading ( #1747 )
...
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
2021-08-29 23:41:02 +09:00
a868c0656e
Changed etaglist_t from string list to new structure etagpairs list
2021-08-16 09:27:12 +09:00
15e89b78de
Add a partial page_list unit test ( #1735 )
2021-08-04 07:36:32 +09:00
66006ba48d
Add dedicated upper- and lower-case hex functions ( #1734 )
...
This makes the call sites more readable than a boolean parameter.
2021-08-04 07:28:51 +09:00
18e9c62087
Make string constants read-only const ( #1733 )
...
This removes some global constructors. Also use a consistent ALL_CAPS
style.
2021-08-03 00:10:27 +09:00
34f89e5936
Added fake_diskfree option to deceive free disk space for test
2021-08-02 11:04:42 +09:00
e1f3b9d8c1
Add support for AWS-style environment variables ( #1729 )
...
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.
The old environment variables are still supported, but they are
deprecated and no longer documented.
Close #1708
2021-07-26 23:29:45 +09:00
d3278f4886
Loosen CheckBucket to check only the bucket ( #1728 )
...
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460 . Fixes #1687 .
2021-07-25 12:29:00 +09:00
Eryu Guan
Takeshi Nakatani
Andrew Gaul
LiuBingrun
Alex Fishman
Eryu Guan
Tan Guofu
Ottavia Balducci
Leon
Garen Chan
Carsten Grohmann
Petr Vaněk
Naoki Ikeguchi
LiuBingrun
Kamil Jakrzewski
Noah Meyerhans