octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-12-23 09:18:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,217 Commits 1 Branch 23 Tags 13 MiB
e8f1e3473c
Commit Graph

1249 Commits

Author SHA1 Message Date
Takeshi Nakatani
ca9a257eec
Allow mount points without directory objects by compat_dir (#2153) 2023-05-07 09:15:27 +09:00
Takeshi Nakatani
6d4bb59865
Corresponded to upload in case of calling release without flush (#2150) 2023-04-23 22:59:04 +09:00
Ottavia Balducci
9b75abfbe6
New option: bucket_size (#2148) 
* Added bucket_size option
 2023-04-23 14:04:38 +09:00
Takeshi Nakatani
c4f95f14cb
Fixed a bug about attributes of mount point (#2147) 2023-04-15 14:34:39 +09:00
Takeshi Nakatani
9c74014443
Fixed a bug in handling file names containing CR(0x1D) (#2136) 2023-03-26 13:19:16 +09:00
Takeshi Nakatani
a25cb9e07a
Fixed a bug when reading a reduced file without flushing (#2133) 2023-03-26 11:45:21 +09:00
Takeshi Nakatani
9648eba5bb
Changed to rename cache files when renaming large files (#2135) 2023-03-21 14:15:34 +09:00
Takeshi Nakatani
18495c44aa
Added cache clearing on upload error (#2127) 2023-03-13 21:22:13 +09:00
Leon
4f22354aae
Fix an error to calculate no data page bytes (#2122) 
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2023-03-11 17:23:56 +09:00
Garen Chan
9d00b8d4a8
Fixed segmentation fault caused by file write failure (#2123) 2023-03-11 16:45:56 +09:00
Takeshi Nakatani
526700f2de
Updated files for support ALPINE v3.17 (#2116) 2023-03-05 13:30:59 +09:00
Takeshi Nakatani
a25a9450a6
Fixed to fail when the mount point directory is not existed (#2114) 2023-03-05 13:03:48 +09:00
Takeshi Nakatani
f8a825e9b9
multipart upload id is converted by url encode (#2097) 2023-02-18 11:40:25 +09:00
Takeshi Nakatani
dbddd762a9
Fixed token update process using ext-cred library (#2101) 2023-02-18 09:49:05 +09:00
Takeshi Nakatani
e715b77307
Added the function to update mtime/ctime of the parent directory (#2016) 2023-02-12 17:59:40 +09:00
Takeshi Nakatani
d1388ff446 Added proxy and proxy_cred_file option 2023-02-08 21:53:34 +09:00
Takeshi Nakatani
38e8a830c9 Fixed a deadlock bug when specifying iam role(IAMv2) 2023-01-31 20:53:39 +09:00
Takeshi Nakatani
7605c2e8fb Updated help contents and man pages for streamupload and etc 2023-01-31 20:53:20 +09:00
Takeshi Nakatani
16bc44948e Retry BucketCheck containing directory paths 2023-01-09 21:18:47 +09:00
Takeshi Nakatani
e4f85c1e08 Fixed renaming bug with SSE 2023-01-09 21:17:48 +09:00
Takeshi Nakatani
222110e153 Fixed a bug about multipart upload with SSE 2023-01-07 09:12:55 +05:30
Andrew Gaul
0ba49518e9
Make some methods and parameters const (#2078) 
This requires making some locks mutable.
 2023-01-04 20:23:39 +09:00
Takeshi Nakatani
23a8124c51 Compatible with OpenSSL 3.0 2022-11-27 21:46:08 +09:00
Takeshi Nakatani
d8e4e34b74 Fixed a warning by compiling on fedora 37 2022-11-23 21:57:31 +09:00
Andrew Gaul
956eb77369
Updates for clang-tidy 15 (#2058) 2022-11-21 23:29:35 +09:00
Takeshi Nakatani
465986e397
Fixed a bug when end of path for mount point is multi slash (#2057) 2022-11-19 15:07:27 +09:00
Takeshi Nakatani
aec7efc3af
Updated comments in curl.cpp (related to #1827) (#2052) 2022-11-05 11:45:28 +09:00
Andrew Gaul
30353f1a83
Remove unnecessary calls to dup (#1827) 
These functions already use pread and keep track of an explicit offset
for the SHA-256 calculations.
 2022-11-05 10:24:41 +09:00
Takeshi Nakatani
2e77920943
Added support for xattr as POSIX ACL (#2039) 2022-10-22 21:48:02 +09:00
Takeshi Nakatani
4a813aec42
Support the object under no directory object path by compat_dir (#2023) 2022-10-22 15:12:00 +09:00
Takeshi Nakatani
4304ec63bb
Added stat information to the mount point (#1964) 2022-10-22 11:46:13 +09:00
Takeshi Nakatani
6e89e69bba
Enabled to load shared library for Credential and Token (#1927) 2022-10-22 10:42:07 +09:00
Takeshi Nakatani
9a4282208e Improved fdcache_page performance 2022-09-25 17:14:35 +09:00
Takeshi Nakatani
238fc0799e Fixed statfs(free inodes) for macos 2022-09-25 17:10:37 +09:00
Takeshi Nakatani
5b95a0fcb6
Fixed errors reported by cppcheck 2.9 (#2038) 2022-09-24 15:45:13 +09:00
Andrew Gaul
2186317676 Include missing ctime header 
This caused compilation errors on Fedora 36.
 2022-09-01 22:40:16 +09:00
Takeshi Nakatani
5ab1037094 Strictly reviewed the exclusive control of PseudoFdInfo class(additional change) 2022-08-08 07:51:49 +09:00
Takeshi Nakatani
f6d7ff1084 Strictly reviewed the exclusive control of PseudoFdInfo class 2022-08-07 20:41:35 +09:00
Andrew Gaul
3e242d0bad
Avoid C-style casts (#2015) 
Prefer more precise C++-style casts.
 2022-07-30 22:35:27 +09:00
Takeshi Nakatani
c491fbeabc Replace uses of lock_already_held flag with AutoLock::Type 2022-07-30 16:20:43 +09:00
Takeshi Nakatani
e654e8ec8a Removed const_cast in set_bucket 2022-07-30 16:18:52 +09:00
Andrew Gaul
48e9e51f4f
Remove more unneeded headers identified by IWYU (#2011) 2022-07-30 12:06:47 +09:00
Takeshi Nakatani
5a2172dc56 Fixed data race and memory leaks in PseudoFdInfo 2022-07-30 06:51:16 +09:00
Andrew Gaul
07535ec3ec
Address clang-tidy warnings (#2010) 2022-07-29 22:00:07 +09:00
Takeshi Nakatani
b8dd466988 Fixed data race in ThreadPoolMan 2022-07-29 13:24:09 +09:00
Andrew Gaul
01a92476e6
Remove unneeded headers identified by IWYU (#2007) 2022-07-28 23:38:38 +09:00
Andrew Gaul
3928a7e359
Remove more uses of const_cast (#2006) 
Follows on to #2004.
 2022-07-28 23:37:15 +09:00
Andrew Gaul
f9f614a474
Avoid mutating mybasename and mydirname parameters (#2004) 
basename and dirname mutate their inputs but const_cast hid this
behavior.  Also shuffle helpers to avoid unnecessary std::string.
Follows on to 404c284440.
 2022-07-28 23:34:01 +09:00
Takeshi Nakatani
e30a5939d0 Fixed deadlock in S3fsCurl::DestroyCurlHandle 2022-07-28 13:47:27 +09:00
Takeshi Nakatani
4b2f3fecb5 Set mtime/ctime/atime of all objects as nanosecond 2022-07-28 13:47:03 +09:00
Takeshi Nakatani
ccfc119e45 Improved to avoid unnecessary head request 2022-07-27 21:51:28 +09:00
Andrew Gaul
38b5018bab
Remove unneedd binary_function inheritance (#1998) 
GCC 12 warns that C++17 removes this.
 2022-07-24 22:41:58 +09:00
Takeshi Nakatani
404c284440 Fixed race condition in dirname and basename call 2022-07-24 18:14:58 +09:00
Andrew Gaul
e0655008b3
Protect pending_status in UploadPending (#1992) 
This requires avoiding double-locking in RowFlush.  References #1991.
 2022-07-22 23:30:04 +09:00
Takeshi Nakatani
22f2392fca Fixed bugs about stream upload 2022-07-19 21:29:56 +09:00
Takeshi Nakatani
faddb4900f Merged the code corresponding to the mknod fix(f11eb7d) 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
6ca7d5ec27 Fixed an error which reported by cppcheck 2.8 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
6f679a9e78 Reflect the result of the review in the code again 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
b0eeaa6679 Reflected the result of the review in the code 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
d22e1dc018 Add the stream upload which starts uploading parts before Flush 2022-07-17 22:20:45 +09:00
Andrew Gaul
4e163b2888
Allow listing implicit directories (#1986) 
This fixes an issue when using -o notsup_compat_dir flag, dating back
to its introduction 3ac39d61f8.  The new
default exposed this in my local testing but a test runner bug masked
a test failure in test_external_directory_creation.  References #927.
References #1984.
 2022-07-17 01:54:29 +09:00
Andrew Gaul
e7ed01b35f
Consume return code from get_object_attribute (#1976) 
Found via clang-analyzer.
 2022-07-09 16:45:23 +09:00
Takeshi Nakatani
3f55c98a3f Fixed a bug when the pool of curl handler exceeds the upper limit 2022-07-08 21:14:29 +09:00
Takeshi Nakatani
f11eb7d69b Fixed a bug that regular files could not be created by mknod 2022-06-29 16:56:19 +09:00
Takeshi Nakatani
73b49c1038 Fixed a bug that regular files could not be created by mknod 2022-06-29 16:56:19 +09:00
Andrew Gaul
8b90cd6ba1
Enable notsup_compat_dir by default (#1970) 
Few applications create the dir_$folder$ objects and users can enable
compat_dir if required.  This commit reduces readdir latency by 33%.
Also remove notsup_compat_dir from tests since these directories are
never created.  Fixes #927.  References #1643.
 2022-06-28 07:56:06 +09:00
Andrew Gaul
b52f916af6
Avoid extended initializer lists in C++03 (#1960) 
C++11 introduces these.
 2022-06-12 23:52:16 +09:00
Takeshi Nakatani
ec7810f08e Fixed a bug could not change the mode while the file was opened 2022-06-12 13:48:42 +09:00
Andrew Gaul
904682b856
Fix double-iteration in curl_slist_remove (#1951) 
Also backfill tests.  Fixes #1948.
 2022-05-27 22:56:20 +09:00
Takeshi Nakatani
92fd5bc3e1
Fixed errors reported by cppcheck 2.8 (#1949) 2022-05-26 01:48:53 +09:00
Takeshi Nakatani
d75c6d6538
Changed the message level(ERR to INFO) of skip cleanup cache (#1946) 2022-05-22 14:36:56 +09:00
Andrew Gaul
5c57e17b77
Enable noobj_cache by default (#1922) 
This should improve performance in many situations.  s3fs already
enables the stat cache by default so memorizing noobj makes this more
consistent.  Fixes #1901.
 2022-03-12 16:57:31 +09:00
Andrew Gaul
deaa85c40e
Document glacier_ir storage class (#1921) 
Announcement:
https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-s3-glacier-instant-retrieval-storage-class/

This does not require code changes due to
bbcccd6e98.
 2022-03-12 16:20:38 +09:00
Takeshi Nakatani
d842d45b2b Fixed a bug about truncation for shrinking file 2022-03-02 22:41:10 +09:00
Takeshi Nakatani
684ced5a41 Changed handling the credential in S3fsCred more robust 2022-03-02 22:39:15 +09:00
Carsten Grohmann
afb0897553 Typos 2022-02-24 19:15:00 +09:00
Andrew Gaul
8a5c4306f5
Preserve sub-second precision where possible (#1915) 2022-02-23 23:58:51 +09:00
Andrew Gaul
08adffd2fe
Fix typos (#1916) 2022-02-23 23:31:52 +09:00
Andrew Gaul
81ed2bd91e
Propagate deferred exit status from main (#1912) 
Previously s3fs always returned zero when the bucket did not mount.
Fixes #1911.
 2022-02-23 10:09:12 +09:00
Carsten Grohmann
a716c72d37 Update notsup_compat_dir in --help 2022-02-21 19:29:15 +09:00
Takeshi Nakatani
d31cbda7b6 Fixed a bug about checking credential 2022-02-19 23:22:15 +09:00
Takeshi Nakatani
b64dc7749c Moved parameter analysis processing to S3fsCred class 2022-02-19 17:23:40 +09:00
Takeshi Nakatani
839a33de49
Fixed not to call Flush even if the file size is increased (#1887) 
Changed s3fs_truncate function.
This change reduces the number of file uploads if the file size is changed.

On macOS, I have found that the truncate call when "size=0" cannot reflect the file size.(This reason is not understood...)
To avoid this, only when "size=0", the flush method is called as before.

Other than that, I found a bug in FdEntity::Open() and fixed it.

Fixes #1875.
 2022-02-15 21:29:07 +09:00
Petr Vaněk
4dfe2bfdd7 Include climits to support musl libc 
PATH_MAX constant is not visible from any of currently included header
files in system with musl libc, where compilation fails with an error
below. The constant is defined in limits.h which is directly include via
climits header file.

fdcache.cpp: In static member function 'static FILE* FdManager::MakeTempFile()':
fdcache.cpp:381:14: error: 'PATH_MAX' was not declared in this scope
  381 |     char cfn[PATH_MAX];
      |              ^~~~~~~~

Fixes: d67b83e671 ("Allow configuration for temporary files directory")
 2022-02-14 09:19:30 +09:00
Takeshi Nakatani
1678803566 Added S3fsCred class and moved Credential related processing in it 2022-02-13 21:38:30 +09:00
Takeshi Nakatani
d7e929e0a8
Fixed some Github Actions errors. (#1886) 
- Fix knownConditionTrueFalse cppcheck(2.7) error on MacOS
- Fixed package installing failure of appstream download on centos8
 2022-02-13 14:23:35 +09:00
Andrew Gaul
9224f792f0
Use CLOCK_REALTIME for UTIME_NOW (#1881) 
Previously s3fs_utimens used CLOCK_MONOTONIC_COARSE which was not
1970-based.  Found via pjdfstest.  References #1589.
 2022-01-30 22:19:15 +09:00
Andrew Gaul
f6ed972926
Always flush open files with O_CREAT flag (#1879) 
Previously s3fs only created files that had dirty data and not those
with zero-bytes.  Regression from
771bbfeac5.  References #1013.  Found
via pjdfstest.  References #1589.
 2022-01-30 22:02:37 +09:00
Andrew Gaul
0c75a63184
Preserve sub-second precision with utimens (#1880) 
Found via pjdfstest.  References #1589.
 2022-01-30 21:45:51 +09:00
Takeshi Nakatani
e452ef3940 Fixed the fault tolerance when time stamp getting fails 2022-01-30 18:31:36 +09:00
Andrew Gaul
cd5a69b9eb
Handle UTIME_NOW and UTIME_OMIT special values (#1868) 
FUSE 3 will require this behavior.  References #1159.
 2022-01-29 11:35:37 +09:00
Andrew Gaul
662882d2f0
Always call clock_gettime(2) (#1871) 
e01ded9e27 introduced this compatibility
shim but macOS 10.12 (2016) added this:
https://stackoverflow.com/a/39801564 .  Also remove fallback to
time(3) which loses precision.
 2022-01-25 08:36:27 +09:00
Andrew Gaul
de0c87c801
Convert S3FS_LOW_LOGPRN from a macro to a function (#1869) 
This shrinks the binary size from 770 to 540 KB and reduces compile
times.
 2022-01-23 23:10:09 +09:00
Naoki Ikeguchi
41aaa4184f Avoid double setting values in statfs 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
581f5c0356 Move strptime polyfill to string_util 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
e5f6f112db Fix typo 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
b3cef944b2 Fix test_page_list_SOURCES has no if MSYS clause 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
6edb6067f3 Remove strcasestr polyfill 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
b2c659c0a6 Disable compiling polyfills in not MSYS2 env 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
3ac9f571f5 Use std::get_time instead in strptime polyfill 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
19303a546e Fix the statfs issue, using f_frsize instead 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
6370e150dd Disable features that causes problems on Windows 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
b14e39815b Use polyfills in MSYS2 environment 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
6aaf9433a5 Add polyfills for MSYS2 environment 2022-01-23 21:49:51 +09:00
Andrew Gaul
93d1c30d4d
Use XML parsing with PUT HTTP 200 responses (#1858) 
This works around the missing strcasestr on win32.  References #728.
 2022-01-14 16:10:22 +09:00
Andrew Gaul
25012f3839 Fix typo in -o enable_unsigned_payload 2022-01-12 22:50:49 +09:00
Andrew Gaul
e734763002
Remove createbucket option (#1841) 
AWS CLI can do this.  Fixes #1840.
 2022-01-05 01:59:31 +09:00
LiuBingrun
85ca2a3e45
fix mixupload return EntityTooSmall while a copypart is less than 5MB after split (#1809) 
* fix  mixupload return EntityTooSmall while a copypart is less than 5MB after split
* fix possible part exceeds 5GB when multipart_copy_size is set to 5120MB
* Update curl.cpp
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
 2021-11-27 16:53:26 +09:00
Andrew Gaul
3cf00626a2
Add option to allow unsigned payloads (#1801) 
This reduces CPU usage of sigv4.  This reduces test run-time by 7
seconds per flag.
 2021-11-01 23:33:55 +09:00
Takeshi Nakatani
7273d561f5 Added exclusive control of statc variables in s3fs xml parser 2021-10-27 08:18:19 +09:00
Takeshi Nakatani
78126aea0b Added exclusive control of statc variables in s3fs xml parser 2021-10-27 08:18:19 +09:00
Takeshi Nakatani
7892eee207 Fixed a bug that copied without considering the length of xmlChar 2021-10-27 08:18:19 +09:00
Takeshi Nakatani
023aaf7dff Fixed wrong stat of cache after new creation file 
And added a test for stat of cache after new creation file
 2021-10-17 16:10:14 +09:00
Takeshi Nakatani
2f412804e2 Fixed forgetting to clear the dirty flag for meta information 
Addressed an error in macos cpp check
 2021-10-15 22:54:55 +09:00
Carsten Grohmann
be0b17329a
Fix wrong function name in log message (#1774) 2021-10-10 11:08:32 +09:00
Kamil Jakrzewski
b4edad86d6 remove Expect: 100-continue header when requesting an IMDSv2 access token 2021-09-09 08:12:36 +09:00
Noah Meyerhans
9d1552a54e fix IAM role retrieval from IMDSv2 
AWS IMDSv2 support was added in #1462, but the implementation did not
cover the addional IMDS access that occurs with the iam_role=auto
configuration.  This change implements IMDSv2 support for the IMDS
call to determine the instance's role name.

See also
https://stackoverflow.com/questions/69031023/how-to-make-s3fs-use-imds-v2-when-mounting-s3-buckets-from-ec2-instance
 2021-09-03 20:36:34 +09:00
Andrew Gaul
47ebfcc60a
Consume return value from curl_easy_setopt (#1759) 
Found via Coverity.
 2021-09-02 08:07:06 +09:00
Andrew Gaul
beecf32dff
fclose(FILE*) instead of close(fileno(FILE*)) (#1758) 
This is the same thing but confuses Coverity.
 2021-09-01 19:41:55 +09:00
Andrew Gaul
57b2e4a4f1
Fix 32-bit compilation issues (#1757) 2021-08-31 19:36:02 +09:00
Andrew Gaul
48817d849f
Require explicit length in s3fs_decode64 (#1755) 
This is available from std::string::size in callers.
 2021-08-31 09:22:10 +09:00
VVoidV
d9f2d17040
1. fix RowFlush can not upload last part smaller than 5MB using NoCacheMultipartPost; (#1753) 
2. fix deadlock in UploadPendingMeta
 2021-08-31 00:41:47 +09:00
Andrew Gaul
cd98afdd7b
Do not NUL terminate base64 decoded output (#1752) 
This is binary data and must use the explicit length.
 2021-08-31 00:15:47 +09:00
Andrew Gaul
dac6885fb0
Don't over-allocate in base64 encoding and decoding (#1751) 2021-08-30 00:03:10 +09:00
VVoidV
fcd180891b
fix misuse of IsUploading (#1747) 
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
 2021-08-29 23:41:02 +09:00
Takeshi Nakatani
a868c0656e Changed etaglist_t from string list to new structure etagpairs list 2021-08-16 09:27:12 +09:00
Andrew Gaul
15e89b78de
Add a partial page_list unit test (#1735) 2021-08-04 07:36:32 +09:00
Andrew Gaul
66006ba48d
Add dedicated upper- and lower-case hex functions (#1734) 
This makes the call sites more readable than a boolean parameter.
 2021-08-04 07:28:51 +09:00
Andrew Gaul
18e9c62087
Make string constants read-only const (#1733) 
This removes some global constructors.  Also use a consistent ALL_CAPS
style.
 2021-08-03 00:10:27 +09:00
Takeshi Nakatani
34f89e5936 Added fake_diskfree option to deceive free disk space for test 2021-08-02 11:04:42 +09:00
Carsten Grohmann
e1f3b9d8c1
Add support for AWS-style environment variables (#1729) 
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.

The old environment variables are still supported, but they are
deprecated and no longer documented.

Close #1708
 2021-07-26 23:29:45 +09:00
Andrew Gaul
d3278f4886
Loosen CheckBucket to check only the bucket (#1728) 
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460.  Fixes #1687.
 2021-07-25 12:29:00 +09:00
Andrew Gaul
77f0b75d2f
Add jitter to avoid thundering herd (#1723) 2021-07-25 10:22:19 +09:00
Takeshi Nakatani
199b3d4709 Fixed a bug in disk free space calculation 2021-07-18 14:52:11 +09:00
Andrew Gaul
7890989cbb
Address cppcheck 2.5 warnings (#1720) 
This slightly simplifies memory management.
 2021-07-14 22:18:09 +09:00
Takeshi Nakatani
945cc2ac54 Added UntreatedParts class instead of untreated upload info members in PseudoFdInfo 2021-07-12 21:45:29 +09:00
Takeshi Nakatani
c30acbbf90 Splited some methods of FdEntity class by uploading mode 
- Splited FdEntity::Write method by uploading mode
- Splited FdEntity::RowFlush method by uplading mode
 2021-07-10 17:24:34 +09:00
Martynov Maxim
881025cc9e
Add initiator pid, uid and gid to a debug log (#1716) 2021-07-10 16:23:20 +09:00
Takeshi Nakatani
4cc210c5ab Prevents the etag string buffer from being destroyed during use 
and absorbed MPPART_INFO structure into filepart structure
 2021-07-03 22:42:18 +09:00
Andrew Gaul
fe44355d25
Report s3fs space as 18.4 EB (#1709) 
256 TB probably seemed large in 2007 but in 2021 it is small enough to
confuse some users.
 2021-07-02 07:56:05 +09:00
Andrew Gaul
487df27008
Return EREMOTE when reading Glacier objects (#1705) 
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466.
 2021-06-30 09:25:36 +09:00
Andrew Gaul
1965916f7a
Convert storage class values to uppercase (#1704) 
AWS requires uppercase values.  Fixes a regression from
bbcccd6e98.  References #1613.
 2021-06-30 09:03:31 +09:00
Andrew Gaul
8948eded09
Address clang-tidy warnings (#1703) 2021-06-30 08:42:44 +09:00
Andrew Gaul
2f59cb5a0a
Remove unneeded volatile qualifiers (#1702) 
We should use proper locking instead.  Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
 2021-06-30 08:20:44 +09:00
Andrew Gaul
f505c8224e
Explicitly specify lock type (#1701) 
This makes it more clear and type-safe if the caller already has the
lock.  Follows on to 84174c560d.
 2021-06-27 15:15:48 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Takeshi Nakatani
f9e80f995d Fixed a bug about rename existing file of different sizes by mpcopy 2021-06-26 00:04:58 +09:00
Jan Stastny
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating (#1681) 
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/.

Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-06-25 20:26:12 +09:00
Takeshi Nakatani
6edf3d6427 Updated to clearly output message about the file descriptor type(pseudo/physical) 2021-06-25 19:18:08 +09:00
Andrew Gaul
20281737b1
Set explicit Content-Length: 0 when initiating MPU (#1696) 
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661.
 2021-06-25 18:36:40 +09:00
Takeshi Nakatani
5b6684ca19 Fixed a bug utimens is calling before flush 2021-06-21 17:20:24 +09:00
Carsten Grohmann
c4ac923b4c
Ensure NUL-terminated result after strncpy (#1694) 
Long symlinks may cause that the result buffer is filled and not proper
terminated with a null byte.
 2021-06-21 08:08:56 +09:00
Takeshi Nakatani
cc022a68f4 Fixed a bug in s3fs_write 2021-06-20 23:13:44 +09:00
Takeshi Nakatani
858562ed53 Fixed comparison warning for size_t variable 2021-06-20 16:53:25 +09:00
Andrew Gaul
b8724425d3
Require explicit base parameter in cvt_strtoofft (#1683) 
Also convert most callers of cvt_strtoofft to base 10 which avoid the
magical behavior of interpreting a leading 0 as octal.
References #1682.
 2021-06-20 11:00:15 +09:00
Dmitrii Vasilev
b9ce0faee2 Set Content-MD5 header even for empty objects 
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
 2021-06-19 16:10:58 +09:00
Carsten Grohmann
1e0e2752bf Improve warning for terminated threads 
with non-zero return code.
 2021-06-18 07:54:38 +09:00
Carsten Grohmann
7a488b93d0 Set decimal base for converting subseconds to int 
The leading 0 triggers an automatic conversion as an octal value. This
fails because it is a decimal value.

Setting the base to 10 prevents this automatism and treats the value
as a decimal value.

Fixes: [WAN] string_util.cpp:cvt_strtoofft(96): something error is occurred in convert std::string(017080564) to off_t, thus return 0 as default.

Related to #1676
 2021-06-17 06:27:02 +09:00
Carsten Grohmann
d67b83e671 Allow configuration for temporary files directory 2021-06-16 21:29:58 +09:00
Andrew Gaul
a100be9dce
Prefer std::string::empty over length == 0 (#1679) 
This is more concise.
 2021-06-13 20:03:10 +09:00
Andrew Gaul
4d39ea887e
Protect orgmeta with fdent_lock (#1678) 
Fixes a crash seen with the CentOS 7 builder and when running with
AddressSanitizer.  Regression introduced by
ac578d188e.  Fixes #1677.
 2021-06-13 16:14:24 +09:00
Andrew Gaul
7638b5b3e3
Prefer std::string::rbegin over operator[] (#1673) 
This is more concise and safer due to not repeating the variable name.
We cannot use std::string::back since it is not available in C++03.
 2021-06-13 13:26:38 +09:00
Andrew Gaul
600cee118d
Fix implicit narrowing conversions (#1672) 
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
 2021-06-13 12:50:07 +09:00
Takeshi Nakatani
c2c56d0263 Added info object about multipart uploading for each pseudo fd 
(and fixed typo about method name)
 2021-06-04 22:42:58 +09:00
Takeshi Nakatani
ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
Andrew Gaul
53dfd48f59
Correct usage of istringstream (#1670) 
Previously this looped one more time than necessary due to the eof
check:

https://isocpp.org/wiki/faq/input-output#istream-and-eof

Remove now redundant empty check.
 2021-06-02 07:14:32 +09:00
Andrew Gaul
84174c560d
Fix data races caused by incorrect locking (#1668) 
Found via Threadsanitizer.  Fixes #1471.
 2021-05-29 00:11:55 +09:00
Carsten Grohmann
9bf525ee7a Ensuring multipart size even when storage is low 
When the temporary storage was full, the old implementation started an
upload even if there was not enough data to completely fill the minimum
multipart size or fill the user-selected multipart size.

The new implementation ensures the minimum multipart size by forcing
the user-selected multipart size.

Fixes #1591
 2021-05-27 22:55:52 +09:00
Takeshi Nakatani
4b69d4b1bb Fixed a bug when the disk capacity was insufficient in RowFlush 2021-05-22 23:11:46 +09:00
Andrew Gaul
bbcccd6e98
Make storage class a string (#1663) 
This allows non-standard storage classes like Google Cloud Storage
Nearline.  Fixes #1613.
 2021-05-21 23:34:31 +09:00
Takeshi Nakatani
b5fef788da Additional bug fixing for not creating zero-byte object 2021-05-09 17:33:53 +09:00
Andrew Gaul
42f5965d8a
Fix off_t to int narrowing in PageList::GetSparseFilePages (#1654) 
Found via clang-tidy.
 2021-05-09 14:35:00 +09:00
Andrew Gaul
d904d91252
Prefer std::map::insert over find and operator[] (#1653) 
This avoids a duplicate lookups.
 2021-05-09 14:11:35 +09:00
Andrew Gaul
9abe3fa662
Use same time when creating directory objects (#1652) 
This avoids mismatched times and is slightly faster.
 2021-05-09 13:44:12 +09:00
Andrew Gaul
bb6d2b1b74
Replace snprintf with string and ostringstream (#1649) 
These uses are probably safe from a buffer overflow perspective but
can cause data race issues in logging due to static buffers.
 2021-05-08 02:48:47 +09:00
Andrew Gaul
096a230b70
Allow arbitrary size AWS secret keys (#1648) 
Previously s3fs limited these to 123 characters.  Fixes #1626.
 2021-05-06 22:24:38 +09:00
Andrew Gaul
8ef01d37a9
Fix a few nits (#1645) 
Make some strings more const, initialize members, and abort if lock
initialization fails.  Partially found via clang-tidy.
 2021-05-06 19:40:35 +09:00
Andrew Gaul
cb9148f6cd
Update for clang-tidy 12 (#1644) 2021-05-03 09:43:18 +09:00
Andrew Gaul
771bbfeac5
Do not create zero-byte object when creating file (#1640) 
Previously s3fs created this object to store metadata and overwrote it
when flushing.  This prevented use with object stores which do not
allow overwrites like HDS.  Instead only create an in-memory
representation which reduces the time to create small files.
Fixes #1013.
 2021-04-30 19:56:33 +09:00
Andrew Gaul
3694786112
Propagate errno instead of EIO (#1638) 
This improves error fidelity.  Follows on to
b70f8db037.  References #1523.
 2021-04-30 07:09:00 +09:00
Andrew Gaul
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses (#1635) 
S3 can emit these in unusual situations.  Fixes #1317.
 2021-04-25 13:18:11 +09:00
Andrew Gaul
8a51a26819 Allow truncation of open and modified files 
Regression introduced in f5bf41cf11.
Fixes #1575.
 2021-04-25 12:35:35 +09:00
Takeshi Nakatani
1838f52e19 Declare undefined symbols for fallocate function 2021-04-25 10:12:25 +09:00
Andrew Gaul
e9eb248f2f
Fail CheckBucket when S3 returns PermanentRedirect (#1630) 
Previously s3fs allowed mounting but all operations returned EIO.
References #693.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-04-24 18:46:24 +09:00
Ambika Nair
77581eda59 Change in ibm iam endpoint 2021-04-21 21:42:55 +09:00
Andrew Gaul
7f3e423bbe
Preserve sub-second time precision (#1624) 
Found via pjdfstests.  References #897.  References #1589.
 2021-04-18 13:11:12 +09:00
Andrew Gaul
706e3cbebd
Update ctime and mtime when opening with O_TRUNC (#1623) 
Found via pjdfstests.  References #1589.
 2021-04-15 07:08:49 +09:00
Takeshi Nakatani
2effffd8e2 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Takeshi Nakatani
2908878988 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Andrew Gaul
01a26a9011
Do not allow open with O_RDONLY and O_TRUNC (#1620) 
Found via pjdfstests.  References #1589.
 2021-04-13 07:28:10 +09:00
Andrew Gaul
23e1fbf7b9
Update ctime when truncating file (#1619) 
Found via pjdfstests.  References #1589.
 2021-04-13 07:08:06 +09:00
Andrew Gaul
ed2e877bb6
Return EACESS when lacking required executable bit (#1617) 
This makes the check consistent with read and write.  Found via
pjdfstests.  References #1551.  References #1589.
 2021-04-11 22:39:27 +09:00
Andrew Gaul
3663082a01
Do not allow renaming to a non-empty directory (#1604) 
Found via pjdfstests.  References #1589.
 2021-03-28 19:10:52 +09:00
Andrew Gaul
9645d57c05
Translate KeyTooLongError to ENAMETOOLONG (#1602) 
AWS does not support keys longer than 1024 characters.  Add special
handling for HeadObject which does not have a response body.
Found via pjdfstests.  References #1589.
 2021-03-28 13:17:41 +09:00
Andrew Gaul
f6fbd75320
Return EFBIG when file exceeds multipart limit (#1600) 
Found via pjdfstests.  References #1589.
 2021-03-28 12:33:01 +09:00
Carsten Grohmann
ef079f4e94 Don't ignore nomultipart when storage is low 
When the temporary storage filled up, the old implementation uploaded
all data with multipart uploads, even if "nomultipart" was set.

The new implementation emits a warning and returns -ENOSPC instead.

Fixes #1595
 2021-03-10 08:23:54 +09:00
Takeshi Nakatani
b589ebec23 Output version and command parameters at startup 2021-03-06 19:11:57 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583) 
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
 2021-02-23 09:45:13 +09:00
Takeshi Nakatani
5b5bc3114a Fixed a bug that padded null bytes when changing xattr 2021-02-21 12:23:36 +09:00
Takeshi Nakatani
81e267d421 Fixed the same bug in chmod/chown/utimens and added tests 2021-02-20 23:30:50 +09:00