octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-06-27 04:43:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,994 Commits 1 Branch 22 Tags 13 MiB
9c74014443
Commit Graph

1095 Commits

Author SHA1 Message Date
Takeshi Nakatani
22f2392fca Fixed bugs about stream upload 2022-07-19 21:29:56 +09:00
Takeshi Nakatani
faddb4900f Merged the code corresponding to the mknod fix(f11eb7d) 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
6ca7d5ec27 Fixed an error which reported by cppcheck 2.8 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
6f679a9e78 Reflect the result of the review in the code again 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
b0eeaa6679 Reflected the result of the review in the code 2022-07-17 22:20:45 +09:00
Takeshi Nakatani
d22e1dc018 Add the stream upload which starts uploading parts before Flush 2022-07-17 22:20:45 +09:00
Andrew Gaul
4e163b2888
Allow listing implicit directories (#1986) 
This fixes an issue when using -o notsup_compat_dir flag, dating back
to its introduction 3ac39d61f8.  The new
default exposed this in my local testing but a test runner bug masked
a test failure in test_external_directory_creation.  References #927.
References #1984.
 2022-07-17 01:54:29 +09:00
Andrew Gaul
e7ed01b35f
Consume return code from get_object_attribute (#1976) 
Found via clang-analyzer.
 2022-07-09 16:45:23 +09:00
Takeshi Nakatani
3f55c98a3f Fixed a bug when the pool of curl handler exceeds the upper limit 2022-07-08 21:14:29 +09:00
Takeshi Nakatani
f11eb7d69b Fixed a bug that regular files could not be created by mknod 2022-06-29 16:56:19 +09:00
Takeshi Nakatani
73b49c1038 Fixed a bug that regular files could not be created by mknod 2022-06-29 16:56:19 +09:00
Andrew Gaul
8b90cd6ba1
Enable notsup_compat_dir by default (#1970) 
Few applications create the dir_$folder$ objects and users can enable
compat_dir if required.  This commit reduces readdir latency by 33%.
Also remove notsup_compat_dir from tests since these directories are
never created.  Fixes #927.  References #1643.
 2022-06-28 07:56:06 +09:00
Andrew Gaul
b52f916af6
Avoid extended initializer lists in C++03 (#1960) 
C++11 introduces these.
 2022-06-12 23:52:16 +09:00
Takeshi Nakatani
ec7810f08e Fixed a bug could not change the mode while the file was opened 2022-06-12 13:48:42 +09:00
Andrew Gaul
904682b856
Fix double-iteration in curl_slist_remove (#1951) 
Also backfill tests.  Fixes #1948.
 2022-05-27 22:56:20 +09:00
Takeshi Nakatani
92fd5bc3e1
Fixed errors reported by cppcheck 2.8 (#1949) 2022-05-26 01:48:53 +09:00
Takeshi Nakatani
d75c6d6538
Changed the message level(ERR to INFO) of skip cleanup cache (#1946) 2022-05-22 14:36:56 +09:00
Andrew Gaul
5c57e17b77
Enable noobj_cache by default (#1922) 
This should improve performance in many situations.  s3fs already
enables the stat cache by default so memorizing noobj makes this more
consistent.  Fixes #1901.
 2022-03-12 16:57:31 +09:00
Andrew Gaul
deaa85c40e
Document glacier_ir storage class (#1921) 
Announcement:
https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-s3-glacier-instant-retrieval-storage-class/

This does not require code changes due to
bbcccd6e98.
 2022-03-12 16:20:38 +09:00
Takeshi Nakatani
d842d45b2b Fixed a bug about truncation for shrinking file 2022-03-02 22:41:10 +09:00
Takeshi Nakatani
684ced5a41 Changed handling the credential in S3fsCred more robust 2022-03-02 22:39:15 +09:00
Carsten Grohmann
afb0897553 Typos 2022-02-24 19:15:00 +09:00
Andrew Gaul
8a5c4306f5
Preserve sub-second precision where possible (#1915) 2022-02-23 23:58:51 +09:00
Andrew Gaul
08adffd2fe
Fix typos (#1916) 2022-02-23 23:31:52 +09:00
Andrew Gaul
81ed2bd91e
Propagate deferred exit status from main (#1912) 
Previously s3fs always returned zero when the bucket did not mount.
Fixes #1911.
 2022-02-23 10:09:12 +09:00
Carsten Grohmann
a716c72d37 Update notsup_compat_dir in --help 2022-02-21 19:29:15 +09:00
Takeshi Nakatani
d31cbda7b6 Fixed a bug about checking credential 2022-02-19 23:22:15 +09:00
Takeshi Nakatani
b64dc7749c Moved parameter analysis processing to S3fsCred class 2022-02-19 17:23:40 +09:00
Takeshi Nakatani
839a33de49
Fixed not to call Flush even if the file size is increased (#1887) 
Changed s3fs_truncate function.
This change reduces the number of file uploads if the file size is changed.

On macOS, I have found that the truncate call when "size=0" cannot reflect the file size.(This reason is not understood...)
To avoid this, only when "size=0", the flush method is called as before.

Other than that, I found a bug in FdEntity::Open() and fixed it.

Fixes #1875.
 2022-02-15 21:29:07 +09:00
Petr Vaněk
4dfe2bfdd7 Include climits to support musl libc 
PATH_MAX constant is not visible from any of currently included header
files in system with musl libc, where compilation fails with an error
below. The constant is defined in limits.h which is directly include via
climits header file.

fdcache.cpp: In static member function 'static FILE* FdManager::MakeTempFile()':
fdcache.cpp:381:14: error: 'PATH_MAX' was not declared in this scope
  381 |     char cfn[PATH_MAX];
      |              ^~~~~~~~

Fixes: d67b83e671 ("Allow configuration for temporary files directory")
 2022-02-14 09:19:30 +09:00
Takeshi Nakatani
1678803566 Added S3fsCred class and moved Credential related processing in it 2022-02-13 21:38:30 +09:00
Takeshi Nakatani
d7e929e0a8
Fixed some Github Actions errors. (#1886) 
- Fix knownConditionTrueFalse cppcheck(2.7) error on MacOS
- Fixed package installing failure of appstream download on centos8
 2022-02-13 14:23:35 +09:00
Andrew Gaul
9224f792f0
Use CLOCK_REALTIME for UTIME_NOW (#1881) 
Previously s3fs_utimens used CLOCK_MONOTONIC_COARSE which was not
1970-based.  Found via pjdfstest.  References #1589.
 2022-01-30 22:19:15 +09:00
Andrew Gaul
f6ed972926
Always flush open files with O_CREAT flag (#1879) 
Previously s3fs only created files that had dirty data and not those
with zero-bytes.  Regression from
771bbfeac5.  References #1013.  Found
via pjdfstest.  References #1589.
 2022-01-30 22:02:37 +09:00
Andrew Gaul
0c75a63184
Preserve sub-second precision with utimens (#1880) 
Found via pjdfstest.  References #1589.
 2022-01-30 21:45:51 +09:00
Takeshi Nakatani
e452ef3940 Fixed the fault tolerance when time stamp getting fails 2022-01-30 18:31:36 +09:00
Andrew Gaul
cd5a69b9eb
Handle UTIME_NOW and UTIME_OMIT special values (#1868) 
FUSE 3 will require this behavior.  References #1159.
 2022-01-29 11:35:37 +09:00
Andrew Gaul
662882d2f0
Always call clock_gettime(2) (#1871) 
e01ded9e27 introduced this compatibility
shim but macOS 10.12 (2016) added this:
https://stackoverflow.com/a/39801564 .  Also remove fallback to
time(3) which loses precision.
 2022-01-25 08:36:27 +09:00
Andrew Gaul
de0c87c801
Convert S3FS_LOW_LOGPRN from a macro to a function (#1869) 
This shrinks the binary size from 770 to 540 KB and reduces compile
times.
 2022-01-23 23:10:09 +09:00
Naoki Ikeguchi
41aaa4184f Avoid double setting values in statfs 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
581f5c0356 Move strptime polyfill to string_util 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
e5f6f112db Fix typo 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
b3cef944b2 Fix test_page_list_SOURCES has no if MSYS clause 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
6edb6067f3 Remove strcasestr polyfill 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
b2c659c0a6 Disable compiling polyfills in not MSYS2 env 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
3ac9f571f5 Use std::get_time instead in strptime polyfill 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
19303a546e Fix the statfs issue, using f_frsize instead 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
6370e150dd Disable features that causes problems on Windows 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
b14e39815b Use polyfills in MSYS2 environment 2022-01-23 21:49:51 +09:00
Naoki Ikeguchi
6aaf9433a5 Add polyfills for MSYS2 environment 2022-01-23 21:49:51 +09:00
Andrew Gaul
93d1c30d4d
Use XML parsing with PUT HTTP 200 responses (#1858) 
This works around the missing strcasestr on win32.  References #728.
 2022-01-14 16:10:22 +09:00
Andrew Gaul
25012f3839 Fix typo in -o enable_unsigned_payload 2022-01-12 22:50:49 +09:00
Andrew Gaul
e734763002
Remove createbucket option (#1841) 
AWS CLI can do this.  Fixes #1840.
 2022-01-05 01:59:31 +09:00
LiuBingrun
85ca2a3e45
fix mixupload return EntityTooSmall while a copypart is less than 5MB after split (#1809) 
* fix  mixupload return EntityTooSmall while a copypart is less than 5MB after split
* fix possible part exceeds 5GB when multipart_copy_size is set to 5120MB
* Update curl.cpp
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
 2021-11-27 16:53:26 +09:00
Andrew Gaul
3cf00626a2
Add option to allow unsigned payloads (#1801) 
This reduces CPU usage of sigv4.  This reduces test run-time by 7
seconds per flag.
 2021-11-01 23:33:55 +09:00
Takeshi Nakatani
7273d561f5 Added exclusive control of statc variables in s3fs xml parser 2021-10-27 08:18:19 +09:00
Takeshi Nakatani
78126aea0b Added exclusive control of statc variables in s3fs xml parser 2021-10-27 08:18:19 +09:00
Takeshi Nakatani
7892eee207 Fixed a bug that copied without considering the length of xmlChar 2021-10-27 08:18:19 +09:00
Takeshi Nakatani
023aaf7dff Fixed wrong stat of cache after new creation file 
And added a test for stat of cache after new creation file
 2021-10-17 16:10:14 +09:00
Takeshi Nakatani
2f412804e2 Fixed forgetting to clear the dirty flag for meta information 
Addressed an error in macos cpp check
 2021-10-15 22:54:55 +09:00
Carsten Grohmann
be0b17329a
Fix wrong function name in log message (#1774) 2021-10-10 11:08:32 +09:00
Kamil Jakrzewski
b4edad86d6 remove Expect: 100-continue header when requesting an IMDSv2 access token 2021-09-09 08:12:36 +09:00
Noah Meyerhans
9d1552a54e fix IAM role retrieval from IMDSv2 
AWS IMDSv2 support was added in #1462, but the implementation did not
cover the addional IMDS access that occurs with the iam_role=auto
configuration.  This change implements IMDSv2 support for the IMDS
call to determine the instance's role name.

See also
https://stackoverflow.com/questions/69031023/how-to-make-s3fs-use-imds-v2-when-mounting-s3-buckets-from-ec2-instance
 2021-09-03 20:36:34 +09:00
Andrew Gaul
47ebfcc60a
Consume return value from curl_easy_setopt (#1759) 
Found via Coverity.
 2021-09-02 08:07:06 +09:00
Andrew Gaul
beecf32dff
fclose(FILE*) instead of close(fileno(FILE*)) (#1758) 
This is the same thing but confuses Coverity.
 2021-09-01 19:41:55 +09:00
Andrew Gaul
57b2e4a4f1
Fix 32-bit compilation issues (#1757) 2021-08-31 19:36:02 +09:00
Andrew Gaul
48817d849f
Require explicit length in s3fs_decode64 (#1755) 
This is available from std::string::size in callers.
 2021-08-31 09:22:10 +09:00
VVoidV
d9f2d17040
1. fix RowFlush can not upload last part smaller than 5MB using NoCacheMultipartPost; (#1753) 
2. fix deadlock in UploadPendingMeta
 2021-08-31 00:41:47 +09:00
Andrew Gaul
cd98afdd7b
Do not NUL terminate base64 decoded output (#1752) 
This is binary data and must use the explicit length.
 2021-08-31 00:15:47 +09:00
Andrew Gaul
dac6885fb0
Don't over-allocate in base64 encoding and decoding (#1751) 2021-08-30 00:03:10 +09:00
VVoidV
fcd180891b
fix misuse of IsUploading (#1747) 
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
 2021-08-29 23:41:02 +09:00
Takeshi Nakatani
a868c0656e Changed etaglist_t from string list to new structure etagpairs list 2021-08-16 09:27:12 +09:00
Andrew Gaul
15e89b78de
Add a partial page_list unit test (#1735) 2021-08-04 07:36:32 +09:00
Andrew Gaul
66006ba48d
Add dedicated upper- and lower-case hex functions (#1734) 
This makes the call sites more readable than a boolean parameter.
 2021-08-04 07:28:51 +09:00
Andrew Gaul
18e9c62087
Make string constants read-only const (#1733) 
This removes some global constructors.  Also use a consistent ALL_CAPS
style.
 2021-08-03 00:10:27 +09:00
Takeshi Nakatani
34f89e5936 Added fake_diskfree option to deceive free disk space for test 2021-08-02 11:04:42 +09:00
Carsten Grohmann
e1f3b9d8c1
Add support for AWS-style environment variables (#1729) 
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.

The old environment variables are still supported, but they are
deprecated and no longer documented.

Close #1708
 2021-07-26 23:29:45 +09:00
Andrew Gaul
d3278f4886
Loosen CheckBucket to check only the bucket (#1728) 
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460.  Fixes #1687.
 2021-07-25 12:29:00 +09:00
Andrew Gaul
77f0b75d2f
Add jitter to avoid thundering herd (#1723) 2021-07-25 10:22:19 +09:00
Takeshi Nakatani
199b3d4709 Fixed a bug in disk free space calculation 2021-07-18 14:52:11 +09:00
Andrew Gaul
7890989cbb
Address cppcheck 2.5 warnings (#1720) 
This slightly simplifies memory management.
 2021-07-14 22:18:09 +09:00
Takeshi Nakatani
945cc2ac54 Added UntreatedParts class instead of untreated upload info members in PseudoFdInfo 2021-07-12 21:45:29 +09:00
Takeshi Nakatani
c30acbbf90 Splited some methods of FdEntity class by uploading mode 
- Splited FdEntity::Write method by uploading mode
- Splited FdEntity::RowFlush method by uplading mode
 2021-07-10 17:24:34 +09:00
Martynov Maxim
881025cc9e
Add initiator pid, uid and gid to a debug log (#1716) 2021-07-10 16:23:20 +09:00
Takeshi Nakatani
4cc210c5ab Prevents the etag string buffer from being destroyed during use 
and absorbed MPPART_INFO structure into filepart structure
 2021-07-03 22:42:18 +09:00
Andrew Gaul
fe44355d25
Report s3fs space as 18.4 EB (#1709) 
256 TB probably seemed large in 2007 but in 2021 it is small enough to
confuse some users.
 2021-07-02 07:56:05 +09:00
Andrew Gaul
487df27008
Return EREMOTE when reading Glacier objects (#1705) 
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466.
 2021-06-30 09:25:36 +09:00
Andrew Gaul
1965916f7a
Convert storage class values to uppercase (#1704) 
AWS requires uppercase values.  Fixes a regression from
bbcccd6e98.  References #1613.
 2021-06-30 09:03:31 +09:00
Andrew Gaul
8948eded09
Address clang-tidy warnings (#1703) 2021-06-30 08:42:44 +09:00
Andrew Gaul
2f59cb5a0a
Remove unneeded volatile qualifiers (#1702) 
We should use proper locking instead.  Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
 2021-06-30 08:20:44 +09:00
Andrew Gaul
f505c8224e
Explicitly specify lock type (#1701) 
This makes it more clear and type-safe if the caller already has the
lock.  Follows on to 84174c560d.
 2021-06-27 15:15:48 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Takeshi Nakatani
f9e80f995d Fixed a bug about rename existing file of different sizes by mpcopy 2021-06-26 00:04:58 +09:00
Jan Stastny
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating (#1681) 
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/.

Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-06-25 20:26:12 +09:00
Takeshi Nakatani
6edf3d6427 Updated to clearly output message about the file descriptor type(pseudo/physical) 2021-06-25 19:18:08 +09:00
Andrew Gaul
20281737b1
Set explicit Content-Length: 0 when initiating MPU (#1696) 
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661.
 2021-06-25 18:36:40 +09:00
Takeshi Nakatani
5b6684ca19 Fixed a bug utimens is calling before flush 2021-06-21 17:20:24 +09:00
Carsten Grohmann
c4ac923b4c
Ensure NUL-terminated result after strncpy (#1694) 
Long symlinks may cause that the result buffer is filled and not proper
terminated with a null byte.
 2021-06-21 08:08:56 +09:00
Takeshi Nakatani
cc022a68f4 Fixed a bug in s3fs_write 2021-06-20 23:13:44 +09:00
Takeshi Nakatani
858562ed53 Fixed comparison warning for size_t variable 2021-06-20 16:53:25 +09:00