9a4282208e
Improved fdcache_page performance
2022-09-25 17:14:35 +09:00
238fc0799e
Fixed statfs(free inodes) for macos
2022-09-25 17:10:37 +09:00
5b95a0fcb6
Fixed errors reported by cppcheck 2.9 ( #2038 )
2022-09-24 15:45:13 +09:00
2186317676
Include missing ctime header
...
This caused compilation errors on Fedora 36.
2022-09-01 22:40:16 +09:00
5ab1037094
Strictly reviewed the exclusive control of PseudoFdInfo class(additional change)
2022-08-08 07:51:49 +09:00
f6d7ff1084
Strictly reviewed the exclusive control of PseudoFdInfo class
2022-08-07 20:41:35 +09:00
3e242d0bad
Avoid C-style casts ( #2015 )
...
Prefer more precise C++-style casts.
2022-07-30 22:35:27 +09:00
c491fbeabc
Replace uses of lock_already_held flag with AutoLock::Type
2022-07-30 16:20:43 +09:00
e654e8ec8a
Removed const_cast in set_bucket
2022-07-30 16:18:52 +09:00
48e9e51f4f
Remove more unneeded headers identified by IWYU ( #2011 )
2022-07-30 12:06:47 +09:00
5a2172dc56
Fixed data race and memory leaks in PseudoFdInfo
2022-07-30 06:51:16 +09:00
07535ec3ec
Address clang-tidy warnings ( #2010 )
2022-07-29 22:00:07 +09:00
b8dd466988
Fixed data race in ThreadPoolMan
2022-07-29 13:24:09 +09:00
01a92476e6
Remove unneeded headers identified by IWYU ( #2007 )
2022-07-28 23:38:38 +09:00
3928a7e359
Remove more uses of const_cast ( #2006 )
...
Follows on to #2004 .
2022-07-28 23:37:15 +09:00
f9f614a474
Avoid mutating mybasename and mydirname parameters ( #2004 )
...
basename and dirname mutate their inputs but const_cast hid this
behavior. Also shuffle helpers to avoid unnecessary std::string.
Follows on to 404c284440
2022-07-28 23:34:01 +09:00
e30a5939d0
Fixed deadlock in S3fsCurl::DestroyCurlHandle
2022-07-28 13:47:27 +09:00
4b2f3fecb5
Set mtime/ctime/atime of all objects as nanosecond
2022-07-28 13:47:03 +09:00
ccfc119e45
Improved to avoid unnecessary head request
2022-07-27 21:51:28 +09:00
38b5018bab
Remove unneedd binary_function inheritance ( #1998 )
...
GCC 12 warns that C++17 removes this.
2022-07-24 22:41:58 +09:00
404c284440
Fixed race condition in dirname and basename call
2022-07-24 18:14:58 +09:00
e0655008b3
Protect pending_status in UploadPending ( #1992 )
...
This requires avoiding double-locking in RowFlush. References #1991 .
2022-07-22 23:30:04 +09:00
22f2392fca
Fixed bugs about stream upload
2022-07-19 21:29:56 +09:00
faddb4900f
Merged the code corresponding to the mknod fix( f11eb7d)
2022-07-17 22:20:45 +09:00
6ca7d5ec27
Fixed an error which reported by cppcheck 2.8
2022-07-17 22:20:45 +09:00
6f679a9e78
Reflect the result of the review in the code again
2022-07-17 22:20:45 +09:00
b0eeaa6679
Reflected the result of the review in the code
2022-07-17 22:20:45 +09:00
d22e1dc018
Add the stream upload which starts uploading parts before Flush
2022-07-17 22:20:45 +09:00
4e163b2888
Allow listing implicit directories ( #1986 )
...
This fixes an issue when using -o notsup_compat_dir flag, dating back
to its introduction 3ac39d61f8#927 .
References #1984 .
2022-07-17 01:54:29 +09:00
e7ed01b35f
Consume return code from get_object_attribute ( #1976 )
...
Found via clang-analyzer.
2022-07-09 16:45:23 +09:00
3f55c98a3f
Fixed a bug when the pool of curl handler exceeds the upper limit
2022-07-08 21:14:29 +09:00
f11eb7d69b
Fixed a bug that regular files could not be created by mknod
2022-06-29 16:56:19 +09:00
73b49c1038
Fixed a bug that regular files could not be created by mknod
2022-06-29 16:56:19 +09:00
8b90cd6ba1
Enable notsup_compat_dir by default ( #1970 )
...
Few applications create the dir_$folder$ objects and users can enable
compat_dir if required. This commit reduces readdir latency by 33%.
Also remove notsup_compat_dir from tests since these directories are
never created. Fixes #927 . References #1643 .
2022-06-28 07:56:06 +09:00
b52f916af6
Avoid extended initializer lists in C++03 ( #1960 )
...
C++11 introduces these.
2022-06-12 23:52:16 +09:00
ec7810f08e
Fixed a bug could not change the mode while the file was opened
2022-06-12 13:48:42 +09:00
904682b856
Fix double-iteration in curl_slist_remove ( #1951 )
...
Also backfill tests. Fixes #1948 .
2022-05-27 22:56:20 +09:00
92fd5bc3e1
Fixed errors reported by cppcheck 2.8 ( #1949 )
2022-05-26 01:48:53 +09:00
d75c6d6538
Changed the message level(ERR to INFO) of skip cleanup cache ( #1946 )
2022-05-22 14:36:56 +09:00
5c57e17b77
Enable noobj_cache by default ( #1922 )
...
This should improve performance in many situations. s3fs already
enables the stat cache by default so memorizing noobj makes this more
consistent. Fixes #1901 .
2022-03-12 16:57:31 +09:00
deaa85c40e
Document glacier_ir storage class ( #1921 )
...
Announcement:
https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-s3-glacier-instant-retrieval-storage-class/
This does not require code changes due to
bbcccd6e98
2022-03-12 16:20:38 +09:00
d842d45b2b
Fixed a bug about truncation for shrinking file
2022-03-02 22:41:10 +09:00
684ced5a41
Changed handling the credential in S3fsCred more robust
2022-03-02 22:39:15 +09:00
afb0897553
Typos
2022-02-24 19:15:00 +09:00
8a5c4306f5
Preserve sub-second precision where possible ( #1915 )
2022-02-23 23:58:51 +09:00
08adffd2fe
Fix typos ( #1916 )
2022-02-23 23:31:52 +09:00
81ed2bd91e
Propagate deferred exit status from main ( #1912 )
...
Previously s3fs always returned zero when the bucket did not mount.
Fixes #1911 .
2022-02-23 10:09:12 +09:00
a716c72d37
Update notsup_compat_dir in --help
2022-02-21 19:29:15 +09:00
d31cbda7b6
Fixed a bug about checking credential
2022-02-19 23:22:15 +09:00
b64dc7749c
Moved parameter analysis processing to S3fsCred class
2022-02-19 17:23:40 +09:00
839a33de49
Fixed not to call Flush even if the file size is increased ( #1887 )
...
Changed s3fs_truncate function.
This change reduces the number of file uploads if the file size is changed.
On macOS, I have found that the truncate call when "size=0" cannot reflect the file size.(This reason is not understood...)
To avoid this, only when "size=0", the flush method is called as before.
Other than that, I found a bug in FdEntity::Open() and fixed it.
Fixes #1875 .
2022-02-15 21:29:07 +09:00
4dfe2bfdd7
Include climits to support musl libc
...
PATH_MAX constant is not visible from any of currently included header
files in system with musl libc, where compilation fails with an error
below. The constant is defined in limits.h which is directly include via
climits header file.
fdcache.cpp: In static member function 'static FILE* FdManager::MakeTempFile()':
fdcache.cpp:381:14: error: 'PATH_MAX' was not declared in this scope
381 | char cfn[PATH_MAX];
| ^~~~~~~~
Fixes: d67b83e671
2022-02-14 09:19:30 +09:00
1678803566
Added S3fsCred class and moved Credential related processing in it
2022-02-13 21:38:30 +09:00
d7e929e0a8
Fixed some Github Actions errors. ( #1886 )
...
- Fix knownConditionTrueFalse cppcheck(2.7) error on MacOS
- Fixed package installing failure of appstream download on centos8
2022-02-13 14:23:35 +09:00
9224f792f0
Use CLOCK_REALTIME for UTIME_NOW ( #1881 )
...
Previously s3fs_utimens used CLOCK_MONOTONIC_COARSE which was not
1970-based. Found via pjdfstest. References #1589 .
2022-01-30 22:19:15 +09:00
f6ed972926
Always flush open files with O_CREAT flag ( #1879 )
...
Previously s3fs only created files that had dirty data and not those
with zero-bytes. Regression from
771bbfeac5#1013 . Found
via pjdfstest. References #1589 .
2022-01-30 22:02:37 +09:00
0c75a63184
Preserve sub-second precision with utimens ( #1880 )
...
Found via pjdfstest. References #1589 .
2022-01-30 21:45:51 +09:00
e452ef3940
Fixed the fault tolerance when time stamp getting fails
2022-01-30 18:31:36 +09:00
cd5a69b9eb
Handle UTIME_NOW and UTIME_OMIT special values ( #1868 )
...
FUSE 3 will require this behavior. References #1159 .
2022-01-29 11:35:37 +09:00
662882d2f0
Always call clock_gettime(2) ( #1871 )
...
e01ded9e27https://stackoverflow.com/a/39801564 . Also remove fallback to
time(3) which loses precision.
2022-01-25 08:36:27 +09:00
de0c87c801
Convert S3FS_LOW_LOGPRN from a macro to a function ( #1869 )
...
This shrinks the binary size from 770 to 540 KB and reduces compile
times.
2022-01-23 23:10:09 +09:00
41aaa4184f
Avoid double setting values in statfs
2022-01-23 21:49:51 +09:00
581f5c0356
Move strptime polyfill to string_util
2022-01-23 21:49:51 +09:00
e5f6f112db
Fix typo
2022-01-23 21:49:51 +09:00
b3cef944b2
Fix test_page_list_SOURCES has no if MSYS clause
2022-01-23 21:49:51 +09:00
6edb6067f3
Remove strcasestr polyfill
2022-01-23 21:49:51 +09:00
b2c659c0a6
Disable compiling polyfills in not MSYS2 env
2022-01-23 21:49:51 +09:00
3ac9f571f5
Use std::get_time instead in strptime polyfill
2022-01-23 21:49:51 +09:00
19303a546e
Fix the statfs issue, using f_frsize instead
2022-01-23 21:49:51 +09:00
6370e150dd
Disable features that causes problems on Windows
2022-01-23 21:49:51 +09:00
b14e39815b
Use polyfills in MSYS2 environment
2022-01-23 21:49:51 +09:00
6aaf9433a5
Add polyfills for MSYS2 environment
2022-01-23 21:49:51 +09:00
93d1c30d4d
Use XML parsing with PUT HTTP 200 responses ( #1858 )
...
This works around the missing strcasestr on win32. References #728 .
2022-01-14 16:10:22 +09:00
25012f3839
Fix typo in -o enable_unsigned_payload
2022-01-12 22:50:49 +09:00
e734763002
Remove createbucket option ( #1841 )
...
AWS CLI can do this. Fixes #1840 .
2022-01-05 01:59:31 +09:00
85ca2a3e45
fix mixupload return EntityTooSmall while a copypart is less than 5MB after split ( #1809 )
...
* fix mixupload return EntityTooSmall while a copypart is less than 5MB after split
* fix possible part exceeds 5GB when multipart_copy_size is set to 5120MB
* Update curl.cpp
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
2021-11-27 16:53:26 +09:00
3cf00626a2
Add option to allow unsigned payloads ( #1801 )
...
This reduces CPU usage of sigv4. This reduces test run-time by 7
seconds per flag.
2021-11-01 23:33:55 +09:00
7273d561f5
Added exclusive control of statc variables in s3fs xml parser
2021-10-27 08:18:19 +09:00
78126aea0b
Added exclusive control of statc variables in s3fs xml parser
2021-10-27 08:18:19 +09:00
7892eee207
Fixed a bug that copied without considering the length of xmlChar
2021-10-27 08:18:19 +09:00
023aaf7dff
Fixed wrong stat of cache after new creation file
...
And added a test for stat of cache after new creation file
2021-10-17 16:10:14 +09:00
2f412804e2
Fixed forgetting to clear the dirty flag for meta information
...
Addressed an error in macos cpp check
2021-10-15 22:54:55 +09:00
be0b17329a
Fix wrong function name in log message ( #1774 )
2021-10-10 11:08:32 +09:00
b4edad86d6
remove Expect: 100-continue header when requesting an IMDSv2 access token
2021-09-09 08:12:36 +09:00
9d1552a54e
fix IAM role retrieval from IMDSv2
...
AWS IMDSv2 support was added in #1462 , but the implementation did not
cover the addional IMDS access that occurs with the iam_role=auto
configuration. This change implements IMDSv2 support for the IMDS
call to determine the instance's role name.
See also
https://stackoverflow.com/questions/69031023/how-to-make-s3fs-use-imds-v2-when-mounting-s3-buckets-from-ec2-instance
2021-09-03 20:36:34 +09:00
47ebfcc60a
Consume return value from curl_easy_setopt ( #1759 )
...
Found via Coverity.
2021-09-02 08:07:06 +09:00
beecf32dff
fclose(FILE*) instead of close(fileno(FILE*)) ( #1758 )
...
This is the same thing but confuses Coverity.
2021-09-01 19:41:55 +09:00
57b2e4a4f1
Fix 32-bit compilation issues ( #1757 )
2021-08-31 19:36:02 +09:00
48817d849f
Require explicit length in s3fs_decode64 ( #1755 )
...
This is available from std::string::size in callers.
2021-08-31 09:22:10 +09:00
d9f2d17040
1. fix RowFlush can not upload last part smaller than 5MB using NoCacheMultipartPost; ( #1753 )
...
2. fix deadlock in UploadPendingMeta
2021-08-31 00:41:47 +09:00
cd98afdd7b
Do not NUL terminate base64 decoded output ( #1752 )
...
This is binary data and must use the explicit length.
2021-08-31 00:15:47 +09:00
dac6885fb0
Don't over-allocate in base64 encoding and decoding ( #1751 )
2021-08-30 00:03:10 +09:00
fcd180891b
fix misuse of IsUploading ( #1747 )
...
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
2021-08-29 23:41:02 +09:00
a868c0656e
Changed etaglist_t from string list to new structure etagpairs list
2021-08-16 09:27:12 +09:00
15e89b78de
Add a partial page_list unit test ( #1735 )
2021-08-04 07:36:32 +09:00
66006ba48d
Add dedicated upper- and lower-case hex functions ( #1734 )
...
This makes the call sites more readable than a boolean parameter.
2021-08-04 07:28:51 +09:00
18e9c62087
Make string constants read-only const ( #1733 )
...
This removes some global constructors. Also use a consistent ALL_CAPS
style.
2021-08-03 00:10:27 +09:00
34f89e5936
Added fake_diskfree option to deceive free disk space for test
2021-08-02 11:04:42 +09:00
e1f3b9d8c1
Add support for AWS-style environment variables ( #1729 )
...
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.
The old environment variables are still supported, but they are
deprecated and no longer documented.
Close #1708
2021-07-26 23:29:45 +09:00
d3278f4886
Loosen CheckBucket to check only the bucket ( #1728 )
...
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460 . Fixes #1687 .
2021-07-25 12:29:00 +09:00
77f0b75d2f
Add jitter to avoid thundering herd ( #1723 )
2021-07-25 10:22:19 +09:00
199b3d4709
Fixed a bug in disk free space calculation
2021-07-18 14:52:11 +09:00
7890989cbb
Address cppcheck 2.5 warnings ( #1720 )
...
This slightly simplifies memory management.
2021-07-14 22:18:09 +09:00
945cc2ac54
Added UntreatedParts class instead of untreated upload info members in PseudoFdInfo
2021-07-12 21:45:29 +09:00
c30acbbf90
Splited some methods of FdEntity class by uploading mode
...
- Splited FdEntity::Write method by uploading mode
- Splited FdEntity::RowFlush method by uplading mode
2021-07-10 17:24:34 +09:00
881025cc9e
Add initiator pid, uid and gid to a debug log ( #1716 )
2021-07-10 16:23:20 +09:00
4cc210c5ab
Prevents the etag string buffer from being destroyed during use
...
and absorbed MPPART_INFO structure into filepart structure
2021-07-03 22:42:18 +09:00
fe44355d25
Report s3fs space as 18.4 EB ( #1709 )
...
256 TB probably seemed large in 2007 but in 2021 it is small enough to
confuse some users.
2021-07-02 07:56:05 +09:00
487df27008
Return EREMOTE when reading Glacier objects ( #1705 )
...
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466 .
2021-06-30 09:25:36 +09:00
1965916f7a
Convert storage class values to uppercase ( #1704 )
...
AWS requires uppercase values. Fixes a regression from
bbcccd6e98#1613 .
2021-06-30 09:03:31 +09:00
8948eded09
Address clang-tidy warnings ( #1703 )
2021-06-30 08:42:44 +09:00
2f59cb5a0a
Remove unneeded volatile qualifiers ( #1702 )
...
We should use proper locking instead. Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
2021-06-30 08:20:44 +09:00
f505c8224e
Explicitly specify lock type ( #1701 )
...
This makes it more clear and type-safe if the caller already has the
lock. Follows on to 84174c560d
2021-06-27 15:15:48 +09:00
2154e898bc
Fix typos ( #1700 )
2021-06-27 11:22:33 +09:00
f9e80f995d
Fixed a bug about rename existing file of different sizes by mpcopy
2021-06-26 00:04:58 +09:00
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating ( #1681 )
...
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/ .
Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-06-25 20:26:12 +09:00
6edf3d6427
Updated to clearly output message about the file descriptor type(pseudo/physical)
2021-06-25 19:18:08 +09:00
20281737b1
Set explicit Content-Length: 0 when initiating MPU ( #1696 )
...
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661 .
2021-06-25 18:36:40 +09:00
5b6684ca19
Fixed a bug utimens is calling before flush
2021-06-21 17:20:24 +09:00
c4ac923b4c
Ensure NUL-terminated result after strncpy ( #1694 )
...
Long symlinks may cause that the result buffer is filled and not proper
terminated with a null byte.
2021-06-21 08:08:56 +09:00
cc022a68f4
Fixed a bug in s3fs_write
2021-06-20 23:13:44 +09:00
858562ed53
Fixed comparison warning for size_t variable
2021-06-20 16:53:25 +09:00
b8724425d3
Require explicit base parameter in cvt_strtoofft ( #1683 )
...
Also convert most callers of cvt_strtoofft to base 10 which avoid the
magical behavior of interpreting a leading 0 as octal.
References #1682 .
2021-06-20 11:00:15 +09:00
b9ce0faee2
Set Content-MD5 header even for empty objects
...
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
2021-06-19 16:10:58 +09:00
1e0e2752bf
Improve warning for terminated threads
...
with non-zero return code.
2021-06-18 07:54:38 +09:00
7a488b93d0
Set decimal base for converting subseconds to int
...
The leading 0 triggers an automatic conversion as an octal value. This
fails because it is a decimal value.
Setting the base to 10 prevents this automatism and treats the value
as a decimal value.
Fixes: [WAN] string_util.cpp:cvt_strtoofft(96): something error is occurred in convert std::string(017080564) to off_t, thus return 0 as default.
Related to #1676
2021-06-17 06:27:02 +09:00
d67b83e671
Allow configuration for temporary files directory
2021-06-16 21:29:58 +09:00
a100be9dce
Prefer std::string::empty over length == 0 ( #1679 )
...
This is more concise.
2021-06-13 20:03:10 +09:00
4d39ea887e
Protect orgmeta with fdent_lock ( #1678 )
...
Fixes a crash seen with the CentOS 7 builder and when running with
AddressSanitizer. Regression introduced by
ac578d188eFixes #1677 .
2021-06-13 16:14:24 +09:00
7638b5b3e3
Prefer std::string::rbegin over operator[] ( #1673 )
...
This is more concise and safer due to not repeating the variable name.
We cannot use std::string::back since it is not available in C++03.
2021-06-13 13:26:38 +09:00
600cee118d
Fix implicit narrowing conversions ( #1672 )
...
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
2021-06-13 12:50:07 +09:00
c2c56d0263
Added info object about multipart uploading for each pseudo fd
...
(and fixed typo about method name)
2021-06-04 22:42:58 +09:00
ac578d188e
Introduced pseudo fd and separated fd for each file opening
2021-06-04 22:42:58 +09:00
53dfd48f59
Correct usage of istringstream ( #1670 )
...
Previously this looped one more time than necessary due to the eof
check:
https://isocpp.org/wiki/faq/input-output#istream-and-eof
Remove now redundant empty check.
2021-06-02 07:14:32 +09:00
84174c560d
Fix data races caused by incorrect locking ( #1668 )
...
Found via Threadsanitizer. Fixes #1471 .
2021-05-29 00:11:55 +09:00
9bf525ee7a
Ensuring multipart size even when storage is low
...
When the temporary storage was full, the old implementation started an
upload even if there was not enough data to completely fill the minimum
multipart size or fill the user-selected multipart size.
The new implementation ensures the minimum multipart size by forcing
the user-selected multipart size.
Fixes #1591
2021-05-27 22:55:52 +09:00
4b69d4b1bb
Fixed a bug when the disk capacity was insufficient in RowFlush
2021-05-22 23:11:46 +09:00
bbcccd6e98
Make storage class a string ( #1663 )
...
This allows non-standard storage classes like Google Cloud Storage
Nearline. Fixes #1613 .
2021-05-21 23:34:31 +09:00
b5fef788da
Additional bug fixing for not creating zero-byte object
2021-05-09 17:33:53 +09:00
42f5965d8a
Fix off_t to int narrowing in PageList::GetSparseFilePages ( #1654 )
...
Found via clang-tidy.
2021-05-09 14:35:00 +09:00
d904d91252
Prefer std::map::insert over find and operator[] ( #1653 )
...
This avoids a duplicate lookups.
2021-05-09 14:11:35 +09:00
9abe3fa662
Use same time when creating directory objects ( #1652 )
...
This avoids mismatched times and is slightly faster.
2021-05-09 13:44:12 +09:00
bb6d2b1b74
Replace snprintf with string and ostringstream ( #1649 )
...
These uses are probably safe from a buffer overflow perspective but
can cause data race issues in logging due to static buffers.
2021-05-08 02:48:47 +09:00
096a230b70
Allow arbitrary size AWS secret keys ( #1648 )
...
Previously s3fs limited these to 123 characters. Fixes #1626 .
2021-05-06 22:24:38 +09:00
8ef01d37a9
Fix a few nits ( #1645 )
...
Make some strings more const, initialize members, and abort if lock
initialization fails. Partially found via clang-tidy.
2021-05-06 19:40:35 +09:00
cb9148f6cd
Update for clang-tidy 12 ( #1644 )
2021-05-03 09:43:18 +09:00
771bbfeac5
Do not create zero-byte object when creating file ( #1640 )
...
Previously s3fs created this object to store metadata and overwrote it
when flushing. This prevented use with object stores which do not
allow overwrites like HDS. Instead only create an in-memory
representation which reduces the time to create small files.
Fixes #1013 .
2021-04-30 19:56:33 +09:00
3694786112
Propagate errno instead of EIO ( #1638 )
...
This improves error fidelity. Follows on to
b70f8db037#1523 .
2021-04-30 07:09:00 +09:00
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses ( #1635 )
...
S3 can emit these in unusual situations. Fixes #1317 .
2021-04-25 13:18:11 +09:00
8a51a26819
Allow truncation of open and modified files
...
Regression introduced in f5bf41cf11Fixes #1575 .
2021-04-25 12:35:35 +09:00
1838f52e19
Declare undefined symbols for fallocate function
2021-04-25 10:12:25 +09:00
e9eb248f2f
Fail CheckBucket when S3 returns PermanentRedirect ( #1630 )
...
Previously s3fs allowed mounting but all operations returned EIO.
References #693 .
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-04-24 18:46:24 +09:00
77581eda59
Change in ibm iam endpoint
2021-04-21 21:42:55 +09:00
7f3e423bbe
Preserve sub-second time precision ( #1624 )
...
Found via pjdfstests. References #897 . References #1589 .
2021-04-18 13:11:12 +09:00
706e3cbebd
Update ctime and mtime when opening with O_TRUNC ( #1623 )
...
Found via pjdfstests. References #1589 .
2021-04-15 07:08:49 +09:00
2effffd8e2
Fixed codes for cppcheck 2.4.1
2021-04-13 21:12:35 +09:00
2908878988
Fixed codes for cppcheck 2.4.1
2021-04-13 21:12:35 +09:00
01a26a9011
Do not allow open with O_RDONLY and O_TRUNC ( #1620 )
...
Found via pjdfstests. References #1589 .
2021-04-13 07:28:10 +09:00
23e1fbf7b9
Update ctime when truncating file ( #1619 )
...
Found via pjdfstests. References #1589 .
2021-04-13 07:08:06 +09:00
ed2e877bb6
Return EACESS when lacking required executable bit ( #1617 )
...
This makes the check consistent with read and write. Found via
pjdfstests. References #1551 . References #1589 .
2021-04-11 22:39:27 +09:00
3663082a01
Do not allow renaming to a non-empty directory ( #1604 )
...
Found via pjdfstests. References #1589 .
2021-03-28 19:10:52 +09:00
9645d57c05
Translate KeyTooLongError to ENAMETOOLONG ( #1602 )
...
AWS does not support keys longer than 1024 characters. Add special
handling for HeadObject which does not have a response body.
Found via pjdfstests. References #1589 .
2021-03-28 13:17:41 +09:00
f6fbd75320
Return EFBIG when file exceeds multipart limit ( #1600 )
...
Found via pjdfstests. References #1589 .
2021-03-28 12:33:01 +09:00
ef079f4e94
Don't ignore nomultipart when storage is low
...
When the temporary storage filled up, the old implementation uploaded
all data with multipart uploads, even if "nomultipart" was set.
The new implementation emits a warning and returns -ENOSPC instead.
Fixes #1595
2021-03-10 08:23:54 +09:00
b589ebec23
Output version and command parameters at startup
2021-03-06 19:11:57 +09:00
032fcf2a47
Allow optional issuing of ListObjectsV2 ( #1583 )
...
This allows use of s3fs on object stores that do not implement
the V1 API. Fixes #1573 .
2021-02-23 09:45:13 +09:00
5b5bc3114a
Fixed a bug that padded null bytes when changing xattr
2021-02-21 12:23:36 +09:00
81e267d421
Fixed the same bug in chmod/chown/utimens and added tests
2021-02-20 23:30:50 +09:00
4fc92d59f3
Fixed a bug that filling NULL bytes when changing the attribute after renaming
2021-02-20 23:30:50 +09:00
a0f347b10f
Added no_time_stamp_ms option mainly for testing
2021-02-20 20:24:26 +09:00
4da56acdcc
Improved processing when HEAD response is 400
2021-02-13 17:00:13 +09:00
493802a605
Fixed osx mistype declaration and typo warnings
2021-02-13 12:04:35 +09:00
22b0ae9d51
set IsExpireTime to true by default
...
fixes #1563
2021-02-12 12:20:47 +09:00
134a54b32f
Allow configuring the multipart threshold ( #1562 )
...
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-11 23:35:46 +09:00
7f6fbb0021
Correct singlepart_copy_limit unit ( #1561 )
...
This is documented to be in MB not KB.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-11 22:53:50 +09:00
ec8bae9827
Do not periodically flush with nomultipartupload ( #1559 )
...
s3fs periodically flushes data to S3 and hole-punches the written
data, marking it as clean. Without multipart copy, s3fs would write
the now-zeroed data along with any new data. s3fs should not attempt
to periodically flush dirty data at all without multipart copy
support. Fixes #1542 .
2021-02-11 22:24:39 +09:00
a4d916af13
Allow configuring multipart copy size ( #1555 )
...
Also align configuration with mixupload.
2021-02-08 20:32:12 +09:00
32f096fa3f
Use clock_gettime instead of gettimeofday
2021-02-08 10:19:28 +09:00
c692093921
Copy at most 5 GB per multipart copy part request ( #1553 )
...
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes. This symptom was seen more frequently due
to periodic flushing of dirty data. Fixes #1547 .
2021-02-07 23:10:07 +09:00
1a6d0826b5
Allow -1 value to disable max_dirty_data ( #1552 )
...
This matches the man page.
2021-02-07 22:14:31 +09:00
bf33fe7f55
Increase buffer size to avoid warning ( #1550 )
...
This can be up to 54 bytes.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-07 12:26:58 +09:00
ff9d6a75c7
Parse more specific AWS error codes ( #1549 )
...
Fixes #1545 .
2021-02-07 11:29:08 +09:00
4c6690f5f0
Add timestamps to debug log and cache integrity check ( #1540 )
...
* Add UTC timestamp to debug output
for easier correlation with external events
Old output:
$ s3fs mybucket /bucket/ -o curldbg,dbglevel=debug -d -d -f
[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from...
[INF] s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=0, ...
New output:
$ s3fs mybucket /bucket/ -o curldbg,dbglevel=debug -d -d -f
2021-01-28T21:09:16.264Z [CRT] s3fs_logger.cpp:LowSetLogLevel(202):...
2021-01-28T21:09:16.264Z [INF] s3fs.cpp:set_mountpoint_attrib...
* Add UTC timestamp to cache integrity check
for easier correlation with external events.
$ s3fs mybucket /mybucket -oset_check_cache_sigusr1=/tmp/check.cache
Old output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
------------------------------------------------------------
Check cache file and its stats file consistency
------------------------------------------------------------
------------------------------------------------------------
Summary - Total files: 0
Detected error files: 0
Detected error directories: 0
------------------------------------------------------------
New output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
---------------------------------------------------------------------------
Check cache file and its stats file consistency at 2021-01-30T13:04:14.111Z
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Summary - Total files: 0
Detected error files: 0
Detected error directories: 0
---------------------------------------------------------------------------
* Fix indentation of S3fsLog::GetCurrentTime()
2021-02-04 10:41:29 +09:00
67b9381825
Remove limit for single-part objects
...
AWS enforces a 5 GB limit for single-part objects but other S3
implementations like GCS support larger objects. Fixes #1542 .
2021-02-04 09:19:37 +09:00
4b53d4bf6b
Suppress spurious SHA256 error message ( #1544 )
2021-02-04 07:28:51 +09:00
ed85b72bf5
Add warning for failing metadata updates of large files
...
Metadata updates fail for large files if "-o nocopyapi" or
"-o nomultipart" is set.
This change is related to #1528
2021-01-28 10:20:32 +09:00
17fda89ae9
Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors ( #1534 )
...
Follows on to 4d833a4fb9
2021-01-25 19:08:14 +09:00
d019dda4f7
Simplify substr manipulations with erase ( #1532 )
...
This avoids creating a new std::string.
2021-01-25 18:02:32 +09:00
dc9255bc5f
Prefer static_cast where possible ( #1531 )
2021-01-25 08:15:17 +09:00
b0e8758b63
Use result instead of res for consistency ( #1530 )
2021-01-25 07:56:10 +09:00
b70f8db037
Propagate errno from get_local_fent ( #1525 )
...
Follows on to 4d833a4fb9#1523 .
2021-01-20 23:21:14 +09:00
4d833a4fb9
Return more specific errno when available ( #1520 )
...
Previously s3fs threw away some function return values and returned
EIO instead. This was due to not trusting the mix of -1 and errno
return codes. Correct the obviously incorrect ones via visual
inspection. Stronger typing may find more occurrences. Fixes #1519 .
2021-01-18 18:50:49 +09:00
168e588ac7
fix: Add reset offset ( #1503 )
2021-01-04 22:57:56 +09:00
bd0fadbe5f
Remove authorization header when remaking handle ( #1505 )
...
This avoids including Authorization in SignedHeaders. s3fs will
recreate the Authorization header before sending the request.
2021-01-04 21:37:34 +09:00
d1c638ab7a
fix 404 error message
2020-12-30 14:10:17 +09:00
51f65d7b14
fix: miss header when retry
2020-12-23 22:16:07 +09:00
a16d00d673
Simply curl_slist_sort_insert ( #1494 )
...
Some good taste from Linus:
https://github.com/mkirchner/linked-list-good-taste
Also avoid an allocation when replacing a value and tighten up tests.
2020-12-23 20:29:33 +09:00
d0363b118e
Add tests for curl_util ( #1481 )
2020-11-24 21:37:09 +09:00
533322859d
Added comment for nullPointerRedundantCheck/cppcheck2.2
2020-11-21 20:06:14 +09:00
f8d5b76edb
Simplify handling of returned ETag ( #1479 )
...
This works around lifetime warnings uncovered by cppcheck.
References #1478 .
2020-11-21 06:56:05 +09:00
Takeshi Nakatani
Andrew Gaul
Carsten Grohmann
Petr Vaněk
Naoki Ikeguchi
LiuBingrun
Kamil Jakrzewski
Noah Meyerhans
Martynov Maxim
Jan Stastny
Dmitrii Vasilev
Ambika Nair
林千里
fly3366
kontrollanten