839a33de49
Fixed not to call Flush even if the file size is increased ( #1887 )
...
Changed s3fs_truncate function.
This change reduces the number of file uploads if the file size is changed.
On macOS, I have found that the truncate call when "size=0" cannot reflect the file size.(This reason is not understood...)
To avoid this, only when "size=0", the flush method is called as before.
Other than that, I found a bug in FdEntity::Open() and fixed it.
Fixes #1875 .
2022-02-15 21:29:07 +09:00
4dfe2bfdd7
Include climits to support musl libc
...
PATH_MAX constant is not visible from any of currently included header
files in system with musl libc, where compilation fails with an error
below. The constant is defined in limits.h which is directly include via
climits header file.
fdcache.cpp: In static member function 'static FILE* FdManager::MakeTempFile()':
fdcache.cpp:381:14: error: 'PATH_MAX' was not declared in this scope
381 | char cfn[PATH_MAX];
| ^~~~~~~~
Fixes: d67b83e671
2022-02-14 09:19:30 +09:00
1678803566
Added S3fsCred class and moved Credential related processing in it
2022-02-13 21:38:30 +09:00
d7e929e0a8
Fixed some Github Actions errors. ( #1886 )
...
- Fix knownConditionTrueFalse cppcheck(2.7) error on MacOS
- Fixed package installing failure of appstream download on centos8
2022-02-13 14:23:35 +09:00
9224f792f0
Use CLOCK_REALTIME for UTIME_NOW ( #1881 )
...
Previously s3fs_utimens used CLOCK_MONOTONIC_COARSE which was not
1970-based. Found via pjdfstest. References #1589 .
2022-01-30 22:19:15 +09:00
f6ed972926
Always flush open files with O_CREAT flag ( #1879 )
...
Previously s3fs only created files that had dirty data and not those
with zero-bytes. Regression from
771bbfeac5#1013 . Found
via pjdfstest. References #1589 .
2022-01-30 22:02:37 +09:00
0c75a63184
Preserve sub-second precision with utimens ( #1880 )
...
Found via pjdfstest. References #1589 .
2022-01-30 21:45:51 +09:00
e452ef3940
Fixed the fault tolerance when time stamp getting fails
2022-01-30 18:31:36 +09:00
cd5a69b9eb
Handle UTIME_NOW and UTIME_OMIT special values ( #1868 )
...
FUSE 3 will require this behavior. References #1159 .
2022-01-29 11:35:37 +09:00
662882d2f0
Always call clock_gettime(2) ( #1871 )
...
e01ded9e27https://stackoverflow.com/a/39801564 . Also remove fallback to
time(3) which loses precision.
2022-01-25 08:36:27 +09:00
de0c87c801
Convert S3FS_LOW_LOGPRN from a macro to a function ( #1869 )
...
This shrinks the binary size from 770 to 540 KB and reduces compile
times.
2022-01-23 23:10:09 +09:00
41aaa4184f
Avoid double setting values in statfs
2022-01-23 21:49:51 +09:00
581f5c0356
Move strptime polyfill to string_util
2022-01-23 21:49:51 +09:00
e5f6f112db
Fix typo
2022-01-23 21:49:51 +09:00
b3cef944b2
Fix test_page_list_SOURCES has no if MSYS clause
2022-01-23 21:49:51 +09:00
6edb6067f3
Remove strcasestr polyfill
2022-01-23 21:49:51 +09:00
b2c659c0a6
Disable compiling polyfills in not MSYS2 env
2022-01-23 21:49:51 +09:00
3ac9f571f5
Use std::get_time instead in strptime polyfill
2022-01-23 21:49:51 +09:00
19303a546e
Fix the statfs issue, using f_frsize instead
2022-01-23 21:49:51 +09:00
6370e150dd
Disable features that causes problems on Windows
2022-01-23 21:49:51 +09:00
b14e39815b
Use polyfills in MSYS2 environment
2022-01-23 21:49:51 +09:00
6aaf9433a5
Add polyfills for MSYS2 environment
2022-01-23 21:49:51 +09:00
93d1c30d4d
Use XML parsing with PUT HTTP 200 responses ( #1858 )
...
This works around the missing strcasestr on win32. References #728 .
2022-01-14 16:10:22 +09:00
25012f3839
Fix typo in -o enable_unsigned_payload
2022-01-12 22:50:49 +09:00
e734763002
Remove createbucket option ( #1841 )
...
AWS CLI can do this. Fixes #1840 .
2022-01-05 01:59:31 +09:00
85ca2a3e45
fix mixupload return EntityTooSmall while a copypart is less than 5MB after split ( #1809 )
...
* fix mixupload return EntityTooSmall while a copypart is less than 5MB after split
* fix possible part exceeds 5GB when multipart_copy_size is set to 5120MB
* Update curl.cpp
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
2021-11-27 16:53:26 +09:00
3cf00626a2
Add option to allow unsigned payloads ( #1801 )
...
This reduces CPU usage of sigv4. This reduces test run-time by 7
seconds per flag.
2021-11-01 23:33:55 +09:00
7273d561f5
Added exclusive control of statc variables in s3fs xml parser
2021-10-27 08:18:19 +09:00
78126aea0b
Added exclusive control of statc variables in s3fs xml parser
2021-10-27 08:18:19 +09:00
7892eee207
Fixed a bug that copied without considering the length of xmlChar
2021-10-27 08:18:19 +09:00
023aaf7dff
Fixed wrong stat of cache after new creation file
...
And added a test for stat of cache after new creation file
2021-10-17 16:10:14 +09:00
2f412804e2
Fixed forgetting to clear the dirty flag for meta information
...
Addressed an error in macos cpp check
2021-10-15 22:54:55 +09:00
be0b17329a
Fix wrong function name in log message ( #1774 )
2021-10-10 11:08:32 +09:00
b4edad86d6
remove Expect: 100-continue header when requesting an IMDSv2 access token
2021-09-09 08:12:36 +09:00
9d1552a54e
fix IAM role retrieval from IMDSv2
...
AWS IMDSv2 support was added in #1462 , but the implementation did not
cover the addional IMDS access that occurs with the iam_role=auto
configuration. This change implements IMDSv2 support for the IMDS
call to determine the instance's role name.
See also
https://stackoverflow.com/questions/69031023/how-to-make-s3fs-use-imds-v2-when-mounting-s3-buckets-from-ec2-instance
2021-09-03 20:36:34 +09:00
47ebfcc60a
Consume return value from curl_easy_setopt ( #1759 )
...
Found via Coverity.
2021-09-02 08:07:06 +09:00
beecf32dff
fclose(FILE*) instead of close(fileno(FILE*)) ( #1758 )
...
This is the same thing but confuses Coverity.
2021-09-01 19:41:55 +09:00
57b2e4a4f1
Fix 32-bit compilation issues ( #1757 )
2021-08-31 19:36:02 +09:00
48817d849f
Require explicit length in s3fs_decode64 ( #1755 )
...
This is available from std::string::size in callers.
2021-08-31 09:22:10 +09:00
d9f2d17040
1. fix RowFlush can not upload last part smaller than 5MB using NoCacheMultipartPost; ( #1753 )
...
2. fix deadlock in UploadPendingMeta
2021-08-31 00:41:47 +09:00
cd98afdd7b
Do not NUL terminate base64 decoded output ( #1752 )
...
This is binary data and must use the explicit length.
2021-08-31 00:15:47 +09:00
dac6885fb0
Don't over-allocate in base64 encoding and decoding ( #1751 )
2021-08-30 00:03:10 +09:00
fcd180891b
fix misuse of IsUploading ( #1747 )
...
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
2021-08-29 23:41:02 +09:00
a868c0656e
Changed etaglist_t from string list to new structure etagpairs list
2021-08-16 09:27:12 +09:00
15e89b78de
Add a partial page_list unit test ( #1735 )
2021-08-04 07:36:32 +09:00
66006ba48d
Add dedicated upper- and lower-case hex functions ( #1734 )
...
This makes the call sites more readable than a boolean parameter.
2021-08-04 07:28:51 +09:00
18e9c62087
Make string constants read-only const ( #1733 )
...
This removes some global constructors. Also use a consistent ALL_CAPS
style.
2021-08-03 00:10:27 +09:00
34f89e5936
Added fake_diskfree option to deceive free disk space for test
2021-08-02 11:04:42 +09:00
e1f3b9d8c1
Add support for AWS-style environment variables ( #1729 )
...
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.
The old environment variables are still supported, but they are
deprecated and no longer documented.
Close #1708
2021-07-26 23:29:45 +09:00
d3278f4886
Loosen CheckBucket to check only the bucket ( #1728 )
...
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460 . Fixes #1687 .
2021-07-25 12:29:00 +09:00
77f0b75d2f
Add jitter to avoid thundering herd ( #1723 )
2021-07-25 10:22:19 +09:00
199b3d4709
Fixed a bug in disk free space calculation
2021-07-18 14:52:11 +09:00
7890989cbb
Address cppcheck 2.5 warnings ( #1720 )
...
This slightly simplifies memory management.
2021-07-14 22:18:09 +09:00
945cc2ac54
Added UntreatedParts class instead of untreated upload info members in PseudoFdInfo
2021-07-12 21:45:29 +09:00
c30acbbf90
Splited some methods of FdEntity class by uploading mode
...
- Splited FdEntity::Write method by uploading mode
- Splited FdEntity::RowFlush method by uplading mode
2021-07-10 17:24:34 +09:00
881025cc9e
Add initiator pid, uid and gid to a debug log ( #1716 )
2021-07-10 16:23:20 +09:00
4cc210c5ab
Prevents the etag string buffer from being destroyed during use
...
and absorbed MPPART_INFO structure into filepart structure
2021-07-03 22:42:18 +09:00
fe44355d25
Report s3fs space as 18.4 EB ( #1709 )
...
256 TB probably seemed large in 2007 but in 2021 it is small enough to
confuse some users.
2021-07-02 07:56:05 +09:00
487df27008
Return EREMOTE when reading Glacier objects ( #1705 )
...
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466 .
2021-06-30 09:25:36 +09:00
1965916f7a
Convert storage class values to uppercase ( #1704 )
...
AWS requires uppercase values. Fixes a regression from
bbcccd6e98#1613 .
2021-06-30 09:03:31 +09:00
8948eded09
Address clang-tidy warnings ( #1703 )
2021-06-30 08:42:44 +09:00
2f59cb5a0a
Remove unneeded volatile qualifiers ( #1702 )
...
We should use proper locking instead. Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
2021-06-30 08:20:44 +09:00
f505c8224e
Explicitly specify lock type ( #1701 )
...
This makes it more clear and type-safe if the caller already has the
lock. Follows on to 84174c560d
2021-06-27 15:15:48 +09:00
2154e898bc
Fix typos ( #1700 )
2021-06-27 11:22:33 +09:00
f9e80f995d
Fixed a bug about rename existing file of different sizes by mpcopy
2021-06-26 00:04:58 +09:00
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating ( #1681 )
...
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/ .
Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-06-25 20:26:12 +09:00
6edf3d6427
Updated to clearly output message about the file descriptor type(pseudo/physical)
2021-06-25 19:18:08 +09:00
20281737b1
Set explicit Content-Length: 0 when initiating MPU ( #1696 )
...
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661 .
2021-06-25 18:36:40 +09:00
5b6684ca19
Fixed a bug utimens is calling before flush
2021-06-21 17:20:24 +09:00
c4ac923b4c
Ensure NUL-terminated result after strncpy ( #1694 )
...
Long symlinks may cause that the result buffer is filled and not proper
terminated with a null byte.
2021-06-21 08:08:56 +09:00
cc022a68f4
Fixed a bug in s3fs_write
2021-06-20 23:13:44 +09:00
858562ed53
Fixed comparison warning for size_t variable
2021-06-20 16:53:25 +09:00
b8724425d3
Require explicit base parameter in cvt_strtoofft ( #1683 )
...
Also convert most callers of cvt_strtoofft to base 10 which avoid the
magical behavior of interpreting a leading 0 as octal.
References #1682 .
2021-06-20 11:00:15 +09:00
b9ce0faee2
Set Content-MD5 header even for empty objects
...
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
2021-06-19 16:10:58 +09:00
1e0e2752bf
Improve warning for terminated threads
...
with non-zero return code.
2021-06-18 07:54:38 +09:00
7a488b93d0
Set decimal base for converting subseconds to int
...
The leading 0 triggers an automatic conversion as an octal value. This
fails because it is a decimal value.
Setting the base to 10 prevents this automatism and treats the value
as a decimal value.
Fixes: [WAN] string_util.cpp:cvt_strtoofft(96): something error is occurred in convert std::string(017080564) to off_t, thus return 0 as default.
Related to #1676
2021-06-17 06:27:02 +09:00
d67b83e671
Allow configuration for temporary files directory
2021-06-16 21:29:58 +09:00
a100be9dce
Prefer std::string::empty over length == 0 ( #1679 )
...
This is more concise.
2021-06-13 20:03:10 +09:00
4d39ea887e
Protect orgmeta with fdent_lock ( #1678 )
...
Fixes a crash seen with the CentOS 7 builder and when running with
AddressSanitizer. Regression introduced by
ac578d188eFixes #1677 .
2021-06-13 16:14:24 +09:00
7638b5b3e3
Prefer std::string::rbegin over operator[] ( #1673 )
...
This is more concise and safer due to not repeating the variable name.
We cannot use std::string::back since it is not available in C++03.
2021-06-13 13:26:38 +09:00
600cee118d
Fix implicit narrowing conversions ( #1672 )
...
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
2021-06-13 12:50:07 +09:00
c2c56d0263
Added info object about multipart uploading for each pseudo fd
...
(and fixed typo about method name)
2021-06-04 22:42:58 +09:00
ac578d188e
Introduced pseudo fd and separated fd for each file opening
2021-06-04 22:42:58 +09:00
53dfd48f59
Correct usage of istringstream ( #1670 )
...
Previously this looped one more time than necessary due to the eof
check:
https://isocpp.org/wiki/faq/input-output#istream-and-eof
Remove now redundant empty check.
2021-06-02 07:14:32 +09:00
84174c560d
Fix data races caused by incorrect locking ( #1668 )
...
Found via Threadsanitizer. Fixes #1471 .
2021-05-29 00:11:55 +09:00
9bf525ee7a
Ensuring multipart size even when storage is low
...
When the temporary storage was full, the old implementation started an
upload even if there was not enough data to completely fill the minimum
multipart size or fill the user-selected multipart size.
The new implementation ensures the minimum multipart size by forcing
the user-selected multipart size.
Fixes #1591
2021-05-27 22:55:52 +09:00
4b69d4b1bb
Fixed a bug when the disk capacity was insufficient in RowFlush
2021-05-22 23:11:46 +09:00
bbcccd6e98
Make storage class a string ( #1663 )
...
This allows non-standard storage classes like Google Cloud Storage
Nearline. Fixes #1613 .
2021-05-21 23:34:31 +09:00
b5fef788da
Additional bug fixing for not creating zero-byte object
2021-05-09 17:33:53 +09:00
42f5965d8a
Fix off_t to int narrowing in PageList::GetSparseFilePages ( #1654 )
...
Found via clang-tidy.
2021-05-09 14:35:00 +09:00
d904d91252
Prefer std::map::insert over find and operator[] ( #1653 )
...
This avoids a duplicate lookups.
2021-05-09 14:11:35 +09:00
9abe3fa662
Use same time when creating directory objects ( #1652 )
...
This avoids mismatched times and is slightly faster.
2021-05-09 13:44:12 +09:00
bb6d2b1b74
Replace snprintf with string and ostringstream ( #1649 )
...
These uses are probably safe from a buffer overflow perspective but
can cause data race issues in logging due to static buffers.
2021-05-08 02:48:47 +09:00
096a230b70
Allow arbitrary size AWS secret keys ( #1648 )
...
Previously s3fs limited these to 123 characters. Fixes #1626 .
2021-05-06 22:24:38 +09:00
8ef01d37a9
Fix a few nits ( #1645 )
...
Make some strings more const, initialize members, and abort if lock
initialization fails. Partially found via clang-tidy.
2021-05-06 19:40:35 +09:00
cb9148f6cd
Update for clang-tidy 12 ( #1644 )
2021-05-03 09:43:18 +09:00
771bbfeac5
Do not create zero-byte object when creating file ( #1640 )
...
Previously s3fs created this object to store metadata and overwrote it
when flushing. This prevented use with object stores which do not
allow overwrites like HDS. Instead only create an in-memory
representation which reduces the time to create small files.
Fixes #1013 .
2021-04-30 19:56:33 +09:00
3694786112
Propagate errno instead of EIO ( #1638 )
...
This improves error fidelity. Follows on to
b70f8db037#1523 .
2021-04-30 07:09:00 +09:00
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses ( #1635 )
...
S3 can emit these in unusual situations. Fixes #1317 .
2021-04-25 13:18:11 +09:00
8a51a26819
Allow truncation of open and modified files
...
Regression introduced in f5bf41cf11Fixes #1575 .
2021-04-25 12:35:35 +09:00
1838f52e19
Declare undefined symbols for fallocate function
2021-04-25 10:12:25 +09:00
e9eb248f2f
Fail CheckBucket when S3 returns PermanentRedirect ( #1630 )
...
Previously s3fs allowed mounting but all operations returned EIO.
References #693 .
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-04-24 18:46:24 +09:00
77581eda59
Change in ibm iam endpoint
2021-04-21 21:42:55 +09:00
7f3e423bbe
Preserve sub-second time precision ( #1624 )
...
Found via pjdfstests. References #897 . References #1589 .
2021-04-18 13:11:12 +09:00
706e3cbebd
Update ctime and mtime when opening with O_TRUNC ( #1623 )
...
Found via pjdfstests. References #1589 .
2021-04-15 07:08:49 +09:00
2effffd8e2
Fixed codes for cppcheck 2.4.1
2021-04-13 21:12:35 +09:00
2908878988
Fixed codes for cppcheck 2.4.1
2021-04-13 21:12:35 +09:00
01a26a9011
Do not allow open with O_RDONLY and O_TRUNC ( #1620 )
...
Found via pjdfstests. References #1589 .
2021-04-13 07:28:10 +09:00
23e1fbf7b9
Update ctime when truncating file ( #1619 )
...
Found via pjdfstests. References #1589 .
2021-04-13 07:08:06 +09:00
ed2e877bb6
Return EACESS when lacking required executable bit ( #1617 )
...
This makes the check consistent with read and write. Found via
pjdfstests. References #1551 . References #1589 .
2021-04-11 22:39:27 +09:00
3663082a01
Do not allow renaming to a non-empty directory ( #1604 )
...
Found via pjdfstests. References #1589 .
2021-03-28 19:10:52 +09:00
9645d57c05
Translate KeyTooLongError to ENAMETOOLONG ( #1602 )
...
AWS does not support keys longer than 1024 characters. Add special
handling for HeadObject which does not have a response body.
Found via pjdfstests. References #1589 .
2021-03-28 13:17:41 +09:00
f6fbd75320
Return EFBIG when file exceeds multipart limit ( #1600 )
...
Found via pjdfstests. References #1589 .
2021-03-28 12:33:01 +09:00
ef079f4e94
Don't ignore nomultipart when storage is low
...
When the temporary storage filled up, the old implementation uploaded
all data with multipart uploads, even if "nomultipart" was set.
The new implementation emits a warning and returns -ENOSPC instead.
Fixes #1595
2021-03-10 08:23:54 +09:00
b589ebec23
Output version and command parameters at startup
2021-03-06 19:11:57 +09:00
032fcf2a47
Allow optional issuing of ListObjectsV2 ( #1583 )
...
This allows use of s3fs on object stores that do not implement
the V1 API. Fixes #1573 .
2021-02-23 09:45:13 +09:00
5b5bc3114a
Fixed a bug that padded null bytes when changing xattr
2021-02-21 12:23:36 +09:00
81e267d421
Fixed the same bug in chmod/chown/utimens and added tests
2021-02-20 23:30:50 +09:00
4fc92d59f3
Fixed a bug that filling NULL bytes when changing the attribute after renaming
2021-02-20 23:30:50 +09:00
a0f347b10f
Added no_time_stamp_ms option mainly for testing
2021-02-20 20:24:26 +09:00
4da56acdcc
Improved processing when HEAD response is 400
2021-02-13 17:00:13 +09:00
493802a605
Fixed osx mistype declaration and typo warnings
2021-02-13 12:04:35 +09:00
22b0ae9d51
set IsExpireTime to true by default
...
fixes #1563
2021-02-12 12:20:47 +09:00
134a54b32f
Allow configuring the multipart threshold ( #1562 )
...
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-11 23:35:46 +09:00
7f6fbb0021
Correct singlepart_copy_limit unit ( #1561 )
...
This is documented to be in MB not KB.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-11 22:53:50 +09:00
ec8bae9827
Do not periodically flush with nomultipartupload ( #1559 )
...
s3fs periodically flushes data to S3 and hole-punches the written
data, marking it as clean. Without multipart copy, s3fs would write
the now-zeroed data along with any new data. s3fs should not attempt
to periodically flush dirty data at all without multipart copy
support. Fixes #1542 .
2021-02-11 22:24:39 +09:00
a4d916af13
Allow configuring multipart copy size ( #1555 )
...
Also align configuration with mixupload.
2021-02-08 20:32:12 +09:00
32f096fa3f
Use clock_gettime instead of gettimeofday
2021-02-08 10:19:28 +09:00
c692093921
Copy at most 5 GB per multipart copy part request ( #1553 )
...
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes. This symptom was seen more frequently due
to periodic flushing of dirty data. Fixes #1547 .
2021-02-07 23:10:07 +09:00
1a6d0826b5
Allow -1 value to disable max_dirty_data ( #1552 )
...
This matches the man page.
2021-02-07 22:14:31 +09:00
bf33fe7f55
Increase buffer size to avoid warning ( #1550 )
...
This can be up to 54 bytes.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-07 12:26:58 +09:00
ff9d6a75c7
Parse more specific AWS error codes ( #1549 )
...
Fixes #1545 .
2021-02-07 11:29:08 +09:00
4c6690f5f0
Add timestamps to debug log and cache integrity check ( #1540 )
...
* Add UTC timestamp to debug output
for easier correlation with external events
Old output:
$ s3fs mybucket /bucket/ -o curldbg,dbglevel=debug -d -d -f
[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from...
[INF] s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=0, ...
New output:
$ s3fs mybucket /bucket/ -o curldbg,dbglevel=debug -d -d -f
2021-01-28T21:09:16.264Z [CRT] s3fs_logger.cpp:LowSetLogLevel(202):...
2021-01-28T21:09:16.264Z [INF] s3fs.cpp:set_mountpoint_attrib...
* Add UTC timestamp to cache integrity check
for easier correlation with external events.
$ s3fs mybucket /mybucket -oset_check_cache_sigusr1=/tmp/check.cache
Old output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
------------------------------------------------------------
Check cache file and its stats file consistency
------------------------------------------------------------
------------------------------------------------------------
Summary - Total files: 0
Detected error files: 0
Detected error directories: 0
------------------------------------------------------------
New output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
---------------------------------------------------------------------------
Check cache file and its stats file consistency at 2021-01-30T13:04:14.111Z
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Summary - Total files: 0
Detected error files: 0
Detected error directories: 0
---------------------------------------------------------------------------
* Fix indentation of S3fsLog::GetCurrentTime()
2021-02-04 10:41:29 +09:00
67b9381825
Remove limit for single-part objects
...
AWS enforces a 5 GB limit for single-part objects but other S3
implementations like GCS support larger objects. Fixes #1542 .
2021-02-04 09:19:37 +09:00
4b53d4bf6b
Suppress spurious SHA256 error message ( #1544 )
2021-02-04 07:28:51 +09:00
ed85b72bf5
Add warning for failing metadata updates of large files
...
Metadata updates fail for large files if "-o nocopyapi" or
"-o nomultipart" is set.
This change is related to #1528
2021-01-28 10:20:32 +09:00
17fda89ae9
Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors ( #1534 )
...
Follows on to 4d833a4fb9
2021-01-25 19:08:14 +09:00
d019dda4f7
Simplify substr manipulations with erase ( #1532 )
...
This avoids creating a new std::string.
2021-01-25 18:02:32 +09:00
dc9255bc5f
Prefer static_cast where possible ( #1531 )
2021-01-25 08:15:17 +09:00
b0e8758b63
Use result instead of res for consistency ( #1530 )
2021-01-25 07:56:10 +09:00
b70f8db037
Propagate errno from get_local_fent ( #1525 )
...
Follows on to 4d833a4fb9#1523 .
2021-01-20 23:21:14 +09:00
4d833a4fb9
Return more specific errno when available ( #1520 )
...
Previously s3fs threw away some function return values and returned
EIO instead. This was due to not trusting the mix of -1 and errno
return codes. Correct the obviously incorrect ones via visual
inspection. Stronger typing may find more occurrences. Fixes #1519 .
2021-01-18 18:50:49 +09:00
168e588ac7
fix: Add reset offset ( #1503 )
2021-01-04 22:57:56 +09:00
bd0fadbe5f
Remove authorization header when remaking handle ( #1505 )
...
This avoids including Authorization in SignedHeaders. s3fs will
recreate the Authorization header before sending the request.
2021-01-04 21:37:34 +09:00
d1c638ab7a
fix 404 error message
2020-12-30 14:10:17 +09:00
51f65d7b14
fix: miss header when retry
2020-12-23 22:16:07 +09:00
a16d00d673
Simply curl_slist_sort_insert ( #1494 )
...
Some good taste from Linus:
https://github.com/mkirchner/linked-list-good-taste
Also avoid an allocation when replacing a value and tighten up tests.
2020-12-23 20:29:33 +09:00
d0363b118e
Add tests for curl_util ( #1481 )
2020-11-24 21:37:09 +09:00
533322859d
Added comment for nullPointerRedundantCheck/cppcheck2.2
2020-11-21 20:06:14 +09:00
f8d5b76edb
Simplify handling of returned ETag ( #1479 )
...
This works around lifetime warnings uncovered by cppcheck.
References #1478 .
2020-11-21 06:56:05 +09:00
d9f6469b7b
Fixed flushing dirty data and compressed the cache size
2020-11-14 16:45:37 +09:00
543231c9f2
Address warnings from clang-tidy 11 ( #1470 )
2020-11-09 21:15:20 +09:00
d96a08d4ad
fixup! Implement AWS IMDSv2 support
2020-11-07 14:48:52 +09:00
f2f930300a
Implement AWS IMDSv2 support
...
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2. It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.
With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials. In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode. It can also be selected explicitly via
the imdsv1only mount option.
More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/
Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
2020-11-07 14:48:52 +09:00
81ad3ce0ae
Fix directory authority
2020-11-07 14:47:53 +09:00
38e1eaa8a3
Added logfile option for non-syslog logging
2020-10-19 20:10:19 +09:00
6aa786b886
Flush dirty data after a writing a number of bytes ( #1448 )
...
This allows s3fs to write large files without consuming a large amount
of temporary local storage but can slow uploads due to server-side
copies. References #617 . Fixed #1056 . Fixes #1257 .
2020-10-13 22:30:42 +09:00
58750cc441
Fixed a bug that symlink could not be read after restarting s3fs
2020-10-11 22:14:36 +09:00
910255745e
Allow relative path with use_path
...
Fixes #1161 .
2020-10-10 18:13:23 +09:00
cf86fa51b0
Merge pull request #1441 from gaul/coverity
...
Suppress false positive from Coverity
2020-10-04 23:44:51 +09:00
0492f75197
Merge pull request #1439 from gaul/run-once
...
Protect curl_warnings_once with a mutex
2020-10-04 22:43:56 +09:00
493cf20f95
Merge pull request #1435 from gaul/no-exceptions
...
Remove exceptions from s3fs_strtoofft
2020-10-04 22:19:25 +09:00
3553fb65a0
Suppress false positive from Coverity
2020-10-04 14:58:13 +09:00
059cc57ba6
Added atime and Corrected atime/mtime/ctime operations
2020-10-04 13:54:01 +09:00
4df4ffe06f
Protect curl_warnings_once with a mutex
...
This allows ThreadSanitizer to run without warnings.
2020-10-03 21:09:35 +09:00
462347256d
Changed the message level in the AutoFdEntity::GetFdEntity
2020-10-03 20:41:17 +09:00
133feb67c3
Merge pull request #1434 from gaul/const-string
...
Use const std::string& where possible
2020-10-03 10:02:03 +09:00
2cf195741c
Fixed signature error due to case of hex string
2020-10-02 18:09:13 +00:00
f61baada46
Merge pull request #1433 from gaul/hex
...
Further simplify and centralize hex conversion
2020-10-03 00:34:02 +09:00
6e1e0d1d31
Merge pull request #1432 from gaul/v2-or-v4
...
Add sigv4 flag and fix v4 regression
2020-10-03 00:09:55 +09:00
66419e7292
Merge pull request #1431 from gaul/string/find_first_of
...
Prefer string::find_first_of(char) where possible
2020-10-02 23:42:17 +09:00
9e998877e9
Merge pull request #1430 from gaul/ostringstream
...
Simplify use of ostringstream
2020-10-02 23:26:41 +09:00
3628b9d1e2
Remove exceptions from s3fs_strtoofft
...
Explicit return codes are simpler and safer.
2020-10-02 22:53:12 +09:00
8d2bd874d7
Use const std::string& where possible
...
This hints to callers on usage.
2020-10-02 08:48:43 +09:00
7e27c6cf7d
Further simplify and centralize hex conversion
2020-10-01 23:31:06 +09:00
b8ff4ede49
Sign GET requests with V4 signature correctly
...
This bug was previously masked by v2 fallback. Fixes regression from
81805715bd
2020-10-01 20:03:14 +09:00
081d6c1245
Allow setting signature V4-only
...
Default to allowing V2 or V4.
2020-10-01 20:03:14 +09:00
eb8004c355
Prefer string::find_first_of(char) where possible
2020-10-01 18:24:45 +09:00
b3bf9f8f54
Simplify use of ostringstream
...
Resetting this does not re-use the underlying buffer so creating a new
ostringstream has similar efficiency.
2020-10-01 17:55:34 +09:00
503c86bb8a
Call is_prefix instead of compare and substr
2020-09-27 22:19:54 +09:00
757f4caee8
Merge pull request #1425 from gaul/create-bucket/sse
...
Do not send SSE headers during bucket creation
2020-09-27 11:34:15 +09:00
95fabd1f3a
Merge pull request #1423 from gaul/nettle
...
Fix MD5 when using gnutls with nettle
2020-09-27 11:17:55 +09:00
0b42e08636
Merge pull request #1422 from gaul/c++/append-assign-at
...
Remove calls to append, assign, and at
2020-09-27 10:59:48 +09:00
d5e4f99e72
Merge pull request #1413 from liuyongqing/master
...
not call put headers if not exist pending meta
2020-09-27 10:37:22 +09:00
f35fe850c0
Do not send SSE headers during bucket creation
...
This allows integration tests to pass with use_sse.
2020-09-26 13:25:18 +09:00
9a55c9fd9f
Fix MD5 when using gnutls with nettle
...
This matches the non-nettle definition of s3fs_md5_fd.
2020-09-25 23:07:50 +09:00
041b4ec05c
enforce exact key match in IBM IAM response
2020-09-25 22:41:19 +09:00
2438066d52
Remove calls to append, assign, and at
...
operator+, operator=, and operator[] are more idiomatic and consistent
with the code base.
2020-09-25 12:19:51 +09:00
e98827ec6f
not call put headers if not exist pending meta
2020-09-21 14:12:07 +08:00
05863a3178
Fix clang-tidy warnings
2020-09-21 07:03:42 +09:00
075d161bb1
Merge pull request #1408 from gaul/env
...
Ensure environment variable is set when using ECS
2020-09-20 13:01:17 +09:00
6f40503328
Merge pull request #1406 from gaul/strcmp
...
Remove unnecessary calls to strcmp(str.c_str())
2020-09-20 11:28:41 +09:00
3440c3348c
Merge pull request #1405 from gaul/is-prefix
...
Rename STR2NCMP to is_prefix for clarity
2020-09-20 10:03:53 +09:00
853be26612
Merge pull request #1404 from liuyongqing/master
...
fix dead lock in disk insufficient and optimize code
2020-09-20 01:34:02 +09:00
44d2cc15f7
Ensure environment variable is set when using ECS
...
Previously s3fs crashed when AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
was not set. References #1162 .
2020-09-18 08:14:58 +09:00
81805715bd
Remove unnecessary calls to strcmp(str.c_str())
2020-09-17 20:00:45 +09:00
ebcbb02d16
Rename STR2NCMP to is_prefix for clarity
...
This may prevent confusion seen in
39ec8e242e
2020-09-17 19:45:54 +09:00
93cf67c65d
Merge pull request #1399 from gaul/md5-sha256
...
Use more descriptive names for digest functions
2020-09-17 17:11:19 +09:00
211cc0f5f2
fix dead lock in disk insufficient and optimize code
2020-09-16 22:45:28 +08:00
c58c91fc4f
Merge pull request #1398 from gaul/memset
...
Remove unneeded memset calls
2020-09-15 22:09:54 +09:00
b7187352e1
Use more descriptive names for digest functions
...
Previously they used hex to refer to binary data which is confusing
since other std::string return types are actually hex. Also remove
unused s3fs_md5sum.
2020-09-15 21:18:58 +09:00
1520ca6220
Merge pull request #1397 from gaul/hex
...
Centralize hex conversion
2020-09-15 21:14:23 +09:00
f7a63d5c97
Merge pull request #1396 from gaul/32-bit-read
...
Use off_t for read sizes
2020-09-15 20:05:19 +09:00
3958450c05
Added a class for automating fdentity reference counts
2020-09-14 22:02:28 +09:00
5121c73ed1
Remove unneeded memset calls
...
Also use constant for array lengths.
2020-09-14 19:51:36 +09:00
44eaac8471
Centralize hex conversion
2020-09-14 19:28:59 +09:00
77501c3600
Use off_t for read sizes
...
This fixes an issue with large multipart_size on 32-bit platforms like
Raspberry Pi.
2020-09-14 18:12:23 +09:00
8205607716
Fixed incorrect Signal related message at s3fs initialization and termination
2020-09-13 18:54:10 +09:00
1043e08dfa
Remove uses of implicit namespace std
...
Fixed via:
sed -i '/using namespace std/{N;d}' src/*.cpp
sed -i 's/ string/ std::string/g' src/*.cpp
sed -i 's/(string/(std::string/g' src/*.cpp
sed -i 's/\[string/\[std::string/g' src/*.cpp
sed -i 's/^string/std::string/g' src/*.cpp
sed -i 's/ ifstream/ std::ifstream/g' src/*.cpp
sed -i 's/ istringstream/ std::istringstream/g' src/*.cpp
sed -i 's/ ostringstream/ std::ostringstream/g' src/*.cpp
sed -i 's/ max(/ std::max(/g' src/*.cpp
sed -i 's/ min(/ std::min(/g' src/*.cpp
sed -i 's/ endl/ std::endl/g' src/*.cpp
2020-09-13 11:57:20 +09:00
e5f4f9b69e
Merge pull request #1393 from gaul/32-bit-casting
...
Fix casting warnings on 32-bit platforms
2020-09-13 10:40:56 +09:00
1e3c10d803
Merge pull request #1392 from gaul/large-singlepart-32-bit
...
Use 64-bit off_t when computing digests
2020-09-13 10:39:34 +09:00
6d1e704e34
Fix casting warnings on 32-bit platforms
2020-09-12 18:22:32 +09:00
ffc33a447f
Use 64-bit off_t when computing digests
...
This allows 32-bit platforms like Raspberry Pi to upload single-part
objects larger than 2 GB.
2020-09-12 15:00:23 +09:00
39ec8e242e
Remove incorrect string prefix comparisons
...
Previously flags like -o nouafoobar were parsed as -o noua. Found
via:
grep STR2NCMP src/s3fs.cpp | grep -v '="'
2020-09-11 17:38:20 +09:00
0d4e39ad1c
return 0 after parsing "use_session_token" arg ( #1388 )
...
return 0 after parsing "use_session_token" arg
I hope this might fix #651 .
There's clearly an open bug in #651 confirmed by several users related to the parsing of the `-o use_session_token`. Looking at the source, I noticed that there was a return value everywhere except here, so I suspect this may be responsible.
2020-09-11 08:09:36 +09:00
986fab8738
close FdEntity to avoid leakage
2020-09-07 20:48:16 +08:00
b5ffd419d8
Source file division and set 4 spaces and cleanup
2020-08-26 17:43:50 +09:00
c6e23212bb
Merge pull request #1375 from gaul/cppcheck
...
Add const where possible
2020-08-21 07:32:39 +09:00
1ec8528502
Add const where possible
...
Found via cppcheck 2.1
2020-08-20 23:46:11 +09:00
892e7129c5
Add support for deep archive storage class
2020-08-20 23:42:45 +09:00
bdea2ee5c8
Merge pull request #1373 from gaul/enum
...
Use scoped enums for acl_t, sse_type_t, and storage_class_t
2020-08-20 23:06:52 +09:00
a5186c73c2
Merge pull request #1365 from ggtakec/check_lseek
...
Dynamically determine whether lseek extended options are supported
2020-08-20 22:36:56 +09:00
60d456a993
Use scoped enums for acl_t, sse_type_t, and storage_class_t
...
This prevents some kinds of implicit conversions. Also deduplicate
str/from_str logic. References #1371 .
2020-08-20 18:46:18 +09:00
bcf6838e86
Merge pull request #1370 from gaul/vim/modeline
...
Correct vim modeline to 2-space indentation
2020-08-19 22:44:40 +09:00
958ad83a4b
Correct vim modeline to 2-space indentation
2020-08-19 21:03:46 +09:00
ba61470bae
Prefer named locks in curl
...
This removes the confusing and unsafe pthread_mutex_t array.
2020-08-18 22:20:41 +09:00
8210a1b2f2
Consistently lock curl_times and curl_progress
...
References #1362 .
2020-08-17 09:47:38 +09:00
55cb8920d5
Dynamically determine whether lseek extended options are supported
2020-08-16 12:37:11 +00:00
46acbf10ba
Merge pull request #1363 from gaul/is-modified-race
...
Lock fdent_data_lock before accessing pagelist
2020-08-16 20:25:24 +09:00
2c0408b95a
Merge pull request #1357 from gaul/readdir
...
Call readdir instead of readdir_r
2020-08-16 18:50:38 +09:00
057da86d87
Merge pull request #1356 from gaul/printf-size_t
...
Prefer %zd and %zu for ssize_t and size_t
2020-08-16 18:16:59 +09:00
8de6cb3504
Lock fdent_data_lock before accessing pagelist
...
Found via ThreadSanitizer. References #1353 . References #1362 .
2020-08-16 17:44:03 +09:00
2bb745cdd7
Fixed a bug about move file over limit of ensure space
2020-08-16 17:42:20 +09:00
35090ba4d5
Call readdir instead of readdir_r
...
Only a single thread uses this directory stream. Further, modern
implementations are thread-safe by default and deprecated this call:
https://man7.org/linux/man-pages/man3/readdir_r.3.html
2020-08-16 13:33:23 +09:00
132a1bebbb
Prefer %zd and %zu for ssize_t and size_t
...
This removes unnecessary casting.
2020-08-15 17:30:23 +09:00
cbd925c56f
Moved the SIGUSR2 handler to S3fsSignals class
2020-07-28 14:54:35 +09:00
63bbb47378
Merge pull request #1341 from gaul/stat-cache/default
...
Change default stat_cache_expire
2020-07-26 23:54:14 +09:00
0fbd0eac80
Change default stat_cache_expire
...
Previously s3fs cached files forever which confused users with
creating objects using another client.
2020-07-26 23:04:43 +09:00
ad1961417d
Added SIGUSR1 option for cache file integrity test
2020-07-26 21:04:11 +09:00
700e288718
Put similar processing together into method GetCacheFileStatTopDir
2020-07-12 08:12:02 +00:00
dc365b65a0
Fix NULL pointer deference
...
Found via cppcheck 2.1.
2020-07-09 21:40:23 +09:00
1f796d432d
Fixed upload error about mixuploading sparse file and truncating file
2020-06-27 22:44:19 +09:00
35006e318f
Fixed about ParallelMixMultipartUpload
2020-06-24 12:48:55 +09:00
7d0c66e08a
Add support for glacier storage class.
...
Just a copy of what have been done in PR #271 .
2020-06-23 11:23:21 +09:00
f324d8e04f
Fixed a bug about serializing from cache file
2020-06-19 12:57:27 +00:00
f16ee96d7e
Merge pull request #1306 from gaul/http/500
...
Retry with exponential backoff during 500 error
2020-06-06 15:30:22 +09:00
d8766b2051
Retry with exponential backoff during 500 error
...
Amazon suggests retrying on both 500 and 503:
https://aws.amazon.com/premiumsupport/knowledge-center/http-5xx-errors-s3/
Fixes #1251 .
2020-06-05 21:01:30 +09:00
9db70bab63
Ignore case when comparing ETags
...
This allows multipart upload to work with Alibaba OSS.
References #1297 .
2020-06-05 18:17:52 +09:00
0cb057dadd
Merge pull request #1303 from gaul/rename/use_cache
...
Relink cache stats file atomically via rename
2020-06-01 00:10:33 +09:00
0f5db0d1bf
Merge pull request #1302 from gaul/rename/nocopy
...
Fix renames of open files with nocopyapi option
2020-05-31 23:46:46 +09:00
Takeshi Nakatani
Petr Vaněk
Andrew Gaul
Naoki Ikeguchi
LiuBingrun
Carsten Grohmann
Kamil Jakrzewski
Noah Meyerhans
Martynov Maxim
Jan Stastny
Dmitrii Vasilev
Ambika Nair
林千里
fly3366
kontrollanten
yoshihiko
Neeraj Kumar Kashyap
yongqingliu
Ben Mares
Eric Vantillard