octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-16 17:25:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,037 Commits 1 Branch 23 Tags 13 MiB
6344d74ae3
Commit Graph

2037 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Andrew Gaul
66006ba48d
Add dedicated upper- and lower-case hex functions (#1734) 
This makes the call sites more readable than a boolean parameter.
 2021-08-04 07:28:51 +09:00
Andrew Gaul
18e9c62087
Make string constants read-only const (#1733) 
This removes some global constructors.  Also use a consistent ALL_CAPS
style.
 2021-08-03 00:10:27 +09:00
Takeshi Nakatani
34f89e5936 Added fake_diskfree option to deceive free disk space for test 2021-08-02 11:04:42 +09:00
Carsten Grohmann
e1f3b9d8c1
Add support for AWS-style environment variables (#1729) 
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.

The old environment variables are still supported, but they are
deprecated and no longer documented.

Close #1708
 2021-07-26 23:29:45 +09:00
Andrew Gaul
d3278f4886
Loosen CheckBucket to check only the bucket (#1728) 
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460.  Fixes #1687.
 2021-07-25 12:29:00 +09:00
Andrew Gaul
77f0b75d2f
Add jitter to avoid thundering herd (#1723) 2021-07-25 10:22:19 +09:00
Andrew Gaul
012e6dd8a2
Fix README nit (#1726) 2021-07-22 13:54:20 +09:00
Takeshi Nakatani
199b3d4709 Fixed a bug in disk free space calculation 2021-07-18 14:52:11 +09:00
Andrew Gaul
7890989cbb
Address cppcheck 2.5 warnings (#1720) 
This slightly simplifies memory management.
 2021-07-14 22:18:09 +09:00
Takeshi Nakatani
945cc2ac54 Added UntreatedParts class instead of untreated upload info members in PseudoFdInfo 2021-07-12 21:45:29 +09:00
Takeshi Nakatani
c30acbbf90 Splited some methods of FdEntity class by uploading mode 
- Splited FdEntity::Write method by uploading mode
- Splited FdEntity::RowFlush method by uplading mode
 2021-07-10 17:24:34 +09:00
Martynov Maxim
881025cc9e
Add initiator pid, uid and gid to a debug log (#1716) 2021-07-10 16:23:20 +09:00
Takeshi Nakatani
4cc210c5ab Prevents the etag string buffer from being destroyed during use 
and absorbed MPPART_INFO structure into filepart structure
 2021-07-03 22:42:18 +09:00
Takeshi Nakatani
2d5316a334 Fixed 1710 - Added a flag for check in Linux 2021-07-02 11:34:51 +09:00
Andrew Gaul
9cfa177af0
Add flag to run all tests (#1710) 
This allows CI to run the full set of tests while making local
development faster.
 2021-07-02 08:41:47 +09:00
Andrew Gaul
fe44355d25
Report s3fs space as 18.4 EB (#1709) 
256 TB probably seemed large in 2007 but in 2021 it is small enough to
confuse some users.
 2021-07-02 07:56:05 +09:00
Andrew Gaul
487df27008
Return EREMOTE when reading Glacier objects (#1705) 
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466.
 2021-06-30 09:25:36 +09:00
Andrew Gaul
1965916f7a
Convert storage class values to uppercase (#1704) 
AWS requires uppercase values.  Fixes a regression from
bbcccd6e98.  References #1613.
 2021-06-30 09:03:31 +09:00
Andrew Gaul
8948eded09
Address clang-tidy warnings (#1703) 2021-06-30 08:42:44 +09:00
Andrew Gaul
2f59cb5a0a
Remove unneeded volatile qualifiers (#1702) 
We should use proper locking instead.  Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
 2021-06-30 08:20:44 +09:00
Andrew Gaul
f505c8224e
Explicitly specify lock type (#1701) 
This makes it more clear and type-safe if the caller already has the
lock.  Follows on to 84174c560d.
 2021-06-27 15:15:48 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Takeshi Nakatani
f9e80f995d Fixed a bug about rename existing file of different sizes by mpcopy 2021-06-26 00:04:58 +09:00
Jan Stastny
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating (#1681) 
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/.

Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-06-25 20:26:12 +09:00
Takeshi Nakatani
6edf3d6427 Updated to clearly output message about the file descriptor type(pseudo/physical) 2021-06-25 19:18:08 +09:00
Andrew Gaul
944c1fd148
Clean up litter in ut_test (#1697) 2021-06-25 19:15:03 +09:00
Andrew Gaul
20281737b1
Set explicit Content-Length: 0 when initiating MPU (#1696) 
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661.
 2021-06-25 18:36:40 +09:00
Takeshi Nakatani
0555c4216e Added test case for utimens is calling during multipart upload 2021-06-21 17:20:24 +09:00
Takeshi Nakatani
5b6684ca19 Fixed a bug utimens is calling before flush 2021-06-21 17:20:24 +09:00
Carsten Grohmann
c4ac923b4c
Ensure NUL-terminated result after strncpy (#1694) 
Long symlinks may cause that the result buffer is filled and not proper
terminated with a null byte.
 2021-06-21 08:08:56 +09:00
Takeshi Nakatani
cc022a68f4 Fixed a bug in s3fs_write 2021-06-20 23:13:44 +09:00
Andrew Gaul
1571379304
Set LC_ALL=en_US.UTF-8 in tests (#1691) 
This allows tests to pass using the S3Proxy filesystem provider.
References #1665.
 2021-06-20 19:27:25 +09:00
Takeshi Nakatani
858562ed53 Fixed comparison warning for size_t variable 2021-06-20 16:53:25 +09:00
Andrew Gaul
b8724425d3
Require explicit base parameter in cvt_strtoofft (#1683) 
Also convert most callers of cvt_strtoofft to base 10 which avoid the
magical behavior of interpreting a leading 0 as octal.
References #1682.
 2021-06-20 11:00:15 +09:00
Dmitrii Vasilev
b9ce0faee2 Set Content-MD5 header even for empty objects 
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
 2021-06-19 16:10:58 +09:00
Carsten Grohmann
1e0e2752bf Improve warning for terminated threads 
with non-zero return code.
 2021-06-18 07:54:38 +09:00
Carsten Grohmann
7a488b93d0 Set decimal base for converting subseconds to int 
The leading 0 triggers an automatic conversion as an octal value. This
fails because it is a decimal value.

Setting the base to 10 prevents this automatism and treats the value
as a decimal value.

Fixes: [WAN] string_util.cpp:cvt_strtoofft(96): something error is occurred in convert std::string(017080564) to off_t, thus return 0 as default.

Related to #1676
 2021-06-17 06:27:02 +09:00
Carsten Grohmann
d67b83e671 Allow configuration for temporary files directory 2021-06-16 21:29:58 +09:00
Andrew Gaul
a100be9dce
Prefer std::string::empty over length == 0 (#1679) 
This is more concise.
 2021-06-13 20:03:10 +09:00
Andrew Gaul
4d39ea887e
Protect orgmeta with fdent_lock (#1678) 
Fixes a crash seen with the CentOS 7 builder and when running with
AddressSanitizer.  Regression introduced by
ac578d188e.  Fixes #1677.
 2021-06-13 16:14:24 +09:00
Andrew Gaul
7638b5b3e3
Prefer std::string::rbegin over operator[] (#1673) 
This is more concise and safer due to not repeating the variable name.
We cannot use std::string::back since it is not available in C++03.
 2021-06-13 13:26:38 +09:00
Andrew Gaul
600cee118d
Fix implicit narrowing conversions (#1672) 
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
 2021-06-13 12:50:07 +09:00
Takeshi Nakatani
c2c56d0263 Added info object about multipart uploading for each pseudo fd 
(and fixed typo about method name)
 2021-06-04 22:42:58 +09:00
Takeshi Nakatani
ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
Andrew Gaul
53dfd48f59
Correct usage of istringstream (#1670) 
Previously this looped one more time than necessary due to the eof
check:

https://isocpp.org/wiki/faq/input-output#istream-and-eof

Remove now redundant empty check.
 2021-06-02 07:14:32 +09:00
Andrew Gaul
f5701fa9ad
Correct typo in undefinedsanitizer config (#1669) 2021-05-29 22:07:33 +09:00
Andrew Gaul
84174c560d
Fix data races caused by incorrect locking (#1668) 
Found via Threadsanitizer.  Fixes #1471.
 2021-05-29 00:11:55 +09:00
Carsten Grohmann
9bf525ee7a Ensuring multipart size even when storage is low 
When the temporary storage was full, the old implementation started an
upload even if there was not enough data to completely fill the minimum
multipart size or fill the user-selected multipart size.

The new implementation ensures the minimum multipart size by forcing
the user-selected multipart size.

Fixes #1591
 2021-05-27 22:55:52 +09:00
Takeshi Nakatani
4b69d4b1bb Fixed a bug when the disk capacity was insufficient in RowFlush 2021-05-22 23:11:46 +09:00
Andrew Gaul
bbcccd6e98
Make storage class a string (#1663) 
This allows non-standard storage classes like Google Cloud Storage
Nearline.  Fixes #1613.
 2021-05-21 23:34:31 +09:00