octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-18 02:05:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,775 Commits 1 Branch 23 Tags 13 MiB
4cc210c5ab
Commit Graph

1775 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Takeshi Nakatani
4cc210c5ab Prevents the etag string buffer from being destroyed during use 
and absorbed MPPART_INFO structure into filepart structure
 2021-07-03 22:42:18 +09:00
Takeshi Nakatani
2d5316a334 Fixed 1710 - Added a flag for check in Linux 2021-07-02 11:34:51 +09:00
Andrew Gaul
9cfa177af0
Add flag to run all tests (#1710) 
This allows CI to run the full set of tests while making local
development faster.
 2021-07-02 08:41:47 +09:00
Andrew Gaul
fe44355d25
Report s3fs space as 18.4 EB (#1709) 
256 TB probably seemed large in 2007 but in 2021 it is small enough to
confuse some users.
 2021-07-02 07:56:05 +09:00
Andrew Gaul
487df27008
Return EREMOTE when reading Glacier objects (#1705) 
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466.
 2021-06-30 09:25:36 +09:00
Andrew Gaul
1965916f7a
Convert storage class values to uppercase (#1704) 
AWS requires uppercase values.  Fixes a regression from
bbcccd6e98.  References #1613.
 2021-06-30 09:03:31 +09:00
Andrew Gaul
8948eded09
Address clang-tidy warnings (#1703) 2021-06-30 08:42:44 +09:00
Andrew Gaul
2f59cb5a0a
Remove unneeded volatile qualifiers (#1702) 
We should use proper locking instead.  Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
 2021-06-30 08:20:44 +09:00
Andrew Gaul
f505c8224e
Explicitly specify lock type (#1701) 
This makes it more clear and type-safe if the caller already has the
lock.  Follows on to 84174c560d.
 2021-06-27 15:15:48 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Takeshi Nakatani
f9e80f995d Fixed a bug about rename existing file of different sizes by mpcopy 2021-06-26 00:04:58 +09:00
Jan Stastny
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating (#1681) 
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/.

Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-06-25 20:26:12 +09:00
Takeshi Nakatani
6edf3d6427 Updated to clearly output message about the file descriptor type(pseudo/physical) 2021-06-25 19:18:08 +09:00
Andrew Gaul
944c1fd148
Clean up litter in ut_test (#1697) 2021-06-25 19:15:03 +09:00
Andrew Gaul
20281737b1
Set explicit Content-Length: 0 when initiating MPU (#1696) 
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661.
 2021-06-25 18:36:40 +09:00
Takeshi Nakatani
0555c4216e Added test case for utimens is calling during multipart upload 2021-06-21 17:20:24 +09:00
Takeshi Nakatani
5b6684ca19 Fixed a bug utimens is calling before flush 2021-06-21 17:20:24 +09:00
Carsten Grohmann
c4ac923b4c
Ensure NUL-terminated result after strncpy (#1694) 
Long symlinks may cause that the result buffer is filled and not proper
terminated with a null byte.
 2021-06-21 08:08:56 +09:00
Takeshi Nakatani
cc022a68f4 Fixed a bug in s3fs_write 2021-06-20 23:13:44 +09:00
Andrew Gaul
1571379304
Set LC_ALL=en_US.UTF-8 in tests (#1691) 
This allows tests to pass using the S3Proxy filesystem provider.
References #1665.
 2021-06-20 19:27:25 +09:00
Takeshi Nakatani
858562ed53 Fixed comparison warning for size_t variable 2021-06-20 16:53:25 +09:00
Andrew Gaul
b8724425d3
Require explicit base parameter in cvt_strtoofft (#1683) 
Also convert most callers of cvt_strtoofft to base 10 which avoid the
magical behavior of interpreting a leading 0 as octal.
References #1682.
 2021-06-20 11:00:15 +09:00
Dmitrii Vasilev
b9ce0faee2 Set Content-MD5 header even for empty objects 
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
 2021-06-19 16:10:58 +09:00
Carsten Grohmann
1e0e2752bf Improve warning for terminated threads 
with non-zero return code.
 2021-06-18 07:54:38 +09:00
Carsten Grohmann
7a488b93d0 Set decimal base for converting subseconds to int 
The leading 0 triggers an automatic conversion as an octal value. This
fails because it is a decimal value.

Setting the base to 10 prevents this automatism and treats the value
as a decimal value.

Fixes: [WAN] string_util.cpp:cvt_strtoofft(96): something error is occurred in convert std::string(017080564) to off_t, thus return 0 as default.

Related to #1676
 2021-06-17 06:27:02 +09:00
Carsten Grohmann
d67b83e671 Allow configuration for temporary files directory 2021-06-16 21:29:58 +09:00
Andrew Gaul
a100be9dce
Prefer std::string::empty over length == 0 (#1679) 
This is more concise.
 2021-06-13 20:03:10 +09:00
Andrew Gaul
4d39ea887e
Protect orgmeta with fdent_lock (#1678) 
Fixes a crash seen with the CentOS 7 builder and when running with
AddressSanitizer.  Regression introduced by
ac578d188e.  Fixes #1677.
 2021-06-13 16:14:24 +09:00
Andrew Gaul
7638b5b3e3
Prefer std::string::rbegin over operator[] (#1673) 
This is more concise and safer due to not repeating the variable name.
We cannot use std::string::back since it is not available in C++03.
 2021-06-13 13:26:38 +09:00
Andrew Gaul
600cee118d
Fix implicit narrowing conversions (#1672) 
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
 2021-06-13 12:50:07 +09:00
Takeshi Nakatani
c2c56d0263 Added info object about multipart uploading for each pseudo fd 
(and fixed typo about method name)
 2021-06-04 22:42:58 +09:00
Takeshi Nakatani
ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
Andrew Gaul
53dfd48f59
Correct usage of istringstream (#1670) 
Previously this looped one more time than necessary due to the eof
check:

https://isocpp.org/wiki/faq/input-output#istream-and-eof

Remove now redundant empty check.
 2021-06-02 07:14:32 +09:00
Andrew Gaul
f5701fa9ad
Correct typo in undefinedsanitizer config (#1669) 2021-05-29 22:07:33 +09:00
Andrew Gaul
84174c560d
Fix data races caused by incorrect locking (#1668) 
Found via Threadsanitizer.  Fixes #1471.
 2021-05-29 00:11:55 +09:00
Carsten Grohmann
9bf525ee7a Ensuring multipart size even when storage is low 
When the temporary storage was full, the old implementation started an
upload even if there was not enough data to completely fill the minimum
multipart size or fill the user-selected multipart size.

The new implementation ensures the minimum multipart size by forcing
the user-selected multipart size.

Fixes #1591
 2021-05-27 22:55:52 +09:00
Takeshi Nakatani
4b69d4b1bb Fixed a bug when the disk capacity was insufficient in RowFlush 2021-05-22 23:11:46 +09:00
Andrew Gaul
bbcccd6e98
Make storage class a string (#1663) 
This allows non-standard storage classes like Google Cloud Storage
Nearline.  Fixes #1613.
 2021-05-21 23:34:31 +09:00
Carsten Grohmann
c0bcb41175
Increase test robustness by adding LC_ALL=C (#1660) 
Use C locale, because some tests check for English expressions

Fixes #1658
 2021-05-20 21:19:50 +09:00
Takeshi Nakatani
b5fef788da Additional bug fixing for not creating zero-byte object 2021-05-09 17:33:53 +09:00
Andrew Gaul
42f5965d8a
Fix off_t to int narrowing in PageList::GetSparseFilePages (#1654) 
Found via clang-tidy.
 2021-05-09 14:35:00 +09:00
Andrew Gaul
d904d91252
Prefer std::map::insert over find and operator[] (#1653) 
This avoids a duplicate lookups.
 2021-05-09 14:11:35 +09:00
Andrew Gaul
4d81a4bf68
Use same time when creating directory objects (#1652) 
This avoids mismatched times and is slightly faster.
 2021-05-09 13:44:59 +09:00
Andrew Gaul
9abe3fa662
Use same time when creating directory objects (#1652) 
This avoids mismatched times and is slightly faster.
 2021-05-09 13:44:12 +09:00
Takeshi Nakatani
9d8f1b00f7 Fixed make_random_string test code being slow on macos 2021-05-08 13:55:39 +09:00
Andrew Gaul
bb6d2b1b74
Replace snprintf with string and ostringstream (#1649) 
These uses are probably safe from a buffer overflow perspective but
can cause data race issues in logging due to static buffers.
 2021-05-08 02:48:47 +09:00
Andrew Gaul
096a230b70
Allow arbitrary size AWS secret keys (#1648) 
Previously s3fs limited these to 123 characters.  Fixes #1626.
 2021-05-06 22:24:38 +09:00
Andrew Gaul
8ef01d37a9
Fix a few nits (#1645) 
Make some strings more const, initialize members, and abort if lock
initialization fails.  Partially found via clang-tidy.
 2021-05-06 19:40:35 +09:00
Andrew Gaul
cb9148f6cd
Update for clang-tidy 12 (#1644) 2021-05-03 09:43:18 +09:00
Carsten Grohmann
c4fa53ec8b
Add section about storage consumption to manpage (#1639) 
References s3fs-fuse#1591. References s3fs-fuse#1595.
 2021-05-01 15:09:29 +09:00