octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-16 17:25:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,654 Commits 1 Branch 23 Tags 13 MiB
d246b9e8bf
Commit Graph

870 Commits

Author SHA1 Message Date
Takeshi Nakatani
93cf67c65d
Merge pull request #1399 from gaul/md5-sha256 
Use more descriptive names for digest functions
 2020-09-17 17:11:19 +09:00
yongqingliu
211cc0f5f2 fix dead lock in disk insufficient and optimize code 2020-09-16 22:45:28 +08:00
Takeshi Nakatani
c58c91fc4f
Merge pull request #1398 from gaul/memset 
Remove unneeded memset calls
 2020-09-15 22:09:54 +09:00
Andrew Gaul
b7187352e1 Use more descriptive names for digest functions 
Previously they used hex to refer to binary data which is confusing
since other std::string return types are actually hex.  Also remove
unused s3fs_md5sum.
 2020-09-15 21:18:58 +09:00
Takeshi Nakatani
1520ca6220
Merge pull request #1397 from gaul/hex 
Centralize hex conversion
 2020-09-15 21:14:23 +09:00
Takeshi Nakatani
f7a63d5c97
Merge pull request #1396 from gaul/32-bit-read 
Use off_t for read sizes
 2020-09-15 20:05:19 +09:00
Takeshi Nakatani
3958450c05 Added a class for automating fdentity reference counts 2020-09-14 22:02:28 +09:00
Andrew Gaul
5121c73ed1 Remove unneeded memset calls 
Also use constant for array lengths.
 2020-09-14 19:51:36 +09:00
Andrew Gaul
44eaac8471 Centralize hex conversion 2020-09-14 19:28:59 +09:00
Andrew Gaul
77501c3600 Use off_t for read sizes 
This fixes an issue with large multipart_size on 32-bit platforms like
Raspberry Pi.
 2020-09-14 18:12:23 +09:00
Takeshi Nakatani
8205607716 Fixed incorrect Signal related message at s3fs initialization and termination 2020-09-13 18:54:10 +09:00
Andrew Gaul
1043e08dfa Remove uses of implicit namespace std 
Fixed via:

sed -i '/using namespace std/{N;d}' src/*.cpp
sed -i 's/ string/ std::string/g' src/*.cpp
sed -i 's/(string/(std::string/g' src/*.cpp
sed -i 's/\[string/\[std::string/g' src/*.cpp
sed -i 's/^string/std::string/g' src/*.cpp
sed -i 's/ ifstream/ std::ifstream/g' src/*.cpp
sed -i 's/ istringstream/ std::istringstream/g' src/*.cpp
sed -i 's/ ostringstream/ std::ostringstream/g' src/*.cpp
sed -i 's/ max(/ std::max(/g' src/*.cpp
sed -i 's/ min(/ std::min(/g' src/*.cpp
sed -i 's/ endl/ std::endl/g' src/*.cpp
 2020-09-13 11:57:20 +09:00
Takeshi Nakatani
e5f4f9b69e
Merge pull request #1393 from gaul/32-bit-casting 
Fix casting warnings on 32-bit platforms
 2020-09-13 10:40:56 +09:00
Takeshi Nakatani
1e3c10d803
Merge pull request #1392 from gaul/large-singlepart-32-bit 
Use 64-bit off_t when computing digests
 2020-09-13 10:39:34 +09:00
Andrew Gaul
6d1e704e34 Fix casting warnings on 32-bit platforms 2020-09-12 18:22:32 +09:00
Andrew Gaul
ffc33a447f Use 64-bit off_t when computing digests 
This allows 32-bit platforms like Raspberry Pi to upload single-part
objects larger than 2 GB.
 2020-09-12 15:00:23 +09:00
Andrew Gaul
39ec8e242e Remove incorrect string prefix comparisons 
Previously flags like -o nouafoobar were parsed as -o noua.  Found
via:

grep STR2NCMP src/s3fs.cpp | grep -v '="'
 2020-09-11 17:38:20 +09:00
Ben Mares
0d4e39ad1c
return 0 after parsing "use_session_token" arg (#1388) 
return 0 after parsing "use_session_token" arg

I hope this might fix #651.

There's clearly an open bug in #651 confirmed by several users related to the parsing of the `-o use_session_token`. Looking at the source, I noticed that there was a return value everywhere except here, so I suspect this may be responsible.
 2020-09-11 08:09:36 +09:00
yongqingliu
986fab8738 close FdEntity to avoid leakage 2020-09-07 20:48:16 +08:00
Takeshi Nakatani
b5ffd419d8 Source file division and set 4 spaces and cleanup 2020-08-26 17:43:50 +09:00
Takeshi Nakatani
c6e23212bb
Merge pull request #1375 from gaul/cppcheck 
Add const where possible
 2020-08-21 07:32:39 +09:00
Andrew Gaul
1ec8528502 Add const where possible 
Found via cppcheck 2.1
 2020-08-20 23:46:11 +09:00
Andrew Gaul
892e7129c5 Add support for deep archive storage class 2020-08-20 23:42:45 +09:00
Takeshi Nakatani
bdea2ee5c8
Merge pull request #1373 from gaul/enum 
Use scoped enums for acl_t, sse_type_t, and storage_class_t
 2020-08-20 23:06:52 +09:00
Takeshi Nakatani
a5186c73c2
Merge pull request #1365 from ggtakec/check_lseek 
Dynamically determine whether lseek extended options are supported
 2020-08-20 22:36:56 +09:00
Andrew Gaul
60d456a993 Use scoped enums for acl_t, sse_type_t, and storage_class_t 
This prevents some kinds of implicit conversions.  Also deduplicate
str/from_str logic.  References #1371.
 2020-08-20 18:46:18 +09:00
Takeshi Nakatani
bcf6838e86
Merge pull request #1370 from gaul/vim/modeline 
Correct vim modeline to 2-space indentation
 2020-08-19 22:44:40 +09:00
Andrew Gaul
958ad83a4b Correct vim modeline to 2-space indentation 2020-08-19 21:03:46 +09:00
Andrew Gaul
ba61470bae Prefer named locks in curl 
This removes the confusing and unsafe pthread_mutex_t array.
 2020-08-18 22:20:41 +09:00
Andrew Gaul
8210a1b2f2 Consistently lock curl_times and curl_progress 
References #1362.
 2020-08-17 09:47:38 +09:00
Takeshi Nakatani
55cb8920d5 Dynamically determine whether lseek extended options are supported 2020-08-16 12:37:11 +00:00
Takeshi Nakatani
46acbf10ba
Merge pull request #1363 from gaul/is-modified-race 
Lock fdent_data_lock before accessing pagelist
 2020-08-16 20:25:24 +09:00
Takeshi Nakatani
2c0408b95a
Merge pull request #1357 from gaul/readdir 
Call readdir instead of readdir_r
 2020-08-16 18:50:38 +09:00
Takeshi Nakatani
057da86d87
Merge pull request #1356 from gaul/printf-size_t 
Prefer %zd and %zu for ssize_t and size_t
 2020-08-16 18:16:59 +09:00
Andrew Gaul
8de6cb3504 Lock fdent_data_lock before accessing pagelist 
Found via ThreadSanitizer.  References #1353.  References #1362.
 2020-08-16 17:44:03 +09:00
Takeshi Nakatani
2bb745cdd7 Fixed a bug about move file over limit of ensure space 2020-08-16 17:42:20 +09:00
Andrew Gaul
35090ba4d5 Call readdir instead of readdir_r 
Only a single thread uses this directory stream.  Further, modern
implementations are thread-safe by default and deprecated this call:

https://man7.org/linux/man-pages/man3/readdir_r.3.html
 2020-08-16 13:33:23 +09:00
Andrew Gaul
132a1bebbb Prefer %zd and %zu for ssize_t and size_t 
This removes unnecessary casting.
 2020-08-15 17:30:23 +09:00
Takeshi Nakatani
cbd925c56f Moved the SIGUSR2 handler to S3fsSignals class 2020-07-28 14:54:35 +09:00
Takeshi Nakatani
63bbb47378
Merge pull request #1341 from gaul/stat-cache/default 
Change default stat_cache_expire
 2020-07-26 23:54:14 +09:00
Andrew Gaul
0fbd0eac80 Change default stat_cache_expire 
Previously s3fs cached files forever which confused users with
creating objects using another client.
 2020-07-26 23:04:43 +09:00
Takeshi Nakatani
ad1961417d Added SIGUSR1 option for cache file integrity test 2020-07-26 21:04:11 +09:00
Takeshi Nakatani
700e288718 Put similar processing together into method GetCacheFileStatTopDir 2020-07-12 08:12:02 +00:00
Takeshi Nakatani
dc365b65a0 Fix NULL pointer deference 
Found via cppcheck 2.1.
 2020-07-09 21:40:23 +09:00
Takeshi Nakatani
1f796d432d Fixed upload error about mixuploading sparse file and truncating file 2020-06-27 22:44:19 +09:00
Takeshi Nakatani
35006e318f Fixed about ParallelMixMultipartUpload 2020-06-24 12:48:55 +09:00
Eric Vantillard
7d0c66e08a Add support for glacier storage class. 
Just a copy of what have been done in PR #271.
 2020-06-23 11:23:21 +09:00
Takeshi Nakatani
f324d8e04f Fixed a bug about serializing from cache file 2020-06-19 12:57:27 +00:00
Takeshi Nakatani
f16ee96d7e
Merge pull request #1306 from gaul/http/500 
Retry with exponential backoff during 500 error
 2020-06-06 15:30:22 +09:00
Andrew Gaul
d8766b2051 Retry with exponential backoff during 500 error 
Amazon suggests retrying on both 500 and 503:

https://aws.amazon.com/premiumsupport/knowledge-center/http-5xx-errors-s3/

Fixes #1251.
 2020-06-05 21:01:30 +09:00
Andrew Gaul
9db70bab63 Ignore case when comparing ETags 
This allows multipart upload to work with Alibaba OSS.
References #1297.
 2020-06-05 18:17:52 +09:00
Takeshi Nakatani
0cb057dadd
Merge pull request #1303 from gaul/rename/use_cache 
Relink cache stats file atomically via rename
 2020-06-01 00:10:33 +09:00
Takeshi Nakatani
0f5db0d1bf
Merge pull request #1302 from gaul/rename/nocopy 
Fix renames of open files with nocopyapi option
 2020-05-31 23:46:46 +09:00
Andrew Gaul
274321524c Relink cache stats file atomically via rename 
The new file may already exist so link may fail.  Further link/unlink
is not atomic.  Addresses an error when renaming an open with with
use_cache.  References #1296.
 2020-05-31 23:09:58 +09:00
Andrew Gaul
40f7007263 Check results from pthread mutex calls 
Also remove some unnecessary exception handling.
 2020-05-30 16:37:55 +09:00
Andrew Gaul
66597ec5f2 Fix renames of open files with nocopyapi option 
References #1296.
 2020-05-30 15:45:43 +09:00
Takeshi Nakatani
75e72385cc Added a parameter to output body to curldbg option 2020-05-25 08:49:01 +09:00
Andrew Gaul
8b15db6dcb Do not allow zero retries 
Retries actually means tries, e.g., if the user sets zero, s3fs will
never try an operation at all.
 2020-05-23 10:05:23 +09:00
Takeshi Nakatani
80c11b6c12 Not abort process by exception threw from s3fs_strtoofft 2020-05-03 13:46:05 +00:00
Andrew Gaul
8945e98d8b Support Google Cloud Storage headers 
This allows s3fs to interpret objects created by gsutil.
 2020-05-03 18:33:13 +09:00
Takeshi Nakatani
97c249d5b9 Not abort process by exception threw from s3fs_strtoofft 2020-05-03 08:08:28 +00:00
Andrew Gaul
2cc88b933f Warn about missing MIME types instead of exiting 
s3fs uses the MIME types file to set Content-Type for uploaded
objects.  Most distribution packages should install this via
recommended (not required) dependencies.  Users compiling from source
may not have this installed and s3fs should not prevent launching
since most users do not care about Content-Type.  Instead warn about
MIME types absence.  Fixes #1270.
 2020-04-29 20:03:50 +09:00
Andrew Gaul
005a684600 Fix typos 2020-04-22 21:49:11 +09:00
Takeshi Nakatani
f26a0aa71d Fixed insufficient upload size for mix multipart upload 2020-04-22 09:31:22 +09:00
Andrew Gaul
cf529e0af7 Add handler for HTTP 416 
This prevents retries when the server indicates an unsatisfiable MPU
copy part request.  References #1220.
 2020-04-21 19:45:10 +09:00
Takeshi Nakatani
4da02d023b Improved strictness of cache file stats(file) 2020-04-21 19:45:03 +09:00
Takeshi Nakatani
811ea0cb85 Fixed the truncation bug of stat file for cache file 2020-04-19 07:08:49 +00:00
Andrew Gaul
48a872e285 Address cppcheck 1.90 warning 2020-04-12 22:20:44 +09:00
Takeshi Nakatani
c44a60f3f5 Fixed a bug of stats cache compression 2020-04-12 18:33:00 +09:00
Takeshi Nakatani
9e01d5b8d1
Merge pull request #1254 from ggtakec/modify_mimetypes 
Added mime option for strict checking of mime types file
 2020-04-11 14:48:47 +09:00
Takeshi Nakatani
7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
Andrew Gaul
56141557dc Avoid unneeded string copy 
Found by clang-tidy 10.
 2020-03-28 08:49:49 +09:00
yongqingliu
b72f4b43a4 use correct content-type when complete multipart upload 2020-02-10 16:58:28 +09:00
Andrew Gaul
924eeb3587 Document host and servicepath 
The various bits of host, url, and servicepath seem to overlap.
References #1203.
 2020-02-04 21:13:29 +09:00
Andrew Gaul
bc9126d774 Set directory MIME type to application/x-directory 
Previously s3fs auto-detected the MIME type of directories like
"TOYOTA TRUCK 8.2.2" as application/x-troff-man.  This caused get_mode
to not set S_IFDIR which failed directory creation.  Instead force all
object names ending in / to application/x-directory.  Fixes #1183.
 2020-02-04 20:03:21 +09:00
rallister
4e26728cbf
break recursion when calling GetIAMCredentials (#1233) 
break recursion when calling GetIAMCredentials
 2020-01-31 16:48:37 +09:00
Takeshi Nakatani
018ccb9a11 Suppressed a lot of message output about cache cleanup 2020-01-30 14:42:28 +00:00
yongqingliu
b762a0a85b fix deadlock due to fdmanager_lock and fdent_data_lock 2020-01-29 12:03:53 +08:00
Andrew Gaul
e0712f444d Update source for clang-tidy 9.0.0 2020-01-13 20:56:45 +09:00
Andrei Precup
39102608aa Add intelligent tiering as option for storage class 
Resolves Issue #1219
 2020-01-13 20:24:35 +09:00
Takeshi Nakatani
23945a0130 In memory cache for softlinks with cache out 2020-01-13 20:23:10 +09:00
Alvaro Huarte
bdfb9ee815 Document requester_pays option 2019-11-25 22:53:50 +09:00
Alvaro Huarte
a24f78f5a4 Adds requester_pays option to enable requests involving Requester Pays buckets 2019-11-18 12:38:16 +01:00
Takeshi Nakatani
a07a5336f6
Merge pull request #1185 from bryceml/master 
handle edge case when bucket name and first part of endpoint match
 2019-10-31 23:42:47 +09:00
Bryce Larson
0c42a74a8a
handle edge case when bucket name and first part of endpoint match 2019-10-26 00:51:19 -06:00
Takeshi Nakatani
cf3e82d10a Fixed a rename bug when enable_noobj_cache 2019-10-23 12:39:42 +00:00
Takeshi Nakatani
1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00
Takeshi Nakatani
bedd648d47 Fixed build error by cppcheck 1.89 2019-09-23 10:49:49 +00:00
Takeshi Nakatani
58b3cce320
Merge pull request #1066 from gaul/sign-requests 
Sign request immediately before sending
 2019-09-23 15:58:21 +09:00
Takeshi Nakatani
81102a5963
Merge pull request #1149 from ggtakec/fix_retry 
Fixed a bug that type and op are not inherited
 2019-09-08 20:32:43 +09:00
Takeshi Nakatani
42fb30852b
Merge pull request #1145 from gaul/bool 
Fix a few bool types
 2019-09-08 20:04:30 +09:00
Takeshi Nakatani
e51361cb94
Merge pull request #1143 from gaul/doc/umask 
Document umask flag
 2019-09-08 19:49:58 +09:00
Takeshi Nakatani
e2129001eb Fixed a bug that type and op are not inherited 2019-09-08 09:59:04 +00:00
Andrew Gaul
805cc064af Fix a few bool types 
Found via clang-tidy.
 2019-09-05 10:42:42 -07:00
Andrew Gaul
3c2279db39 Document umask flag 2019-09-04 20:53:58 -07:00
Andrew Gaul
412876ca33 Expose base parameter in s3fs_strtoofft 
This fixes a regression from ccf3e7bfa2
which caused the misparsing of octal inputs for the mp_umask and umask
flags.  It also allows some callers to be more precise about their
decimal inputs.
 2019-09-04 20:41:47 -07:00
Andrew Gaul
461a346bf4 Sign request immediately before sending 
Previously s3fs could create a long list of pre-signed requests which
could take longer than the default S3 clock skew limit of 15 minutes.
This also changes SHA-256 computation from single- to multi-threaded
since this is now computed in the worker threads.  Regression from
88cd8feb05.  Fixes #1019.
 2019-08-29 15:35:27 -07:00
liuyongqing
ae4bcd405c use correct content-type when overwriting and copying files 2019-08-28 22:25:09 -07:00
Takeshi Nakatani
0536dc1112 Refixed strict processing of MultiRead method 2019-08-25 20:46:01 -07:00
Takeshi Nakatani
1c3507ede1
Merge pull request #1135 from gaul/curl-timeout 
Remove useless CURLE_OPERATION_TIMEDOUT check
 2019-08-25 17:53:02 +09:00