octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-12-25 09:59:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,557 Commits 1 Branch 23 Tags 13 MiB
b5ffd419d8
Commit Graph

801 Commits

Author SHA1 Message Date
Takeshi Nakatani
9e01d5b8d1
Merge pull request #1254 from ggtakec/modify_mimetypes 
Added mime option for strict checking of mime types file
 2020-04-11 14:48:47 +09:00
Takeshi Nakatani
7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
Andrew Gaul
56141557dc Avoid unneeded string copy 
Found by clang-tidy 10.
 2020-03-28 08:49:49 +09:00
yongqingliu
b72f4b43a4 use correct content-type when complete multipart upload 2020-02-10 16:58:28 +09:00
Andrew Gaul
924eeb3587 Document host and servicepath 
The various bits of host, url, and servicepath seem to overlap.
References #1203.
 2020-02-04 21:13:29 +09:00
Andrew Gaul
bc9126d774 Set directory MIME type to application/x-directory 
Previously s3fs auto-detected the MIME type of directories like
"TOYOTA TRUCK 8.2.2" as application/x-troff-man.  This caused get_mode
to not set S_IFDIR which failed directory creation.  Instead force all
object names ending in / to application/x-directory.  Fixes #1183.
 2020-02-04 20:03:21 +09:00
rallister
4e26728cbf
break recursion when calling GetIAMCredentials (#1233) 
break recursion when calling GetIAMCredentials
 2020-01-31 16:48:37 +09:00
Takeshi Nakatani
018ccb9a11 Suppressed a lot of message output about cache cleanup 2020-01-30 14:42:28 +00:00
yongqingliu
b762a0a85b fix deadlock due to fdmanager_lock and fdent_data_lock 2020-01-29 12:03:53 +08:00
Andrew Gaul
e0712f444d Update source for clang-tidy 9.0.0 2020-01-13 20:56:45 +09:00
Andrei Precup
39102608aa Add intelligent tiering as option for storage class 
Resolves Issue #1219
 2020-01-13 20:24:35 +09:00
Takeshi Nakatani
23945a0130 In memory cache for softlinks with cache out 2020-01-13 20:23:10 +09:00
Alvaro Huarte
bdfb9ee815 Document requester_pays option 2019-11-25 22:53:50 +09:00
Alvaro Huarte
a24f78f5a4 Adds requester_pays option to enable requests involving Requester Pays buckets 2019-11-18 12:38:16 +01:00
Takeshi Nakatani
a07a5336f6
Merge pull request #1185 from bryceml/master 
handle edge case when bucket name and first part of endpoint match
 2019-10-31 23:42:47 +09:00
Bryce Larson
0c42a74a8a
handle edge case when bucket name and first part of endpoint match 2019-10-26 00:51:19 -06:00
Takeshi Nakatani
cf3e82d10a Fixed a rename bug when enable_noobj_cache 2019-10-23 12:39:42 +00:00
Takeshi Nakatani
1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00
Takeshi Nakatani
bedd648d47 Fixed build error by cppcheck 1.89 2019-09-23 10:49:49 +00:00
Takeshi Nakatani
58b3cce320
Merge pull request #1066 from gaul/sign-requests 
Sign request immediately before sending
 2019-09-23 15:58:21 +09:00
Takeshi Nakatani
81102a5963
Merge pull request #1149 from ggtakec/fix_retry 
Fixed a bug that type and op are not inherited
 2019-09-08 20:32:43 +09:00
Takeshi Nakatani
42fb30852b
Merge pull request #1145 from gaul/bool 
Fix a few bool types
 2019-09-08 20:04:30 +09:00
Takeshi Nakatani
e51361cb94
Merge pull request #1143 from gaul/doc/umask 
Document umask flag
 2019-09-08 19:49:58 +09:00
Takeshi Nakatani
e2129001eb Fixed a bug that type and op are not inherited 2019-09-08 09:59:04 +00:00
Andrew Gaul
805cc064af Fix a few bool types 
Found via clang-tidy.
 2019-09-05 10:42:42 -07:00
Andrew Gaul
3c2279db39 Document umask flag 2019-09-04 20:53:58 -07:00
Andrew Gaul
412876ca33 Expose base parameter in s3fs_strtoofft 
This fixes a regression from ccf3e7bfa2
which caused the misparsing of octal inputs for the mp_umask and umask
flags.  It also allows some callers to be more precise about their
decimal inputs.
 2019-09-04 20:41:47 -07:00
Andrew Gaul
461a346bf4 Sign request immediately before sending 
Previously s3fs could create a long list of pre-signed requests which
could take longer than the default S3 clock skew limit of 15 minutes.
This also changes SHA-256 computation from single- to multi-threaded
since this is now computed in the worker threads.  Regression from
88cd8feb05.  Fixes #1019.
 2019-08-29 15:35:27 -07:00
liuyongqing
ae4bcd405c use correct content-type when overwriting and copying files 2019-08-28 22:25:09 -07:00
Takeshi Nakatani
0536dc1112 Refixed strict processing of MultiRead method 2019-08-25 20:46:01 -07:00
Takeshi Nakatani
1c3507ede1
Merge pull request #1135 from gaul/curl-timeout 
Remove useless CURLE_OPERATION_TIMEDOUT check
 2019-08-25 17:53:02 +09:00
Andrew Gaul
01ac815346 Remove useless CURLE_OPERATION_TIMEDOUT check 
CheckBucket only returns errno and HTTP status codes.
 2019-08-20 18:44:48 -07:00
Andrew Gaul
cc2eed84a5 Promote default ACL to enum 
This sanity checks ACLs during initialization and also omits sending
the header when set to PRIVATE which is the default.
 2019-08-20 15:05:58 -07:00
Takeshi Nakatani
873e376098 Strict processing of MultiRead method 2019-08-13 15:23:13 +00:00
Takeshi Nakatani
4a0c23258e Fixed MultiRead method to exit after waiting other parts at error 2019-08-12 14:45:06 -07:00
Takeshi Nakatani
5c4a0a862a Fixed multipart copy and its retry handler 2019-08-11 16:21:26 +00:00
Takeshi Nakatani
f5bf41cf11 Fixed simple copy with nocpoyapi and modify flag in fdcache 2019-08-11 07:42:48 +00:00
Takeshi Nakatani
f74c7407db
Merge pull request #1118 from gaul/atoi 
Prefer s3fs_strtoofft over atoi and strtol
 2019-08-06 21:23:37 +09:00
Takeshi Nakatani
433c04af26
Merge pull request #1116 from gaul/bodydata 
Do not heap allocate BodyData
 2019-08-06 21:08:02 +09:00
Takeshi Nakatani
2e51a339a9
Merge pull request #1115 from gaul/hash-length 
Use length from s3fs_HMAC256 for consistency
 2019-08-06 20:49:22 +09:00
Takeshi Nakatani
14d3e12103
Merge pull request #1114 from gaul/flush/return-value 
Consume return value from FdEntity::Flush
 2019-08-06 14:12:25 +09:00
Andrew Gaul
ccf3e7bfa2 Prefer s3fs_strtoofft over atoi and strtol 
The former propagates errors consistently.
 2019-08-03 16:13:48 -07:00
Andrew Gaul
7ecfba811e Do not heap allocate BodyData 
Callers already manage lifetimes via Clear().
 2019-08-03 15:50:46 -07:00
Takeshi Nakatani
1e7330e499
Merge pull request #1113 from gaul/unused 
Remove unused field
 2019-08-04 01:31:56 +09:00
Takeshi Nakatani
3d42d0515d
Merge pull request #1109 from gaul/lock-already-held 
Pass lock_already_held state to DelStat
 2019-08-03 22:13:22 +09:00
Takeshi Nakatani
3d70e8966f
Merge pull request #1108 from gaul/lock/diskspace 
Always hold lock when accessing free_disk_space
 2019-08-03 22:01:31 +09:00
Andrew Gaul
a5c20175a1 Use length from s3fs_HMAC256 for consistency 
EVP_MAX_MD_SIZE can range up to 64.  Found by Coverity.
 2019-08-02 10:40:55 -07:00
Andrew Gaul
43d1439420 Consume return value from FdEntity::Flush 
Found via Coverity.
 2019-08-02 10:01:06 -07:00
Andrew Gaul
d8cf26bd50 Remove unused field 
Found via clang.
 2019-08-01 16:42:38 -07:00
Andrew Gaul
f528a86219 Pass lock_already_held state to DelStat 2019-08-01 11:07:56 -07:00
Andrew Gaul
5b15c7c4e9 Always hold lock when accessing free_disk_space 
Slightly reorder locks to avoid double locking.  Found via
ThreadSanitizer.
 2019-07-30 08:36:54 -07:00
Andrew Gaul
afd438d363 Access FdEntity metadata while holding fdent_lock 
Create copies to avoid thread races.  Found via ThreadSanitizer.
Follows on to ecf13a8cb9.
 2019-07-29 13:54:57 -07:00
Andrew Gaul
fb937635f5 Eagerly initialize sysconf variables 
Previously s3fs had races updating these shared variables.  Found via
ThreadSanitizer.
 2019-07-17 09:08:13 -07:00
Takeshi Nakatani
3ad1c95e86
Merge pull request #1103 from gaul/config/readwrite-timeout 
Use consistent default for readwrite_timeout
 2019-07-17 22:37:02 +09:00
Andrew Gaul
bacd15714a Use consistent default for readwrite_timeout 
Previously curl threads mutated this shared state without a lock.
Found via ThreadSanitizer.
 2019-07-16 22:03:56 -07:00
Andrew Gaul
99aace4fc9 Avoid race when using thread-unsafe gmtime 
Found via ThreadSanitizer.
 2019-07-16 07:28:34 -07:00
Takeshi Nakatani
513f41fddf
Merge pull request #1097 from ggtakec/fix_lock 
Add Lock() func parameter in RowFlush()
 2019-07-15 11:56:21 +09:00
Takeshi Nakatani
543aed2a32
Merge pull request #1096 from gaul/pthread/lock-checking 
Expand error checking to all pthread_mutex_t
 2019-07-15 11:43:56 +09:00
Takeshi Nakatani
20ea96328c
Merge pull request #1095 from gaul/intmax_t 
Avoid narrowing time_t and off_t args in logging
 2019-07-15 11:33:44 +09:00
Takeshi Nakatani
f78bcc5229 Add Lock() func parameter in RowFlush() 2019-07-15 01:52:29 +00:00
Andrew Gaul
43ec064fb9 Expand error checking to all pthread_mutex_t 2019-07-14 18:21:23 -07:00
Andrew Gaul
ffac4c8417 Avoid narrowing time_t and off_t args in logging 
This displays correct results on 32-bit platforms.
 2019-07-14 17:02:36 -07:00
Andrew Gaul
4adcd4a6c8 Implement s3fs_strtoofft via strtoll 
This tightens error checking and aligns s3fs with known good behavior.
 2019-07-14 15:15:06 -07:00
Takeshi Nakatani
e936854493
Merge pull request #1094 from larsch/master 
Fix multipart upload not setting curl options
 2019-07-15 05:13:37 +09:00
Takeshi Nakatani
850a813171
Merge pull request #1092 from liuyongqing/master 
fix coredump caused by fd_manager_lock locking
 2019-07-15 04:53:10 +09:00
Takeshi Nakatani
5bbcd3b981
Merge pull request #1089 from gaul/null-dereference 
Avoid null dereference
 2019-07-15 04:17:10 +09:00
Takeshi Nakatani
a337c32840
Merge pull request #1088 from gaul/clang-tidy/misplaced-const 
Avoid misplaced const via removing unneeded typedef
 2019-07-15 04:08:51 +09:00
Takeshi Nakatani
d39e4e4b1f
Merge pull request #1087 from gaul/clang-tidy/deprecated-headers 
Prefer modern C headers
 2019-07-15 03:50:15 +09:00
Lars Christensen
58037da061 Fix multipart upload not setting curl options 2019-07-14 14:09:37 +02:00
Andrew Gaul
1eb266588e Add a missing lock to SetCtime 
Also lock before log statements which touch member variables.
 2019-07-13 17:53:38 -07:00
root
4e351c59e3 fix coredump caused by fd_manager_lock locking 2019-07-13 12:03:27 +08:00
Andrew Gaul
eb597289cb Avoid null dereference 
Found via clang-tidy.
 2019-07-12 18:40:24 -07:00
Andrew Gaul
6fd42d9fe4 Prefer modern C headers 
Found and fixed via clang-tidy.
 2019-07-12 03:50:59 -07:00
Andrew Gaul
efff9c01a6 Avoid misplaced const via removing unneeded typedef 
Found via clang-tidy.
 2019-07-12 03:41:19 -07:00
Andrew Gaul
a83d5baa90 Remove recursive locking 
Recursive locking is frowned upon and is incompatible with
PTHREAD_MUTEX_ERRORCHECK.  Also clean up pthread_mutex_lock error
checking.
 2019-07-10 12:39:00 -07:00
Takeshi Nakatani
50d13255e4
Merge pull request #1081 from gaul/write-blocks-readdir 
Break FdEntity locks into data and metadata
 2019-07-11 03:40:26 +09:00
Takeshi Nakatani
e5e63d6ac3
Merge pull request #1067 from gaul/mismatched-free 
Fix mismatched new[]/free
 2019-07-07 16:55:07 +09:00
Takeshi Nakatani
7a65a414c3
Merge pull request #1080 from gaul/xml/simplification 
Add simple XML parsing wrapper
 2019-07-07 16:44:18 +09:00
Andrew Gaul
944d21cabb Fix mismatched new[]/free 
Found via Valgrind.
 2019-07-07 00:36:48 -07:00
Takeshi Nakatani
d267212289
Merge pull request #1079 from gaul/cppcheck/shadowing 
Avoid shadowing variable in FdEntity::Open
 2019-07-07 16:29:42 +09:00
Takeshi Nakatani
58d8e5586a
Merge pull request #1078 from gaul/multipart/2x 
Issue multipart when object size exceeds part size
 2019-07-07 16:21:22 +09:00
Takeshi Nakatani
ce803daf4a
Merge pull request #1077 from gaul/openssl/locking 
Annotate OpenSSL locking functions as unused
 2019-07-07 16:10:14 +09:00
Takeshi Nakatani
9bf34e2fda
Merge pull request #1076 from gaul/doc/man-help 
Add documentation for use_session_token and use_rrs
 2019-07-07 15:45:42 +09:00
Takeshi Nakatani
6bd1a7eac0
Merge pull request #1074 from gaul/clang-tidy 
Configure clang-tidy target
 2019-07-07 15:28:08 +09:00
Andrew Gaul
8aa06d621a Add documentation for use_session_token and use_rrs 
Fixes #929.
 2019-07-06 10:04:43 -07:00
Andrew Gaul
ecf13a8cb9 Break FdEntity locks into data and metadata 
Previously long-running data operations like RowFlush would block
metadata operations like GetStats and thus user readdir.  Fixes #928.
 2019-07-05 23:12:24 -07:00
Andrew Gaul
b8ff6a647e Add simple XML parsing wrapper 
Also simplify check_region_error.
 2019-07-05 17:39:09 -07:00
Andrew Gaul
49110c671d Avoid shadowing variable in FdEntity::Open 
Found via cppcheck 1.88.
 2019-07-05 15:06:18 -07:00
Andrew Gaul
febaf6849f Issue multipart when object size exceeds part size 
Previously s3fs issued multipart uploads when the object size was
twice the part size.  Conjoining this with the part size was confusing
and s3fs should add a separate tunable for this if needed, similar to
singlepart_copy_limit.  Fixes #1058.
 2019-07-05 12:14:56 -07:00
Andrew Gaul
4893174652 Annotate OpenSSL locking functions as unused 
OpenSSL 1.1.0 removed these and generates warnings with newer
versions.
 2019-07-05 10:45:57 -07:00
Andrew Gaul
4f23f38583 Individually test multiple s3fs flags 
Remove unneeded comments; single part limits ensure that the tests
exercise multipart code paths even with smaller files.
References #971.
 2019-07-03 21:09:40 -07:00
Andrew Gaul
bbfa91141a Configure clang-tidy target 
Also fix nits.
 2019-07-03 14:04:11 -07:00
Takeshi Nakatani
21321a9d96
Merge pull request #1065 from gaul/doc/ssl-verify-hostname 
Document ssl_verify_hostname
 2019-07-03 22:15:33 +09:00
Takeshi Nakatani
f03b50fd13
Merge pull request #1063 from gaul/truncate-2nd-open-fd 
Flush file when opening second fd
 2019-07-03 21:14:03 +09:00
Andrew Gaul
f02105c346 Document ssl_verify_hostname 
References #1064.
 2019-07-02 10:26:29 -07:00
Andrew Gaul
c596441f58 Flush file when opening second fd 
Previously when s3fs had dirty local data and an application opened a
second fd it would remove the stat cache entry, query S3 for the size
which was still zero, then reinitialize FdEntry with this incorrect
size.  Instead flush local data to S3 so this query works.  It is
possible that a more involved patch could do this with a less
heavyweight approach but this requires changing open.  This does not
completely fix git clone but allows it to proceed further.  Also make
some cache methods const-correct.  References #839.
 2019-07-02 01:12:09 -07:00
Andrew Gaul
455e29cbea Make fdpage a value type in fdpage_list_t 
This simplifies memory management.
 2019-07-01 09:30:48 -07:00
Takeshi Nakatani
511d223468
Merge pull request #1059 from gaul/casting 
Fix casting warning on 32-bit
 2019-07-01 22:26:14 +09:00
Andrew Gaul
d7f77a6282 Fix casting warning on 32-bit 2019-06-29 20:57:42 -07:00
Andrew Gaul
f1ad626b46 Log flags as hexadecimal 2019-06-24 16:13:07 -07:00