798 Commits

Author	SHA1	Message	Date
yongqingliu	b72f4b43a4	use correct content-type when complete multipart upload	2020-02-10 16:58:28 +09:00
Andrew Gaul	924eeb3587	Document host and servicepath ... The various bits of host, url, and servicepath seem to overlap. References #1203.	2020-02-04 21:13:29 +09:00
Andrew Gaul	bc9126d774	Set directory MIME type to application/x-directory ... Previously s3fs auto-detected the MIME type of directories like "TOYOTA TRUCK 8.2.2" as application/x-troff-man. This caused get_mode to not set S_IFDIR which failed directory creation. Instead force all object names ending in / to application/x-directory. Fixes #1183.	2020-02-04 20:03:21 +09:00
rallister	4e26728cbf	break recursion when calling GetIAMCredentials (#1233) ... break recursion when calling GetIAMCredentials	2020-01-31 16:48:37 +09:00
Takeshi Nakatani	018ccb9a11	Suppressed a lot of message output about cache cleanup	2020-01-30 14:42:28 +00:00
yongqingliu	b762a0a85b	fix deadlock due to fdmanager_lock and fdent_data_lock	2020-01-29 12:03:53 +08:00
Andrew Gaul	e0712f444d	Update source for clang-tidy 9.0.0	2020-01-13 20:56:45 +09:00
Andrei Precup	39102608aa	Add intelligent tiering as option for storage class ... Resolves Issue #1219	2020-01-13 20:24:35 +09:00
Takeshi Nakatani	23945a0130	In memory cache for softlinks with cache out	2020-01-13 20:23:10 +09:00
Alvaro Huarte	bdfb9ee815	Document requester_pays option	2019-11-25 22:53:50 +09:00
Alvaro Huarte	a24f78f5a4	Adds requester_pays option to enable requests involving Requester Pays buckets	2019-11-18 12:38:16 +01:00
Takeshi Nakatani	a07a5336f6	Merge pull request #1185 from bryceml/master ... handle edge case when bucket name and first part of endpoint match	2019-10-31 23:42:47 +09:00
Bryce Larson	0c42a74a8a	handle edge case when bucket name and first part of endpoint match	2019-10-26 00:51:19 -06:00
Takeshi Nakatani	cf3e82d10a	Fixed a rename bug when enable_noobj_cache	2019-10-23 12:39:42 +00:00
Takeshi Nakatani	1db94a0b30	Fixed to use copy api in multipart upload	2019-09-25 19:30:58 -07:00
Takeshi Nakatani	bedd648d47	Fixed build error by cppcheck 1.89	2019-09-23 10:49:49 +00:00
Takeshi Nakatani	58b3cce320	Merge pull request #1066 from gaul/sign-requests ... Sign request immediately before sending	2019-09-23 15:58:21 +09:00
Takeshi Nakatani	81102a5963	Merge pull request #1149 from ggtakec/fix_retry ... Fixed a bug that type and op are not inherited	2019-09-08 20:32:43 +09:00
Takeshi Nakatani	42fb30852b	Merge pull request #1145 from gaul/bool ... Fix a few bool types	2019-09-08 20:04:30 +09:00
Takeshi Nakatani	e51361cb94	Merge pull request #1143 from gaul/doc/umask ... Document umask flag	2019-09-08 19:49:58 +09:00
Takeshi Nakatani	e2129001eb	Fixed a bug that type and op are not inherited	2019-09-08 09:59:04 +00:00
Andrew Gaul	805cc064af	Fix a few bool types ... Found via clang-tidy.	2019-09-05 10:42:42 -07:00
Andrew Gaul	3c2279db39	Document umask flag	2019-09-04 20:53:58 -07:00
Andrew Gaul	412876ca33	Expose base parameter in s3fs_strtoofft ... This fixes a regression from ccf3e7bfa2 which caused the misparsing of octal inputs for the mp_umask and umask flags. It also allows some callers to be more precise about their decimal inputs.	2019-09-04 20:41:47 -07:00
Andrew Gaul	461a346bf4	Sign request immediately before sending ... Previously s3fs could create a long list of pre-signed requests which could take longer than the default S3 clock skew limit of 15 minutes. This also changes SHA-256 computation from single- to multi-threaded since this is now computed in the worker threads. Regression from 88cd8feb05. Fixes #1019.	2019-08-29 15:35:27 -07:00
liuyongqing	ae4bcd405c	use correct content-type when overwriting and copying files	2019-08-28 22:25:09 -07:00
Takeshi Nakatani	0536dc1112	Refixed strict processing of MultiRead method	2019-08-25 20:46:01 -07:00
Takeshi Nakatani	1c3507ede1	Merge pull request #1135 from gaul/curl-timeout ... Remove useless CURLE_OPERATION_TIMEDOUT check	2019-08-25 17:53:02 +09:00
Andrew Gaul	01ac815346	Remove useless CURLE_OPERATION_TIMEDOUT check ... CheckBucket only returns errno and HTTP status codes.	2019-08-20 18:44:48 -07:00
Andrew Gaul	cc2eed84a5	Promote default ACL to enum ... This sanity checks ACLs during initialization and also omits sending the header when set to PRIVATE which is the default.	2019-08-20 15:05:58 -07:00
Takeshi Nakatani	873e376098	Strict processing of MultiRead method	2019-08-13 15:23:13 +00:00
Takeshi Nakatani	4a0c23258e	Fixed MultiRead method to exit after waiting other parts at error	2019-08-12 14:45:06 -07:00
Takeshi Nakatani	5c4a0a862a	Fixed multipart copy and its retry handler	2019-08-11 16:21:26 +00:00
Takeshi Nakatani	f5bf41cf11	Fixed simple copy with nocpoyapi and modify flag in fdcache	2019-08-11 07:42:48 +00:00
Takeshi Nakatani	f74c7407db	Merge pull request #1118 from gaul/atoi ... Prefer s3fs_strtoofft over atoi and strtol	2019-08-06 21:23:37 +09:00
Takeshi Nakatani	433c04af26	Merge pull request #1116 from gaul/bodydata ... Do not heap allocate BodyData	2019-08-06 21:08:02 +09:00
Takeshi Nakatani	2e51a339a9	Merge pull request #1115 from gaul/hash-length ... Use length from s3fs_HMAC256 for consistency	2019-08-06 20:49:22 +09:00
Takeshi Nakatani	14d3e12103	Merge pull request #1114 from gaul/flush/return-value ... Consume return value from FdEntity::Flush	2019-08-06 14:12:25 +09:00
Andrew Gaul	ccf3e7bfa2	Prefer s3fs_strtoofft over atoi and strtol ... The former propagates errors consistently.	2019-08-03 16:13:48 -07:00
Andrew Gaul	7ecfba811e	Do not heap allocate BodyData ... Callers already manage lifetimes via Clear().	2019-08-03 15:50:46 -07:00
Takeshi Nakatani	1e7330e499	Merge pull request #1113 from gaul/unused ... Remove unused field	2019-08-04 01:31:56 +09:00
Takeshi Nakatani	3d42d0515d	Merge pull request #1109 from gaul/lock-already-held ... Pass lock_already_held state to DelStat	2019-08-03 22:13:22 +09:00
Takeshi Nakatani	3d70e8966f	Merge pull request #1108 from gaul/lock/diskspace ... Always hold lock when accessing free_disk_space	2019-08-03 22:01:31 +09:00
Andrew Gaul	a5c20175a1	Use length from s3fs_HMAC256 for consistency ... EVP_MAX_MD_SIZE can range up to 64. Found by Coverity.	2019-08-02 10:40:55 -07:00
Andrew Gaul	43d1439420	Consume return value from FdEntity::Flush ... Found via Coverity.	2019-08-02 10:01:06 -07:00
Andrew Gaul	d8cf26bd50	Remove unused field ... Found via clang.	2019-08-01 16:42:38 -07:00
Andrew Gaul	f528a86219	Pass lock_already_held state to DelStat	2019-08-01 11:07:56 -07:00
Andrew Gaul	5b15c7c4e9	Always hold lock when accessing free_disk_space ... Slightly reorder locks to avoid double locking. Found via ThreadSanitizer.	2019-07-30 08:36:54 -07:00
Andrew Gaul	afd438d363	Access FdEntity metadata while holding fdent_lock ... Create copies to avoid thread races. Found via ThreadSanitizer. Follows on to ecf13a8cb9.	2019-07-29 13:54:57 -07:00
Andrew Gaul	fb937635f5	Eagerly initialize sysconf variables ... Previously s3fs had races updating these shared variables. Found via ThreadSanitizer.	2019-07-17 09:08:13 -07:00
Takeshi Nakatani	3ad1c95e86	Merge pull request #1103 from gaul/config/readwrite-timeout ... Use consistent default for readwrite_timeout	2019-07-17 22:37:02 +09:00
Andrew Gaul	bacd15714a	Use consistent default for readwrite_timeout ... Previously curl threads mutated this shared state without a lock. Found via ThreadSanitizer.	2019-07-16 22:03:56 -07:00
Andrew Gaul	99aace4fc9	Avoid race when using thread-unsafe gmtime ... Found via ThreadSanitizer.	2019-07-16 07:28:34 -07:00
Takeshi Nakatani	513f41fddf	Merge pull request #1097 from ggtakec/fix_lock ... Add Lock() func parameter in RowFlush()	2019-07-15 11:56:21 +09:00
Takeshi Nakatani	543aed2a32	Merge pull request #1096 from gaul/pthread/lock-checking ... Expand error checking to all pthread_mutex_t	2019-07-15 11:43:56 +09:00
Takeshi Nakatani	20ea96328c	Merge pull request #1095 from gaul/intmax_t ... Avoid narrowing time_t and off_t args in logging	2019-07-15 11:33:44 +09:00
Takeshi Nakatani	f78bcc5229	Add Lock() func parameter in RowFlush()	2019-07-15 01:52:29 +00:00
Andrew Gaul	43ec064fb9	Expand error checking to all pthread_mutex_t	2019-07-14 18:21:23 -07:00
Andrew Gaul	ffac4c8417	Avoid narrowing time_t and off_t args in logging ... This displays correct results on 32-bit platforms.	2019-07-14 17:02:36 -07:00
Andrew Gaul	4adcd4a6c8	Implement s3fs_strtoofft via strtoll ... This tightens error checking and aligns s3fs with known good behavior.	2019-07-14 15:15:06 -07:00
Takeshi Nakatani	e936854493	Merge pull request #1094 from larsch/master ... Fix multipart upload not setting curl options	2019-07-15 05:13:37 +09:00
Takeshi Nakatani	850a813171	Merge pull request #1092 from liuyongqing/master ... fix coredump caused by fd_manager_lock locking	2019-07-15 04:53:10 +09:00
Takeshi Nakatani	5bbcd3b981	Merge pull request #1089 from gaul/null-dereference ... Avoid null dereference	2019-07-15 04:17:10 +09:00
Takeshi Nakatani	a337c32840	Merge pull request #1088 from gaul/clang-tidy/misplaced-const ... Avoid misplaced const via removing unneeded typedef	2019-07-15 04:08:51 +09:00
Takeshi Nakatani	d39e4e4b1f	Merge pull request #1087 from gaul/clang-tidy/deprecated-headers ... Prefer modern C headers	2019-07-15 03:50:15 +09:00
Lars Christensen	58037da061	Fix multipart upload not setting curl options	2019-07-14 14:09:37 +02:00
Andrew Gaul	1eb266588e	Add a missing lock to SetCtime ... Also lock before log statements which touch member variables.	2019-07-13 17:53:38 -07:00
root	4e351c59e3	fix coredump caused by fd_manager_lock locking	2019-07-13 12:03:27 +08:00
Andrew Gaul	eb597289cb	Avoid null dereference ... Found via clang-tidy.	2019-07-12 18:40:24 -07:00
Andrew Gaul	6fd42d9fe4	Prefer modern C headers ... Found and fixed via clang-tidy.	2019-07-12 03:50:59 -07:00
Andrew Gaul	efff9c01a6	Avoid misplaced const via removing unneeded typedef ... Found via clang-tidy.	2019-07-12 03:41:19 -07:00
Andrew Gaul	a83d5baa90	Remove recursive locking ... Recursive locking is frowned upon and is incompatible with PTHREAD_MUTEX_ERRORCHECK. Also clean up pthread_mutex_lock error checking.	2019-07-10 12:39:00 -07:00
Takeshi Nakatani	50d13255e4	Merge pull request #1081 from gaul/write-blocks-readdir ... Break FdEntity locks into data and metadata	2019-07-11 03:40:26 +09:00
Takeshi Nakatani	e5e63d6ac3	Merge pull request #1067 from gaul/mismatched-free ... Fix mismatched new[]/free	2019-07-07 16:55:07 +09:00
Takeshi Nakatani	7a65a414c3	Merge pull request #1080 from gaul/xml/simplification ... Add simple XML parsing wrapper	2019-07-07 16:44:18 +09:00
Andrew Gaul	944d21cabb	Fix mismatched new[]/free ... Found via Valgrind.	2019-07-07 00:36:48 -07:00
Takeshi Nakatani	d267212289	Merge pull request #1079 from gaul/cppcheck/shadowing ... Avoid shadowing variable in FdEntity::Open	2019-07-07 16:29:42 +09:00
Takeshi Nakatani	58d8e5586a	Merge pull request #1078 from gaul/multipart/2x ... Issue multipart when object size exceeds part size	2019-07-07 16:21:22 +09:00
Takeshi Nakatani	ce803daf4a	Merge pull request #1077 from gaul/openssl/locking ... Annotate OpenSSL locking functions as unused	2019-07-07 16:10:14 +09:00
Takeshi Nakatani	9bf34e2fda	Merge pull request #1076 from gaul/doc/man-help ... Add documentation for use_session_token and use_rrs	2019-07-07 15:45:42 +09:00
Takeshi Nakatani	6bd1a7eac0	Merge pull request #1074 from gaul/clang-tidy ... Configure clang-tidy target	2019-07-07 15:28:08 +09:00
Andrew Gaul	8aa06d621a	Add documentation for use_session_token and use_rrs ... Fixes #929.	2019-07-06 10:04:43 -07:00
Andrew Gaul	ecf13a8cb9	Break FdEntity locks into data and metadata ... Previously long-running data operations like RowFlush would block metadata operations like GetStats and thus user readdir. Fixes #928.	2019-07-05 23:12:24 -07:00
Andrew Gaul	b8ff6a647e	Add simple XML parsing wrapper ... Also simplify check_region_error.	2019-07-05 17:39:09 -07:00
Andrew Gaul	49110c671d	Avoid shadowing variable in FdEntity::Open ... Found via cppcheck 1.88.	2019-07-05 15:06:18 -07:00
Andrew Gaul	febaf6849f	Issue multipart when object size exceeds part size ... Previously s3fs issued multipart uploads when the object size was twice the part size. Conjoining this with the part size was confusing and s3fs should add a separate tunable for this if needed, similar to singlepart_copy_limit. Fixes #1058.	2019-07-05 12:14:56 -07:00
Andrew Gaul	4893174652	Annotate OpenSSL locking functions as unused ... OpenSSL 1.1.0 removed these and generates warnings with newer versions.	2019-07-05 10:45:57 -07:00
Andrew Gaul	4f23f38583	Individually test multiple s3fs flags ... Remove unneeded comments; single part limits ensure that the tests exercise multipart code paths even with smaller files. References #971.	2019-07-03 21:09:40 -07:00
Andrew Gaul	bbfa91141a	Configure clang-tidy target ... Also fix nits.	2019-07-03 14:04:11 -07:00
Takeshi Nakatani	21321a9d96	Merge pull request #1065 from gaul/doc/ssl-verify-hostname ... Document ssl_verify_hostname	2019-07-03 22:15:33 +09:00
Takeshi Nakatani	f03b50fd13	Merge pull request #1063 from gaul/truncate-2nd-open-fd ... Flush file when opening second fd	2019-07-03 21:14:03 +09:00
Andrew Gaul	f02105c346	Document ssl_verify_hostname ... References #1064.	2019-07-02 10:26:29 -07:00
Andrew Gaul	c596441f58	Flush file when opening second fd ... Previously when s3fs had dirty local data and an application opened a second fd it would remove the stat cache entry, query S3 for the size which was still zero, then reinitialize FdEntry with this incorrect size. Instead flush local data to S3 so this query works. It is possible that a more involved patch could do this with a less heavyweight approach but this requires changing open. This does not completely fix git clone but allows it to proceed further. Also make some cache methods const-correct. References #839.	2019-07-02 01:12:09 -07:00
Andrew Gaul	455e29cbea	Make fdpage a value type in fdpage_list_t ... This simplifies memory management.	2019-07-01 09:30:48 -07:00
Takeshi Nakatani	511d223468	Merge pull request #1059 from gaul/casting ... Fix casting warning on 32-bit	2019-07-01 22:26:14 +09:00
Andrew Gaul	d7f77a6282	Fix casting warning on 32-bit	2019-06-29 20:57:42 -07:00
Andrew Gaul	f1ad626b46	Log flags as hexadecimal	2019-06-24 16:13:07 -07:00
Takeshi Nakatani	a78d8d1da4	Merge pull request #1049 from gaul/external-modication ... Remove cache file when object time differs	2019-06-24 00:51:54 +09:00
Takeshi Nakatani	fbebc6fa57	Merge pull request #1048 from gaul/log/curl ... Emit older curl warnings only once	2019-06-24 00:42:44 +09:00
Andrew Gaul	9e5eaad79b	Remove cache file when object time differs ... Check the modification times to determine whether an object has updated. This relies on low clock skew between s3fs and the S3 server; a more robust approach could use the ETag. Fixes #1047.	2019-06-22 19:09:00 -07:00