912 Commits

Author	SHA1	Message	Date
Takeshi Nakatani	80c11b6c12	Not abort process by exception threw from s3fs_strtoofft	2020-05-03 13:46:05 +00:00
Andrew Gaul	8945e98d8b	Support Google Cloud Storage headers ... This allows s3fs to interpret objects created by gsutil.	2020-05-03 18:33:13 +09:00
Takeshi Nakatani	97c249d5b9	Not abort process by exception threw from s3fs_strtoofft	2020-05-03 08:08:28 +00:00
Andrew Gaul	2cc88b933f	Warn about missing MIME types instead of exiting ... s3fs uses the MIME types file to set Content-Type for uploaded objects. Most distribution packages should install this via recommended (not required) dependencies. Users compiling from source may not have this installed and s3fs should not prevent launching since most users do not care about Content-Type. Instead warn about MIME types absence. Fixes #1270.	2020-04-29 20:03:50 +09:00
Andrew Gaul	005a684600	Fix typos	2020-04-22 21:49:11 +09:00
Takeshi Nakatani	f26a0aa71d	Fixed insufficient upload size for mix multipart upload	2020-04-22 09:31:22 +09:00
Andrew Gaul	cf529e0af7	Add handler for HTTP 416 ... This prevents retries when the server indicates an unsatisfiable MPU copy part request. References #1220.	2020-04-21 19:45:10 +09:00
Takeshi Nakatani	4da02d023b	Improved strictness of cache file stats(file)	2020-04-21 19:45:03 +09:00
Takeshi Nakatani	811ea0cb85	Fixed the truncation bug of stat file for cache file	2020-04-19 07:08:49 +00:00
Andrew Gaul	48a872e285	Address cppcheck 1.90 warning	2020-04-12 22:20:44 +09:00
Takeshi Nakatani	c44a60f3f5	Fixed a bug of stats cache compression	2020-04-12 18:33:00 +09:00
Takeshi Nakatani	9e01d5b8d1	Merge pull request #1254 from ggtakec/modify_mimetypes ... Added mime option for strict checking of mime types file	2020-04-11 14:48:47 +09:00
Takeshi Nakatani	7fbda230f5	Added mime option for strict checking of mime types file	2020-03-30 14:41:18 +00:00
Andrew Gaul	56141557dc	Avoid unneeded string copy ... Found by clang-tidy 10.	2020-03-28 08:49:49 +09:00
yongqingliu	b72f4b43a4	use correct content-type when complete multipart upload	2020-02-10 16:58:28 +09:00
Andrew Gaul	924eeb3587	Document host and servicepath ... The various bits of host, url, and servicepath seem to overlap. References #1203.	2020-02-04 21:13:29 +09:00
Andrew Gaul	bc9126d774	Set directory MIME type to application/x-directory ... Previously s3fs auto-detected the MIME type of directories like "TOYOTA TRUCK 8.2.2" as application/x-troff-man. This caused get_mode to not set S_IFDIR which failed directory creation. Instead force all object names ending in / to application/x-directory. Fixes #1183.	2020-02-04 20:03:21 +09:00
rallister	4e26728cbf	break recursion when calling GetIAMCredentials (#1233) ... break recursion when calling GetIAMCredentials	2020-01-31 16:48:37 +09:00
Takeshi Nakatani	018ccb9a11	Suppressed a lot of message output about cache cleanup	2020-01-30 14:42:28 +00:00
yongqingliu	b762a0a85b	fix deadlock due to fdmanager_lock and fdent_data_lock	2020-01-29 12:03:53 +08:00
Andrew Gaul	e0712f444d	Update source for clang-tidy 9.0.0	2020-01-13 20:56:45 +09:00
Andrei Precup	39102608aa	Add intelligent tiering as option for storage class ... Resolves Issue #1219	2020-01-13 20:24:35 +09:00
Takeshi Nakatani	23945a0130	In memory cache for softlinks with cache out	2020-01-13 20:23:10 +09:00
Alvaro Huarte	bdfb9ee815	Document requester_pays option	2019-11-25 22:53:50 +09:00
Alvaro Huarte	a24f78f5a4	Adds requester_pays option to enable requests involving Requester Pays buckets	2019-11-18 12:38:16 +01:00
Takeshi Nakatani	a07a5336f6	Merge pull request #1185 from bryceml/master ... handle edge case when bucket name and first part of endpoint match	2019-10-31 23:42:47 +09:00
Bryce Larson	0c42a74a8a	handle edge case when bucket name and first part of endpoint match	2019-10-26 00:51:19 -06:00
Takeshi Nakatani	cf3e82d10a	Fixed a rename bug when enable_noobj_cache	2019-10-23 12:39:42 +00:00
Takeshi Nakatani	1db94a0b30	Fixed to use copy api in multipart upload	2019-09-25 19:30:58 -07:00
Takeshi Nakatani	bedd648d47	Fixed build error by cppcheck 1.89	2019-09-23 10:49:49 +00:00
Takeshi Nakatani	58b3cce320	Merge pull request #1066 from gaul/sign-requests ... Sign request immediately before sending	2019-09-23 15:58:21 +09:00
Takeshi Nakatani	81102a5963	Merge pull request #1149 from ggtakec/fix_retry ... Fixed a bug that type and op are not inherited	2019-09-08 20:32:43 +09:00
Takeshi Nakatani	42fb30852b	Merge pull request #1145 from gaul/bool ... Fix a few bool types	2019-09-08 20:04:30 +09:00
Takeshi Nakatani	e51361cb94	Merge pull request #1143 from gaul/doc/umask ... Document umask flag	2019-09-08 19:49:58 +09:00
Takeshi Nakatani	e2129001eb	Fixed a bug that type and op are not inherited	2019-09-08 09:59:04 +00:00
Andrew Gaul	805cc064af	Fix a few bool types ... Found via clang-tidy.	2019-09-05 10:42:42 -07:00
Andrew Gaul	3c2279db39	Document umask flag	2019-09-04 20:53:58 -07:00
Andrew Gaul	412876ca33	Expose base parameter in s3fs_strtoofft ... This fixes a regression from ccf3e7bfa2 which caused the misparsing of octal inputs for the mp_umask and umask flags. It also allows some callers to be more precise about their decimal inputs.	2019-09-04 20:41:47 -07:00
Andrew Gaul	461a346bf4	Sign request immediately before sending ... Previously s3fs could create a long list of pre-signed requests which could take longer than the default S3 clock skew limit of 15 minutes. This also changes SHA-256 computation from single- to multi-threaded since this is now computed in the worker threads. Regression from 88cd8feb05. Fixes #1019.	2019-08-29 15:35:27 -07:00
liuyongqing	ae4bcd405c	use correct content-type when overwriting and copying files	2019-08-28 22:25:09 -07:00
Takeshi Nakatani	0536dc1112	Refixed strict processing of MultiRead method	2019-08-25 20:46:01 -07:00
Takeshi Nakatani	1c3507ede1	Merge pull request #1135 from gaul/curl-timeout ... Remove useless CURLE_OPERATION_TIMEDOUT check	2019-08-25 17:53:02 +09:00
Andrew Gaul	01ac815346	Remove useless CURLE_OPERATION_TIMEDOUT check ... CheckBucket only returns errno and HTTP status codes.	2019-08-20 18:44:48 -07:00
Andrew Gaul	cc2eed84a5	Promote default ACL to enum ... This sanity checks ACLs during initialization and also omits sending the header when set to PRIVATE which is the default.	2019-08-20 15:05:58 -07:00
Takeshi Nakatani	873e376098	Strict processing of MultiRead method	2019-08-13 15:23:13 +00:00
Takeshi Nakatani	4a0c23258e	Fixed MultiRead method to exit after waiting other parts at error	2019-08-12 14:45:06 -07:00
Takeshi Nakatani	5c4a0a862a	Fixed multipart copy and its retry handler	2019-08-11 16:21:26 +00:00
Takeshi Nakatani	f5bf41cf11	Fixed simple copy with nocpoyapi and modify flag in fdcache	2019-08-11 07:42:48 +00:00
Takeshi Nakatani	f74c7407db	Merge pull request #1118 from gaul/atoi ... Prefer s3fs_strtoofft over atoi and strtol	2019-08-06 21:23:37 +09:00
Takeshi Nakatani	433c04af26	Merge pull request #1116 from gaul/bodydata ... Do not heap allocate BodyData	2019-08-06 21:08:02 +09:00
Takeshi Nakatani	2e51a339a9	Merge pull request #1115 from gaul/hash-length ... Use length from s3fs_HMAC256 for consistency	2019-08-06 20:49:22 +09:00
Takeshi Nakatani	14d3e12103	Merge pull request #1114 from gaul/flush/return-value ... Consume return value from FdEntity::Flush	2019-08-06 14:12:25 +09:00
Andrew Gaul	ccf3e7bfa2	Prefer s3fs_strtoofft over atoi and strtol ... The former propagates errors consistently.	2019-08-03 16:13:48 -07:00
Andrew Gaul	7ecfba811e	Do not heap allocate BodyData ... Callers already manage lifetimes via Clear().	2019-08-03 15:50:46 -07:00
Takeshi Nakatani	1e7330e499	Merge pull request #1113 from gaul/unused ... Remove unused field	2019-08-04 01:31:56 +09:00
Takeshi Nakatani	3d42d0515d	Merge pull request #1109 from gaul/lock-already-held ... Pass lock_already_held state to DelStat	2019-08-03 22:13:22 +09:00
Takeshi Nakatani	3d70e8966f	Merge pull request #1108 from gaul/lock/diskspace ... Always hold lock when accessing free_disk_space	2019-08-03 22:01:31 +09:00
Andrew Gaul	a5c20175a1	Use length from s3fs_HMAC256 for consistency ... EVP_MAX_MD_SIZE can range up to 64. Found by Coverity.	2019-08-02 10:40:55 -07:00
Andrew Gaul	43d1439420	Consume return value from FdEntity::Flush ... Found via Coverity.	2019-08-02 10:01:06 -07:00
Andrew Gaul	d8cf26bd50	Remove unused field ... Found via clang.	2019-08-01 16:42:38 -07:00
Andrew Gaul	f528a86219	Pass lock_already_held state to DelStat	2019-08-01 11:07:56 -07:00
Andrew Gaul	5b15c7c4e9	Always hold lock when accessing free_disk_space ... Slightly reorder locks to avoid double locking. Found via ThreadSanitizer.	2019-07-30 08:36:54 -07:00
Andrew Gaul	afd438d363	Access FdEntity metadata while holding fdent_lock ... Create copies to avoid thread races. Found via ThreadSanitizer. Follows on to ecf13a8cb9.	2019-07-29 13:54:57 -07:00
Andrew Gaul	fb937635f5	Eagerly initialize sysconf variables ... Previously s3fs had races updating these shared variables. Found via ThreadSanitizer.	2019-07-17 09:08:13 -07:00
Takeshi Nakatani	3ad1c95e86	Merge pull request #1103 from gaul/config/readwrite-timeout ... Use consistent default for readwrite_timeout	2019-07-17 22:37:02 +09:00
Andrew Gaul	bacd15714a	Use consistent default for readwrite_timeout ... Previously curl threads mutated this shared state without a lock. Found via ThreadSanitizer.	2019-07-16 22:03:56 -07:00
Andrew Gaul	99aace4fc9	Avoid race when using thread-unsafe gmtime ... Found via ThreadSanitizer.	2019-07-16 07:28:34 -07:00
Takeshi Nakatani	513f41fddf	Merge pull request #1097 from ggtakec/fix_lock ... Add Lock() func parameter in RowFlush()	2019-07-15 11:56:21 +09:00
Takeshi Nakatani	543aed2a32	Merge pull request #1096 from gaul/pthread/lock-checking ... Expand error checking to all pthread_mutex_t	2019-07-15 11:43:56 +09:00
Takeshi Nakatani	20ea96328c	Merge pull request #1095 from gaul/intmax_t ... Avoid narrowing time_t and off_t args in logging	2019-07-15 11:33:44 +09:00
Takeshi Nakatani	f78bcc5229	Add Lock() func parameter in RowFlush()	2019-07-15 01:52:29 +00:00
Andrew Gaul	43ec064fb9	Expand error checking to all pthread_mutex_t	2019-07-14 18:21:23 -07:00
Andrew Gaul	ffac4c8417	Avoid narrowing time_t and off_t args in logging ... This displays correct results on 32-bit platforms.	2019-07-14 17:02:36 -07:00
Andrew Gaul	4adcd4a6c8	Implement s3fs_strtoofft via strtoll ... This tightens error checking and aligns s3fs with known good behavior.	2019-07-14 15:15:06 -07:00
Takeshi Nakatani	e936854493	Merge pull request #1094 from larsch/master ... Fix multipart upload not setting curl options	2019-07-15 05:13:37 +09:00
Takeshi Nakatani	850a813171	Merge pull request #1092 from liuyongqing/master ... fix coredump caused by fd_manager_lock locking	2019-07-15 04:53:10 +09:00
Takeshi Nakatani	5bbcd3b981	Merge pull request #1089 from gaul/null-dereference ... Avoid null dereference	2019-07-15 04:17:10 +09:00
Takeshi Nakatani	a337c32840	Merge pull request #1088 from gaul/clang-tidy/misplaced-const ... Avoid misplaced const via removing unneeded typedef	2019-07-15 04:08:51 +09:00
Takeshi Nakatani	d39e4e4b1f	Merge pull request #1087 from gaul/clang-tidy/deprecated-headers ... Prefer modern C headers	2019-07-15 03:50:15 +09:00
Lars Christensen	58037da061	Fix multipart upload not setting curl options	2019-07-14 14:09:37 +02:00
Andrew Gaul	1eb266588e	Add a missing lock to SetCtime ... Also lock before log statements which touch member variables.	2019-07-13 17:53:38 -07:00
root	4e351c59e3	fix coredump caused by fd_manager_lock locking	2019-07-13 12:03:27 +08:00
Andrew Gaul	eb597289cb	Avoid null dereference ... Found via clang-tidy.	2019-07-12 18:40:24 -07:00
Andrew Gaul	6fd42d9fe4	Prefer modern C headers ... Found and fixed via clang-tidy.	2019-07-12 03:50:59 -07:00
Andrew Gaul	efff9c01a6	Avoid misplaced const via removing unneeded typedef ... Found via clang-tidy.	2019-07-12 03:41:19 -07:00
Andrew Gaul	a83d5baa90	Remove recursive locking ... Recursive locking is frowned upon and is incompatible with PTHREAD_MUTEX_ERRORCHECK. Also clean up pthread_mutex_lock error checking.	2019-07-10 12:39:00 -07:00
Takeshi Nakatani	50d13255e4	Merge pull request #1081 from gaul/write-blocks-readdir ... Break FdEntity locks into data and metadata	2019-07-11 03:40:26 +09:00
Takeshi Nakatani	e5e63d6ac3	Merge pull request #1067 from gaul/mismatched-free ... Fix mismatched new[]/free	2019-07-07 16:55:07 +09:00
Takeshi Nakatani	7a65a414c3	Merge pull request #1080 from gaul/xml/simplification ... Add simple XML parsing wrapper	2019-07-07 16:44:18 +09:00
Andrew Gaul	944d21cabb	Fix mismatched new[]/free ... Found via Valgrind.	2019-07-07 00:36:48 -07:00
Takeshi Nakatani	d267212289	Merge pull request #1079 from gaul/cppcheck/shadowing ... Avoid shadowing variable in FdEntity::Open	2019-07-07 16:29:42 +09:00
Takeshi Nakatani	58d8e5586a	Merge pull request #1078 from gaul/multipart/2x ... Issue multipart when object size exceeds part size	2019-07-07 16:21:22 +09:00
Takeshi Nakatani	ce803daf4a	Merge pull request #1077 from gaul/openssl/locking ... Annotate OpenSSL locking functions as unused	2019-07-07 16:10:14 +09:00
Takeshi Nakatani	9bf34e2fda	Merge pull request #1076 from gaul/doc/man-help ... Add documentation for use_session_token and use_rrs	2019-07-07 15:45:42 +09:00
Takeshi Nakatani	6bd1a7eac0	Merge pull request #1074 from gaul/clang-tidy ... Configure clang-tidy target	2019-07-07 15:28:08 +09:00
Andrew Gaul	8aa06d621a	Add documentation for use_session_token and use_rrs ... Fixes #929.	2019-07-06 10:04:43 -07:00
Andrew Gaul	ecf13a8cb9	Break FdEntity locks into data and metadata ... Previously long-running data operations like RowFlush would block metadata operations like GetStats and thus user readdir. Fixes #928.	2019-07-05 23:12:24 -07:00
Andrew Gaul	b8ff6a647e	Add simple XML parsing wrapper ... Also simplify check_region_error.	2019-07-05 17:39:09 -07:00
Andrew Gaul	49110c671d	Avoid shadowing variable in FdEntity::Open ... Found via cppcheck 1.88.	2019-07-05 15:06:18 -07:00
Andrew Gaul	febaf6849f	Issue multipart when object size exceeds part size ... Previously s3fs issued multipart uploads when the object size was twice the part size. Conjoining this with the part size was confusing and s3fs should add a separate tunable for this if needed, similar to singlepart_copy_limit. Fixes #1058.	2019-07-05 12:14:56 -07:00
Andrew Gaul	4893174652	Annotate OpenSSL locking functions as unused ... OpenSSL 1.1.0 removed these and generates warnings with newer versions.	2019-07-05 10:45:57 -07:00
Andrew Gaul	4f23f38583	Individually test multiple s3fs flags ... Remove unneeded comments; single part limits ensure that the tests exercise multipart code paths even with smaller files. References #971.	2019-07-03 21:09:40 -07:00
Andrew Gaul	bbfa91141a	Configure clang-tidy target ... Also fix nits.	2019-07-03 14:04:11 -07:00
Takeshi Nakatani	21321a9d96	Merge pull request #1065 from gaul/doc/ssl-verify-hostname ... Document ssl_verify_hostname	2019-07-03 22:15:33 +09:00
Takeshi Nakatani	f03b50fd13	Merge pull request #1063 from gaul/truncate-2nd-open-fd ... Flush file when opening second fd	2019-07-03 21:14:03 +09:00
Andrew Gaul	f02105c346	Document ssl_verify_hostname ... References #1064.	2019-07-02 10:26:29 -07:00
Andrew Gaul	c596441f58	Flush file when opening second fd ... Previously when s3fs had dirty local data and an application opened a second fd it would remove the stat cache entry, query S3 for the size which was still zero, then reinitialize FdEntry with this incorrect size. Instead flush local data to S3 so this query works. It is possible that a more involved patch could do this with a less heavyweight approach but this requires changing open. This does not completely fix git clone but allows it to proceed further. Also make some cache methods const-correct. References #839.	2019-07-02 01:12:09 -07:00
Andrew Gaul	455e29cbea	Make fdpage a value type in fdpage_list_t ... This simplifies memory management.	2019-07-01 09:30:48 -07:00
Takeshi Nakatani	511d223468	Merge pull request #1059 from gaul/casting ... Fix casting warning on 32-bit	2019-07-01 22:26:14 +09:00
Andrew Gaul	d7f77a6282	Fix casting warning on 32-bit	2019-06-29 20:57:42 -07:00
Andrew Gaul	f1ad626b46	Log flags as hexadecimal	2019-06-24 16:13:07 -07:00
Takeshi Nakatani	a78d8d1da4	Merge pull request #1049 from gaul/external-modication ... Remove cache file when object time differs	2019-06-24 00:51:54 +09:00
Takeshi Nakatani	fbebc6fa57	Merge pull request #1048 from gaul/log/curl ... Emit older curl warnings only once	2019-06-24 00:42:44 +09:00
Andrew Gaul	9e5eaad79b	Remove cache file when object time differs ... Check the modification times to determine whether an object has updated. This relies on low clock skew between s3fs and the S3 server; a more robust approach could use the ETag. Fixes #1047.	2019-06-22 19:09:00 -07:00
Andrew Gaul	738eaadcbf	Emit older curl warnings only once ... This makes Travis logs actually readable.	2019-06-22 10:01:20 -07:00
Andrew Gaul	15b7450713	Convert file offsets from size_t to off_t ... The latter is 64-bits on 32-bit platforms when specifying -D_FILE_OFFSET_BITS=64. This allows early Raspberry Pis to use files larger than 2 GB. It also cleans up some ugly casting. Fixes #620. Fixes #656.	2019-06-15 17:05:37 -07:00
Andrew Gaul	059ab1f0f4	Update ctime during nocopyapi operations ... Follows on to 2c43b1e12b. Fixes #971.	2019-04-30 18:59:51 +09:00
Michal Lula	0d4847596e	fix issue with aws session token provided inside ~/.aws/credentials file	2019-04-25 14:29:35 +02:00
Takeshi Nakatani	8e86ef1634	Merge pull request #1015 from gaul/c++03 ... Remove uses of std::map::at	2019-04-22 23:01:29 +09:00
Andrew Gaul	9e4f9d4bdc	Remove uses of std::map::at ... This provides compatibility with C++03 and removes duplicate lookups.	2019-04-18 16:06:59 +09:00
Andrew Gaul	0677a096a9	Prefer simple over compound statements in macros ... This prohibits missing semicolons.	2019-04-17 21:46:24 +09:00
Michal Lula	f9cd43b684	add session token support	2019-04-16 16:53:05 +02:00
Andrew Gaul	8a18806a57	Make man page and --help more consistent ... Mostly small fixes but also some reordering. References #929. Found via: diff -u <(man --no-hyphenation doc/man/s3fs.1 | tr -s ' ' '\n' | sed '/^-o$/d' ) <(src/s3fs --help | tr -s ' ' '\n' | sed '/^-$/d')	2019-04-14 14:34:24 +09:00
Takeshi Nakatani	0a99470369	Merge pull request #1002 from gaul/sse-c ... Correctly calculate MD5 hash for SSE-C keys	2019-04-09 00:17:12 +09:00
Takeshi Nakatani	cd280d8702	Merge pull request #1001 from gaul/missing-braces ... Add missing braces	2019-04-08 23:55:02 +09:00
Andrew Gaul	6f8ecb28c5	Correctly calculate MD5 hash for SSE-C keys ... Previously s3fs calculated the strlen of a binary input instead of using the stored length. This yielded IO errors when interacting with SSE-encrypted objects. Fixes #696.	2019-04-08 21:49:42 +09:00
Andrew Gaul	042332bcec	Add missing braces ... Found via clang-tidy. Also fix errant indentation.	2019-04-07 23:12:27 +09:00
Takeshi Nakatani	071cd0f849	Modified to discard _netdev option etc	2019-04-07 11:51:55 +00:00
Andrew Gaul	6f6a67807b	Prefer new over malloc ... The former cannot return NULL and will allow use of scoped pointers.	2019-04-07 16:54:24 +09:00
Takeshi Nakatani	ea517c80a4	Fixed a bug about overwrite metadata at updating directory stats	2019-03-29 15:30:30 +00:00
Takeshi Nakatani	71766039ff	Support undefined CURLoption in libcurl library used in build	2019-03-22 10:47:42 +00:00
Andrew Gaul	058706014b	Simplify string comparison ... Found via newer cppcheck.	2019-03-14 22:19:33 +09:00
Takeshi Nakatani	62dcda6a56	Fixed ref-count when error occurred.	2019-03-10 06:04:19 +00:00
Takeshi Nakatani	cbf072bc55	Merge pull request #967 from gaul/reference-count ... Increase FdEntity reference count when returning	2019-03-10 14:27:10 +09:00
Takeshi Nakatani	b5ca400500	Merge pull request #970 from gaul/nocopyapi ... Remove from file from stat cache during rename	2019-03-10 12:56:04 +09:00
Andrew Gaul	08b132ddb9	Remove from file from stat cache during rename ... This addresses failures with test_rename_before_close when using nocopyapi. Note that test_update_time still fails.	2019-03-09 17:53:35 +09:00
Andrew Gaul	1e86cc643d	Fix lazy typo	2019-03-09 17:46:02 +09:00
Andrew Gaul	f53503438c	Increase FdEntity reference count when returning ... Previously s3fs had a race condition where one client could delete FdEntity that another client was using. Add a simple concurrent test which previously failed but now succeeds. Fixes #964.	2019-03-09 15:55:04 +09:00
Takeshi Nakatani	0d43d070cc	Merge pull request #965 from ggtakec/fix_multi_x ... Improvement of curl session pool for multipart	2019-02-27 22:44:25 +09:00
Takeshi Nakatani	0791fdca2a	Merge pull request #960 from kristjanvalur/wtf8 ... Wtf8	2019-02-27 21:21:38 +09:00
Kristján Valur Jónsson	6e8678d5e3	remove lines that wer accidentally duplicated	2019-02-25 16:02:01 +00:00
Takeshi Nakatani	10d9f75366	Improvement of curl session pool for multipart ... Improvement of curl session pool for multipart(2) Improvement of curl session pool for multipart(3)	2019-02-25 14:46:24 +00:00
Andrew Gaul	74d8671e54	Work around cppcheck warnings ... Follows on to eeb839242b.	2019-02-20 21:55:41 +09:00
Kristján Valur Jónsson	4c41eac29c	fix documentation and man page	2019-02-20 11:24:29 +00:00
Kristján Valur Jónsson	3c97c1b251	merged main	2019-02-19 10:53:00 +00:00
Kristján Valur Jónsson	84c671a81a	fix indentation	2019-02-19 10:37:43 +00:00
Kristján Valur Jónsson	f336bdebcc	add command line flag and documentation	2019-02-19 10:32:37 +00:00
Kristján Valur Jónsson	e5b8377202	fix comments and code	2019-02-19 10:32:10 +00:00
Kristján Valur Jónsson	4f42f4ab0c	Enable s3fs encoding and decoding in the fuse interface	2019-02-18 13:36:48 +00:00
Kristján Valur Jónsson	11b385820d	more robust wtf8 encoding	2019-02-18 12:27:44 +00:00
Takeshi Nakatani	eeb839242b	Fixed code for latest cppcheck error on OSX	2019-02-17 13:59:11 +00:00
Kristján Valur Jónsson	ca2d1d873d	Adding utility functions to convert invalid utf8 to wtf8 encoding ... This is to deal with windows clients who pass in cp1252 as if it were utf8	2019-02-15 15:57:03 +00:00
Takeshi Nakatani	231fd001d9	Added a non-interactive option to utility mode ... Rebase for resolving conflicts Deleted interactive mode on utility mode Fixed minor mistakes	2019-02-10 17:36:18 +00:00
Takeshi Nakatani	314dc5a398	Merge pull request #955 from gaul/clang-tidy/misc ... Address miscellaneous clang-tidy warnings	2019-02-06 21:21:54 +09:00
Takeshi Nakatani	e07cb020cc	Merge pull request #954 from gaul/clear-iter ... Clear containers instead of individual erases	2019-02-06 21:12:36 +09:00
Andrew Gaul	e87e40b3b4	Address miscellaneous clang-tidy warnings ... Also add configuration with suppressions.	2019-02-04 22:58:40 -08:00
Takeshi Nakatani	bd66b57ad3	Merge pull request #952 from gaul/multipart/failed ... Automatically abort failed multipart requests	2019-02-04 22:19:27 +09:00
Andrew Gaul	a1d3ff9766	Automatically abort failed multipart requests ... This can avoid dangling parts. However, many transfers fail due to network errors so we still need other mechanisms to handle these parts.	2019-02-03 10:29:20 -08:00
Andrew Gaul	4d0bef1e90	Clear containers instead of individual erases ... This has O(n) runtime instead of O(n log n).	2019-02-02 23:58:43 -08:00
Takeshi Nakatani	960823fb40	Added S3FS_MALLOC_TRIM build switch	2019-02-03 07:36:17 +00:00
Takeshi Nakatani	c04e8e7a9d	Merge pull request #949 from gaul/503-slow-down ... Implement exponential backoff for 503	2019-02-03 15:19:28 +09:00
Takeshi Nakatani	fb6debd986	Merge pull request #948 from gaul/too-many-parts ... Add logging for too many parts	2019-02-03 14:55:54 +09:00
Takeshi Nakatani	d8185a25aa	Merge pull request #946 from gaul/async/completed-tids ... Simplify async request completion code	2019-02-03 14:08:49 +09:00
Takeshi Nakatani	53337a0a28	Merge pull request #944 from gaul/utility-mode ... Repair utility mode	2019-02-03 13:59:56 +09:00
Takeshi Nakatani	ae51556d04	Merge pull request #943 from gaul/hard-link ... Return not supported when hard linking	2019-02-03 12:45:21 +09:00
Takeshi Nakatani	055ecf6ea7	Merge pull request #940 from gaul/parallel-multipart-copy ... Copy parts in parallel	2019-02-03 11:44:15 +09:00
Takeshi Nakatani	c603680e02	Merge pull request #939 from gaul/stringstream-specific ... Prefer specific [io]stringstream where possible	2019-02-03 11:06:57 +09:00
Takeshi Nakatani	814aadd7e3	Merge pull request #938 from gaul/clang-tidy/redundant-void ... Remove unneeded void parameter	2019-02-03 10:57:56 +09:00
Andrew Gaul	8ff05d8e38	Implement exponential backoff for 503 ... Amazon returns SlowDown when overloaded. Also return ENOTSUP for 501 and immediately return EIO for 500 instead of retrying. Fixes #603.	2019-02-02 00:03:50 -08:00
Andrew Gaul	dfa84b82a8	Add logging for too many parts ... References #610.	2019-02-01 19:34:26 -08:00
Andrew Gaul	8c527c3616	Simplify async request completion code ... Workers now notify the master thread when they complete, unifying the Linux and macOS code paths. This also avoids excessive pthread_tryjoin_np calls. Follows on to 88cd8feb05.	2019-01-31 22:55:23 -08:00
Andrew Gaul	54a074647e	Repair utility mode ... This deinitialized S3fsCurl twice and incorrectly calculated V4 signatures.	2019-01-31 18:45:39 -08:00
Andrew Gaul	c5ebf5d328	Copy parts in parallel ... S3 can copy multipart much faster than single part due to IO parallelization. Renaming a 4 GB file reduces from 72 to 20 seconds with bigger gains with larger files.	2019-01-31 10:21:39 -08:00
Andrew Gaul	43c6ef560e	Return not supported when hard linking ... This is more correct than permission denied.	2019-01-30 16:43:04 -08:00
Andrew Gaul	3076abc744	Disable malloc_trim ... This avoids walking the entire heap multiple times for complex operations like readdir. This does not entirely eliminate the observed performance regression but does dramatically reduce s3fs CPU usage. References #935. Fixes #936.	2019-01-29 15:29:07 -08:00
Andrew Gaul	07636c8a8d	Prefer specific [io]stringstream where possible ... These better communicate intent and are slightly more efficient.	2019-01-29 10:44:33 -08:00
Andrew Gaul	35d55ee513	Remove unneeded void parameter ... This is implicit in C++. Found and fixed via clang-tidy.	2019-01-28 23:22:27 -08:00
Takeshi Nakatani	c0cf90cf8b	Checked and corrected all typo	2019-01-27 12:04:29 +00:00
Takeshi Nakatani	3b1cc3b197	Merge pull request #933 from gaul/cache/remove-mirror-path ... Remove mirror path when deleting cache	2019-01-27 16:15:49 +09:00
Takeshi Nakatani	a0c1f30ae7	Merge pull request #932 from gaul/autolock ... Prefer AutoLock for synchronization	2019-01-27 15:59:18 +09:00
Takeshi Nakatani	8822a86709	Merge pull request #931 from gaul/doc/typo ... Correct sigv2 typo	2019-01-27 15:48:24 +09:00
Takeshi Nakatani	98f397de0e	Merge pull request #930 from gaul/doc/md5-multipart ... Correct enable_content_md5 docs	2019-01-27 15:36:56 +09:00
Takeshi Nakatani	fd4d23f8f7	Merge pull request #926 from kzidane/master ... Accept paths with : in them	2019-01-27 15:23:24 +09:00
Takeshi Nakatani	4820f0a42b	Merge pull request #925 from gaul/clang-tidy/delete-null ... Remove redundant null checks before delete	2019-01-27 15:15:31 +09:00
Andrew Gaul	a93e500b44	Remove mirror path when deleting cache ... Fixes #827.	2019-01-25 18:10:03 -08:00
Andrew Gaul	92d3114584	Prefer AutoLock for synchronization ... This simplifies the code and fixes an issue with unlocked access. Also use a recursive lock for StatCache to avoid races between lock..unlock..lock sequences.	2019-01-25 15:28:41 -08:00
Andrew Gaul	5062d6fbd9	Correct sigv2 typo	2019-01-25 14:30:30 -08:00
Andrew Gaul	7d14ebaf09	Correct enable_content_md5 docs ... Both S3fsCurl::PutRequest and S3fsCurl::UploadMultipartPostSetup can calculate and send Content-MD5 to the server. Remove spurious comment about large files and make man page and help consistent. References #929.	2019-01-25 14:27:52 -08:00
Kareem Zidane	cd794a6985	Accept paths with : in them	2019-01-23 14:56:25 -05:00
Andrew Gaul	84b421d6ef	Prefer empty over size checks ... Found and fixed via clang-tidy.	2019-01-23 11:30:28 -08:00
Andrew Gaul	8316da5bbe	Remove redundant null checks before delete ... Found by clang-tidy.	2019-01-23 11:25:25 -08:00
Takeshi Nakatani	caaf4cac55	Reverted #912(Automatic region change) and added message	2019-01-23 13:23:03 +00:00
Takeshi Nakatani	010276ceab	Merge pull request #921 from gaul/clang-tidy/redundant-string-init ... Remove redundant string initializations	2019-01-23 19:44:59 +09:00
Takeshi Nakatani	f219817eb3	Merge pull request #920 from gaul/clang-tidy/string-copy ... Remove unnecessary string copies	2019-01-23 19:30:55 +09:00
Andrew Gaul	1fc25e8c3f	Remove redundant string initializations ... Found and fixed via clang-tidy.	2019-01-22 23:16:37 -08:00
Andrew Gaul	61ecafd426	Remove unnecessary string copies ... Found via clang-tidy.	2019-01-22 23:09:37 -08:00
Andrew Gaul	5f5da4b2cb	Load tail range during overwrite ... Previously s3fs experienced data loss when writing to the middle of a file. Corrupt files would have the expected data from 0..offset+size but unexpected NUL bytes from offset+size..EOF. References #808.	2019-01-22 22:02:40 -08:00
Takeshi Nakatani	fada95f58e	Merge pull request #914 from gaul/readdir/head-of-line ... Issue readdir HEAD requests without batching	2019-01-21 22:10:50 +09:00
Takeshi Nakatani	014b8c5982	Merge pull request #913 from gaul/assert ... Prefer abort over assert(false)	2019-01-21 21:56:41 +09:00
Andrew Gaul	46d79c5bc2	Issue readdir HEAD requests without batching ... Previously s3fs would issue a batch of HEAD requests and wait for all to succeed before issuing the next batch. Now it issues the first batch and only waits for a single call to succeed before issuing the next call. This can improve performance when one call lags due to network errors. I measured 25% improvement with the same level of parallelism. This commit also reparents parallelism knobs for consistency. Follows on to 88cd8feb05. Fixes #223.	2019-01-20 18:07:22 -08:00