912 Commits

Author	SHA1	Message	Date
Takeshi Nakatani	2e51a339a9	Merge pull request #1115 from gaul/hash-length ... Use length from s3fs_HMAC256 for consistency	2019-08-06 20:49:22 +09:00
Takeshi Nakatani	14d3e12103	Merge pull request #1114 from gaul/flush/return-value ... Consume return value from FdEntity::Flush	2019-08-06 14:12:25 +09:00
Andrew Gaul	ccf3e7bfa2	Prefer s3fs_strtoofft over atoi and strtol ... The former propagates errors consistently.	2019-08-03 16:13:48 -07:00
Andrew Gaul	7ecfba811e	Do not heap allocate BodyData ... Callers already manage lifetimes via Clear().	2019-08-03 15:50:46 -07:00
Takeshi Nakatani	1e7330e499	Merge pull request #1113 from gaul/unused ... Remove unused field	2019-08-04 01:31:56 +09:00
Takeshi Nakatani	3d42d0515d	Merge pull request #1109 from gaul/lock-already-held ... Pass lock_already_held state to DelStat	2019-08-03 22:13:22 +09:00
Takeshi Nakatani	3d70e8966f	Merge pull request #1108 from gaul/lock/diskspace ... Always hold lock when accessing free_disk_space	2019-08-03 22:01:31 +09:00
Andrew Gaul	a5c20175a1	Use length from s3fs_HMAC256 for consistency ... EVP_MAX_MD_SIZE can range up to 64. Found by Coverity.	2019-08-02 10:40:55 -07:00
Andrew Gaul	43d1439420	Consume return value from FdEntity::Flush ... Found via Coverity.	2019-08-02 10:01:06 -07:00
Andrew Gaul	d8cf26bd50	Remove unused field ... Found via clang.	2019-08-01 16:42:38 -07:00
Andrew Gaul	f528a86219	Pass lock_already_held state to DelStat	2019-08-01 11:07:56 -07:00
Andrew Gaul	5b15c7c4e9	Always hold lock when accessing free_disk_space ... Slightly reorder locks to avoid double locking. Found via ThreadSanitizer.	2019-07-30 08:36:54 -07:00
Andrew Gaul	afd438d363	Access FdEntity metadata while holding fdent_lock ... Create copies to avoid thread races. Found via ThreadSanitizer. Follows on to ecf13a8cb9.	2019-07-29 13:54:57 -07:00
Andrew Gaul	fb937635f5	Eagerly initialize sysconf variables ... Previously s3fs had races updating these shared variables. Found via ThreadSanitizer.	2019-07-17 09:08:13 -07:00
Takeshi Nakatani	3ad1c95e86	Merge pull request #1103 from gaul/config/readwrite-timeout ... Use consistent default for readwrite_timeout	2019-07-17 22:37:02 +09:00
Andrew Gaul	bacd15714a	Use consistent default for readwrite_timeout ... Previously curl threads mutated this shared state without a lock. Found via ThreadSanitizer.	2019-07-16 22:03:56 -07:00
Andrew Gaul	99aace4fc9	Avoid race when using thread-unsafe gmtime ... Found via ThreadSanitizer.	2019-07-16 07:28:34 -07:00
Takeshi Nakatani	513f41fddf	Merge pull request #1097 from ggtakec/fix_lock ... Add Lock() func parameter in RowFlush()	2019-07-15 11:56:21 +09:00
Takeshi Nakatani	543aed2a32	Merge pull request #1096 from gaul/pthread/lock-checking ... Expand error checking to all pthread_mutex_t	2019-07-15 11:43:56 +09:00
Takeshi Nakatani	20ea96328c	Merge pull request #1095 from gaul/intmax_t ... Avoid narrowing time_t and off_t args in logging	2019-07-15 11:33:44 +09:00
Takeshi Nakatani	f78bcc5229	Add Lock() func parameter in RowFlush()	2019-07-15 01:52:29 +00:00
Andrew Gaul	43ec064fb9	Expand error checking to all pthread_mutex_t	2019-07-14 18:21:23 -07:00
Andrew Gaul	ffac4c8417	Avoid narrowing time_t and off_t args in logging ... This displays correct results on 32-bit platforms.	2019-07-14 17:02:36 -07:00
Andrew Gaul	4adcd4a6c8	Implement s3fs_strtoofft via strtoll ... This tightens error checking and aligns s3fs with known good behavior.	2019-07-14 15:15:06 -07:00
Takeshi Nakatani	e936854493	Merge pull request #1094 from larsch/master ... Fix multipart upload not setting curl options	2019-07-15 05:13:37 +09:00
Takeshi Nakatani	850a813171	Merge pull request #1092 from liuyongqing/master ... fix coredump caused by fd_manager_lock locking	2019-07-15 04:53:10 +09:00
Takeshi Nakatani	5bbcd3b981	Merge pull request #1089 from gaul/null-dereference ... Avoid null dereference	2019-07-15 04:17:10 +09:00
Takeshi Nakatani	a337c32840	Merge pull request #1088 from gaul/clang-tidy/misplaced-const ... Avoid misplaced const via removing unneeded typedef	2019-07-15 04:08:51 +09:00
Takeshi Nakatani	d39e4e4b1f	Merge pull request #1087 from gaul/clang-tidy/deprecated-headers ... Prefer modern C headers	2019-07-15 03:50:15 +09:00
Lars Christensen	58037da061	Fix multipart upload not setting curl options	2019-07-14 14:09:37 +02:00
Andrew Gaul	1eb266588e	Add a missing lock to SetCtime ... Also lock before log statements which touch member variables.	2019-07-13 17:53:38 -07:00
root	4e351c59e3	fix coredump caused by fd_manager_lock locking	2019-07-13 12:03:27 +08:00
Andrew Gaul	eb597289cb	Avoid null dereference ... Found via clang-tidy.	2019-07-12 18:40:24 -07:00
Andrew Gaul	6fd42d9fe4	Prefer modern C headers ... Found and fixed via clang-tidy.	2019-07-12 03:50:59 -07:00
Andrew Gaul	efff9c01a6	Avoid misplaced const via removing unneeded typedef ... Found via clang-tidy.	2019-07-12 03:41:19 -07:00
Andrew Gaul	a83d5baa90	Remove recursive locking ... Recursive locking is frowned upon and is incompatible with PTHREAD_MUTEX_ERRORCHECK. Also clean up pthread_mutex_lock error checking.	2019-07-10 12:39:00 -07:00
Takeshi Nakatani	50d13255e4	Merge pull request #1081 from gaul/write-blocks-readdir ... Break FdEntity locks into data and metadata	2019-07-11 03:40:26 +09:00
Takeshi Nakatani	e5e63d6ac3	Merge pull request #1067 from gaul/mismatched-free ... Fix mismatched new[]/free	2019-07-07 16:55:07 +09:00
Takeshi Nakatani	7a65a414c3	Merge pull request #1080 from gaul/xml/simplification ... Add simple XML parsing wrapper	2019-07-07 16:44:18 +09:00
Andrew Gaul	944d21cabb	Fix mismatched new[]/free ... Found via Valgrind.	2019-07-07 00:36:48 -07:00
Takeshi Nakatani	d267212289	Merge pull request #1079 from gaul/cppcheck/shadowing ... Avoid shadowing variable in FdEntity::Open	2019-07-07 16:29:42 +09:00
Takeshi Nakatani	58d8e5586a	Merge pull request #1078 from gaul/multipart/2x ... Issue multipart when object size exceeds part size	2019-07-07 16:21:22 +09:00
Takeshi Nakatani	ce803daf4a	Merge pull request #1077 from gaul/openssl/locking ... Annotate OpenSSL locking functions as unused	2019-07-07 16:10:14 +09:00
Takeshi Nakatani	9bf34e2fda	Merge pull request #1076 from gaul/doc/man-help ... Add documentation for use_session_token and use_rrs	2019-07-07 15:45:42 +09:00
Takeshi Nakatani	6bd1a7eac0	Merge pull request #1074 from gaul/clang-tidy ... Configure clang-tidy target	2019-07-07 15:28:08 +09:00
Andrew Gaul	8aa06d621a	Add documentation for use_session_token and use_rrs ... Fixes #929.	2019-07-06 10:04:43 -07:00
Andrew Gaul	ecf13a8cb9	Break FdEntity locks into data and metadata ... Previously long-running data operations like RowFlush would block metadata operations like GetStats and thus user readdir. Fixes #928.	2019-07-05 23:12:24 -07:00
Andrew Gaul	b8ff6a647e	Add simple XML parsing wrapper ... Also simplify check_region_error.	2019-07-05 17:39:09 -07:00
Andrew Gaul	49110c671d	Avoid shadowing variable in FdEntity::Open ... Found via cppcheck 1.88.	2019-07-05 15:06:18 -07:00
Andrew Gaul	febaf6849f	Issue multipart when object size exceeds part size ... Previously s3fs issued multipart uploads when the object size was twice the part size. Conjoining this with the part size was confusing and s3fs should add a separate tunable for this if needed, similar to singlepart_copy_limit. Fixes #1058.	2019-07-05 12:14:56 -07:00
Andrew Gaul	4893174652	Annotate OpenSSL locking functions as unused ... OpenSSL 1.1.0 removed these and generates warnings with newer versions.	2019-07-05 10:45:57 -07:00
Andrew Gaul	4f23f38583	Individually test multiple s3fs flags ... Remove unneeded comments; single part limits ensure that the tests exercise multipart code paths even with smaller files. References #971.	2019-07-03 21:09:40 -07:00
Andrew Gaul	bbfa91141a	Configure clang-tidy target ... Also fix nits.	2019-07-03 14:04:11 -07:00
Takeshi Nakatani	21321a9d96	Merge pull request #1065 from gaul/doc/ssl-verify-hostname ... Document ssl_verify_hostname	2019-07-03 22:15:33 +09:00
Takeshi Nakatani	f03b50fd13	Merge pull request #1063 from gaul/truncate-2nd-open-fd ... Flush file when opening second fd	2019-07-03 21:14:03 +09:00
Andrew Gaul	f02105c346	Document ssl_verify_hostname ... References #1064.	2019-07-02 10:26:29 -07:00
Andrew Gaul	c596441f58	Flush file when opening second fd ... Previously when s3fs had dirty local data and an application opened a second fd it would remove the stat cache entry, query S3 for the size which was still zero, then reinitialize FdEntry with this incorrect size. Instead flush local data to S3 so this query works. It is possible that a more involved patch could do this with a less heavyweight approach but this requires changing open. This does not completely fix git clone but allows it to proceed further. Also make some cache methods const-correct. References #839.	2019-07-02 01:12:09 -07:00
Andrew Gaul	455e29cbea	Make fdpage a value type in fdpage_list_t ... This simplifies memory management.	2019-07-01 09:30:48 -07:00
Takeshi Nakatani	511d223468	Merge pull request #1059 from gaul/casting ... Fix casting warning on 32-bit	2019-07-01 22:26:14 +09:00
Andrew Gaul	d7f77a6282	Fix casting warning on 32-bit	2019-06-29 20:57:42 -07:00
Andrew Gaul	f1ad626b46	Log flags as hexadecimal	2019-06-24 16:13:07 -07:00
Takeshi Nakatani	a78d8d1da4	Merge pull request #1049 from gaul/external-modication ... Remove cache file when object time differs	2019-06-24 00:51:54 +09:00
Takeshi Nakatani	fbebc6fa57	Merge pull request #1048 from gaul/log/curl ... Emit older curl warnings only once	2019-06-24 00:42:44 +09:00
Andrew Gaul	9e5eaad79b	Remove cache file when object time differs ... Check the modification times to determine whether an object has updated. This relies on low clock skew between s3fs and the S3 server; a more robust approach could use the ETag. Fixes #1047.	2019-06-22 19:09:00 -07:00
Andrew Gaul	738eaadcbf	Emit older curl warnings only once ... This makes Travis logs actually readable.	2019-06-22 10:01:20 -07:00
Andrew Gaul	15b7450713	Convert file offsets from size_t to off_t ... The latter is 64-bits on 32-bit platforms when specifying -D_FILE_OFFSET_BITS=64. This allows early Raspberry Pis to use files larger than 2 GB. It also cleans up some ugly casting. Fixes #620. Fixes #656.	2019-06-15 17:05:37 -07:00
Andrew Gaul	059ab1f0f4	Update ctime during nocopyapi operations ... Follows on to 2c43b1e12b. Fixes #971.	2019-04-30 18:59:51 +09:00
Michal Lula	0d4847596e	fix issue with aws session token provided inside ~/.aws/credentials file	2019-04-25 14:29:35 +02:00
Takeshi Nakatani	8e86ef1634	Merge pull request #1015 from gaul/c++03 ... Remove uses of std::map::at	2019-04-22 23:01:29 +09:00
Andrew Gaul	9e4f9d4bdc	Remove uses of std::map::at ... This provides compatibility with C++03 and removes duplicate lookups.	2019-04-18 16:06:59 +09:00
Andrew Gaul	0677a096a9	Prefer simple over compound statements in macros ... This prohibits missing semicolons.	2019-04-17 21:46:24 +09:00
Michal Lula	f9cd43b684	add session token support	2019-04-16 16:53:05 +02:00
Andrew Gaul	8a18806a57	Make man page and --help more consistent ... Mostly small fixes but also some reordering. References #929. Found via: diff -u <(man --no-hyphenation doc/man/s3fs.1 | tr -s ' ' '\n' | sed '/^-o$/d' ) <(src/s3fs --help | tr -s ' ' '\n' | sed '/^-$/d')	2019-04-14 14:34:24 +09:00
Takeshi Nakatani	0a99470369	Merge pull request #1002 from gaul/sse-c ... Correctly calculate MD5 hash for SSE-C keys	2019-04-09 00:17:12 +09:00
Takeshi Nakatani	cd280d8702	Merge pull request #1001 from gaul/missing-braces ... Add missing braces	2019-04-08 23:55:02 +09:00
Andrew Gaul	6f8ecb28c5	Correctly calculate MD5 hash for SSE-C keys ... Previously s3fs calculated the strlen of a binary input instead of using the stored length. This yielded IO errors when interacting with SSE-encrypted objects. Fixes #696.	2019-04-08 21:49:42 +09:00
Andrew Gaul	042332bcec	Add missing braces ... Found via clang-tidy. Also fix errant indentation.	2019-04-07 23:12:27 +09:00
Takeshi Nakatani	071cd0f849	Modified to discard _netdev option etc	2019-04-07 11:51:55 +00:00
Andrew Gaul	6f6a67807b	Prefer new over malloc ... The former cannot return NULL and will allow use of scoped pointers.	2019-04-07 16:54:24 +09:00
Takeshi Nakatani	ea517c80a4	Fixed a bug about overwrite metadata at updating directory stats	2019-03-29 15:30:30 +00:00
Takeshi Nakatani	71766039ff	Support undefined CURLoption in libcurl library used in build	2019-03-22 10:47:42 +00:00
Andrew Gaul	058706014b	Simplify string comparison ... Found via newer cppcheck.	2019-03-14 22:19:33 +09:00
Takeshi Nakatani	62dcda6a56	Fixed ref-count when error occurred.	2019-03-10 06:04:19 +00:00
Takeshi Nakatani	cbf072bc55	Merge pull request #967 from gaul/reference-count ... Increase FdEntity reference count when returning	2019-03-10 14:27:10 +09:00
Takeshi Nakatani	b5ca400500	Merge pull request #970 from gaul/nocopyapi ... Remove from file from stat cache during rename	2019-03-10 12:56:04 +09:00
Andrew Gaul	08b132ddb9	Remove from file from stat cache during rename ... This addresses failures with test_rename_before_close when using nocopyapi. Note that test_update_time still fails.	2019-03-09 17:53:35 +09:00
Andrew Gaul	1e86cc643d	Fix lazy typo	2019-03-09 17:46:02 +09:00
Andrew Gaul	f53503438c	Increase FdEntity reference count when returning ... Previously s3fs had a race condition where one client could delete FdEntity that another client was using. Add a simple concurrent test which previously failed but now succeeds. Fixes #964.	2019-03-09 15:55:04 +09:00
Takeshi Nakatani	0d43d070cc	Merge pull request #965 from ggtakec/fix_multi_x ... Improvement of curl session pool for multipart	2019-02-27 22:44:25 +09:00
Takeshi Nakatani	0791fdca2a	Merge pull request #960 from kristjanvalur/wtf8 ... Wtf8	2019-02-27 21:21:38 +09:00
Kristján Valur Jónsson	6e8678d5e3	remove lines that wer accidentally duplicated	2019-02-25 16:02:01 +00:00
Takeshi Nakatani	10d9f75366	Improvement of curl session pool for multipart ... Improvement of curl session pool for multipart(2) Improvement of curl session pool for multipart(3)	2019-02-25 14:46:24 +00:00
Andrew Gaul	74d8671e54	Work around cppcheck warnings ... Follows on to eeb839242b.	2019-02-20 21:55:41 +09:00
Kristján Valur Jónsson	4c41eac29c	fix documentation and man page	2019-02-20 11:24:29 +00:00
Kristján Valur Jónsson	3c97c1b251	merged main	2019-02-19 10:53:00 +00:00
Kristján Valur Jónsson	84c671a81a	fix indentation	2019-02-19 10:37:43 +00:00
Kristján Valur Jónsson	f336bdebcc	add command line flag and documentation	2019-02-19 10:32:37 +00:00
Kristján Valur Jónsson	e5b8377202	fix comments and code	2019-02-19 10:32:10 +00:00
Kristján Valur Jónsson	4f42f4ab0c	Enable s3fs encoding and decoding in the fuse interface	2019-02-18 13:36:48 +00:00
Kristján Valur Jónsson	11b385820d	more robust wtf8 encoding	2019-02-18 12:27:44 +00:00
Takeshi Nakatani	eeb839242b	Fixed code for latest cppcheck error on OSX	2019-02-17 13:59:11 +00:00
Kristján Valur Jónsson	ca2d1d873d	Adding utility functions to convert invalid utf8 to wtf8 encoding ... This is to deal with windows clients who pass in cp1252 as if it were utf8	2019-02-15 15:57:03 +00:00
Takeshi Nakatani	231fd001d9	Added a non-interactive option to utility mode ... Rebase for resolving conflicts Deleted interactive mode on utility mode Fixed minor mistakes	2019-02-10 17:36:18 +00:00
Takeshi Nakatani	314dc5a398	Merge pull request #955 from gaul/clang-tidy/misc ... Address miscellaneous clang-tidy warnings	2019-02-06 21:21:54 +09:00
Takeshi Nakatani	e07cb020cc	Merge pull request #954 from gaul/clear-iter ... Clear containers instead of individual erases	2019-02-06 21:12:36 +09:00
Andrew Gaul	e87e40b3b4	Address miscellaneous clang-tidy warnings ... Also add configuration with suppressions.	2019-02-04 22:58:40 -08:00
Takeshi Nakatani	bd66b57ad3	Merge pull request #952 from gaul/multipart/failed ... Automatically abort failed multipart requests	2019-02-04 22:19:27 +09:00
Andrew Gaul	a1d3ff9766	Automatically abort failed multipart requests ... This can avoid dangling parts. However, many transfers fail due to network errors so we still need other mechanisms to handle these parts.	2019-02-03 10:29:20 -08:00
Andrew Gaul	4d0bef1e90	Clear containers instead of individual erases ... This has O(n) runtime instead of O(n log n).	2019-02-02 23:58:43 -08:00
Takeshi Nakatani	960823fb40	Added S3FS_MALLOC_TRIM build switch	2019-02-03 07:36:17 +00:00
Takeshi Nakatani	c04e8e7a9d	Merge pull request #949 from gaul/503-slow-down ... Implement exponential backoff for 503	2019-02-03 15:19:28 +09:00
Takeshi Nakatani	fb6debd986	Merge pull request #948 from gaul/too-many-parts ... Add logging for too many parts	2019-02-03 14:55:54 +09:00
Takeshi Nakatani	d8185a25aa	Merge pull request #946 from gaul/async/completed-tids ... Simplify async request completion code	2019-02-03 14:08:49 +09:00
Takeshi Nakatani	53337a0a28	Merge pull request #944 from gaul/utility-mode ... Repair utility mode	2019-02-03 13:59:56 +09:00
Takeshi Nakatani	ae51556d04	Merge pull request #943 from gaul/hard-link ... Return not supported when hard linking	2019-02-03 12:45:21 +09:00
Takeshi Nakatani	055ecf6ea7	Merge pull request #940 from gaul/parallel-multipart-copy ... Copy parts in parallel	2019-02-03 11:44:15 +09:00
Takeshi Nakatani	c603680e02	Merge pull request #939 from gaul/stringstream-specific ... Prefer specific [io]stringstream where possible	2019-02-03 11:06:57 +09:00
Takeshi Nakatani	814aadd7e3	Merge pull request #938 from gaul/clang-tidy/redundant-void ... Remove unneeded void parameter	2019-02-03 10:57:56 +09:00
Andrew Gaul	8ff05d8e38	Implement exponential backoff for 503 ... Amazon returns SlowDown when overloaded. Also return ENOTSUP for 501 and immediately return EIO for 500 instead of retrying. Fixes #603.	2019-02-02 00:03:50 -08:00
Andrew Gaul	dfa84b82a8	Add logging for too many parts ... References #610.	2019-02-01 19:34:26 -08:00
Andrew Gaul	8c527c3616	Simplify async request completion code ... Workers now notify the master thread when they complete, unifying the Linux and macOS code paths. This also avoids excessive pthread_tryjoin_np calls. Follows on to 88cd8feb05.	2019-01-31 22:55:23 -08:00
Andrew Gaul	54a074647e	Repair utility mode ... This deinitialized S3fsCurl twice and incorrectly calculated V4 signatures.	2019-01-31 18:45:39 -08:00
Andrew Gaul	c5ebf5d328	Copy parts in parallel ... S3 can copy multipart much faster than single part due to IO parallelization. Renaming a 4 GB file reduces from 72 to 20 seconds with bigger gains with larger files.	2019-01-31 10:21:39 -08:00
Andrew Gaul	43c6ef560e	Return not supported when hard linking ... This is more correct than permission denied.	2019-01-30 16:43:04 -08:00
Andrew Gaul	3076abc744	Disable malloc_trim ... This avoids walking the entire heap multiple times for complex operations like readdir. This does not entirely eliminate the observed performance regression but does dramatically reduce s3fs CPU usage. References #935. Fixes #936.	2019-01-29 15:29:07 -08:00
Andrew Gaul	07636c8a8d	Prefer specific [io]stringstream where possible ... These better communicate intent and are slightly more efficient.	2019-01-29 10:44:33 -08:00
Andrew Gaul	35d55ee513	Remove unneeded void parameter ... This is implicit in C++. Found and fixed via clang-tidy.	2019-01-28 23:22:27 -08:00
Takeshi Nakatani	c0cf90cf8b	Checked and corrected all typo	2019-01-27 12:04:29 +00:00
Takeshi Nakatani	3b1cc3b197	Merge pull request #933 from gaul/cache/remove-mirror-path ... Remove mirror path when deleting cache	2019-01-27 16:15:49 +09:00
Takeshi Nakatani	a0c1f30ae7	Merge pull request #932 from gaul/autolock ... Prefer AutoLock for synchronization	2019-01-27 15:59:18 +09:00
Takeshi Nakatani	8822a86709	Merge pull request #931 from gaul/doc/typo ... Correct sigv2 typo	2019-01-27 15:48:24 +09:00
Takeshi Nakatani	98f397de0e	Merge pull request #930 from gaul/doc/md5-multipart ... Correct enable_content_md5 docs	2019-01-27 15:36:56 +09:00
Takeshi Nakatani	fd4d23f8f7	Merge pull request #926 from kzidane/master ... Accept paths with : in them	2019-01-27 15:23:24 +09:00
Takeshi Nakatani	4820f0a42b	Merge pull request #925 from gaul/clang-tidy/delete-null ... Remove redundant null checks before delete	2019-01-27 15:15:31 +09:00
Andrew Gaul	a93e500b44	Remove mirror path when deleting cache ... Fixes #827.	2019-01-25 18:10:03 -08:00
Andrew Gaul	92d3114584	Prefer AutoLock for synchronization ... This simplifies the code and fixes an issue with unlocked access. Also use a recursive lock for StatCache to avoid races between lock..unlock..lock sequences.	2019-01-25 15:28:41 -08:00
Andrew Gaul	5062d6fbd9	Correct sigv2 typo	2019-01-25 14:30:30 -08:00
Andrew Gaul	7d14ebaf09	Correct enable_content_md5 docs ... Both S3fsCurl::PutRequest and S3fsCurl::UploadMultipartPostSetup can calculate and send Content-MD5 to the server. Remove spurious comment about large files and make man page and help consistent. References #929.	2019-01-25 14:27:52 -08:00
Kareem Zidane	cd794a6985	Accept paths with : in them	2019-01-23 14:56:25 -05:00
Andrew Gaul	84b421d6ef	Prefer empty over size checks ... Found and fixed via clang-tidy.	2019-01-23 11:30:28 -08:00
Andrew Gaul	8316da5bbe	Remove redundant null checks before delete ... Found by clang-tidy.	2019-01-23 11:25:25 -08:00
Takeshi Nakatani	caaf4cac55	Reverted #912(Automatic region change) and added message	2019-01-23 13:23:03 +00:00
Takeshi Nakatani	010276ceab	Merge pull request #921 from gaul/clang-tidy/redundant-string-init ... Remove redundant string initializations	2019-01-23 19:44:59 +09:00
Takeshi Nakatani	f219817eb3	Merge pull request #920 from gaul/clang-tidy/string-copy ... Remove unnecessary string copies	2019-01-23 19:30:55 +09:00
Andrew Gaul	1fc25e8c3f	Remove redundant string initializations ... Found and fixed via clang-tidy.	2019-01-22 23:16:37 -08:00
Andrew Gaul	61ecafd426	Remove unnecessary string copies ... Found via clang-tidy.	2019-01-22 23:09:37 -08:00
Andrew Gaul	5f5da4b2cb	Load tail range during overwrite ... Previously s3fs experienced data loss when writing to the middle of a file. Corrupt files would have the expected data from 0..offset+size but unexpected NUL bytes from offset+size..EOF. References #808.	2019-01-22 22:02:40 -08:00
Takeshi Nakatani	fada95f58e	Merge pull request #914 from gaul/readdir/head-of-line ... Issue readdir HEAD requests without batching	2019-01-21 22:10:50 +09:00
Takeshi Nakatani	014b8c5982	Merge pull request #913 from gaul/assert ... Prefer abort over assert(false)	2019-01-21 21:56:41 +09:00
Andrew Gaul	46d79c5bc2	Issue readdir HEAD requests without batching ... Previously s3fs would issue a batch of HEAD requests and wait for all to succeed before issuing the next batch. Now it issues the first batch and only waits for a single call to succeed before issuing the next call. This can improve performance when one call lags due to network errors. I measured 25% improvement with the same level of parallelism. This commit also reparents parallelism knobs for consistency. Follows on to 88cd8feb05. Fixes #223.	2019-01-20 18:07:22 -08:00
Andrew Gaul	40ba3b44a1	Prefer abort over assert(false) ... The compiler can remove the latter when compiled with NDEBUG which may cause unintended control flow.	2019-01-20 12:30:27 -08:00
Takeshi Nakatani	2887f8916b	Automatic region change made possible other than us-east-1(default)	2019-01-20 10:51:49 +00:00
Takeshi Nakatani	ac72431195	Added detail error message when HTTP 301/307 status	2019-01-20 10:07:58 +00:00
Takeshi Nakatani	f2184e34dd	Ignore after period character of floating point in x-amz-meta-mtime	2019-01-20 08:28:06 +00:00
Takeshi Nakatani	36a4903843	Added an error message when HTTP 301 status	2019-01-20 07:17:40 +00:00
Takeshi Nakatani	c83a3e67c9	Merge pull request #885 from LutzFinsterle2019/master ... Update s3fs_util.cpp for correspondence of Nextcloud contype	2019-01-20 15:32:58 +09:00
Takeshi Nakatani	05014c49c8	Merge pull request #906 from gaul/doc/https ... Prefer HTTPS links where possible	2019-01-20 15:10:51 +09:00
Takeshi Nakatani	aa69107165	Merge pull request #905 from gaul/clang-tidy/redundant ... Fix comparison in s3fs_strtoofft	2019-01-20 15:01:40 +09:00
Takeshi Nakatani	d373b0eca3	Merge pull request #904 from gaul/clang-tidy/c-str ... Remove unnecessary calls to std::string::c_str	2019-01-20 14:13:46 +09:00
Takeshi Nakatani	6aa40b2747	Merge pull request #903 from gaul/clang-tidy/find_char ... Prefer find(char) over find(const char *)	2019-01-20 14:05:06 +09:00
Takeshi Nakatani	34c3bfe408	Merge pull request #902 from gaul/clang-tidy/pass-by-value ... Avoid pass-by-value when not necessary	2019-01-20 13:45:44 +09:00
Takeshi Nakatani	6ac56e722d	Merge pull request #901 from gaul/clang-tidy/leaks ... Plug memory leaks	2019-01-20 12:41:08 +09:00
Takeshi Nakatani	61dc7f0a70	Merge pull request #900 from gaul/leak ... Plug memory leak	2019-01-20 12:30:31 +09:00
Takeshi Nakatani	9f000957dd	Merge pull request #899 from gaul/response-code ... Tighten up HTTP response code check	2019-01-20 11:16:23 +09:00
Andrew Gaul	aa9bd1fa3c	Prefer HTTPS links where possible ... Fix a few stale links as well.	2019-01-18 11:09:08 -08:00
Andrew Gaul	5a2dc03a1c	Fix comparison in s3fs_strtoofft ... Also backfill unit tests. Document limitations. Found via clang-tidy.	2019-01-17 22:59:25 -08:00
Andrew Gaul	508fafbe62	Remove unnecessary calls to std::string::c_str ... Found via clang-tidy.	2019-01-17 22:05:16 -08:00
Andrew Gaul	e29548178b	Prefer find(char) over find(const char *) ... The former can be faster. Found via clang-tidy.	2019-01-17 20:24:24 -08:00
Andrew Gaul	ab2f36f202	Plug memory leaks ... Found via clang-tidy.	2019-01-17 18:54:34 -08:00
Andrew Gaul	b8c9fcfd70	Avoid pass-by-value when not necessary ... This requires unnecessary memcpy. Found via clang-tidy.	2019-01-17 18:22:11 -08:00
Andrew Gaul	58ce544e83	Plug memory leak ... Previously this appended to an empty list, zeroed the list, then appended to the list. Instead zero the list first and then append. This also enables sending Content-MD5 which can eagerly detect transmission errors. Found via Valgrind.	2019-01-17 16:24:34 -08:00
Andrew Gaul	e98ce36301	Tighten up HTTP response code check ... Previously s3fs considered 3xx codes as successful. When writing an object to a misconfigured region, s3fs did not propagate the 301 as a write error to the client. References #693.	2019-01-17 15:04:59 -08:00
Andrew Gaul	6401b4ae92	Flush file before renaming ... Previously s3fs could copy the zero-byte stub object without including any pending writes. Fixes #145.	2019-01-17 12:05:10 -08:00
Takeshi Nakatani	25b49e1a2e	Merge pull request #894 from gaul/default-mode ... Default uid/gid/mode when object lacks permissions	2019-01-16 18:48:54 +09:00
Takeshi Nakatani	c7def35b54	Merge pull request #895 from gaul/bucket-name-dot ... Emit more friendly error for buckets with dots	2019-01-16 18:39:47 +09:00
Andrew Gaul	c512516e14	Emit more friendly error for buckets with dots ... These fail SSL certificate checks due to the *.s3.amazon.com wildcard. Fixes #284.	2019-01-14 18:47:36 -08:00
Andrew Gaul	2c43b1e12b	Store and retrieve file change time ... This introduces a new header with the change time; existing objects will report modification time. Fixes #771.	2019-01-14 10:05:11 -08:00
Andrew Gaul	e2d5641d99	Default uid/gid/mode when object lacks permissions ... This addresses a common use case when interacting with objects from both s3fs and other S3 tools. Fixes #890.	2019-01-13 21:57:23 -08:00
LutzFinsterle2019	c985b5e4d0	Corrected Comment to C++ style	2019-01-12 10:19:48 +01:00
LutzFinsterle2019	18cb2e2662	Update s3fs_util.cpp ... Sorry for answering late, have been busy lately. The comment is: "Nextcloud stores Directory objects with this mime type when mounting a Bucket as external Storage"	2019-01-10 07:44:36 +01:00
LutzFinsterle2019	743c706b0a	Update s3fs_util.cpp ... Nextcloud Compatibility in directory mime-types	2019-01-07 07:41:27 +01:00
Takeshi Nakatani	4ed0e5f35a	Merge pull request #882 from earlchew/issue-817 ... [curl] Assume long encryption keys are base64 encoded	2019-01-06 17:29:04 +09:00
Takeshi Nakatani	fd6b37d3da	Merge pull request #877 from gaul/aws/credentials ... Check arguments and environment before .aws/creds	2019-01-06 17:03:11 +09:00
Takeshi Nakatani	56e24de0d4	Merge pull request #870 from gaul/typos ... Correct typos in command-line parsing	2019-01-06 16:21:46 +09:00
Earl Chew	ed5795eead	[curl] Assume long encryption keys are base64 encoded ... Correct tabs and whitespace. Signed-off-by: Earl Chew <earl_chew@yahoo.com>	2019-01-05 21:08:41 +00:00
Andrew Gaul	3d225163f8	Correct typos in command-line parsing ... Also fix stray macOS references. References #869.	2019-01-02 14:40:57 -08:00
Andrew Gaul	0569cec3ea	Check arguments and environment before .aws/creds ... Fixes #857. Fixes #864.	2018-12-20 15:51:32 -08:00
Andrew Gaul	a2f8ac535e	Address cppcheck 1.86 errors ... Lifetime, shadowing, and unused variables. Found via the Travis macOS builder.	2018-12-20 14:56:31 -08:00
Takeshi Nakatani	d9e89deef6	Merge pull request #865 from orozery/multihead_warning_check ... fix multihead warning check	2018-11-29 20:53:47 +09:00
Or Ozeri	b91fc5409e	fix multihead warning check	2018-11-28 09:54:02 +02:00
Michael Chen	3c970646d1	Add 'profile' option to command line help.	2018-11-19 08:26:23 -08:00
Andrew Gaul	88cd8feb05	Upload S3 parts without batching ... Previously s3fs would issue a batch of uploads and wait for all to succeed before issuing the next batch. Now it issues the first batch and only waits for a single part to succeed before uploading the next part. This can improve performance when one part lags due to network errors. Fixes #183.	2018-11-16 18:32:38 -08:00
Takeshi Nakatani	5fba542a29	Merge pull request #852 from gaul/aws-credentials-file ... Allow credentials from ${HOME}/.aws/credentials	2018-11-11 11:37:38 +09:00
Andrew Gaul	9e530c86ae	Allow credentials from ${HOME}/.aws/credentials ... This matches the configuration from popular tools like AWS CLI and allows multiple profile names via -o profile=name. The existing credential mechanisms continue to work. Fixes #822.	2018-11-04 17:47:07 -08:00
Andrew Gaul	664f910083	Correctly compare list_object_max_keys ... Previously this did not allow an argument. Fixes #843. References #793.	2018-11-04 10:49:35 -08:00
Takeshi Nakatani	6fe92d5ed6	Merge pull request #832 from gaul/hex ... Simplify hex conversion	2018-10-08 23:03:42 +09:00
Andrew Gaul	c703fa15c0	Simplify hex conversion ... Addresses GCC 8 warning: common_auth.cpp: In function ‘std::__cxx11::string s3fs_sha256sum(int, off_t, ssize_t)’: common_auth.cpp:84:12: warning: ‘char* strncat(char*, const char*, size_t)’ output may be truncated copying 2 bytes from a string of length 2 [-Wstringop-truncation] strncat(sha256, hexbuf, 2);	2018-10-01 19:08:27 -07:00
Jonatan Nilsson	d9c106cfde	Add support for storage class ONEZONE_IA.	2018-09-30 14:27:31 +02:00
Takeshi Nakatani	c5af62b023	Merge pull request #820 from gaul/big-writes ... Enable big writes if capable	2018-09-17 17:24:06 +09:00
Earl Chew	41c23adb0e	[curl] Assume long encryption keys are base64 encoded ... Amazon SSE-C https://tinyurl.com/ychug4cg writes: > Use this header to provide the 256-bit, base64-encoded encryption key > for Amazon S3 to use to encrypt or decrypt your data. It seems likely that future keys would be as long, or longer, so this change assumes that text longer than 32 characters are keys that are base64 encoded. Signed-off-by: Earl Chew <earl_chew@yahoo.com>	2018-09-16 21:23:22 +00:00
Andrew Gaul	a85183d42c	Enable big writes if capable ... Fixes #813.	2018-09-10 22:21:51 -07:00
Mattia Rizzolo	c376efdd28	Fix typo s/mutliple/multiple/ ... Signed-off-by: Mattia Rizzolo <mattia@mapreri.org>	2018-08-19 00:14:57 +02:00
Takeshi Nakatani	7e0c53dfe9	Added list_object_max_keys option based on #783 PR	2018-07-08 03:49:10 +00:00
root	35d3fce7a0	Review comment: Include the error code being returned	2018-07-06 05:14:32 -04:00
root	4177d8bd3b	Review comment: Include the error code being returned	2018-07-06 03:03:57 -04:00
root	ad5349a488	Changes as per review comments	2018-07-05 05:02:04 -04:00
root	92a4034c5e	Log messages for 5xx and 4xx HTTP response code	2018-07-04 03:50:45 -04:00
Takeshi Nakatani	3e4002df0d	Merge pull request #780 from wjt/initialize-libgcry ... gnutls_auth: initialize libgcrypt	2018-06-24 12:48:08 +09:00
Takeshi Nakatani	1b9ec7f4fc	Merge pull request #774 from nkkashyap/master ... Option for IAM authentication endpoint	2018-06-24 12:36:23 +09:00
Takeshi Nakatani	0d3fb0658a	Fixed a error by cppcheck on OSX	2018-06-24 02:38:59 +00:00
Will Thompson	73cf2ba95d	gnutls_auth: initialize libgcrypt ... Without this change, the following warning appears in the syslog/journal during startup: Libgcrypt warning: missing initialization - please fix the application From the [documentation][0]: > The function `gcry_check_version` initializes some subsystems used by > Libgcrypt and must be invoked before any other function in the > library. Fixes #524, which says: > gnutls is initialized by gnutls_global_init() function and > gcry_check_version() function for initializing libgcry is called from > this gnutls_global_init(). I checked the gnutls source and it hasn't contained a call to gcry_check_version() since the libgcrypt backend was removed in 2011 (commit 8116cdc8f131edd586dad3128ae35dd744cfc32f). In any case, the gcry_check_version() documentation continues: > It is important that these initialization steps are not done by a > library but by the actual application. so it would be incorrect for a library used by s3fs to initialize libgcrypt. [0]: https://www.gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html	2018-06-21 20:55:00 +01:00
Neeraj Kumar Kashyap	5a481e6a01	Option for IBM IAM auth endpoint added return	2018-06-04 16:44:14 +05:30
Neeraj Kumar Kashyap	d8e12839af	Option for IBM IAM auth endpoint	2018-05-31 16:02:48 +05:30
Or Ozeri	d4e86a17d1	Revert "enable FUSE read_sync by default" ... This reverts commit 86b0921ac4. Conflicts: src/s3fs.cpp	2018-05-28 13:49:54 +03:00
Takeshi Nakatani	ae9d8eb734	Fixed memory leak	2018-05-27 10:48:03 +00:00
Takeshi Nakatani	57b2a60172	Merge pull request #764 from orozery/remove_false_multihead_warnings ... Remove false multihead warnings	2018-05-23 22:38:35 +09:00
Takeshi Nakatani	212bbbbdf0	Merge pull request #763 from orozery/cleanup_share_after_handles ... cleanup curl handles before curl share	2018-05-23 22:30:36 +09:00
Takeshi Nakatani	da95afba8a	Merge pull request #756 from orozery/optimize_defaults ... Optimize defaults	2018-05-23 22:05:00 +09:00
Or Ozeri	0bd875eb9e	remove false readdir_multi_head warnings	2018-05-22 17:10:50 +03:00
Or Ozeri	af63a42773	cleanup curl handles before curl share	2018-05-21 13:20:09 +03:00
Or Ozeri	86b0921ac4	enable FUSE read_sync by default	2018-05-06 16:10:36 +03:00
Or Ozeri	4a72b60707	increase default stat cache size from 1000 to 100000	2018-05-06 15:31:07 +03:00
Takeshi Nakatani	e3de6ea458	Added reset curl handle when returning to handle pool	2018-05-06 12:11:53 +00:00
Neeraj Kumar Kashyap	25375a6b48	Validate the URL fixed inefficient usage of find	2018-05-04 11:24:32 +05:30
Neeraj Kumar Kashyap	ca87df7d44	Validate the URL format for http/https	2018-05-03 22:08:28 +05:30
Takeshi Nakatani	d052dc0b9d	Merge pull request #753 from cfz/master ... fix xpath selector in bucket listing	2018-05-02 12:04:12 +09:00
陈方舟	04493de767	fix xpath selector in bucket listing ... the original implementation in get_base_exp() depends on the order of xml return from the server. patriotically, when listing a directory with sub directory(s), the xml document response contains more than 2 <Prefix> nodes(some of them are in <CommonPrefixes> node). the source code arbitrarily select the first one in the documents (nodes->nodeTab[0]->xmlChildrenNode). some s3 compatible service return the list-bucket result in different result, leading the s3fs to a wrong behavior	2018-04-23 15:11:29 +08:00
Or Ozeri	4fdab46617	don't fail mkdir when directory exists	2018-04-08 11:13:47 +03:00
Takeshi Nakatani	1a23b880d5	Merge pull request #739 from orozery/cleanup_failing_curl_handles ... cleanup curl handle state on retries	2018-04-01 22:45:04 +09:00
Or Ozeri	adcf5754ae	cleanup failing curl handles on retries	2018-03-29 13:56:08 +03:00
Tristen Horton	0863672e27	add a more helpful error message for when there are multiple entries for the same bucket in the passwd file	2018-03-13 14:37:34 -07:00
Takeshi Nakatani	0f503ced25	Merge pull request #729 from dmgk/master ... FreeBSD build fixes	2018-03-04 16:36:31 +09:00
Takeshi Nakatani	987a166bf4	Merge pull request #726 from orozery/instance_name_logging ... add an instance_name option for logging	2018-03-04 15:41:12 +09:00
Takeshi Nakatani	57b6f0eeaf	Merge pull request #724 from orozery/dont_fail_multirequest ... don't fail multirequest on single thread error	2018-03-04 15:35:29 +09:00
Takeshi Nakatani	f71a28f9b9	Merge pull request #714 from orozery/reduce_lock_contention ... reduce lock contention on file open	2018-03-04 13:36:08 +09:00
Dmitri Goutnik	c9f4312588	FreeBSD build fixes	2018-03-02 15:58:52 -05:00
Or Ozeri	8b657eee41	add disk space reservation	2018-02-28 19:20:23 +02:00
Takeshi Nakatani	be9d407fa0	Fixed cppcheck error on osx	2018-02-28 12:29:58 +00:00
Takeshi Nakatani	c494e54320	Fixed cppcheck error on osx	2018-02-28 12:06:06 +00:00
Or Ozeri	b52b6f3fc5	add an instance_name option for logging	2018-02-28 09:51:35 +02:00
Or Ozeri	82c9733101	don't fail multirequest on single thread error	2018-02-26 12:06:08 +02:00
Takeshi Nakatani	a45ff6cdaa	Fixed cppcheck error and clean ^M code	2018-02-25 13:08:41 +00:00
Takeshi Nakatani	960d45c853	Fixed cppcheck error on osx	2018-02-25 08:51:19 +00:00
Or Ozeri	0edf056e95	reduce lock contention on file open	2018-02-04 17:13:58 +02:00
Takeshi Nakatani	6b58220009	Merge pull request #697 from pwulff/master ... Fixing race condition in FdEntity::GetStats	2017-12-17 15:46:48 +09:00
Paul Wulff	ee6abea956	Race condition in FdManager::Rename because no mutex is used.	2017-12-15 15:27:51 +01:00
Paul Wulff	cea7d44717	Fixing race condition in FdEntity::GetStats	2017-12-13 10:49:00 +01:00
Or Ozeri	0da87e75fe	fix condition for parallel download	2017-12-04 16:07:33 +02:00
Or Ozeri	bd4bc0e7f1	add support for IBM IAM authentication	2017-11-23 12:01:52 +02:00
Takeshi Nakatani	026260e7a1	Improved use of temporary files	2017-11-23 09:18:11 +00:00