375 Commits

Author	SHA1	Message	Date
Alvaro Huarte	a24f78f5a4	Adds requester_pays option to enable requests involving Requester Pays buckets	2019-11-18 12:38:16 +01:00
Bryce Larson	0c42a74a8a	handle edge case when bucket name and first part of endpoint match	2019-10-26 00:51:19 -06:00
Takeshi Nakatani	1db94a0b30	Fixed to use copy api in multipart upload	2019-09-25 19:30:58 -07:00
Takeshi Nakatani	bedd648d47	Fixed build error by cppcheck 1.89	2019-09-23 10:49:49 +00:00
Takeshi Nakatani	58b3cce320	Merge pull request #1066 from gaul/sign-requests ... Sign request immediately before sending	2019-09-23 15:58:21 +09:00
Takeshi Nakatani	81102a5963	Merge pull request #1149 from ggtakec/fix_retry ... Fixed a bug that type and op are not inherited	2019-09-08 20:32:43 +09:00
Takeshi Nakatani	42fb30852b	Merge pull request #1145 from gaul/bool ... Fix a few bool types	2019-09-08 20:04:30 +09:00
Takeshi Nakatani	e2129001eb	Fixed a bug that type and op are not inherited	2019-09-08 09:59:04 +00:00
Andrew Gaul	805cc064af	Fix a few bool types ... Found via clang-tidy.	2019-09-05 10:42:42 -07:00
Andrew Gaul	412876ca33	Expose base parameter in s3fs_strtoofft ... This fixes a regression from ccf3e7bfa2 which caused the misparsing of octal inputs for the mp_umask and umask flags. It also allows some callers to be more precise about their decimal inputs.	2019-09-04 20:41:47 -07:00
Andrew Gaul	461a346bf4	Sign request immediately before sending ... Previously s3fs could create a long list of pre-signed requests which could take longer than the default S3 clock skew limit of 15 minutes. This also changes SHA-256 computation from single- to multi-threaded since this is now computed in the worker threads. Regression from 88cd8feb05. Fixes #1019.	2019-08-29 15:35:27 -07:00
liuyongqing	ae4bcd405c	use correct content-type when overwriting and copying files	2019-08-28 22:25:09 -07:00
Takeshi Nakatani	0536dc1112	Refixed strict processing of MultiRead method	2019-08-25 20:46:01 -07:00
Andrew Gaul	cc2eed84a5	Promote default ACL to enum ... This sanity checks ACLs during initialization and also omits sending the header when set to PRIVATE which is the default.	2019-08-20 15:05:58 -07:00
Takeshi Nakatani	873e376098	Strict processing of MultiRead method	2019-08-13 15:23:13 +00:00
Takeshi Nakatani	4a0c23258e	Fixed MultiRead method to exit after waiting other parts at error	2019-08-12 14:45:06 -07:00
Takeshi Nakatani	5c4a0a862a	Fixed multipart copy and its retry handler	2019-08-11 16:21:26 +00:00
Takeshi Nakatani	f74c7407db	Merge pull request #1118 from gaul/atoi ... Prefer s3fs_strtoofft over atoi and strtol	2019-08-06 21:23:37 +09:00
Takeshi Nakatani	433c04af26	Merge pull request #1116 from gaul/bodydata ... Do not heap allocate BodyData	2019-08-06 21:08:02 +09:00
Andrew Gaul	ccf3e7bfa2	Prefer s3fs_strtoofft over atoi and strtol ... The former propagates errors consistently.	2019-08-03 16:13:48 -07:00
Andrew Gaul	7ecfba811e	Do not heap allocate BodyData ... Callers already manage lifetimes via Clear().	2019-08-03 15:50:46 -07:00
Andrew Gaul	a5c20175a1	Use length from s3fs_HMAC256 for consistency ... EVP_MAX_MD_SIZE can range up to 64. Found by Coverity.	2019-08-02 10:40:55 -07:00
Andrew Gaul	bacd15714a	Use consistent default for readwrite_timeout ... Previously curl threads mutated this shared state without a lock. Found via ThreadSanitizer.	2019-07-16 22:03:56 -07:00
Takeshi Nakatani	543aed2a32	Merge pull request #1096 from gaul/pthread/lock-checking ... Expand error checking to all pthread_mutex_t	2019-07-15 11:43:56 +09:00
Andrew Gaul	43ec064fb9	Expand error checking to all pthread_mutex_t	2019-07-14 18:21:23 -07:00
Andrew Gaul	ffac4c8417	Avoid narrowing time_t and off_t args in logging ... This displays correct results on 32-bit platforms.	2019-07-14 17:02:36 -07:00
Takeshi Nakatani	e936854493	Merge pull request #1094 from larsch/master ... Fix multipart upload not setting curl options	2019-07-15 05:13:37 +09:00
Lars Christensen	58037da061	Fix multipart upload not setting curl options	2019-07-14 14:09:37 +02:00
Andrew Gaul	6fd42d9fe4	Prefer modern C headers ... Found and fixed via clang-tidy.	2019-07-12 03:50:59 -07:00
Takeshi Nakatani	e5e63d6ac3	Merge pull request #1067 from gaul/mismatched-free ... Fix mismatched new[]/free	2019-07-07 16:55:07 +09:00
Takeshi Nakatani	7a65a414c3	Merge pull request #1080 from gaul/xml/simplification ... Add simple XML parsing wrapper	2019-07-07 16:44:18 +09:00
Andrew Gaul	944d21cabb	Fix mismatched new[]/free ... Found via Valgrind.	2019-07-07 00:36:48 -07:00
Andrew Gaul	b8ff6a647e	Add simple XML parsing wrapper ... Also simplify check_region_error.	2019-07-05 17:39:09 -07:00
Andrew Gaul	bbfa91141a	Configure clang-tidy target ... Also fix nits.	2019-07-03 14:04:11 -07:00
Takeshi Nakatani	fbebc6fa57	Merge pull request #1048 from gaul/log/curl ... Emit older curl warnings only once	2019-06-24 00:42:44 +09:00
Andrew Gaul	738eaadcbf	Emit older curl warnings only once ... This makes Travis logs actually readable.	2019-06-22 10:01:20 -07:00
Andrew Gaul	15b7450713	Convert file offsets from size_t to off_t ... The latter is 64-bits on 32-bit platforms when specifying -D_FILE_OFFSET_BITS=64. This allows early Raspberry Pis to use files larger than 2 GB. It also cleans up some ugly casting. Fixes #620. Fixes #656.	2019-06-15 17:05:37 -07:00
Michal Lula	0d4847596e	fix issue with aws session token provided inside ~/.aws/credentials file	2019-04-25 14:29:35 +02:00
Andrew Gaul	0677a096a9	Prefer simple over compound statements in macros ... This prohibits missing semicolons.	2019-04-17 21:46:24 +09:00
Michal Lula	f9cd43b684	add session token support	2019-04-16 16:53:05 +02:00
Andrew Gaul	6f8ecb28c5	Correctly calculate MD5 hash for SSE-C keys ... Previously s3fs calculated the strlen of a binary input instead of using the stored length. This yielded IO errors when interacting with SSE-encrypted objects. Fixes #696.	2019-04-08 21:49:42 +09:00
Andrew Gaul	6f6a67807b	Prefer new over malloc ... The former cannot return NULL and will allow use of scoped pointers.	2019-04-07 16:54:24 +09:00
Takeshi Nakatani	71766039ff	Support undefined CURLoption in libcurl library used in build	2019-03-22 10:47:42 +00:00
Andrew Gaul	1e86cc643d	Fix lazy typo	2019-03-09 17:46:02 +09:00
Takeshi Nakatani	10d9f75366	Improvement of curl session pool for multipart ... Improvement of curl session pool for multipart(2) Improvement of curl session pool for multipart(3)	2019-02-25 14:46:24 +00:00
Takeshi Nakatani	eeb839242b	Fixed code for latest cppcheck error on OSX	2019-02-17 13:59:11 +00:00
Takeshi Nakatani	e07cb020cc	Merge pull request #954 from gaul/clear-iter ... Clear containers instead of individual erases	2019-02-06 21:12:36 +09:00
Andrew Gaul	a1d3ff9766	Automatically abort failed multipart requests ... This can avoid dangling parts. However, many transfers fail due to network errors so we still need other mechanisms to handle these parts.	2019-02-03 10:29:20 -08:00
Andrew Gaul	4d0bef1e90	Clear containers instead of individual erases ... This has O(n) runtime instead of O(n log n).	2019-02-02 23:58:43 -08:00
Takeshi Nakatani	c04e8e7a9d	Merge pull request #949 from gaul/503-slow-down ... Implement exponential backoff for 503	2019-02-03 15:19:28 +09:00
Takeshi Nakatani	d8185a25aa	Merge pull request #946 from gaul/async/completed-tids ... Simplify async request completion code	2019-02-03 14:08:49 +09:00
Takeshi Nakatani	53337a0a28	Merge pull request #944 from gaul/utility-mode ... Repair utility mode	2019-02-03 13:59:56 +09:00
Takeshi Nakatani	055ecf6ea7	Merge pull request #940 from gaul/parallel-multipart-copy ... Copy parts in parallel	2019-02-03 11:44:15 +09:00
Takeshi Nakatani	c603680e02	Merge pull request #939 from gaul/stringstream-specific ... Prefer specific [io]stringstream where possible	2019-02-03 11:06:57 +09:00
Andrew Gaul	8ff05d8e38	Implement exponential backoff for 503 ... Amazon returns SlowDown when overloaded. Also return ENOTSUP for 501 and immediately return EIO for 500 instead of retrying. Fixes #603.	2019-02-02 00:03:50 -08:00
Andrew Gaul	8c527c3616	Simplify async request completion code ... Workers now notify the master thread when they complete, unifying the Linux and macOS code paths. This also avoids excessive pthread_tryjoin_np calls. Follows on to 88cd8feb05.	2019-01-31 22:55:23 -08:00
Andrew Gaul	54a074647e	Repair utility mode ... This deinitialized S3fsCurl twice and incorrectly calculated V4 signatures.	2019-01-31 18:45:39 -08:00
Andrew Gaul	c5ebf5d328	Copy parts in parallel ... S3 can copy multipart much faster than single part due to IO parallelization. Renaming a 4 GB file reduces from 72 to 20 seconds with bigger gains with larger files.	2019-01-31 10:21:39 -08:00
Andrew Gaul	07636c8a8d	Prefer specific [io]stringstream where possible ... These better communicate intent and are slightly more efficient.	2019-01-29 10:44:33 -08:00
Andrew Gaul	35d55ee513	Remove unneeded void parameter ... This is implicit in C++. Found and fixed via clang-tidy.	2019-01-28 23:22:27 -08:00
Takeshi Nakatani	a0c1f30ae7	Merge pull request #932 from gaul/autolock ... Prefer AutoLock for synchronization	2019-01-27 15:59:18 +09:00
Andrew Gaul	92d3114584	Prefer AutoLock for synchronization ... This simplifies the code and fixes an issue with unlocked access. Also use a recursive lock for StatCache to avoid races between lock..unlock..lock sequences.	2019-01-25 15:28:41 -08:00
Andrew Gaul	84b421d6ef	Prefer empty over size checks ... Found and fixed via clang-tidy.	2019-01-23 11:30:28 -08:00
Takeshi Nakatani	010276ceab	Merge pull request #921 from gaul/clang-tidy/redundant-string-init ... Remove redundant string initializations	2019-01-23 19:44:59 +09:00
Andrew Gaul	1fc25e8c3f	Remove redundant string initializations ... Found and fixed via clang-tidy.	2019-01-22 23:16:37 -08:00
Andrew Gaul	61ecafd426	Remove unnecessary string copies ... Found via clang-tidy.	2019-01-22 23:09:37 -08:00
Takeshi Nakatani	fada95f58e	Merge pull request #914 from gaul/readdir/head-of-line ... Issue readdir HEAD requests without batching	2019-01-21 22:10:50 +09:00
Takeshi Nakatani	014b8c5982	Merge pull request #913 from gaul/assert ... Prefer abort over assert(false)	2019-01-21 21:56:41 +09:00
Andrew Gaul	46d79c5bc2	Issue readdir HEAD requests without batching ... Previously s3fs would issue a batch of HEAD requests and wait for all to succeed before issuing the next batch. Now it issues the first batch and only waits for a single call to succeed before issuing the next call. This can improve performance when one call lags due to network errors. I measured 25% improvement with the same level of parallelism. This commit also reparents parallelism knobs for consistency. Follows on to 88cd8feb05. Fixes #223.	2019-01-20 18:07:22 -08:00
Andrew Gaul	40ba3b44a1	Prefer abort over assert(false) ... The compiler can remove the latter when compiled with NDEBUG which may cause unintended control flow.	2019-01-20 12:30:27 -08:00
Takeshi Nakatani	ac72431195	Added detail error message when HTTP 301/307 status	2019-01-20 10:07:58 +00:00
Takeshi Nakatani	36a4903843	Added an error message when HTTP 301 status	2019-01-20 07:17:40 +00:00
Takeshi Nakatani	05014c49c8	Merge pull request #906 from gaul/doc/https ... Prefer HTTPS links where possible	2019-01-20 15:10:51 +09:00
Takeshi Nakatani	d373b0eca3	Merge pull request #904 from gaul/clang-tidy/c-str ... Remove unnecessary calls to std::string::c_str	2019-01-20 14:13:46 +09:00
Takeshi Nakatani	61dc7f0a70	Merge pull request #900 from gaul/leak ... Plug memory leak	2019-01-20 12:30:31 +09:00
Andrew Gaul	aa9bd1fa3c	Prefer HTTPS links where possible ... Fix a few stale links as well.	2019-01-18 11:09:08 -08:00
Andrew Gaul	508fafbe62	Remove unnecessary calls to std::string::c_str ... Found via clang-tidy.	2019-01-17 22:05:16 -08:00
Andrew Gaul	58ce544e83	Plug memory leak ... Previously this appended to an empty list, zeroed the list, then appended to the list. Instead zero the list first and then append. This also enables sending Content-MD5 which can eagerly detect transmission errors. Found via Valgrind.	2019-01-17 16:24:34 -08:00
Andrew Gaul	e98ce36301	Tighten up HTTP response code check ... Previously s3fs considered 3xx codes as successful. When writing an object to a misconfigured region, s3fs did not propagate the 301 as a write error to the client. References #693.	2019-01-17 15:04:59 -08:00
Takeshi Nakatani	4ed0e5f35a	Merge pull request #882 from earlchew/issue-817 ... [curl] Assume long encryption keys are base64 encoded	2019-01-06 17:29:04 +09:00
Earl Chew	ed5795eead	[curl] Assume long encryption keys are base64 encoded ... Correct tabs and whitespace. Signed-off-by: Earl Chew <earl_chew@yahoo.com>	2019-01-05 21:08:41 +00:00
Andrew Gaul	a2f8ac535e	Address cppcheck 1.86 errors ... Lifetime, shadowing, and unused variables. Found via the Travis macOS builder.	2018-12-20 14:56:31 -08:00
Or Ozeri	b91fc5409e	fix multihead warning check	2018-11-28 09:54:02 +02:00
Andrew Gaul	88cd8feb05	Upload S3 parts without batching ... Previously s3fs would issue a batch of uploads and wait for all to succeed before issuing the next batch. Now it issues the first batch and only waits for a single part to succeed before uploading the next part. This can improve performance when one part lags due to network errors. Fixes #183.	2018-11-16 18:32:38 -08:00
Jonatan Nilsson	d9c106cfde	Add support for storage class ONEZONE_IA.	2018-09-30 14:27:31 +02:00
Earl Chew	41c23adb0e	[curl] Assume long encryption keys are base64 encoded ... Amazon SSE-C https://tinyurl.com/ychug4cg writes: > Use this header to provide the 256-bit, base64-encoded encryption key > for Amazon S3 to use to encrypt or decrypt your data. It seems likely that future keys would be as long, or longer, so this change assumes that text longer than 32 characters are keys that are base64 encoded. Signed-off-by: Earl Chew <earl_chew@yahoo.com>	2018-09-16 21:23:22 +00:00
Takeshi Nakatani	7e0c53dfe9	Added list_object_max_keys option based on #783 PR	2018-07-08 03:49:10 +00:00
root	35d3fce7a0	Review comment: Include the error code being returned	2018-07-06 05:14:32 -04:00
root	4177d8bd3b	Review comment: Include the error code being returned	2018-07-06 03:03:57 -04:00
root	ad5349a488	Changes as per review comments	2018-07-05 05:02:04 -04:00
root	92a4034c5e	Log messages for 5xx and 4xx HTTP response code	2018-07-04 03:50:45 -04:00
Takeshi Nakatani	0d3fb0658a	Fixed a error by cppcheck on OSX	2018-06-24 02:38:59 +00:00
Takeshi Nakatani	ae9d8eb734	Fixed memory leak	2018-05-27 10:48:03 +00:00
Takeshi Nakatani	57b2a60172	Merge pull request #764 from orozery/remove_false_multihead_warnings ... Remove false multihead warnings	2018-05-23 22:38:35 +09:00
Or Ozeri	0bd875eb9e	remove false readdir_multi_head warnings	2018-05-22 17:10:50 +03:00
Or Ozeri	af63a42773	cleanup curl handles before curl share	2018-05-21 13:20:09 +03:00
Takeshi Nakatani	e3de6ea458	Added reset curl handle when returning to handle pool	2018-05-06 12:11:53 +00:00
Or Ozeri	adcf5754ae	cleanup failing curl handles on retries	2018-03-29 13:56:08 +03:00
Takeshi Nakatani	987a166bf4	Merge pull request #726 from orozery/instance_name_logging ... add an instance_name option for logging	2018-03-04 15:41:12 +09:00
Takeshi Nakatani	57b6f0eeaf	Merge pull request #724 from orozery/dont_fail_multirequest ... don't fail multirequest on single thread error	2018-03-04 15:35:29 +09:00
Takeshi Nakatani	be9d407fa0	Fixed cppcheck error on osx	2018-02-28 12:29:58 +00:00
Takeshi Nakatani	c494e54320	Fixed cppcheck error on osx	2018-02-28 12:06:06 +00:00
Or Ozeri	b52b6f3fc5	add an instance_name option for logging	2018-02-28 09:51:35 +02:00
Or Ozeri	82c9733101	don't fail multirequest on single thread error	2018-02-26 12:06:08 +02:00
Takeshi Nakatani	a45ff6cdaa	Fixed cppcheck error and clean ^M code	2018-02-25 13:08:41 +00:00
Takeshi Nakatani	960d45c853	Fixed cppcheck error on osx	2018-02-25 08:51:19 +00:00
Or Ozeri	bd4bc0e7f1	add support for IBM IAM authentication	2017-11-23 12:01:52 +02:00
Takeshi Nakatani	99fe93b7f1	Merge pull request #684 from gaul/signedness ... Correct signedness warning	2017-11-23 17:33:46 +09:00
Or Ozeri	11bd7128d2	remove use of jsoncpp	2017-11-22 13:36:27 +02:00
Andrew Gaul	7cda32664b	Correct signedness warning ... Fixes regression from 0418e53b3c.	2017-11-19 11:14:37 -08:00
Takeshi Nakatani	e1928288fe	Merge pull request #676 from gaul/sstream ... Move str definition from header to implementation	2017-11-19 16:34:15 +09:00
Andrew Gaul	0418e53b3c	Reduce use of preprocessor ... This provides type-safety and avoids token expansion side effects.	2017-11-18 22:40:06 -08:00
Takeshi Nakatani	bad48ab59a	Merge pull request #671 from psyvision/master ... Add support for ECS metadata endpoint	2017-11-19 14:43:06 +09:00
Andrew Gaul	bbad76bb71	Move str definition from header to implementation ... Also clean up some char * to std::string conversions.	2017-11-18 11:34:34 -08:00
Richard Caunt	b95e4acaeb	Remove debug logging statements	2017-11-08 15:52:35 +00:00
Richard Caunt	c238701d09	Corrected ECS headers	2017-11-08 15:21:49 +00:00
Richard Caunt	60d2ac3c7a	Adding x-amz-security-token header	2017-11-08 15:09:59 +00:00
Richard Caunt	967ef4d56b	Corrected fat finger mistakes	2017-11-08 13:14:49 +00:00
Richard Caunt	ad57bdda6c	Corrected keycount check	2017-11-08 13:06:22 +00:00
Richard Caunt	a0b69d1d3d	Corrected keyval[].c_str()	2017-11-08 13:01:52 +00:00
Richard Caunt	5df94d7e33	Add debug messages	2017-11-08 09:50:39 +00:00
Richard Caunt	1cbe9fb7a3	Gotta pass that cppcheckgit add .	2017-11-07 21:41:51 +00:00
Richard Caunt	8660abaea2	Use jsoncpp to parse AWS JSON	2017-11-07 21:20:02 +00:00
Richard Caunt	366f0705a0	ECS credentials bug fixes	2017-11-06 21:45:58 +00:00
Richard Caunt	5d54883e2f	Remove commented out code	2017-11-05 19:25:34 +00:00
Richard Caunt	662f65c3c8	Add support for ECS metadata endpoint	2017-11-05 19:24:02 +00:00
Takeshi Nakatani	5db550a298	Fixed a bug in S3fsCurl::LocateBundle	2017-11-05 11:26:05 +00:00
Or Ozeri	384b4cbafa	auth headers insertion refactoring	2017-10-30 11:52:58 +02:00
Takeshi Nakatani	5957d9ead0	Fixed with unnecessary equal in POST uploads url argment - #643	2017-09-17 10:52:28 +00:00
Takeshi Nakatani	00bc9142c4	Fixed potential atomic violation in S3fsCurl::AddUserAgent - #633	2017-09-17 09:16:05 +00:00
Takeshi Nakatani	a22675bafd	Merge pull request #567 from andrewgaul/default-acl ... Do not send ACL unless overridden	2017-05-09 23:03:27 +09:00
Andrew Gaul	7b30d5d15b	Do not send canned ACL header when empty string ... Some providers such as StorageGRID do not support canned ACLs. Setting to empty allows callers to omit the header. References #125.	2017-05-07 10:52:31 -07:00
Takeshi Nakatani	9d10a5aa70	Changed copyright year format for debian pkg	2017-05-07 11:24:17 +00:00
Takeshi Nakatani	a12e0d5ec4	Fixed failure to upload/copy with SSE_C and SSE_KMS	2017-05-07 09:29:08 +00:00
Takeshi Nakatani	d07c3f38b7	Check errors returned in 200 OK responses for put header request	2017-05-06 02:15:53 +00:00
Takeshi Nakatani	df0ff3a2fd	Merge pull request #556 from orozery/fix_nocache_multipart_upload ... fix multipart upload handling without cache	2017-04-16 19:22:15 +09:00
Takeshi Nakatani	edcf4c6218	Merge pull request #555 from orozery/dont_sign_empty_headers ... don't sign empty headers (as they are discarded by libcurl)	2017-04-16 19:16:47 +09:00
Takeshi Nakatani	efba9bcbc1	Merge pull request #553 from orozery/custom_cipher_suite ... add TLS cipher suites customization	2017-04-16 19:09:27 +09:00
Takeshi Nakatani	6bd179c92b	Merge pull request #552 from orozery/foreground_threads ... switch S3fsMultiCurl to use foreground threads	2017-04-16 19:05:16 +09:00
Or Ozeri	96764b7410	switch S3fsMultiCurl to use foreground threads	2017-04-09 16:56:49 +03:00
Takeshi Nakatani	b4c90d6957	Fixed a bug about multipart uploading at no disk free space related to #509	2017-04-09 04:37:20 +00:00
Or Ozeri	75b59a7c16	switch S3fsMultiCurl to use foreground threads	2017-04-04 15:32:53 +03:00
Or Ozeri	3bcca75a88	don't sign empty headers (as they are discarded by libcurl)	2017-04-04 15:24:20 +03:00
Or Ozeri	8ee95ff7ab	fix multipart upload handling without cache	2017-04-02 10:27:43 +03:00
Andrew Gaul	03217baa99	Address cppcheck 1.77 warnings	2017-03-06 12:41:08 -08:00
Andrew Gaul	915a1321c7	Use server-provided ETag during complete upload ... This avoids calculating the MD5 locally and enables use with object stores which do not use MD5 as ETag.	2016-11-23 18:48:57 -08:00
Andrew Gaul	d375bca0d0	Correct typos	2016-11-19 15:57:41 -08:00
Takeshi Nakatani	edd0a11fb5	Merge pull request #494 from mapreri/typo ... Fix typo s/destroied/destroyed/	2016-11-20 05:53:23 +09:00
Takeshi Nakatani	67a836223a	Merge pull request #495 from driskell/fix_sse_copy ... Fix invalid V4 signature on multipart copy requests	2016-11-20 05:43:10 +09:00
Jason Woods	6f688770fd	Fix invalid V4 signature on multipart copy requests	2016-11-13 13:22:00 +00:00