1221 Commits

Author	SHA1	Message	Date
Andrew Gaul	771bbfeac5	Do not create zero-byte object when creating file (#1640) ... Previously s3fs created this object to store metadata and overwrote it when flushing. This prevented use with object stores which do not allow overwrites like HDS. Instead only create an in-memory representation which reduces the time to create small files. Fixes #1013.	2021-04-30 19:56:33 +09:00
Andrew Gaul	3694786112	Propagate errno instead of EIO (#1638) ... This improves error fidelity. Follows on to b70f8db037. References #1523.	2021-04-30 07:09:00 +09:00
Andrew Gaul	fbf3c83019	Consider S3 errors in HTTP PUT 200 responses (#1635) ... S3 can emit these in unusual situations. Fixes #1317.	2021-04-25 13:18:11 +09:00
Andrew Gaul	8a51a26819	Allow truncation of open and modified files ... Regression introduced in f5bf41cf11. Fixes #1575.	2021-04-25 12:35:35 +09:00
Takeshi Nakatani	1838f52e19	Declare undefined symbols for fallocate function	2021-04-25 10:12:25 +09:00
Andrew Gaul	e9eb248f2f	Fail CheckBucket when S3 returns PermanentRedirect (#1630) ... Previously s3fs allowed mounting but all operations returned EIO. References #693. Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>	2021-04-24 18:46:24 +09:00
Ambika Nair	77581eda59	Change in ibm iam endpoint	2021-04-21 21:42:55 +09:00
Andrew Gaul	7f3e423bbe	Preserve sub-second time precision (#1624) ... Found via pjdfstests. References #897. References #1589.	2021-04-18 13:11:12 +09:00
Andrew Gaul	706e3cbebd	Update ctime and mtime when opening with O_TRUNC (#1623) ... Found via pjdfstests. References #1589.	2021-04-15 07:08:49 +09:00
Takeshi Nakatani	2effffd8e2	Fixed codes for cppcheck 2.4.1	2021-04-13 21:12:35 +09:00
Takeshi Nakatani	2908878988	Fixed codes for cppcheck 2.4.1	2021-04-13 21:12:35 +09:00
Andrew Gaul	01a26a9011	Do not allow open with O_RDONLY and O_TRUNC (#1620) ... Found via pjdfstests. References #1589.	2021-04-13 07:28:10 +09:00
Andrew Gaul	23e1fbf7b9	Update ctime when truncating file (#1619) ... Found via pjdfstests. References #1589.	2021-04-13 07:08:06 +09:00
Andrew Gaul	ed2e877bb6	Return EACESS when lacking required executable bit (#1617) ... This makes the check consistent with read and write. Found via pjdfstests. References #1551. References #1589.	2021-04-11 22:39:27 +09:00
Andrew Gaul	3663082a01	Do not allow renaming to a non-empty directory (#1604) ... Found via pjdfstests. References #1589.	2021-03-28 19:10:52 +09:00
Andrew Gaul	9645d57c05	Translate KeyTooLongError to ENAMETOOLONG (#1602) ... AWS does not support keys longer than 1024 characters. Add special handling for HeadObject which does not have a response body. Found via pjdfstests. References #1589.	2021-03-28 13:17:41 +09:00
Andrew Gaul	f6fbd75320	Return EFBIG when file exceeds multipart limit (#1600) ... Found via pjdfstests. References #1589.	2021-03-28 12:33:01 +09:00
Carsten Grohmann	ef079f4e94	Don't ignore nomultipart when storage is low ... When the temporary storage filled up, the old implementation uploaded all data with multipart uploads, even if "nomultipart" was set. The new implementation emits a warning and returns -ENOSPC instead. Fixes #1595	2021-03-10 08:23:54 +09:00
Takeshi Nakatani	b589ebec23	Output version and command parameters at startup	2021-03-06 19:11:57 +09:00
Andrew Gaul	032fcf2a47	Allow optional issuing of ListObjectsV2 (#1583) ... This allows use of s3fs on object stores that do not implement the V1 API. Fixes #1573.	2021-02-23 09:45:13 +09:00
Takeshi Nakatani	5b5bc3114a	Fixed a bug that padded null bytes when changing xattr	2021-02-21 12:23:36 +09:00
Takeshi Nakatani	81e267d421	Fixed the same bug in chmod/chown/utimens and added tests	2021-02-20 23:30:50 +09:00
Takeshi Nakatani	4fc92d59f3	Fixed a bug that filling NULL bytes when changing the attribute after renaming	2021-02-20 23:30:50 +09:00
Takeshi Nakatani	a0f347b10f	Added no_time_stamp_ms option mainly for testing	2021-02-20 20:24:26 +09:00
Takeshi Nakatani	4da56acdcc	Improved processing when HEAD response is 400	2021-02-13 17:00:13 +09:00
Takeshi Nakatani	493802a605	Fixed osx mistype declaration and typo warnings	2021-02-13 12:04:35 +09:00
林千里	22b0ae9d51	set IsExpireTime to true by default ... fixes #1563	2021-02-12 12:20:47 +09:00
Andrew Gaul	134a54b32f	Allow configuring the multipart threshold (#1562) ... Also change default which improves write performance for files >= 25 MB and <= 5 GB, particularly over lossy networks. Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>	2021-02-11 23:35:46 +09:00
Andrew Gaul	7f6fbb0021	Correct singlepart_copy_limit unit (#1561) ... This is documented to be in MB not KB. Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>	2021-02-11 22:53:50 +09:00
Andrew Gaul	ec8bae9827	Do not periodically flush with nomultipartupload (#1559) ... s3fs periodically flushes data to S3 and hole-punches the written data, marking it as clean. Without multipart copy, s3fs would write the now-zeroed data along with any new data. s3fs should not attempt to periodically flush dirty data at all without multipart copy support. Fixes #1542.	2021-02-11 22:24:39 +09:00
Andrew Gaul	a4d916af13	Allow configuring multipart copy size (#1555) ... Also align configuration with mixupload.	2021-02-08 20:32:12 +09:00
Takeshi Nakatani	32f096fa3f	Use clock_gettime instead of gettimeofday	2021-02-08 10:19:28 +09:00
Andrew Gaul	c692093921	Copy at most 5 GB per multipart copy part request (#1553) ... Previously mixupload failed with larger objects due to the AWS constraint on part sizes. This symptom was seen more frequently due to periodic flushing of dirty data. Fixes #1547.	2021-02-07 23:10:07 +09:00
Andrew Gaul	1a6d0826b5	Allow -1 value to disable max_dirty_data (#1552) ... This matches the man page.	2021-02-07 22:14:31 +09:00
Andrew Gaul	bf33fe7f55	Increase buffer size to avoid warning (#1550) ... This can be up to 54 bytes. Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>	2021-02-07 12:26:58 +09:00
Andrew Gaul	ff9d6a75c7	Parse more specific AWS error codes (#1549) ... Fixes #1545.	2021-02-07 11:29:08 +09:00
Carsten Grohmann	4c6690f5f0	Add timestamps to debug log and cache integrity check (#1540) ... * Add UTC timestamp to debug output for easier correlation with external events Old output: $ s3fs mybucket /bucket/ -o curldbg,dbglevel=debug -d -d -f [CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from... [INF] s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=0, ... New output: $ s3fs mybucket /bucket/ -o curldbg,dbglevel=debug -d -d -f 2021-01-28T21:09:16.264Z [CRT] s3fs_logger.cpp:LowSetLogLevel(202):... 2021-01-28T21:09:16.264Z [INF] s3fs.cpp:set_mountpoint_attrib... * Add UTC timestamp to cache integrity check for easier correlation with external events. $ s3fs mybucket /mybucket -oset_check_cache_sigusr1=/tmp/check.cache Old output: $ kill -s SIGUSR1 $(pgrep s3fs) $ cat /tmp/check.cache ------------------------------------------------------------ Check cache file and its stats file consistency ------------------------------------------------------------ ------------------------------------------------------------ Summary - Total files: 0 Detected error files: 0 Detected error directories: 0 ------------------------------------------------------------ New output: $ kill -s SIGUSR1 $(pgrep s3fs) $ cat /tmp/check.cache --------------------------------------------------------------------------- Check cache file and its stats file consistency at 2021-01-30T13:04:14.111Z --------------------------------------------------------------------------- --------------------------------------------------------------------------- Summary - Total files: 0 Detected error files: 0 Detected error directories: 0 --------------------------------------------------------------------------- * Fix indentation of S3fsLog::GetCurrentTime()	2021-02-04 10:41:29 +09:00
Andrew Gaul	67b9381825	Remove limit for single-part objects ... AWS enforces a 5 GB limit for single-part objects but other S3 implementations like GCS support larger objects. Fixes #1542.	2021-02-04 09:19:37 +09:00
Andrew Gaul	4b53d4bf6b	Suppress spurious SHA256 error message (#1544)	2021-02-04 07:28:51 +09:00
Carsten Grohmann	ed85b72bf5	Add warning for failing metadata updates of large files ... Metadata updates fail for large files if "-o nocopyapi" or "-o nomultipart" is set. This change is related to #1528	2021-01-28 10:20:32 +09:00
Andrew Gaul	17fda89ae9	Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors (#1534) ... Follows on to 4d833a4fb9.	2021-01-25 19:08:14 +09:00
Andrew Gaul	d019dda4f7	Simplify substr manipulations with erase (#1532) ... This avoids creating a new std::string.	2021-01-25 18:02:32 +09:00
Andrew Gaul	dc9255bc5f	Prefer static_cast where possible (#1531)	2021-01-25 08:15:17 +09:00
Andrew Gaul	b0e8758b63	Use result instead of res for consistency (#1530)	2021-01-25 07:56:10 +09:00
Andrew Gaul	b70f8db037	Propagate errno from get_local_fent (#1525) ... Follows on to 4d833a4fb9. References #1523.	2021-01-20 23:21:14 +09:00
Andrew Gaul	4d833a4fb9	Return more specific errno when available (#1520) ... Previously s3fs threw away some function return values and returned EIO instead. This was due to not trusting the mix of -1 and errno return codes. Correct the obviously incorrect ones via visual inspection. Stronger typing may find more occurrences. Fixes #1519.	2021-01-18 18:50:49 +09:00
fly3366	168e588ac7	fix: Add reset offset (#1503)	2021-01-04 22:57:56 +09:00
Andrew Gaul	bd0fadbe5f	Remove authorization header when remaking handle (#1505) ... This avoids including Authorization in SignedHeaders. s3fs will recreate the Authorization header before sending the request.	2021-01-04 21:37:34 +09:00
kontrollanten	d1c638ab7a	fix 404 error message	2020-12-30 14:10:17 +09:00
fly3366	51f65d7b14	fix: miss header when retry	2020-12-23 22:16:07 +09:00
Andrew Gaul	a16d00d673	Simply curl_slist_sort_insert (#1494) ... Some good taste from Linus: https://github.com/mkirchner/linked-list-good-taste Also avoid an allocation when replacing a value and tighten up tests.	2020-12-23 20:29:33 +09:00
Andrew Gaul	d0363b118e	Add tests for curl_util (#1481)	2020-11-24 21:37:09 +09:00
Takeshi Nakatani	533322859d	Added comment for nullPointerRedundantCheck/cppcheck2.2	2020-11-21 20:06:14 +09:00
Andrew Gaul	f8d5b76edb	Simplify handling of returned ETag (#1479) ... This works around lifetime warnings uncovered by cppcheck. References #1478.	2020-11-21 06:56:05 +09:00
Takeshi Nakatani	d9f6469b7b	Fixed flushing dirty data and compressed the cache size	2020-11-14 16:45:37 +09:00
Andrew Gaul	543231c9f2	Address warnings from clang-tidy 11 (#1470)	2020-11-09 21:15:20 +09:00
Noah Meyerhans	d96a08d4ad	fixup! Implement AWS IMDSv2 support	2020-11-07 14:48:52 +09:00
Noah Meyerhans	f2f930300a	Implement AWS IMDSv2 support ... AWS IMDSv2 is a session oriented method for retrieving instance metadata, including IAM credentials, in Amazon EC2. It is enabled by default in non-enforcing mode in AWS (meaning it retains backwards compatibility with existing IMDSv1 clients), but can be switched to enforcing mode, in which clients are required to return API tokens with requests. With this change, we implement support for IMDSv2 and enable it by default when IAM roles are our source for authentication credentials. In the event that s3fs is running in cloud environment offering an IMDSv1-compatible API, we support graceful fallback to that mode. It can also be selected explicitly via the imdsv1only mount option. More details on IMDSv2 are available at https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html and https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/ Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>	2020-11-07 14:48:52 +09:00
yoshihiko	81ad3ce0ae	Fix directory authority	2020-11-07 14:47:53 +09:00
Takeshi Nakatani	38e1eaa8a3	Added logfile option for non-syslog logging	2020-10-19 20:10:19 +09:00
Andrew Gaul	6aa786b886	Flush dirty data after a writing a number of bytes (#1448) ... This allows s3fs to write large files without consuming a large amount of temporary local storage but can slow uploads due to server-side copies. References #617. Fixed #1056. Fixes #1257.	2020-10-13 22:30:42 +09:00
Takeshi Nakatani	58750cc441	Fixed a bug that symlink could not be read after restarting s3fs	2020-10-11 22:14:36 +09:00
Andrew Gaul	910255745e	Allow relative path with use_path ... Fixes #1161.	2020-10-10 18:13:23 +09:00
Takeshi Nakatani	cf86fa51b0	Merge pull request #1441 from gaul/coverity ... Suppress false positive from Coverity	2020-10-04 23:44:51 +09:00
Takeshi Nakatani	0492f75197	Merge pull request #1439 from gaul/run-once ... Protect curl_warnings_once with a mutex	2020-10-04 22:43:56 +09:00
Takeshi Nakatani	493cf20f95	Merge pull request #1435 from gaul/no-exceptions ... Remove exceptions from s3fs_strtoofft	2020-10-04 22:19:25 +09:00
Andrew Gaul	3553fb65a0	Suppress false positive from Coverity	2020-10-04 14:58:13 +09:00
Takeshi Nakatani	059cc57ba6	Added atime and Corrected atime/mtime/ctime operations	2020-10-04 13:54:01 +09:00
Andrew Gaul	4df4ffe06f	Protect curl_warnings_once with a mutex ... This allows ThreadSanitizer to run without warnings.	2020-10-03 21:09:35 +09:00
Takeshi Nakatani	462347256d	Changed the message level in the AutoFdEntity::GetFdEntity	2020-10-03 20:41:17 +09:00
Takeshi Nakatani	133feb67c3	Merge pull request #1434 from gaul/const-string ... Use const std::string& where possible	2020-10-03 10:02:03 +09:00
Takeshi Nakatani	2cf195741c	Fixed signature error due to case of hex string	2020-10-02 18:09:13 +00:00
Takeshi Nakatani	f61baada46	Merge pull request #1433 from gaul/hex ... Further simplify and centralize hex conversion	2020-10-03 00:34:02 +09:00
Takeshi Nakatani	6e1e0d1d31	Merge pull request #1432 from gaul/v2-or-v4 ... Add sigv4 flag and fix v4 regression	2020-10-03 00:09:55 +09:00
Takeshi Nakatani	66419e7292	Merge pull request #1431 from gaul/string/find_first_of ... Prefer string::find_first_of(char) where possible	2020-10-02 23:42:17 +09:00
Takeshi Nakatani	9e998877e9	Merge pull request #1430 from gaul/ostringstream ... Simplify use of ostringstream	2020-10-02 23:26:41 +09:00
Andrew Gaul	3628b9d1e2	Remove exceptions from s3fs_strtoofft ... Explicit return codes are simpler and safer.	2020-10-02 22:53:12 +09:00
Andrew Gaul	8d2bd874d7	Use const std::string& where possible ... This hints to callers on usage.	2020-10-02 08:48:43 +09:00
Andrew Gaul	7e27c6cf7d	Further simplify and centralize hex conversion	2020-10-01 23:31:06 +09:00
Andrew Gaul	b8ff4ede49	Sign GET requests with V4 signature correctly ... This bug was previously masked by v2 fallback. Fixes regression from 81805715bd.	2020-10-01 20:03:14 +09:00
Andrew Gaul	081d6c1245	Allow setting signature V4-only ... Default to allowing V2 or V4.	2020-10-01 20:03:14 +09:00
Andrew Gaul	eb8004c355	Prefer string::find_first_of(char) where possible	2020-10-01 18:24:45 +09:00
Andrew Gaul	b3bf9f8f54	Simplify use of ostringstream ... Resetting this does not re-use the underlying buffer so creating a new ostringstream has similar efficiency.	2020-10-01 17:55:34 +09:00
Andrew Gaul	503c86bb8a	Call is_prefix instead of compare and substr	2020-09-27 22:19:54 +09:00
Takeshi Nakatani	757f4caee8	Merge pull request #1425 from gaul/create-bucket/sse ... Do not send SSE headers during bucket creation	2020-09-27 11:34:15 +09:00
Takeshi Nakatani	95fabd1f3a	Merge pull request #1423 from gaul/nettle ... Fix MD5 when using gnutls with nettle	2020-09-27 11:17:55 +09:00
Takeshi Nakatani	0b42e08636	Merge pull request #1422 from gaul/c++/append-assign-at ... Remove calls to append, assign, and at	2020-09-27 10:59:48 +09:00
Takeshi Nakatani	d5e4f99e72	Merge pull request #1413 from liuyongqing/master ... not call put headers if not exist pending meta	2020-09-27 10:37:22 +09:00
Andrew Gaul	f35fe850c0	Do not send SSE headers during bucket creation ... This allows integration tests to pass with use_sse.	2020-09-26 13:25:18 +09:00
Andrew Gaul	9a55c9fd9f	Fix MD5 when using gnutls with nettle ... This matches the non-nettle definition of s3fs_md5_fd.	2020-09-25 23:07:50 +09:00
Neeraj Kumar Kashyap	041b4ec05c	enforce exact key match in IBM IAM response	2020-09-25 22:41:19 +09:00
Andrew Gaul	2438066d52	Remove calls to append, assign, and at ... operator+, operator=, and operator[] are more idiomatic and consistent with the code base.	2020-09-25 12:19:51 +09:00
yongqingliu	e98827ec6f	not call put headers if not exist pending meta	2020-09-21 14:12:07 +08:00
Andrew Gaul	05863a3178	Fix clang-tidy warnings	2020-09-21 07:03:42 +09:00
Takeshi Nakatani	075d161bb1	Merge pull request #1408 from gaul/env ... Ensure environment variable is set when using ECS	2020-09-20 13:01:17 +09:00
Takeshi Nakatani	6f40503328	Merge pull request #1406 from gaul/strcmp ... Remove unnecessary calls to strcmp(str.c_str())	2020-09-20 11:28:41 +09:00
Takeshi Nakatani	3440c3348c	Merge pull request #1405 from gaul/is-prefix ... Rename STR2NCMP to is_prefix for clarity	2020-09-20 10:03:53 +09:00
Takeshi Nakatani	853be26612	Merge pull request #1404 from liuyongqing/master ... fix dead lock in disk insufficient and optimize code	2020-09-20 01:34:02 +09:00
Andrew Gaul	44d2cc15f7	Ensure environment variable is set when using ECS ... Previously s3fs crashed when AWS_CONTAINER_CREDENTIALS_RELATIVE_URI was not set. References #1162.	2020-09-18 08:14:58 +09:00
Andrew Gaul	81805715bd	Remove unnecessary calls to strcmp(str.c_str())	2020-09-17 20:00:45 +09:00
Andrew Gaul	ebcbb02d16	Rename STR2NCMP to is_prefix for clarity ... This may prevent confusion seen in 39ec8e242e.	2020-09-17 19:45:54 +09:00
Takeshi Nakatani	93cf67c65d	Merge pull request #1399 from gaul/md5-sha256 ... Use more descriptive names for digest functions	2020-09-17 17:11:19 +09:00
yongqingliu	211cc0f5f2	fix dead lock in disk insufficient and optimize code	2020-09-16 22:45:28 +08:00
Takeshi Nakatani	c58c91fc4f	Merge pull request #1398 from gaul/memset ... Remove unneeded memset calls	2020-09-15 22:09:54 +09:00
Andrew Gaul	b7187352e1	Use more descriptive names for digest functions ... Previously they used hex to refer to binary data which is confusing since other std::string return types are actually hex. Also remove unused s3fs_md5sum.	2020-09-15 21:18:58 +09:00
Takeshi Nakatani	1520ca6220	Merge pull request #1397 from gaul/hex ... Centralize hex conversion	2020-09-15 21:14:23 +09:00
Takeshi Nakatani	f7a63d5c97	Merge pull request #1396 from gaul/32-bit-read ... Use off_t for read sizes	2020-09-15 20:05:19 +09:00
Takeshi Nakatani	3958450c05	Added a class for automating fdentity reference counts	2020-09-14 22:02:28 +09:00
Andrew Gaul	5121c73ed1	Remove unneeded memset calls ... Also use constant for array lengths.	2020-09-14 19:51:36 +09:00
Andrew Gaul	44eaac8471	Centralize hex conversion	2020-09-14 19:28:59 +09:00
Andrew Gaul	77501c3600	Use off_t for read sizes ... This fixes an issue with large multipart_size on 32-bit platforms like Raspberry Pi.	2020-09-14 18:12:23 +09:00
Takeshi Nakatani	8205607716	Fixed incorrect Signal related message at s3fs initialization and termination	2020-09-13 18:54:10 +09:00
Andrew Gaul	1043e08dfa	Remove uses of implicit namespace std ... Fixed via: sed -i '/using namespace std/{N;d}' src/*.cpp sed -i 's/ string/ std::string/g' src/*.cpp sed -i 's/(string/(std::string/g' src/*.cpp sed -i 's/\[string/\[std::string/g' src/*.cpp sed -i 's/^string/std::string/g' src/*.cpp sed -i 's/ ifstream/ std::ifstream/g' src/*.cpp sed -i 's/ istringstream/ std::istringstream/g' src/*.cpp sed -i 's/ ostringstream/ std::ostringstream/g' src/*.cpp sed -i 's/ max(/ std::max(/g' src/*.cpp sed -i 's/ min(/ std::min(/g' src/*.cpp sed -i 's/ endl/ std::endl/g' src/*.cpp	2020-09-13 11:57:20 +09:00
Takeshi Nakatani	e5f4f9b69e	Merge pull request #1393 from gaul/32-bit-casting ... Fix casting warnings on 32-bit platforms	2020-09-13 10:40:56 +09:00
Takeshi Nakatani	1e3c10d803	Merge pull request #1392 from gaul/large-singlepart-32-bit ... Use 64-bit off_t when computing digests	2020-09-13 10:39:34 +09:00
Andrew Gaul	6d1e704e34	Fix casting warnings on 32-bit platforms	2020-09-12 18:22:32 +09:00
Andrew Gaul	ffc33a447f	Use 64-bit off_t when computing digests ... This allows 32-bit platforms like Raspberry Pi to upload single-part objects larger than 2 GB.	2020-09-12 15:00:23 +09:00
Andrew Gaul	39ec8e242e	Remove incorrect string prefix comparisons ... Previously flags like -o nouafoobar were parsed as -o noua. Found via: grep STR2NCMP src/s3fs.cpp | grep -v '="'	2020-09-11 17:38:20 +09:00
Ben Mares	0d4e39ad1c	return 0 after parsing "use_session_token" arg (#1388) ... return 0 after parsing "use_session_token" arg I hope this might fix #651. There's clearly an open bug in #651 confirmed by several users related to the parsing of the `-o use_session_token`. Looking at the source, I noticed that there was a return value everywhere except here, so I suspect this may be responsible.	2020-09-11 08:09:36 +09:00
yongqingliu	986fab8738	close FdEntity to avoid leakage	2020-09-07 20:48:16 +08:00
Takeshi Nakatani	b5ffd419d8	Source file division and set 4 spaces and cleanup	2020-08-26 17:43:50 +09:00
Takeshi Nakatani	c6e23212bb	Merge pull request #1375 from gaul/cppcheck ... Add const where possible	2020-08-21 07:32:39 +09:00
Andrew Gaul	1ec8528502	Add const where possible ... Found via cppcheck 2.1	2020-08-20 23:46:11 +09:00
Andrew Gaul	892e7129c5	Add support for deep archive storage class	2020-08-20 23:42:45 +09:00
Takeshi Nakatani	bdea2ee5c8	Merge pull request #1373 from gaul/enum ... Use scoped enums for acl_t, sse_type_t, and storage_class_t	2020-08-20 23:06:52 +09:00
Takeshi Nakatani	a5186c73c2	Merge pull request #1365 from ggtakec/check_lseek ... Dynamically determine whether lseek extended options are supported	2020-08-20 22:36:56 +09:00
Andrew Gaul	60d456a993	Use scoped enums for acl_t, sse_type_t, and storage_class_t ... This prevents some kinds of implicit conversions. Also deduplicate str/from_str logic. References #1371.	2020-08-20 18:46:18 +09:00
Takeshi Nakatani	bcf6838e86	Merge pull request #1370 from gaul/vim/modeline ... Correct vim modeline to 2-space indentation	2020-08-19 22:44:40 +09:00
Andrew Gaul	958ad83a4b	Correct vim modeline to 2-space indentation	2020-08-19 21:03:46 +09:00
Andrew Gaul	ba61470bae	Prefer named locks in curl ... This removes the confusing and unsafe pthread_mutex_t array.	2020-08-18 22:20:41 +09:00
Andrew Gaul	8210a1b2f2	Consistently lock curl_times and curl_progress ... References #1362.	2020-08-17 09:47:38 +09:00
Takeshi Nakatani	55cb8920d5	Dynamically determine whether lseek extended options are supported	2020-08-16 12:37:11 +00:00
Takeshi Nakatani	46acbf10ba	Merge pull request #1363 from gaul/is-modified-race ... Lock fdent_data_lock before accessing pagelist	2020-08-16 20:25:24 +09:00
Takeshi Nakatani	2c0408b95a	Merge pull request #1357 from gaul/readdir ... Call readdir instead of readdir_r	2020-08-16 18:50:38 +09:00
Takeshi Nakatani	057da86d87	Merge pull request #1356 from gaul/printf-size_t ... Prefer %zd and %zu for ssize_t and size_t	2020-08-16 18:16:59 +09:00
Andrew Gaul	8de6cb3504	Lock fdent_data_lock before accessing pagelist ... Found via ThreadSanitizer. References #1353. References #1362.	2020-08-16 17:44:03 +09:00
Takeshi Nakatani	2bb745cdd7	Fixed a bug about move file over limit of ensure space	2020-08-16 17:42:20 +09:00
Andrew Gaul	35090ba4d5	Call readdir instead of readdir_r ... Only a single thread uses this directory stream. Further, modern implementations are thread-safe by default and deprecated this call: https://man7.org/linux/man-pages/man3/readdir_r.3.html	2020-08-16 13:33:23 +09:00
Andrew Gaul	132a1bebbb	Prefer %zd and %zu for ssize_t and size_t ... This removes unnecessary casting.	2020-08-15 17:30:23 +09:00
Takeshi Nakatani	cbd925c56f	Moved the SIGUSR2 handler to S3fsSignals class	2020-07-28 14:54:35 +09:00
Takeshi Nakatani	63bbb47378	Merge pull request #1341 from gaul/stat-cache/default ... Change default stat_cache_expire	2020-07-26 23:54:14 +09:00
Andrew Gaul	0fbd0eac80	Change default stat_cache_expire ... Previously s3fs cached files forever which confused users with creating objects using another client.	2020-07-26 23:04:43 +09:00
Takeshi Nakatani	ad1961417d	Added SIGUSR1 option for cache file integrity test	2020-07-26 21:04:11 +09:00
Takeshi Nakatani	700e288718	Put similar processing together into method GetCacheFileStatTopDir	2020-07-12 08:12:02 +00:00
Takeshi Nakatani	dc365b65a0	Fix NULL pointer deference ... Found via cppcheck 2.1.	2020-07-09 21:40:23 +09:00
Takeshi Nakatani	1f796d432d	Fixed upload error about mixuploading sparse file and truncating file	2020-06-27 22:44:19 +09:00
Takeshi Nakatani	35006e318f	Fixed about ParallelMixMultipartUpload	2020-06-24 12:48:55 +09:00
Eric Vantillard	7d0c66e08a	Add support for glacier storage class. ... Just a copy of what have been done in PR #271.	2020-06-23 11:23:21 +09:00
Takeshi Nakatani	f324d8e04f	Fixed a bug about serializing from cache file	2020-06-19 12:57:27 +00:00
Takeshi Nakatani	f16ee96d7e	Merge pull request #1306 from gaul/http/500 ... Retry with exponential backoff during 500 error	2020-06-06 15:30:22 +09:00
Andrew Gaul	d8766b2051	Retry with exponential backoff during 500 error ... Amazon suggests retrying on both 500 and 503: https://aws.amazon.com/premiumsupport/knowledge-center/http-5xx-errors-s3/ Fixes #1251.	2020-06-05 21:01:30 +09:00
Andrew Gaul	9db70bab63	Ignore case when comparing ETags ... This allows multipart upload to work with Alibaba OSS. References #1297.	2020-06-05 18:17:52 +09:00
Takeshi Nakatani	0cb057dadd	Merge pull request #1303 from gaul/rename/use_cache ... Relink cache stats file atomically via rename	2020-06-01 00:10:33 +09:00
Takeshi Nakatani	0f5db0d1bf	Merge pull request #1302 from gaul/rename/nocopy ... Fix renames of open files with nocopyapi option	2020-05-31 23:46:46 +09:00
Andrew Gaul	274321524c	Relink cache stats file atomically via rename ... The new file may already exist so link may fail. Further link/unlink is not atomic. Addresses an error when renaming an open with with use_cache. References #1296.	2020-05-31 23:09:58 +09:00
Andrew Gaul	40f7007263	Check results from pthread mutex calls ... Also remove some unnecessary exception handling.	2020-05-30 16:37:55 +09:00
Andrew Gaul	66597ec5f2	Fix renames of open files with nocopyapi option ... References #1296.	2020-05-30 15:45:43 +09:00
Takeshi Nakatani	75e72385cc	Added a parameter to output body to curldbg option	2020-05-25 08:49:01 +09:00
Andrew Gaul	8b15db6dcb	Do not allow zero retries ... Retries actually means tries, e.g., if the user sets zero, s3fs will never try an operation at all.	2020-05-23 10:05:23 +09:00
Takeshi Nakatani	80c11b6c12	Not abort process by exception threw from s3fs_strtoofft	2020-05-03 13:46:05 +00:00
Andrew Gaul	8945e98d8b	Support Google Cloud Storage headers ... This allows s3fs to interpret objects created by gsutil.	2020-05-03 18:33:13 +09:00
Takeshi Nakatani	97c249d5b9	Not abort process by exception threw from s3fs_strtoofft	2020-05-03 08:08:28 +00:00
Andrew Gaul	2cc88b933f	Warn about missing MIME types instead of exiting ... s3fs uses the MIME types file to set Content-Type for uploaded objects. Most distribution packages should install this via recommended (not required) dependencies. Users compiling from source may not have this installed and s3fs should not prevent launching since most users do not care about Content-Type. Instead warn about MIME types absence. Fixes #1270.	2020-04-29 20:03:50 +09:00
Andrew Gaul	005a684600	Fix typos	2020-04-22 21:49:11 +09:00
Takeshi Nakatani	f26a0aa71d	Fixed insufficient upload size for mix multipart upload	2020-04-22 09:31:22 +09:00
Andrew Gaul	cf529e0af7	Add handler for HTTP 416 ... This prevents retries when the server indicates an unsatisfiable MPU copy part request. References #1220.	2020-04-21 19:45:10 +09:00
Takeshi Nakatani	4da02d023b	Improved strictness of cache file stats(file)	2020-04-21 19:45:03 +09:00
Takeshi Nakatani	811ea0cb85	Fixed the truncation bug of stat file for cache file	2020-04-19 07:08:49 +00:00
Andrew Gaul	48a872e285	Address cppcheck 1.90 warning	2020-04-12 22:20:44 +09:00
Takeshi Nakatani	c44a60f3f5	Fixed a bug of stats cache compression	2020-04-12 18:33:00 +09:00
Takeshi Nakatani	9e01d5b8d1	Merge pull request #1254 from ggtakec/modify_mimetypes ... Added mime option for strict checking of mime types file	2020-04-11 14:48:47 +09:00
Takeshi Nakatani	7fbda230f5	Added mime option for strict checking of mime types file	2020-03-30 14:41:18 +00:00
Andrew Gaul	56141557dc	Avoid unneeded string copy ... Found by clang-tidy 10.	2020-03-28 08:49:49 +09:00
yongqingliu	b72f4b43a4	use correct content-type when complete multipart upload	2020-02-10 16:58:28 +09:00
Andrew Gaul	924eeb3587	Document host and servicepath ... The various bits of host, url, and servicepath seem to overlap. References #1203.	2020-02-04 21:13:29 +09:00
Andrew Gaul	bc9126d774	Set directory MIME type to application/x-directory ... Previously s3fs auto-detected the MIME type of directories like "TOYOTA TRUCK 8.2.2" as application/x-troff-man. This caused get_mode to not set S_IFDIR which failed directory creation. Instead force all object names ending in / to application/x-directory. Fixes #1183.	2020-02-04 20:03:21 +09:00
rallister	4e26728cbf	break recursion when calling GetIAMCredentials (#1233) ... break recursion when calling GetIAMCredentials	2020-01-31 16:48:37 +09:00
Takeshi Nakatani	018ccb9a11	Suppressed a lot of message output about cache cleanup	2020-01-30 14:42:28 +00:00
yongqingliu	b762a0a85b	fix deadlock due to fdmanager_lock and fdent_data_lock	2020-01-29 12:03:53 +08:00
Andrew Gaul	e0712f444d	Update source for clang-tidy 9.0.0	2020-01-13 20:56:45 +09:00
Andrei Precup	39102608aa	Add intelligent tiering as option for storage class ... Resolves Issue #1219	2020-01-13 20:24:35 +09:00
Takeshi Nakatani	23945a0130	In memory cache for softlinks with cache out	2020-01-13 20:23:10 +09:00
Alvaro Huarte	bdfb9ee815	Document requester_pays option	2019-11-25 22:53:50 +09:00
Alvaro Huarte	a24f78f5a4	Adds requester_pays option to enable requests involving Requester Pays buckets	2019-11-18 12:38:16 +01:00
Takeshi Nakatani	a07a5336f6	Merge pull request #1185 from bryceml/master ... handle edge case when bucket name and first part of endpoint match	2019-10-31 23:42:47 +09:00
Bryce Larson	0c42a74a8a	handle edge case when bucket name and first part of endpoint match	2019-10-26 00:51:19 -06:00
Takeshi Nakatani	cf3e82d10a	Fixed a rename bug when enable_noobj_cache	2019-10-23 12:39:42 +00:00
Takeshi Nakatani	1db94a0b30	Fixed to use copy api in multipart upload	2019-09-25 19:30:58 -07:00
Takeshi Nakatani	bedd648d47	Fixed build error by cppcheck 1.89	2019-09-23 10:49:49 +00:00
Takeshi Nakatani	58b3cce320	Merge pull request #1066 from gaul/sign-requests ... Sign request immediately before sending	2019-09-23 15:58:21 +09:00
Takeshi Nakatani	81102a5963	Merge pull request #1149 from ggtakec/fix_retry ... Fixed a bug that type and op are not inherited	2019-09-08 20:32:43 +09:00
Takeshi Nakatani	42fb30852b	Merge pull request #1145 from gaul/bool ... Fix a few bool types	2019-09-08 20:04:30 +09:00
Takeshi Nakatani	e51361cb94	Merge pull request #1143 from gaul/doc/umask ... Document umask flag	2019-09-08 19:49:58 +09:00
Takeshi Nakatani	e2129001eb	Fixed a bug that type and op are not inherited	2019-09-08 09:59:04 +00:00
Andrew Gaul	805cc064af	Fix a few bool types ... Found via clang-tidy.	2019-09-05 10:42:42 -07:00
Andrew Gaul	3c2279db39	Document umask flag	2019-09-04 20:53:58 -07:00
Andrew Gaul	412876ca33	Expose base parameter in s3fs_strtoofft ... This fixes a regression from ccf3e7bfa2 which caused the misparsing of octal inputs for the mp_umask and umask flags. It also allows some callers to be more precise about their decimal inputs.	2019-09-04 20:41:47 -07:00
Andrew Gaul	461a346bf4	Sign request immediately before sending ... Previously s3fs could create a long list of pre-signed requests which could take longer than the default S3 clock skew limit of 15 minutes. This also changes SHA-256 computation from single- to multi-threaded since this is now computed in the worker threads. Regression from 88cd8feb05. Fixes #1019.	2019-08-29 15:35:27 -07:00
liuyongqing	ae4bcd405c	use correct content-type when overwriting and copying files	2019-08-28 22:25:09 -07:00
Takeshi Nakatani	0536dc1112	Refixed strict processing of MultiRead method	2019-08-25 20:46:01 -07:00
Takeshi Nakatani	1c3507ede1	Merge pull request #1135 from gaul/curl-timeout ... Remove useless CURLE_OPERATION_TIMEDOUT check	2019-08-25 17:53:02 +09:00
Andrew Gaul	01ac815346	Remove useless CURLE_OPERATION_TIMEDOUT check ... CheckBucket only returns errno and HTTP status codes.	2019-08-20 18:44:48 -07:00
Andrew Gaul	cc2eed84a5	Promote default ACL to enum ... This sanity checks ACLs during initialization and also omits sending the header when set to PRIVATE which is the default.	2019-08-20 15:05:58 -07:00
Takeshi Nakatani	873e376098	Strict processing of MultiRead method	2019-08-13 15:23:13 +00:00
Takeshi Nakatani	4a0c23258e	Fixed MultiRead method to exit after waiting other parts at error	2019-08-12 14:45:06 -07:00
Takeshi Nakatani	5c4a0a862a	Fixed multipart copy and its retry handler	2019-08-11 16:21:26 +00:00
Takeshi Nakatani	f5bf41cf11	Fixed simple copy with nocpoyapi and modify flag in fdcache	2019-08-11 07:42:48 +00:00
Takeshi Nakatani	f74c7407db	Merge pull request #1118 from gaul/atoi ... Prefer s3fs_strtoofft over atoi and strtol	2019-08-06 21:23:37 +09:00
Takeshi Nakatani	433c04af26	Merge pull request #1116 from gaul/bodydata ... Do not heap allocate BodyData	2019-08-06 21:08:02 +09:00
Takeshi Nakatani	2e51a339a9	Merge pull request #1115 from gaul/hash-length ... Use length from s3fs_HMAC256 for consistency	2019-08-06 20:49:22 +09:00
Takeshi Nakatani	14d3e12103	Merge pull request #1114 from gaul/flush/return-value ... Consume return value from FdEntity::Flush	2019-08-06 14:12:25 +09:00
Andrew Gaul	ccf3e7bfa2	Prefer s3fs_strtoofft over atoi and strtol ... The former propagates errors consistently.	2019-08-03 16:13:48 -07:00
Andrew Gaul	7ecfba811e	Do not heap allocate BodyData ... Callers already manage lifetimes via Clear().	2019-08-03 15:50:46 -07:00
Takeshi Nakatani	1e7330e499	Merge pull request #1113 from gaul/unused ... Remove unused field	2019-08-04 01:31:56 +09:00
Takeshi Nakatani	3d42d0515d	Merge pull request #1109 from gaul/lock-already-held ... Pass lock_already_held state to DelStat	2019-08-03 22:13:22 +09:00
Takeshi Nakatani	3d70e8966f	Merge pull request #1108 from gaul/lock/diskspace ... Always hold lock when accessing free_disk_space	2019-08-03 22:01:31 +09:00
Andrew Gaul	a5c20175a1	Use length from s3fs_HMAC256 for consistency ... EVP_MAX_MD_SIZE can range up to 64. Found by Coverity.	2019-08-02 10:40:55 -07:00
Andrew Gaul	43d1439420	Consume return value from FdEntity::Flush ... Found via Coverity.	2019-08-02 10:01:06 -07:00
Andrew Gaul	d8cf26bd50	Remove unused field ... Found via clang.	2019-08-01 16:42:38 -07:00
Andrew Gaul	f528a86219	Pass lock_already_held state to DelStat	2019-08-01 11:07:56 -07:00
Andrew Gaul	5b15c7c4e9	Always hold lock when accessing free_disk_space ... Slightly reorder locks to avoid double locking. Found via ThreadSanitizer.	2019-07-30 08:36:54 -07:00
Andrew Gaul	afd438d363	Access FdEntity metadata while holding fdent_lock ... Create copies to avoid thread races. Found via ThreadSanitizer. Follows on to ecf13a8cb9.	2019-07-29 13:54:57 -07:00
Andrew Gaul	fb937635f5	Eagerly initialize sysconf variables ... Previously s3fs had races updating these shared variables. Found via ThreadSanitizer.	2019-07-17 09:08:13 -07:00
Takeshi Nakatani	3ad1c95e86	Merge pull request #1103 from gaul/config/readwrite-timeout ... Use consistent default for readwrite_timeout	2019-07-17 22:37:02 +09:00
Andrew Gaul	bacd15714a	Use consistent default for readwrite_timeout ... Previously curl threads mutated this shared state without a lock. Found via ThreadSanitizer.	2019-07-16 22:03:56 -07:00
Andrew Gaul	99aace4fc9	Avoid race when using thread-unsafe gmtime ... Found via ThreadSanitizer.	2019-07-16 07:28:34 -07:00
Takeshi Nakatani	513f41fddf	Merge pull request #1097 from ggtakec/fix_lock ... Add Lock() func parameter in RowFlush()	2019-07-15 11:56:21 +09:00
Takeshi Nakatani	543aed2a32	Merge pull request #1096 from gaul/pthread/lock-checking ... Expand error checking to all pthread_mutex_t	2019-07-15 11:43:56 +09:00
Takeshi Nakatani	20ea96328c	Merge pull request #1095 from gaul/intmax_t ... Avoid narrowing time_t and off_t args in logging	2019-07-15 11:33:44 +09:00
Takeshi Nakatani	f78bcc5229	Add Lock() func parameter in RowFlush()	2019-07-15 01:52:29 +00:00
Andrew Gaul	43ec064fb9	Expand error checking to all pthread_mutex_t	2019-07-14 18:21:23 -07:00
Andrew Gaul	ffac4c8417	Avoid narrowing time_t and off_t args in logging ... This displays correct results on 32-bit platforms.	2019-07-14 17:02:36 -07:00
Andrew Gaul	4adcd4a6c8	Implement s3fs_strtoofft via strtoll ... This tightens error checking and aligns s3fs with known good behavior.	2019-07-14 15:15:06 -07:00
Takeshi Nakatani	e936854493	Merge pull request #1094 from larsch/master ... Fix multipart upload not setting curl options	2019-07-15 05:13:37 +09:00
Takeshi Nakatani	850a813171	Merge pull request #1092 from liuyongqing/master ... fix coredump caused by fd_manager_lock locking	2019-07-15 04:53:10 +09:00
Takeshi Nakatani	5bbcd3b981	Merge pull request #1089 from gaul/null-dereference ... Avoid null dereference	2019-07-15 04:17:10 +09:00
Takeshi Nakatani	a337c32840	Merge pull request #1088 from gaul/clang-tidy/misplaced-const ... Avoid misplaced const via removing unneeded typedef	2019-07-15 04:08:51 +09:00
Takeshi Nakatani	d39e4e4b1f	Merge pull request #1087 from gaul/clang-tidy/deprecated-headers ... Prefer modern C headers	2019-07-15 03:50:15 +09:00
Lars Christensen	58037da061	Fix multipart upload not setting curl options	2019-07-14 14:09:37 +02:00
Andrew Gaul	1eb266588e	Add a missing lock to SetCtime ... Also lock before log statements which touch member variables.	2019-07-13 17:53:38 -07:00
root	4e351c59e3	fix coredump caused by fd_manager_lock locking	2019-07-13 12:03:27 +08:00
Andrew Gaul	eb597289cb	Avoid null dereference ... Found via clang-tidy.	2019-07-12 18:40:24 -07:00
Andrew Gaul	6fd42d9fe4	Prefer modern C headers ... Found and fixed via clang-tidy.	2019-07-12 03:50:59 -07:00
Andrew Gaul	efff9c01a6	Avoid misplaced const via removing unneeded typedef ... Found via clang-tidy.	2019-07-12 03:41:19 -07:00
Andrew Gaul	a83d5baa90	Remove recursive locking ... Recursive locking is frowned upon and is incompatible with PTHREAD_MUTEX_ERRORCHECK. Also clean up pthread_mutex_lock error checking.	2019-07-10 12:39:00 -07:00
Takeshi Nakatani	50d13255e4	Merge pull request #1081 from gaul/write-blocks-readdir ... Break FdEntity locks into data and metadata	2019-07-11 03:40:26 +09:00
Takeshi Nakatani	e5e63d6ac3	Merge pull request #1067 from gaul/mismatched-free ... Fix mismatched new[]/free	2019-07-07 16:55:07 +09:00
Takeshi Nakatani	7a65a414c3	Merge pull request #1080 from gaul/xml/simplification ... Add simple XML parsing wrapper	2019-07-07 16:44:18 +09:00
Andrew Gaul	944d21cabb	Fix mismatched new[]/free ... Found via Valgrind.	2019-07-07 00:36:48 -07:00
Takeshi Nakatani	d267212289	Merge pull request #1079 from gaul/cppcheck/shadowing ... Avoid shadowing variable in FdEntity::Open	2019-07-07 16:29:42 +09:00
Takeshi Nakatani	58d8e5586a	Merge pull request #1078 from gaul/multipart/2x ... Issue multipart when object size exceeds part size	2019-07-07 16:21:22 +09:00
Takeshi Nakatani	ce803daf4a	Merge pull request #1077 from gaul/openssl/locking ... Annotate OpenSSL locking functions as unused	2019-07-07 16:10:14 +09:00
Takeshi Nakatani	9bf34e2fda	Merge pull request #1076 from gaul/doc/man-help ... Add documentation for use_session_token and use_rrs	2019-07-07 15:45:42 +09:00
Takeshi Nakatani	6bd1a7eac0	Merge pull request #1074 from gaul/clang-tidy ... Configure clang-tidy target	2019-07-07 15:28:08 +09:00
Andrew Gaul	8aa06d621a	Add documentation for use_session_token and use_rrs ... Fixes #929.	2019-07-06 10:04:43 -07:00
Andrew Gaul	ecf13a8cb9	Break FdEntity locks into data and metadata ... Previously long-running data operations like RowFlush would block metadata operations like GetStats and thus user readdir. Fixes #928.	2019-07-05 23:12:24 -07:00
Andrew Gaul	b8ff6a647e	Add simple XML parsing wrapper ... Also simplify check_region_error.	2019-07-05 17:39:09 -07:00
Andrew Gaul	49110c671d	Avoid shadowing variable in FdEntity::Open ... Found via cppcheck 1.88.	2019-07-05 15:06:18 -07:00
Andrew Gaul	febaf6849f	Issue multipart when object size exceeds part size ... Previously s3fs issued multipart uploads when the object size was twice the part size. Conjoining this with the part size was confusing and s3fs should add a separate tunable for this if needed, similar to singlepart_copy_limit. Fixes #1058.	2019-07-05 12:14:56 -07:00
Andrew Gaul	4893174652	Annotate OpenSSL locking functions as unused ... OpenSSL 1.1.0 removed these and generates warnings with newer versions.	2019-07-05 10:45:57 -07:00
Andrew Gaul	4f23f38583	Individually test multiple s3fs flags ... Remove unneeded comments; single part limits ensure that the tests exercise multipart code paths even with smaller files. References #971.	2019-07-03 21:09:40 -07:00
Andrew Gaul	bbfa91141a	Configure clang-tidy target ... Also fix nits.	2019-07-03 14:04:11 -07:00
Takeshi Nakatani	21321a9d96	Merge pull request #1065 from gaul/doc/ssl-verify-hostname ... Document ssl_verify_hostname	2019-07-03 22:15:33 +09:00
Takeshi Nakatani	f03b50fd13	Merge pull request #1063 from gaul/truncate-2nd-open-fd ... Flush file when opening second fd	2019-07-03 21:14:03 +09:00
Andrew Gaul	f02105c346	Document ssl_verify_hostname ... References #1064.	2019-07-02 10:26:29 -07:00
Andrew Gaul	c596441f58	Flush file when opening second fd ... Previously when s3fs had dirty local data and an application opened a second fd it would remove the stat cache entry, query S3 for the size which was still zero, then reinitialize FdEntry with this incorrect size. Instead flush local data to S3 so this query works. It is possible that a more involved patch could do this with a less heavyweight approach but this requires changing open. This does not completely fix git clone but allows it to proceed further. Also make some cache methods const-correct. References #839.	2019-07-02 01:12:09 -07:00
Andrew Gaul	455e29cbea	Make fdpage a value type in fdpage_list_t ... This simplifies memory management.	2019-07-01 09:30:48 -07:00
Takeshi Nakatani	511d223468	Merge pull request #1059 from gaul/casting ... Fix casting warning on 32-bit	2019-07-01 22:26:14 +09:00
Andrew Gaul	d7f77a6282	Fix casting warning on 32-bit	2019-06-29 20:57:42 -07:00
Andrew Gaul	f1ad626b46	Log flags as hexadecimal	2019-06-24 16:13:07 -07:00
Takeshi Nakatani	a78d8d1da4	Merge pull request #1049 from gaul/external-modication ... Remove cache file when object time differs	2019-06-24 00:51:54 +09:00
Takeshi Nakatani	fbebc6fa57	Merge pull request #1048 from gaul/log/curl ... Emit older curl warnings only once	2019-06-24 00:42:44 +09:00
Andrew Gaul	9e5eaad79b	Remove cache file when object time differs ... Check the modification times to determine whether an object has updated. This relies on low clock skew between s3fs and the S3 server; a more robust approach could use the ETag. Fixes #1047.	2019-06-22 19:09:00 -07:00
Andrew Gaul	738eaadcbf	Emit older curl warnings only once ... This makes Travis logs actually readable.	2019-06-22 10:01:20 -07:00
Andrew Gaul	15b7450713	Convert file offsets from size_t to off_t ... The latter is 64-bits on 32-bit platforms when specifying -D_FILE_OFFSET_BITS=64. This allows early Raspberry Pis to use files larger than 2 GB. It also cleans up some ugly casting. Fixes #620. Fixes #656.	2019-06-15 17:05:37 -07:00
Andrew Gaul	059ab1f0f4	Update ctime during nocopyapi operations ... Follows on to 2c43b1e12b. Fixes #971.	2019-04-30 18:59:51 +09:00
Michal Lula	0d4847596e	fix issue with aws session token provided inside ~/.aws/credentials file	2019-04-25 14:29:35 +02:00
Takeshi Nakatani	8e86ef1634	Merge pull request #1015 from gaul/c++03 ... Remove uses of std::map::at	2019-04-22 23:01:29 +09:00
Andrew Gaul	9e4f9d4bdc	Remove uses of std::map::at ... This provides compatibility with C++03 and removes duplicate lookups.	2019-04-18 16:06:59 +09:00
Andrew Gaul	0677a096a9	Prefer simple over compound statements in macros ... This prohibits missing semicolons.	2019-04-17 21:46:24 +09:00
Michal Lula	f9cd43b684	add session token support	2019-04-16 16:53:05 +02:00
Andrew Gaul	8a18806a57	Make man page and --help more consistent ... Mostly small fixes but also some reordering. References #929. Found via: diff -u <(man --no-hyphenation doc/man/s3fs.1 | tr -s ' ' '\n' | sed '/^-o$/d' ) <(src/s3fs --help | tr -s ' ' '\n' | sed '/^-$/d')	2019-04-14 14:34:24 +09:00
Takeshi Nakatani	0a99470369	Merge pull request #1002 from gaul/sse-c ... Correctly calculate MD5 hash for SSE-C keys	2019-04-09 00:17:12 +09:00
Takeshi Nakatani	cd280d8702	Merge pull request #1001 from gaul/missing-braces ... Add missing braces	2019-04-08 23:55:02 +09:00
Andrew Gaul	6f8ecb28c5	Correctly calculate MD5 hash for SSE-C keys ... Previously s3fs calculated the strlen of a binary input instead of using the stored length. This yielded IO errors when interacting with SSE-encrypted objects. Fixes #696.	2019-04-08 21:49:42 +09:00
Andrew Gaul	042332bcec	Add missing braces ... Found via clang-tidy. Also fix errant indentation.	2019-04-07 23:12:27 +09:00
Takeshi Nakatani	071cd0f849	Modified to discard _netdev option etc	2019-04-07 11:51:55 +00:00
Andrew Gaul	6f6a67807b	Prefer new over malloc ... The former cannot return NULL and will allow use of scoped pointers.	2019-04-07 16:54:24 +09:00
Takeshi Nakatani	ea517c80a4	Fixed a bug about overwrite metadata at updating directory stats	2019-03-29 15:30:30 +00:00
Takeshi Nakatani	71766039ff	Support undefined CURLoption in libcurl library used in build	2019-03-22 10:47:42 +00:00
Andrew Gaul	058706014b	Simplify string comparison ... Found via newer cppcheck.	2019-03-14 22:19:33 +09:00
Takeshi Nakatani	62dcda6a56	Fixed ref-count when error occurred.	2019-03-10 06:04:19 +00:00
Takeshi Nakatani	cbf072bc55	Merge pull request #967 from gaul/reference-count ... Increase FdEntity reference count when returning	2019-03-10 14:27:10 +09:00
Takeshi Nakatani	b5ca400500	Merge pull request #970 from gaul/nocopyapi ... Remove from file from stat cache during rename	2019-03-10 12:56:04 +09:00
Andrew Gaul	08b132ddb9	Remove from file from stat cache during rename ... This addresses failures with test_rename_before_close when using nocopyapi. Note that test_update_time still fails.	2019-03-09 17:53:35 +09:00
Andrew Gaul	1e86cc643d	Fix lazy typo	2019-03-09 17:46:02 +09:00
Andrew Gaul	f53503438c	Increase FdEntity reference count when returning ... Previously s3fs had a race condition where one client could delete FdEntity that another client was using. Add a simple concurrent test which previously failed but now succeeds. Fixes #964.	2019-03-09 15:55:04 +09:00
Takeshi Nakatani	0d43d070cc	Merge pull request #965 from ggtakec/fix_multi_x ... Improvement of curl session pool for multipart	2019-02-27 22:44:25 +09:00
Takeshi Nakatani	0791fdca2a	Merge pull request #960 from kristjanvalur/wtf8 ... Wtf8	2019-02-27 21:21:38 +09:00
Kristján Valur Jónsson	6e8678d5e3	remove lines that wer accidentally duplicated	2019-02-25 16:02:01 +00:00
Takeshi Nakatani	10d9f75366	Improvement of curl session pool for multipart ... Improvement of curl session pool for multipart(2) Improvement of curl session pool for multipart(3)	2019-02-25 14:46:24 +00:00
Andrew Gaul	74d8671e54	Work around cppcheck warnings ... Follows on to eeb839242b.	2019-02-20 21:55:41 +09:00
Kristján Valur Jónsson	4c41eac29c	fix documentation and man page	2019-02-20 11:24:29 +00:00
Kristján Valur Jónsson	3c97c1b251	merged main	2019-02-19 10:53:00 +00:00
Kristján Valur Jónsson	84c671a81a	fix indentation	2019-02-19 10:37:43 +00:00
Kristján Valur Jónsson	f336bdebcc	add command line flag and documentation	2019-02-19 10:32:37 +00:00
Kristján Valur Jónsson	e5b8377202	fix comments and code	2019-02-19 10:32:10 +00:00
Kristján Valur Jónsson	4f42f4ab0c	Enable s3fs encoding and decoding in the fuse interface	2019-02-18 13:36:48 +00:00
Kristján Valur Jónsson	11b385820d	more robust wtf8 encoding	2019-02-18 12:27:44 +00:00
Takeshi Nakatani	eeb839242b	Fixed code for latest cppcheck error on OSX	2019-02-17 13:59:11 +00:00
Kristján Valur Jónsson	ca2d1d873d	Adding utility functions to convert invalid utf8 to wtf8 encoding ... This is to deal with windows clients who pass in cp1252 as if it were utf8	2019-02-15 15:57:03 +00:00
Takeshi Nakatani	231fd001d9	Added a non-interactive option to utility mode ... Rebase for resolving conflicts Deleted interactive mode on utility mode Fixed minor mistakes	2019-02-10 17:36:18 +00:00
Takeshi Nakatani	314dc5a398	Merge pull request #955 from gaul/clang-tidy/misc ... Address miscellaneous clang-tidy warnings	2019-02-06 21:21:54 +09:00
Takeshi Nakatani	e07cb020cc	Merge pull request #954 from gaul/clear-iter ... Clear containers instead of individual erases	2019-02-06 21:12:36 +09:00
Andrew Gaul	e87e40b3b4	Address miscellaneous clang-tidy warnings ... Also add configuration with suppressions.	2019-02-04 22:58:40 -08:00
Takeshi Nakatani	bd66b57ad3	Merge pull request #952 from gaul/multipart/failed ... Automatically abort failed multipart requests	2019-02-04 22:19:27 +09:00
Andrew Gaul	a1d3ff9766	Automatically abort failed multipart requests ... This can avoid dangling parts. However, many transfers fail due to network errors so we still need other mechanisms to handle these parts.	2019-02-03 10:29:20 -08:00
Andrew Gaul	4d0bef1e90	Clear containers instead of individual erases ... This has O(n) runtime instead of O(n log n).	2019-02-02 23:58:43 -08:00
Takeshi Nakatani	960823fb40	Added S3FS_MALLOC_TRIM build switch	2019-02-03 07:36:17 +00:00
Takeshi Nakatani	c04e8e7a9d	Merge pull request #949 from gaul/503-slow-down ... Implement exponential backoff for 503	2019-02-03 15:19:28 +09:00
Takeshi Nakatani	fb6debd986	Merge pull request #948 from gaul/too-many-parts ... Add logging for too many parts	2019-02-03 14:55:54 +09:00
Takeshi Nakatani	d8185a25aa	Merge pull request #946 from gaul/async/completed-tids ... Simplify async request completion code	2019-02-03 14:08:49 +09:00
Takeshi Nakatani	53337a0a28	Merge pull request #944 from gaul/utility-mode ... Repair utility mode	2019-02-03 13:59:56 +09:00
Takeshi Nakatani	ae51556d04	Merge pull request #943 from gaul/hard-link ... Return not supported when hard linking	2019-02-03 12:45:21 +09:00
Takeshi Nakatani	055ecf6ea7	Merge pull request #940 from gaul/parallel-multipart-copy ... Copy parts in parallel	2019-02-03 11:44:15 +09:00
Takeshi Nakatani	c603680e02	Merge pull request #939 from gaul/stringstream-specific ... Prefer specific [io]stringstream where possible	2019-02-03 11:06:57 +09:00
Takeshi Nakatani	814aadd7e3	Merge pull request #938 from gaul/clang-tidy/redundant-void ... Remove unneeded void parameter	2019-02-03 10:57:56 +09:00
Andrew Gaul	8ff05d8e38	Implement exponential backoff for 503 ... Amazon returns SlowDown when overloaded. Also return ENOTSUP for 501 and immediately return EIO for 500 instead of retrying. Fixes #603.	2019-02-02 00:03:50 -08:00
Andrew Gaul	dfa84b82a8	Add logging for too many parts ... References #610.	2019-02-01 19:34:26 -08:00
Andrew Gaul	8c527c3616	Simplify async request completion code ... Workers now notify the master thread when they complete, unifying the Linux and macOS code paths. This also avoids excessive pthread_tryjoin_np calls. Follows on to 88cd8feb05.	2019-01-31 22:55:23 -08:00
Andrew Gaul	54a074647e	Repair utility mode ... This deinitialized S3fsCurl twice and incorrectly calculated V4 signatures.	2019-01-31 18:45:39 -08:00
Andrew Gaul	c5ebf5d328	Copy parts in parallel ... S3 can copy multipart much faster than single part due to IO parallelization. Renaming a 4 GB file reduces from 72 to 20 seconds with bigger gains with larger files.	2019-01-31 10:21:39 -08:00
Andrew Gaul	43c6ef560e	Return not supported when hard linking ... This is more correct than permission denied.	2019-01-30 16:43:04 -08:00
Andrew Gaul	3076abc744	Disable malloc_trim ... This avoids walking the entire heap multiple times for complex operations like readdir. This does not entirely eliminate the observed performance regression but does dramatically reduce s3fs CPU usage. References #935. Fixes #936.	2019-01-29 15:29:07 -08:00
Andrew Gaul	07636c8a8d	Prefer specific [io]stringstream where possible ... These better communicate intent and are slightly more efficient.	2019-01-29 10:44:33 -08:00
Andrew Gaul	35d55ee513	Remove unneeded void parameter ... This is implicit in C++. Found and fixed via clang-tidy.	2019-01-28 23:22:27 -08:00
Takeshi Nakatani	c0cf90cf8b	Checked and corrected all typo	2019-01-27 12:04:29 +00:00
Takeshi Nakatani	3b1cc3b197	Merge pull request #933 from gaul/cache/remove-mirror-path ... Remove mirror path when deleting cache	2019-01-27 16:15:49 +09:00
Takeshi Nakatani	a0c1f30ae7	Merge pull request #932 from gaul/autolock ... Prefer AutoLock for synchronization	2019-01-27 15:59:18 +09:00
Takeshi Nakatani	8822a86709	Merge pull request #931 from gaul/doc/typo ... Correct sigv2 typo	2019-01-27 15:48:24 +09:00
Takeshi Nakatani	98f397de0e	Merge pull request #930 from gaul/doc/md5-multipart ... Correct enable_content_md5 docs	2019-01-27 15:36:56 +09:00
Takeshi Nakatani	fd4d23f8f7	Merge pull request #926 from kzidane/master ... Accept paths with : in them	2019-01-27 15:23:24 +09:00
Takeshi Nakatani	4820f0a42b	Merge pull request #925 from gaul/clang-tidy/delete-null ... Remove redundant null checks before delete	2019-01-27 15:15:31 +09:00
Andrew Gaul	a93e500b44	Remove mirror path when deleting cache ... Fixes #827.	2019-01-25 18:10:03 -08:00
Andrew Gaul	92d3114584	Prefer AutoLock for synchronization ... This simplifies the code and fixes an issue with unlocked access. Also use a recursive lock for StatCache to avoid races between lock..unlock..lock sequences.	2019-01-25 15:28:41 -08:00
Andrew Gaul	5062d6fbd9	Correct sigv2 typo	2019-01-25 14:30:30 -08:00
Andrew Gaul	7d14ebaf09	Correct enable_content_md5 docs ... Both S3fsCurl::PutRequest and S3fsCurl::UploadMultipartPostSetup can calculate and send Content-MD5 to the server. Remove spurious comment about large files and make man page and help consistent. References #929.	2019-01-25 14:27:52 -08:00
Kareem Zidane	cd794a6985	Accept paths with : in them	2019-01-23 14:56:25 -05:00
Andrew Gaul	84b421d6ef	Prefer empty over size checks ... Found and fixed via clang-tidy.	2019-01-23 11:30:28 -08:00
Andrew Gaul	8316da5bbe	Remove redundant null checks before delete ... Found by clang-tidy.	2019-01-23 11:25:25 -08:00