octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-06-09 04:12:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,181 Commits 1 Branch 22 Tags 12 MiB
master
Commit Graph

1221 Commits

Author SHA1 Message Date
Andrew Gaul
18e9c62087
Make string constants read-only const (#1733) 
This removes some global constructors.  Also use a consistent ALL_CAPS
style.
 2021-08-03 00:10:27 +09:00
Takeshi Nakatani
34f89e5936 Added fake_diskfree option to deceive free disk space for test 2021-08-02 11:04:42 +09:00
Carsten Grohmann
e1f3b9d8c1
Add support for AWS-style environment variables (#1729) 
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.

The old environment variables are still supported, but they are
deprecated and no longer documented.

Close #1708
 2021-07-26 23:29:45 +09:00
Andrew Gaul
d3278f4886
Loosen CheckBucket to check only the bucket (#1728) 
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460.  Fixes #1687.
 2021-07-25 12:29:00 +09:00
Andrew Gaul
77f0b75d2f
Add jitter to avoid thundering herd (#1723) 2021-07-25 10:22:19 +09:00
Takeshi Nakatani
199b3d4709 Fixed a bug in disk free space calculation 2021-07-18 14:52:11 +09:00
Andrew Gaul
7890989cbb
Address cppcheck 2.5 warnings (#1720) 
This slightly simplifies memory management.
 2021-07-14 22:18:09 +09:00
Takeshi Nakatani
945cc2ac54 Added UntreatedParts class instead of untreated upload info members in PseudoFdInfo 2021-07-12 21:45:29 +09:00
Takeshi Nakatani
c30acbbf90 Splited some methods of FdEntity class by uploading mode 
- Splited FdEntity::Write method by uploading mode
- Splited FdEntity::RowFlush method by uplading mode
 2021-07-10 17:24:34 +09:00
Martynov Maxim
881025cc9e
Add initiator pid, uid and gid to a debug log (#1716) 2021-07-10 16:23:20 +09:00
Takeshi Nakatani
4cc210c5ab Prevents the etag string buffer from being destroyed during use 
and absorbed MPPART_INFO structure into filepart structure
 2021-07-03 22:42:18 +09:00
Andrew Gaul
fe44355d25
Report s3fs space as 18.4 EB (#1709) 
256 TB probably seemed large in 2007 but in 2021 it is small enough to
confuse some users.
 2021-07-02 07:56:05 +09:00
Andrew Gaul
487df27008
Return EREMOTE when reading Glacier objects (#1705) 
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466.
 2021-06-30 09:25:36 +09:00
Andrew Gaul
1965916f7a
Convert storage class values to uppercase (#1704) 
AWS requires uppercase values.  Fixes a regression from
bbcccd6e98.  References #1613.
 2021-06-30 09:03:31 +09:00
Andrew Gaul
8948eded09
Address clang-tidy warnings (#1703) 2021-06-30 08:42:44 +09:00
Andrew Gaul
2f59cb5a0a
Remove unneeded volatile qualifiers (#1702) 
We should use proper locking instead.  Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
 2021-06-30 08:20:44 +09:00
Andrew Gaul
f505c8224e
Explicitly specify lock type (#1701) 
This makes it more clear and type-safe if the caller already has the
lock.  Follows on to 84174c560d.
 2021-06-27 15:15:48 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Takeshi Nakatani
f9e80f995d Fixed a bug about rename existing file of different sizes by mpcopy 2021-06-26 00:04:58 +09:00
Jan Stastny
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating (#1681) 
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/.

Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-06-25 20:26:12 +09:00
Takeshi Nakatani
6edf3d6427 Updated to clearly output message about the file descriptor type(pseudo/physical) 2021-06-25 19:18:08 +09:00
Andrew Gaul
20281737b1
Set explicit Content-Length: 0 when initiating MPU (#1696) 
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661.
 2021-06-25 18:36:40 +09:00
Takeshi Nakatani
5b6684ca19 Fixed a bug utimens is calling before flush 2021-06-21 17:20:24 +09:00
Carsten Grohmann
c4ac923b4c
Ensure NUL-terminated result after strncpy (#1694) 
Long symlinks may cause that the result buffer is filled and not proper
terminated with a null byte.
 2021-06-21 08:08:56 +09:00
Takeshi Nakatani
cc022a68f4 Fixed a bug in s3fs_write 2021-06-20 23:13:44 +09:00
Takeshi Nakatani
858562ed53 Fixed comparison warning for size_t variable 2021-06-20 16:53:25 +09:00
Andrew Gaul
b8724425d3
Require explicit base parameter in cvt_strtoofft (#1683) 
Also convert most callers of cvt_strtoofft to base 10 which avoid the
magical behavior of interpreting a leading 0 as octal.
References #1682.
 2021-06-20 11:00:15 +09:00
Dmitrii Vasilev
b9ce0faee2 Set Content-MD5 header even for empty objects 
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
 2021-06-19 16:10:58 +09:00
Carsten Grohmann
1e0e2752bf Improve warning for terminated threads 
with non-zero return code.
 2021-06-18 07:54:38 +09:00
Carsten Grohmann
7a488b93d0 Set decimal base for converting subseconds to int 
The leading 0 triggers an automatic conversion as an octal value. This
fails because it is a decimal value.

Setting the base to 10 prevents this automatism and treats the value
as a decimal value.

Fixes: [WAN] string_util.cpp:cvt_strtoofft(96): something error is occurred in convert std::string(017080564) to off_t, thus return 0 as default.

Related to #1676
 2021-06-17 06:27:02 +09:00
Carsten Grohmann
d67b83e671 Allow configuration for temporary files directory 2021-06-16 21:29:58 +09:00
Andrew Gaul
a100be9dce
Prefer std::string::empty over length == 0 (#1679) 
This is more concise.
 2021-06-13 20:03:10 +09:00
Andrew Gaul
4d39ea887e
Protect orgmeta with fdent_lock (#1678) 
Fixes a crash seen with the CentOS 7 builder and when running with
AddressSanitizer.  Regression introduced by
ac578d188e.  Fixes #1677.
 2021-06-13 16:14:24 +09:00
Andrew Gaul
7638b5b3e3
Prefer std::string::rbegin over operator[] (#1673) 
This is more concise and safer due to not repeating the variable name.
We cannot use std::string::back since it is not available in C++03.
 2021-06-13 13:26:38 +09:00
Andrew Gaul
600cee118d
Fix implicit narrowing conversions (#1672) 
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
 2021-06-13 12:50:07 +09:00
Takeshi Nakatani
c2c56d0263 Added info object about multipart uploading for each pseudo fd 
(and fixed typo about method name)
 2021-06-04 22:42:58 +09:00
Takeshi Nakatani
ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
Andrew Gaul
53dfd48f59
Correct usage of istringstream (#1670) 
Previously this looped one more time than necessary due to the eof
check:

https://isocpp.org/wiki/faq/input-output#istream-and-eof

Remove now redundant empty check.
 2021-06-02 07:14:32 +09:00
Andrew Gaul
84174c560d
Fix data races caused by incorrect locking (#1668) 
Found via Threadsanitizer.  Fixes #1471.
 2021-05-29 00:11:55 +09:00
Carsten Grohmann
9bf525ee7a Ensuring multipart size even when storage is low 
When the temporary storage was full, the old implementation started an
upload even if there was not enough data to completely fill the minimum
multipart size or fill the user-selected multipart size.

The new implementation ensures the minimum multipart size by forcing
the user-selected multipart size.

Fixes #1591
 2021-05-27 22:55:52 +09:00
Takeshi Nakatani
4b69d4b1bb Fixed a bug when the disk capacity was insufficient in RowFlush 2021-05-22 23:11:46 +09:00
Andrew Gaul
bbcccd6e98
Make storage class a string (#1663) 
This allows non-standard storage classes like Google Cloud Storage
Nearline.  Fixes #1613.
 2021-05-21 23:34:31 +09:00
Takeshi Nakatani
b5fef788da Additional bug fixing for not creating zero-byte object 2021-05-09 17:33:53 +09:00
Andrew Gaul
42f5965d8a
Fix off_t to int narrowing in PageList::GetSparseFilePages (#1654) 
Found via clang-tidy.
 2021-05-09 14:35:00 +09:00
Andrew Gaul
d904d91252
Prefer std::map::insert over find and operator[] (#1653) 
This avoids a duplicate lookups.
 2021-05-09 14:11:35 +09:00
Andrew Gaul
9abe3fa662
Use same time when creating directory objects (#1652) 
This avoids mismatched times and is slightly faster.
 2021-05-09 13:44:12 +09:00
Andrew Gaul
bb6d2b1b74
Replace snprintf with string and ostringstream (#1649) 
These uses are probably safe from a buffer overflow perspective but
can cause data race issues in logging due to static buffers.
 2021-05-08 02:48:47 +09:00
Andrew Gaul
096a230b70
Allow arbitrary size AWS secret keys (#1648) 
Previously s3fs limited these to 123 characters.  Fixes #1626.
 2021-05-06 22:24:38 +09:00
Andrew Gaul
8ef01d37a9
Fix a few nits (#1645) 
Make some strings more const, initialize members, and abort if lock
initialization fails.  Partially found via clang-tidy.
 2021-05-06 19:40:35 +09:00
Andrew Gaul
cb9148f6cd
Update for clang-tidy 12 (#1644) 2021-05-03 09:43:18 +09:00
Andrew Gaul
771bbfeac5
Do not create zero-byte object when creating file (#1640) 
Previously s3fs created this object to store metadata and overwrote it
when flushing.  This prevented use with object stores which do not
allow overwrites like HDS.  Instead only create an in-memory
representation which reduces the time to create small files.
Fixes #1013.
 2021-04-30 19:56:33 +09:00
Andrew Gaul
3694786112
Propagate errno instead of EIO (#1638) 
This improves error fidelity.  Follows on to
b70f8db037.  References #1523.
 2021-04-30 07:09:00 +09:00
Andrew Gaul
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses (#1635) 
S3 can emit these in unusual situations.  Fixes #1317.
 2021-04-25 13:18:11 +09:00
Andrew Gaul
8a51a26819 Allow truncation of open and modified files 
Regression introduced in f5bf41cf11.
Fixes #1575.
 2021-04-25 12:35:35 +09:00
Takeshi Nakatani
1838f52e19 Declare undefined symbols for fallocate function 2021-04-25 10:12:25 +09:00
Andrew Gaul
e9eb248f2f
Fail CheckBucket when S3 returns PermanentRedirect (#1630) 
Previously s3fs allowed mounting but all operations returned EIO.
References #693.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-04-24 18:46:24 +09:00
Ambika Nair
77581eda59 Change in ibm iam endpoint 2021-04-21 21:42:55 +09:00
Andrew Gaul
7f3e423bbe
Preserve sub-second time precision (#1624) 
Found via pjdfstests.  References #897.  References #1589.
 2021-04-18 13:11:12 +09:00
Andrew Gaul
706e3cbebd
Update ctime and mtime when opening with O_TRUNC (#1623) 
Found via pjdfstests.  References #1589.
 2021-04-15 07:08:49 +09:00
Takeshi Nakatani
2effffd8e2 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Takeshi Nakatani
2908878988 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Andrew Gaul
01a26a9011
Do not allow open with O_RDONLY and O_TRUNC (#1620) 
Found via pjdfstests.  References #1589.
 2021-04-13 07:28:10 +09:00
Andrew Gaul
23e1fbf7b9
Update ctime when truncating file (#1619) 
Found via pjdfstests.  References #1589.
 2021-04-13 07:08:06 +09:00
Andrew Gaul
ed2e877bb6
Return EACESS when lacking required executable bit (#1617) 
This makes the check consistent with read and write.  Found via
pjdfstests.  References #1551.  References #1589.
 2021-04-11 22:39:27 +09:00
Andrew Gaul
3663082a01
Do not allow renaming to a non-empty directory (#1604) 
Found via pjdfstests.  References #1589.
 2021-03-28 19:10:52 +09:00
Andrew Gaul
9645d57c05
Translate KeyTooLongError to ENAMETOOLONG (#1602) 
AWS does not support keys longer than 1024 characters.  Add special
handling for HeadObject which does not have a response body.
Found via pjdfstests.  References #1589.
 2021-03-28 13:17:41 +09:00
Andrew Gaul
f6fbd75320
Return EFBIG when file exceeds multipart limit (#1600) 
Found via pjdfstests.  References #1589.
 2021-03-28 12:33:01 +09:00
Carsten Grohmann
ef079f4e94 Don't ignore nomultipart when storage is low 
When the temporary storage filled up, the old implementation uploaded
all data with multipart uploads, even if "nomultipart" was set.

The new implementation emits a warning and returns -ENOSPC instead.

Fixes #1595
 2021-03-10 08:23:54 +09:00
Takeshi Nakatani
b589ebec23 Output version and command parameters at startup 2021-03-06 19:11:57 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583) 
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
 2021-02-23 09:45:13 +09:00
Takeshi Nakatani
5b5bc3114a Fixed a bug that padded null bytes when changing xattr 2021-02-21 12:23:36 +09:00
Takeshi Nakatani
81e267d421 Fixed the same bug in chmod/chown/utimens and added tests 2021-02-20 23:30:50 +09:00
Takeshi Nakatani
4fc92d59f3 Fixed a bug that filling NULL bytes when changing the attribute after renaming 2021-02-20 23:30:50 +09:00
Takeshi Nakatani
a0f347b10f Added no_time_stamp_ms option mainly for testing 2021-02-20 20:24:26 +09:00
Takeshi Nakatani
4da56acdcc Improved processing when HEAD response is 400 2021-02-13 17:00:13 +09:00
Takeshi Nakatani
493802a605 Fixed osx mistype declaration and typo warnings 2021-02-13 12:04:35 +09:00
林千里
22b0ae9d51 set IsExpireTime to true by default 
fixes #1563
 2021-02-12 12:20:47 +09:00
Andrew Gaul
134a54b32f
Allow configuring the multipart threshold (#1562) 
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 23:35:46 +09:00
Andrew Gaul
7f6fbb0021
Correct singlepart_copy_limit unit (#1561) 
This is documented to be in MB not KB.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 22:53:50 +09:00
Andrew Gaul
ec8bae9827
Do not periodically flush with nomultipartupload (#1559) 
s3fs periodically flushes data to S3 and hole-punches the written
data, marking it as clean.  Without multipart copy, s3fs would write
the now-zeroed data along with any new data.  s3fs should not attempt
to periodically flush dirty data at all without multipart copy
support.  Fixes #1542.
 2021-02-11 22:24:39 +09:00
Andrew Gaul
a4d916af13
Allow configuring multipart copy size (#1555) 
Also align configuration with mixupload.
 2021-02-08 20:32:12 +09:00
Takeshi Nakatani
32f096fa3f Use clock_gettime instead of gettimeofday 2021-02-08 10:19:28 +09:00
Andrew Gaul
c692093921
Copy at most 5 GB per multipart copy part request (#1553) 
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes.  This symptom was seen more frequently due
to periodic flushing of dirty data.  Fixes #1547.
 2021-02-07 23:10:07 +09:00
Andrew Gaul
1a6d0826b5
Allow -1 value to disable max_dirty_data (#1552) 
This matches the man page.
 2021-02-07 22:14:31 +09:00
Andrew Gaul
bf33fe7f55
Increase buffer size to avoid warning (#1550) 
This can be up to 54 bytes.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-07 12:26:58 +09:00
Andrew Gaul
ff9d6a75c7
Parse more specific AWS error codes (#1549) 
Fixes #1545.
 2021-02-07 11:29:08 +09:00
Carsten Grohmann
4c6690f5f0
Add timestamps to debug log and cache integrity check (#1540) 
* Add UTC timestamp to debug output

for easier correlation with external events

Old output:
$ s3fs mybucket /bucket/  -o curldbg,dbglevel=debug -d -d -f
[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from...
[INF]     s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=0, ...

New output:
$ s3fs mybucket /bucket/  -o curldbg,dbglevel=debug -d -d -f
2021-01-28T21:09:16.264Z [CRT] s3fs_logger.cpp:LowSetLogLevel(202):...
2021-01-28T21:09:16.264Z [INF]     s3fs.cpp:set_mountpoint_attrib...

* Add UTC timestamp to cache integrity check

for easier correlation with external events.

$ s3fs mybucket /mybucket  -oset_check_cache_sigusr1=/tmp/check.cache

Old output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
------------------------------------------------------------
Check cache file and its stats file consistency
------------------------------------------------------------
------------------------------------------------------------
Summary - Total files:                0
          Detected error files:       0
          Detected error directories: 0
------------------------------------------------------------

New output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
---------------------------------------------------------------------------
Check cache file and its stats file consistency at 2021-01-30T13:04:14.111Z
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Summary - Total files:                0
          Detected error files:       0
          Detected error directories: 0
---------------------------------------------------------------------------

* Fix indentation of S3fsLog::GetCurrentTime()
 2021-02-04 10:41:29 +09:00
Andrew Gaul
67b9381825 Remove limit for single-part objects 
AWS enforces a 5 GB limit for single-part objects but other S3
implementations like GCS support larger objects.  Fixes #1542.
 2021-02-04 09:19:37 +09:00
Andrew Gaul
4b53d4bf6b
Suppress spurious SHA256 error message (#1544) 2021-02-04 07:28:51 +09:00
Carsten Grohmann
ed85b72bf5 Add warning for failing metadata updates of large files 
Metadata updates fail for large files if "-o nocopyapi" or
"-o nomultipart" is set.

This change is related to #1528
 2021-01-28 10:20:32 +09:00
Andrew Gaul
17fda89ae9
Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors (#1534) 
Follows on to 4d833a4fb9.
 2021-01-25 19:08:14 +09:00
Andrew Gaul
d019dda4f7
Simplify substr manipulations with erase (#1532) 
This avoids creating a new std::string.
 2021-01-25 18:02:32 +09:00
Andrew Gaul
dc9255bc5f
Prefer static_cast where possible (#1531) 2021-01-25 08:15:17 +09:00
Andrew Gaul
b0e8758b63
Use result instead of res for consistency (#1530) 2021-01-25 07:56:10 +09:00
Andrew Gaul
b70f8db037
Propagate errno from get_local_fent (#1525) 
Follows on to 4d833a4fb9.
References #1523.
 2021-01-20 23:21:14 +09:00
Andrew Gaul
4d833a4fb9
Return more specific errno when available (#1520) 
Previously s3fs threw away some function return values and returned
EIO instead.  This was due to not trusting the mix of -1 and errno
return codes.  Correct the obviously incorrect ones via visual
inspection.  Stronger typing may find more occurrences.  Fixes #1519.
 2021-01-18 18:50:49 +09:00
fly3366
168e588ac7
fix: Add reset offset (#1503) 2021-01-04 22:57:56 +09:00
Andrew Gaul
bd0fadbe5f
Remove authorization header when remaking handle (#1505) 
This avoids including Authorization in SignedHeaders.  s3fs will
recreate the Authorization header before sending the request.
 2021-01-04 21:37:34 +09:00
kontrollanten
d1c638ab7a fix 404 error message 2020-12-30 14:10:17 +09:00
fly3366
51f65d7b14 fix: miss header when retry 2020-12-23 22:16:07 +09:00
Andrew Gaul
a16d00d673
Simply curl_slist_sort_insert (#1494) 
Some good taste from Linus:
https://github.com/mkirchner/linked-list-good-taste
Also avoid an allocation when replacing a value and tighten up tests.
 2020-12-23 20:29:33 +09:00
Andrew Gaul
d0363b118e
Add tests for curl_util (#1481) 2020-11-24 21:37:09 +09:00
Takeshi Nakatani
533322859d Added comment for nullPointerRedundantCheck/cppcheck2.2 2020-11-21 20:06:14 +09:00
Andrew Gaul
f8d5b76edb
Simplify handling of returned ETag (#1479) 
This works around lifetime warnings uncovered by cppcheck.
References #1478.
 2020-11-21 06:56:05 +09:00
Takeshi Nakatani
d9f6469b7b Fixed flushing dirty data and compressed the cache size 2020-11-14 16:45:37 +09:00
Andrew Gaul
543231c9f2
Address warnings from clang-tidy 11 (#1470) 2020-11-09 21:15:20 +09:00
Noah Meyerhans
d96a08d4ad fixup! Implement AWS IMDSv2 support 2020-11-07 14:48:52 +09:00
Noah Meyerhans
f2f930300a Implement AWS IMDSv2 support 
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2.  It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.

With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials.  In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode.  It can also be selected explicitly via
the imdsv1only mount option.

More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
 2020-11-07 14:48:52 +09:00
yoshihiko
81ad3ce0ae Fix directory authority 2020-11-07 14:47:53 +09:00
Takeshi Nakatani
38e1eaa8a3 Added logfile option for non-syslog logging 2020-10-19 20:10:19 +09:00
Andrew Gaul
6aa786b886
Flush dirty data after a writing a number of bytes (#1448) 
This allows s3fs to write large files without consuming a large amount
of temporary local storage but can slow uploads due to server-side
copies.  References #617.  Fixed #1056.  Fixes #1257.
 2020-10-13 22:30:42 +09:00
Takeshi Nakatani
58750cc441 Fixed a bug that symlink could not be read after restarting s3fs 2020-10-11 22:14:36 +09:00
Andrew Gaul
910255745e Allow relative path with use_path 
Fixes #1161.
 2020-10-10 18:13:23 +09:00
Takeshi Nakatani
cf86fa51b0
Merge pull request #1441 from gaul/coverity 
Suppress false positive from Coverity
 2020-10-04 23:44:51 +09:00
Takeshi Nakatani
0492f75197
Merge pull request #1439 from gaul/run-once 
Protect curl_warnings_once with a mutex
 2020-10-04 22:43:56 +09:00
Takeshi Nakatani
493cf20f95
Merge pull request #1435 from gaul/no-exceptions 
Remove exceptions from s3fs_strtoofft
 2020-10-04 22:19:25 +09:00
Andrew Gaul
3553fb65a0 Suppress false positive from Coverity 2020-10-04 14:58:13 +09:00
Takeshi Nakatani
059cc57ba6 Added atime and Corrected atime/mtime/ctime operations 2020-10-04 13:54:01 +09:00
Andrew Gaul
4df4ffe06f Protect curl_warnings_once with a mutex 
This allows ThreadSanitizer to run without warnings.
 2020-10-03 21:09:35 +09:00
Takeshi Nakatani
462347256d Changed the message level in the AutoFdEntity::GetFdEntity 2020-10-03 20:41:17 +09:00
Takeshi Nakatani
133feb67c3
Merge pull request #1434 from gaul/const-string 
Use const std::string& where possible
 2020-10-03 10:02:03 +09:00
Takeshi Nakatani
2cf195741c Fixed signature error due to case of hex string 2020-10-02 18:09:13 +00:00
Takeshi Nakatani
f61baada46
Merge pull request #1433 from gaul/hex 
Further simplify and centralize hex conversion
 2020-10-03 00:34:02 +09:00
Takeshi Nakatani
6e1e0d1d31
Merge pull request #1432 from gaul/v2-or-v4 
Add sigv4 flag and fix v4 regression
 2020-10-03 00:09:55 +09:00
Takeshi Nakatani
66419e7292
Merge pull request #1431 from gaul/string/find_first_of 
Prefer string::find_first_of(char) where possible
 2020-10-02 23:42:17 +09:00
Takeshi Nakatani
9e998877e9
Merge pull request #1430 from gaul/ostringstream 
Simplify use of ostringstream
 2020-10-02 23:26:41 +09:00
Andrew Gaul
3628b9d1e2 Remove exceptions from s3fs_strtoofft 
Explicit return codes are simpler and safer.
 2020-10-02 22:53:12 +09:00
Andrew Gaul
8d2bd874d7 Use const std::string& where possible 
This hints to callers on usage.
 2020-10-02 08:48:43 +09:00
Andrew Gaul
7e27c6cf7d Further simplify and centralize hex conversion 2020-10-01 23:31:06 +09:00
Andrew Gaul
b8ff4ede49 Sign GET requests with V4 signature correctly 
This bug was previously masked by v2 fallback.  Fixes regression from
81805715bd.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
081d6c1245 Allow setting signature V4-only 
Default to allowing V2 or V4.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
eb8004c355 Prefer string::find_first_of(char) where possible 2020-10-01 18:24:45 +09:00
Andrew Gaul
b3bf9f8f54 Simplify use of ostringstream 
Resetting this does not re-use the underlying buffer so creating a new
ostringstream has similar efficiency.
 2020-10-01 17:55:34 +09:00
Andrew Gaul
503c86bb8a Call is_prefix instead of compare and substr 2020-09-27 22:19:54 +09:00
Takeshi Nakatani
757f4caee8
Merge pull request #1425 from gaul/create-bucket/sse 
Do not send SSE headers during bucket creation
 2020-09-27 11:34:15 +09:00
Takeshi Nakatani
95fabd1f3a
Merge pull request #1423 from gaul/nettle 
Fix MD5 when using gnutls with nettle
 2020-09-27 11:17:55 +09:00
Takeshi Nakatani
0b42e08636
Merge pull request #1422 from gaul/c++/append-assign-at 
Remove calls to append, assign, and at
 2020-09-27 10:59:48 +09:00
Takeshi Nakatani
d5e4f99e72
Merge pull request #1413 from liuyongqing/master 
not call put headers if not exist pending meta
 2020-09-27 10:37:22 +09:00
Andrew Gaul
f35fe850c0 Do not send SSE headers during bucket creation 
This allows integration tests to pass with use_sse.
 2020-09-26 13:25:18 +09:00
Andrew Gaul
9a55c9fd9f Fix MD5 when using gnutls with nettle 
This matches the non-nettle definition of s3fs_md5_fd.
 2020-09-25 23:07:50 +09:00
Neeraj Kumar Kashyap
041b4ec05c enforce exact key match in IBM IAM response 2020-09-25 22:41:19 +09:00
Andrew Gaul
2438066d52 Remove calls to append, assign, and at 
operator+, operator=, and operator[] are more idiomatic and consistent
with the code base.
 2020-09-25 12:19:51 +09:00
yongqingliu
e98827ec6f not call put headers if not exist pending meta 2020-09-21 14:12:07 +08:00
Andrew Gaul
05863a3178 Fix clang-tidy warnings 2020-09-21 07:03:42 +09:00
Takeshi Nakatani
075d161bb1
Merge pull request #1408 from gaul/env 
Ensure environment variable is set when using ECS
 2020-09-20 13:01:17 +09:00
Takeshi Nakatani
6f40503328
Merge pull request #1406 from gaul/strcmp 
Remove unnecessary calls to strcmp(str.c_str())
 2020-09-20 11:28:41 +09:00
Takeshi Nakatani
3440c3348c
Merge pull request #1405 from gaul/is-prefix 
Rename STR2NCMP to is_prefix for clarity
 2020-09-20 10:03:53 +09:00
Takeshi Nakatani
853be26612
Merge pull request #1404 from liuyongqing/master 
fix dead lock in disk insufficient and optimize code
 2020-09-20 01:34:02 +09:00
Andrew Gaul
44d2cc15f7 Ensure environment variable is set when using ECS 
Previously s3fs crashed when AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
was not set.  References #1162.
 2020-09-18 08:14:58 +09:00
Andrew Gaul
81805715bd Remove unnecessary calls to strcmp(str.c_str()) 2020-09-17 20:00:45 +09:00
Andrew Gaul
ebcbb02d16 Rename STR2NCMP to is_prefix for clarity 
This may prevent confusion seen in
39ec8e242e.
 2020-09-17 19:45:54 +09:00
Takeshi Nakatani
93cf67c65d
Merge pull request #1399 from gaul/md5-sha256 
Use more descriptive names for digest functions
 2020-09-17 17:11:19 +09:00
yongqingliu
211cc0f5f2 fix dead lock in disk insufficient and optimize code 2020-09-16 22:45:28 +08:00
Takeshi Nakatani
c58c91fc4f
Merge pull request #1398 from gaul/memset 
Remove unneeded memset calls
 2020-09-15 22:09:54 +09:00
Andrew Gaul
b7187352e1 Use more descriptive names for digest functions 
Previously they used hex to refer to binary data which is confusing
since other std::string return types are actually hex.  Also remove
unused s3fs_md5sum.
 2020-09-15 21:18:58 +09:00
Takeshi Nakatani
1520ca6220
Merge pull request #1397 from gaul/hex 
Centralize hex conversion
 2020-09-15 21:14:23 +09:00
Takeshi Nakatani
f7a63d5c97
Merge pull request #1396 from gaul/32-bit-read 
Use off_t for read sizes
 2020-09-15 20:05:19 +09:00
Takeshi Nakatani
3958450c05 Added a class for automating fdentity reference counts 2020-09-14 22:02:28 +09:00
Andrew Gaul
5121c73ed1 Remove unneeded memset calls 
Also use constant for array lengths.
 2020-09-14 19:51:36 +09:00
Andrew Gaul
44eaac8471 Centralize hex conversion 2020-09-14 19:28:59 +09:00
Andrew Gaul
77501c3600 Use off_t for read sizes 
This fixes an issue with large multipart_size on 32-bit platforms like
Raspberry Pi.
 2020-09-14 18:12:23 +09:00
Takeshi Nakatani
8205607716 Fixed incorrect Signal related message at s3fs initialization and termination 2020-09-13 18:54:10 +09:00
Andrew Gaul
1043e08dfa Remove uses of implicit namespace std 
Fixed via:

sed -i '/using namespace std/{N;d}' src/*.cpp
sed -i 's/ string/ std::string/g' src/*.cpp
sed -i 's/(string/(std::string/g' src/*.cpp
sed -i 's/\[string/\[std::string/g' src/*.cpp
sed -i 's/^string/std::string/g' src/*.cpp
sed -i 's/ ifstream/ std::ifstream/g' src/*.cpp
sed -i 's/ istringstream/ std::istringstream/g' src/*.cpp
sed -i 's/ ostringstream/ std::ostringstream/g' src/*.cpp
sed -i 's/ max(/ std::max(/g' src/*.cpp
sed -i 's/ min(/ std::min(/g' src/*.cpp
sed -i 's/ endl/ std::endl/g' src/*.cpp
 2020-09-13 11:57:20 +09:00
Takeshi Nakatani
e5f4f9b69e
Merge pull request #1393 from gaul/32-bit-casting 
Fix casting warnings on 32-bit platforms
 2020-09-13 10:40:56 +09:00
Takeshi Nakatani
1e3c10d803
Merge pull request #1392 from gaul/large-singlepart-32-bit 
Use 64-bit off_t when computing digests
 2020-09-13 10:39:34 +09:00
Andrew Gaul
6d1e704e34 Fix casting warnings on 32-bit platforms 2020-09-12 18:22:32 +09:00
Andrew Gaul
ffc33a447f Use 64-bit off_t when computing digests 
This allows 32-bit platforms like Raspberry Pi to upload single-part
objects larger than 2 GB.
 2020-09-12 15:00:23 +09:00
Andrew Gaul
39ec8e242e Remove incorrect string prefix comparisons 
Previously flags like -o nouafoobar were parsed as -o noua.  Found
via:

grep STR2NCMP src/s3fs.cpp | grep -v '="'
 2020-09-11 17:38:20 +09:00
Ben Mares
0d4e39ad1c
return 0 after parsing "use_session_token" arg (#1388) 
return 0 after parsing "use_session_token" arg

I hope this might fix #651.

There's clearly an open bug in #651 confirmed by several users related to the parsing of the `-o use_session_token`. Looking at the source, I noticed that there was a return value everywhere except here, so I suspect this may be responsible.
 2020-09-11 08:09:36 +09:00
yongqingliu
986fab8738 close FdEntity to avoid leakage 2020-09-07 20:48:16 +08:00
Takeshi Nakatani
b5ffd419d8 Source file division and set 4 spaces and cleanup 2020-08-26 17:43:50 +09:00
Takeshi Nakatani
c6e23212bb
Merge pull request #1375 from gaul/cppcheck 
Add const where possible
 2020-08-21 07:32:39 +09:00
Andrew Gaul
1ec8528502 Add const where possible 
Found via cppcheck 2.1
 2020-08-20 23:46:11 +09:00
Andrew Gaul
892e7129c5 Add support for deep archive storage class 2020-08-20 23:42:45 +09:00
Takeshi Nakatani
bdea2ee5c8
Merge pull request #1373 from gaul/enum 
Use scoped enums for acl_t, sse_type_t, and storage_class_t
 2020-08-20 23:06:52 +09:00
Takeshi Nakatani
a5186c73c2
Merge pull request #1365 from ggtakec/check_lseek 
Dynamically determine whether lseek extended options are supported
 2020-08-20 22:36:56 +09:00
Andrew Gaul
60d456a993 Use scoped enums for acl_t, sse_type_t, and storage_class_t 
This prevents some kinds of implicit conversions.  Also deduplicate
str/from_str logic.  References #1371.
 2020-08-20 18:46:18 +09:00
Takeshi Nakatani
bcf6838e86
Merge pull request #1370 from gaul/vim/modeline 
Correct vim modeline to 2-space indentation
 2020-08-19 22:44:40 +09:00
Andrew Gaul
958ad83a4b Correct vim modeline to 2-space indentation 2020-08-19 21:03:46 +09:00
Andrew Gaul
ba61470bae Prefer named locks in curl 
This removes the confusing and unsafe pthread_mutex_t array.
 2020-08-18 22:20:41 +09:00
Andrew Gaul
8210a1b2f2 Consistently lock curl_times and curl_progress 
References #1362.
 2020-08-17 09:47:38 +09:00
Takeshi Nakatani
55cb8920d5 Dynamically determine whether lseek extended options are supported 2020-08-16 12:37:11 +00:00
Takeshi Nakatani
46acbf10ba
Merge pull request #1363 from gaul/is-modified-race 
Lock fdent_data_lock before accessing pagelist
 2020-08-16 20:25:24 +09:00
Takeshi Nakatani
2c0408b95a
Merge pull request #1357 from gaul/readdir 
Call readdir instead of readdir_r
 2020-08-16 18:50:38 +09:00
Takeshi Nakatani
057da86d87
Merge pull request #1356 from gaul/printf-size_t 
Prefer %zd and %zu for ssize_t and size_t
 2020-08-16 18:16:59 +09:00
Andrew Gaul
8de6cb3504 Lock fdent_data_lock before accessing pagelist 
Found via ThreadSanitizer.  References #1353.  References #1362.
 2020-08-16 17:44:03 +09:00
Takeshi Nakatani
2bb745cdd7 Fixed a bug about move file over limit of ensure space 2020-08-16 17:42:20 +09:00
Andrew Gaul
35090ba4d5 Call readdir instead of readdir_r 
Only a single thread uses this directory stream.  Further, modern
implementations are thread-safe by default and deprecated this call:

https://man7.org/linux/man-pages/man3/readdir_r.3.html
 2020-08-16 13:33:23 +09:00
Andrew Gaul
132a1bebbb Prefer %zd and %zu for ssize_t and size_t 
This removes unnecessary casting.
 2020-08-15 17:30:23 +09:00
Takeshi Nakatani
cbd925c56f Moved the SIGUSR2 handler to S3fsSignals class 2020-07-28 14:54:35 +09:00
Takeshi Nakatani
63bbb47378
Merge pull request #1341 from gaul/stat-cache/default 
Change default stat_cache_expire
 2020-07-26 23:54:14 +09:00
Andrew Gaul
0fbd0eac80 Change default stat_cache_expire 
Previously s3fs cached files forever which confused users with
creating objects using another client.
 2020-07-26 23:04:43 +09:00
Takeshi Nakatani
ad1961417d Added SIGUSR1 option for cache file integrity test 2020-07-26 21:04:11 +09:00
Takeshi Nakatani
700e288718 Put similar processing together into method GetCacheFileStatTopDir 2020-07-12 08:12:02 +00:00
Takeshi Nakatani
dc365b65a0 Fix NULL pointer deference 
Found via cppcheck 2.1.
 2020-07-09 21:40:23 +09:00
Takeshi Nakatani
1f796d432d Fixed upload error about mixuploading sparse file and truncating file 2020-06-27 22:44:19 +09:00
Takeshi Nakatani
35006e318f Fixed about ParallelMixMultipartUpload 2020-06-24 12:48:55 +09:00
Eric Vantillard
7d0c66e08a Add support for glacier storage class. 
Just a copy of what have been done in PR #271.
 2020-06-23 11:23:21 +09:00
Takeshi Nakatani
f324d8e04f Fixed a bug about serializing from cache file 2020-06-19 12:57:27 +00:00
Takeshi Nakatani
f16ee96d7e
Merge pull request #1306 from gaul/http/500 
Retry with exponential backoff during 500 error
 2020-06-06 15:30:22 +09:00
Andrew Gaul
d8766b2051 Retry with exponential backoff during 500 error 
Amazon suggests retrying on both 500 and 503:

https://aws.amazon.com/premiumsupport/knowledge-center/http-5xx-errors-s3/

Fixes #1251.
 2020-06-05 21:01:30 +09:00
Andrew Gaul
9db70bab63 Ignore case when comparing ETags 
This allows multipart upload to work with Alibaba OSS.
References #1297.
 2020-06-05 18:17:52 +09:00
Takeshi Nakatani
0cb057dadd
Merge pull request #1303 from gaul/rename/use_cache 
Relink cache stats file atomically via rename
 2020-06-01 00:10:33 +09:00
Takeshi Nakatani
0f5db0d1bf
Merge pull request #1302 from gaul/rename/nocopy 
Fix renames of open files with nocopyapi option
 2020-05-31 23:46:46 +09:00
Andrew Gaul
274321524c Relink cache stats file atomically via rename 
The new file may already exist so link may fail.  Further link/unlink
is not atomic.  Addresses an error when renaming an open with with
use_cache.  References #1296.
 2020-05-31 23:09:58 +09:00
Andrew Gaul
40f7007263 Check results from pthread mutex calls 
Also remove some unnecessary exception handling.
 2020-05-30 16:37:55 +09:00
Andrew Gaul
66597ec5f2 Fix renames of open files with nocopyapi option 
References #1296.
 2020-05-30 15:45:43 +09:00
Takeshi Nakatani
75e72385cc Added a parameter to output body to curldbg option 2020-05-25 08:49:01 +09:00
Andrew Gaul
8b15db6dcb Do not allow zero retries 
Retries actually means tries, e.g., if the user sets zero, s3fs will
never try an operation at all.
 2020-05-23 10:05:23 +09:00
Takeshi Nakatani
80c11b6c12 Not abort process by exception threw from s3fs_strtoofft 2020-05-03 13:46:05 +00:00
Andrew Gaul
8945e98d8b Support Google Cloud Storage headers 
This allows s3fs to interpret objects created by gsutil.
 2020-05-03 18:33:13 +09:00
Takeshi Nakatani
97c249d5b9 Not abort process by exception threw from s3fs_strtoofft 2020-05-03 08:08:28 +00:00
Andrew Gaul
2cc88b933f Warn about missing MIME types instead of exiting 
s3fs uses the MIME types file to set Content-Type for uploaded
objects.  Most distribution packages should install this via
recommended (not required) dependencies.  Users compiling from source
may not have this installed and s3fs should not prevent launching
since most users do not care about Content-Type.  Instead warn about
MIME types absence.  Fixes #1270.
 2020-04-29 20:03:50 +09:00
Andrew Gaul
005a684600 Fix typos 2020-04-22 21:49:11 +09:00
Takeshi Nakatani
f26a0aa71d Fixed insufficient upload size for mix multipart upload 2020-04-22 09:31:22 +09:00
Andrew Gaul
cf529e0af7 Add handler for HTTP 416 
This prevents retries when the server indicates an unsatisfiable MPU
copy part request.  References #1220.
 2020-04-21 19:45:10 +09:00
Takeshi Nakatani
4da02d023b Improved strictness of cache file stats(file) 2020-04-21 19:45:03 +09:00
Takeshi Nakatani
811ea0cb85 Fixed the truncation bug of stat file for cache file 2020-04-19 07:08:49 +00:00
Andrew Gaul
48a872e285 Address cppcheck 1.90 warning 2020-04-12 22:20:44 +09:00
Takeshi Nakatani
c44a60f3f5 Fixed a bug of stats cache compression 2020-04-12 18:33:00 +09:00
Takeshi Nakatani
9e01d5b8d1
Merge pull request #1254 from ggtakec/modify_mimetypes 
Added mime option for strict checking of mime types file
 2020-04-11 14:48:47 +09:00
Takeshi Nakatani
7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
Andrew Gaul
56141557dc Avoid unneeded string copy 
Found by clang-tidy 10.
 2020-03-28 08:49:49 +09:00
yongqingliu
b72f4b43a4 use correct content-type when complete multipart upload 2020-02-10 16:58:28 +09:00
Andrew Gaul
924eeb3587 Document host and servicepath 
The various bits of host, url, and servicepath seem to overlap.
References #1203.
 2020-02-04 21:13:29 +09:00
Andrew Gaul
bc9126d774 Set directory MIME type to application/x-directory 
Previously s3fs auto-detected the MIME type of directories like
"TOYOTA TRUCK 8.2.2" as application/x-troff-man.  This caused get_mode
to not set S_IFDIR which failed directory creation.  Instead force all
object names ending in / to application/x-directory.  Fixes #1183.
 2020-02-04 20:03:21 +09:00
rallister
4e26728cbf
break recursion when calling GetIAMCredentials (#1233) 
break recursion when calling GetIAMCredentials
 2020-01-31 16:48:37 +09:00
Takeshi Nakatani
018ccb9a11 Suppressed a lot of message output about cache cleanup 2020-01-30 14:42:28 +00:00
yongqingliu
b762a0a85b fix deadlock due to fdmanager_lock and fdent_data_lock 2020-01-29 12:03:53 +08:00
Andrew Gaul
e0712f444d Update source for clang-tidy 9.0.0 2020-01-13 20:56:45 +09:00
Andrei Precup
39102608aa Add intelligent tiering as option for storage class 
Resolves Issue #1219
 2020-01-13 20:24:35 +09:00
Takeshi Nakatani
23945a0130 In memory cache for softlinks with cache out 2020-01-13 20:23:10 +09:00
Alvaro Huarte
bdfb9ee815 Document requester_pays option 2019-11-25 22:53:50 +09:00
Alvaro Huarte
a24f78f5a4 Adds requester_pays option to enable requests involving Requester Pays buckets 2019-11-18 12:38:16 +01:00
Takeshi Nakatani
a07a5336f6
Merge pull request #1185 from bryceml/master 
handle edge case when bucket name and first part of endpoint match
 2019-10-31 23:42:47 +09:00
Bryce Larson
0c42a74a8a
handle edge case when bucket name and first part of endpoint match 2019-10-26 00:51:19 -06:00
Takeshi Nakatani
cf3e82d10a Fixed a rename bug when enable_noobj_cache 2019-10-23 12:39:42 +00:00
Takeshi Nakatani
1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00
Takeshi Nakatani
bedd648d47 Fixed build error by cppcheck 1.89 2019-09-23 10:49:49 +00:00
Takeshi Nakatani
58b3cce320
Merge pull request #1066 from gaul/sign-requests 
Sign request immediately before sending
 2019-09-23 15:58:21 +09:00
Takeshi Nakatani
81102a5963
Merge pull request #1149 from ggtakec/fix_retry 
Fixed a bug that type and op are not inherited
 2019-09-08 20:32:43 +09:00
Takeshi Nakatani
42fb30852b
Merge pull request #1145 from gaul/bool 
Fix a few bool types
 2019-09-08 20:04:30 +09:00
Takeshi Nakatani
e51361cb94
Merge pull request #1143 from gaul/doc/umask 
Document umask flag
 2019-09-08 19:49:58 +09:00
Takeshi Nakatani
e2129001eb Fixed a bug that type and op are not inherited 2019-09-08 09:59:04 +00:00
Andrew Gaul
805cc064af Fix a few bool types 
Found via clang-tidy.
 2019-09-05 10:42:42 -07:00
Andrew Gaul
3c2279db39 Document umask flag 2019-09-04 20:53:58 -07:00
Andrew Gaul
412876ca33 Expose base parameter in s3fs_strtoofft 
This fixes a regression from ccf3e7bfa2
which caused the misparsing of octal inputs for the mp_umask and umask
flags.  It also allows some callers to be more precise about their
decimal inputs.
 2019-09-04 20:41:47 -07:00
Andrew Gaul
461a346bf4 Sign request immediately before sending 
Previously s3fs could create a long list of pre-signed requests which
could take longer than the default S3 clock skew limit of 15 minutes.
This also changes SHA-256 computation from single- to multi-threaded
since this is now computed in the worker threads.  Regression from
88cd8feb05.  Fixes #1019.
 2019-08-29 15:35:27 -07:00
liuyongqing
ae4bcd405c use correct content-type when overwriting and copying files 2019-08-28 22:25:09 -07:00
Takeshi Nakatani
0536dc1112 Refixed strict processing of MultiRead method 2019-08-25 20:46:01 -07:00
Takeshi Nakatani
1c3507ede1
Merge pull request #1135 from gaul/curl-timeout 
Remove useless CURLE_OPERATION_TIMEDOUT check
 2019-08-25 17:53:02 +09:00
Andrew Gaul
01ac815346 Remove useless CURLE_OPERATION_TIMEDOUT check 
CheckBucket only returns errno and HTTP status codes.
 2019-08-20 18:44:48 -07:00
Andrew Gaul
cc2eed84a5 Promote default ACL to enum 
This sanity checks ACLs during initialization and also omits sending
the header when set to PRIVATE which is the default.
 2019-08-20 15:05:58 -07:00
Takeshi Nakatani
873e376098 Strict processing of MultiRead method 2019-08-13 15:23:13 +00:00
Takeshi Nakatani
4a0c23258e Fixed MultiRead method to exit after waiting other parts at error 2019-08-12 14:45:06 -07:00
Takeshi Nakatani
5c4a0a862a Fixed multipart copy and its retry handler 2019-08-11 16:21:26 +00:00
Takeshi Nakatani
f5bf41cf11 Fixed simple copy with nocpoyapi and modify flag in fdcache 2019-08-11 07:42:48 +00:00
Takeshi Nakatani
f74c7407db
Merge pull request #1118 from gaul/atoi 
Prefer s3fs_strtoofft over atoi and strtol
 2019-08-06 21:23:37 +09:00
Takeshi Nakatani
433c04af26
Merge pull request #1116 from gaul/bodydata 
Do not heap allocate BodyData
 2019-08-06 21:08:02 +09:00
Takeshi Nakatani
2e51a339a9
Merge pull request #1115 from gaul/hash-length 
Use length from s3fs_HMAC256 for consistency
 2019-08-06 20:49:22 +09:00
Takeshi Nakatani
14d3e12103
Merge pull request #1114 from gaul/flush/return-value 
Consume return value from FdEntity::Flush
 2019-08-06 14:12:25 +09:00
Andrew Gaul
ccf3e7bfa2 Prefer s3fs_strtoofft over atoi and strtol 
The former propagates errors consistently.
 2019-08-03 16:13:48 -07:00
Andrew Gaul
7ecfba811e Do not heap allocate BodyData 
Callers already manage lifetimes via Clear().
 2019-08-03 15:50:46 -07:00
Takeshi Nakatani
1e7330e499
Merge pull request #1113 from gaul/unused 
Remove unused field
 2019-08-04 01:31:56 +09:00
Takeshi Nakatani
3d42d0515d
Merge pull request #1109 from gaul/lock-already-held 
Pass lock_already_held state to DelStat
 2019-08-03 22:13:22 +09:00
Takeshi Nakatani
3d70e8966f
Merge pull request #1108 from gaul/lock/diskspace 
Always hold lock when accessing free_disk_space
 2019-08-03 22:01:31 +09:00
Andrew Gaul
a5c20175a1 Use length from s3fs_HMAC256 for consistency 
EVP_MAX_MD_SIZE can range up to 64.  Found by Coverity.
 2019-08-02 10:40:55 -07:00
Andrew Gaul
43d1439420 Consume return value from FdEntity::Flush 
Found via Coverity.
 2019-08-02 10:01:06 -07:00
Andrew Gaul
d8cf26bd50 Remove unused field 
Found via clang.
 2019-08-01 16:42:38 -07:00
Andrew Gaul
f528a86219 Pass lock_already_held state to DelStat 2019-08-01 11:07:56 -07:00
Andrew Gaul
5b15c7c4e9 Always hold lock when accessing free_disk_space 
Slightly reorder locks to avoid double locking.  Found via
ThreadSanitizer.
 2019-07-30 08:36:54 -07:00
Andrew Gaul
afd438d363 Access FdEntity metadata while holding fdent_lock 
Create copies to avoid thread races.  Found via ThreadSanitizer.
Follows on to ecf13a8cb9.
 2019-07-29 13:54:57 -07:00
Andrew Gaul
fb937635f5 Eagerly initialize sysconf variables 
Previously s3fs had races updating these shared variables.  Found via
ThreadSanitizer.
 2019-07-17 09:08:13 -07:00
Takeshi Nakatani
3ad1c95e86
Merge pull request #1103 from gaul/config/readwrite-timeout 
Use consistent default for readwrite_timeout
 2019-07-17 22:37:02 +09:00
Andrew Gaul
bacd15714a Use consistent default for readwrite_timeout 
Previously curl threads mutated this shared state without a lock.
Found via ThreadSanitizer.
 2019-07-16 22:03:56 -07:00
Andrew Gaul
99aace4fc9 Avoid race when using thread-unsafe gmtime 
Found via ThreadSanitizer.
 2019-07-16 07:28:34 -07:00
Takeshi Nakatani
513f41fddf
Merge pull request #1097 from ggtakec/fix_lock 
Add Lock() func parameter in RowFlush()
 2019-07-15 11:56:21 +09:00
Takeshi Nakatani
543aed2a32
Merge pull request #1096 from gaul/pthread/lock-checking 
Expand error checking to all pthread_mutex_t
 2019-07-15 11:43:56 +09:00
Takeshi Nakatani
20ea96328c
Merge pull request #1095 from gaul/intmax_t 
Avoid narrowing time_t and off_t args in logging
 2019-07-15 11:33:44 +09:00
Takeshi Nakatani
f78bcc5229 Add Lock() func parameter in RowFlush() 2019-07-15 01:52:29 +00:00
Andrew Gaul
43ec064fb9 Expand error checking to all pthread_mutex_t 2019-07-14 18:21:23 -07:00
Andrew Gaul
ffac4c8417 Avoid narrowing time_t and off_t args in logging 
This displays correct results on 32-bit platforms.
 2019-07-14 17:02:36 -07:00
Andrew Gaul
4adcd4a6c8 Implement s3fs_strtoofft via strtoll 
This tightens error checking and aligns s3fs with known good behavior.
 2019-07-14 15:15:06 -07:00
Takeshi Nakatani
e936854493
Merge pull request #1094 from larsch/master 
Fix multipart upload not setting curl options
 2019-07-15 05:13:37 +09:00
Takeshi Nakatani
850a813171
Merge pull request #1092 from liuyongqing/master 
fix coredump caused by fd_manager_lock locking
 2019-07-15 04:53:10 +09:00
Takeshi Nakatani
5bbcd3b981
Merge pull request #1089 from gaul/null-dereference 
Avoid null dereference
 2019-07-15 04:17:10 +09:00
Takeshi Nakatani
a337c32840
Merge pull request #1088 from gaul/clang-tidy/misplaced-const 
Avoid misplaced const via removing unneeded typedef
 2019-07-15 04:08:51 +09:00
Takeshi Nakatani
d39e4e4b1f
Merge pull request #1087 from gaul/clang-tidy/deprecated-headers 
Prefer modern C headers
 2019-07-15 03:50:15 +09:00
Lars Christensen
58037da061 Fix multipart upload not setting curl options 2019-07-14 14:09:37 +02:00
Andrew Gaul
1eb266588e Add a missing lock to SetCtime 
Also lock before log statements which touch member variables.
 2019-07-13 17:53:38 -07:00
root
4e351c59e3 fix coredump caused by fd_manager_lock locking 2019-07-13 12:03:27 +08:00
Andrew Gaul
eb597289cb Avoid null dereference 
Found via clang-tidy.
 2019-07-12 18:40:24 -07:00
Andrew Gaul
6fd42d9fe4 Prefer modern C headers 
Found and fixed via clang-tidy.
 2019-07-12 03:50:59 -07:00
Andrew Gaul
efff9c01a6 Avoid misplaced const via removing unneeded typedef 
Found via clang-tidy.
 2019-07-12 03:41:19 -07:00
Andrew Gaul
a83d5baa90 Remove recursive locking 
Recursive locking is frowned upon and is incompatible with
PTHREAD_MUTEX_ERRORCHECK.  Also clean up pthread_mutex_lock error
checking.
 2019-07-10 12:39:00 -07:00
Takeshi Nakatani
50d13255e4
Merge pull request #1081 from gaul/write-blocks-readdir 
Break FdEntity locks into data and metadata
 2019-07-11 03:40:26 +09:00
Takeshi Nakatani
e5e63d6ac3
Merge pull request #1067 from gaul/mismatched-free 
Fix mismatched new[]/free
 2019-07-07 16:55:07 +09:00
Takeshi Nakatani
7a65a414c3
Merge pull request #1080 from gaul/xml/simplification 
Add simple XML parsing wrapper
 2019-07-07 16:44:18 +09:00
Andrew Gaul
944d21cabb Fix mismatched new[]/free 
Found via Valgrind.
 2019-07-07 00:36:48 -07:00
Takeshi Nakatani
d267212289
Merge pull request #1079 from gaul/cppcheck/shadowing 
Avoid shadowing variable in FdEntity::Open
 2019-07-07 16:29:42 +09:00