octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-09-21 13:59:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,966 Commits 1 Branch 22 Tags 12 MiB
956eb77369
Commit Graph

1075 Commits

Author SHA1 Message Date
VVoidV
fcd180891b
fix misuse of IsUploading (#1747) 
Co-authored-by: liubingrun <liubr1@chinatelecom.cn>
 2021-08-29 23:41:02 +09:00
Takeshi Nakatani
a868c0656e Changed etaglist_t from string list to new structure etagpairs list 2021-08-16 09:27:12 +09:00
Andrew Gaul
15e89b78de
Add a partial page_list unit test (#1735) 2021-08-04 07:36:32 +09:00
Andrew Gaul
66006ba48d
Add dedicated upper- and lower-case hex functions (#1734) 
This makes the call sites more readable than a boolean parameter.
 2021-08-04 07:28:51 +09:00
Andrew Gaul
18e9c62087
Make string constants read-only const (#1733) 
This removes some global constructors.  Also use a consistent ALL_CAPS
style.
 2021-08-03 00:10:27 +09:00
Takeshi Nakatani
34f89e5936 Added fake_diskfree option to deceive free disk space for test 2021-08-02 11:04:42 +09:00
Carsten Grohmann
e1f3b9d8c1
Add support for AWS-style environment variables (#1729) 
Support AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN
in addition to the AWSACCESSKEYID, AWSSECRETACCESSKEY and
AWSSESSIONTOKEN.

The old environment variables are still supported, but they are
deprecated and no longer documented.

Close #1708
 2021-07-26 23:29:45 +09:00
Andrew Gaul
d3278f4886
Loosen CheckBucket to check only the bucket (#1728) 
Previously it checked if there was an object present, assuming that
this was a directory object normally created for s3fs directories.
However most S3 clients do not create this object for virtual folders.
Fixes #1460.  Fixes #1687.
 2021-07-25 12:29:00 +09:00
Andrew Gaul
77f0b75d2f
Add jitter to avoid thundering herd (#1723) 2021-07-25 10:22:19 +09:00
Takeshi Nakatani
199b3d4709 Fixed a bug in disk free space calculation 2021-07-18 14:52:11 +09:00
Andrew Gaul
7890989cbb
Address cppcheck 2.5 warnings (#1720) 
This slightly simplifies memory management.
 2021-07-14 22:18:09 +09:00
Takeshi Nakatani
945cc2ac54 Added UntreatedParts class instead of untreated upload info members in PseudoFdInfo 2021-07-12 21:45:29 +09:00
Takeshi Nakatani
c30acbbf90 Splited some methods of FdEntity class by uploading mode 
- Splited FdEntity::Write method by uploading mode
- Splited FdEntity::RowFlush method by uplading mode
 2021-07-10 17:24:34 +09:00
Martynov Maxim
881025cc9e
Add initiator pid, uid and gid to a debug log (#1716) 2021-07-10 16:23:20 +09:00
Takeshi Nakatani
4cc210c5ab Prevents the etag string buffer from being destroyed during use 
and absorbed MPPART_INFO structure into filepart structure
 2021-07-03 22:42:18 +09:00
Andrew Gaul
fe44355d25
Report s3fs space as 18.4 EB (#1709) 
256 TB probably seemed large in 2007 but in 2021 it is small enough to
confuse some users.
 2021-07-02 07:56:05 +09:00
Andrew Gaul
487df27008
Return EREMOTE when reading Glacier objects (#1705) 
Previously s3fs returned EIO which was not helpful to the caller.
Returning a more specific error code allows automatically restoring
the object via RestoreObject in a subsequent commit.
References #1466.
 2021-06-30 09:25:36 +09:00
Andrew Gaul
1965916f7a
Convert storage class values to uppercase (#1704) 
AWS requires uppercase values.  Fixes a regression from
bbcccd6e98.  References #1613.
 2021-06-30 09:03:31 +09:00
Andrew Gaul
8948eded09
Address clang-tidy warnings (#1703) 2021-06-30 08:42:44 +09:00
Andrew Gaul
2f59cb5a0a
Remove unneeded volatile qualifiers (#1702) 
We should use proper locking instead.  Specifically for
is_meta_pending this does not do what was intended due to the
read-modify-write of the member.
 2021-06-30 08:20:44 +09:00
Andrew Gaul
f505c8224e
Explicitly specify lock type (#1701) 
This makes it more clear and type-safe if the caller already has the
lock.  Follows on to 84174c560d.
 2021-06-27 15:15:48 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Takeshi Nakatani
f9e80f995d Fixed a bug about rename existing file of different sizes by mpcopy 2021-06-26 00:04:58 +09:00
Jan Stastny
a5c1915772
Set CURLOPT_UNRESTRICTED_AUTH when authenticating (#1681) 
This is necessary for authentication to work with AWS when it responds with 307 to the check bucket request when mounting. This happens to newly created buckets according to https://aws.amazon.com/premiumsupport/knowledge-center/s3-http-307-response/.

Prior to this, curl would follow the redirect, but would not include the `Authorization` header which would end up with `404`.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-06-25 20:26:12 +09:00
Takeshi Nakatani
6edf3d6427 Updated to clearly output message about the file descriptor type(pseudo/physical) 2021-06-25 19:18:08 +09:00
Andrew Gaul
20281737b1
Set explicit Content-Length: 0 when initiating MPU (#1696) 
This works around a GCS quirk that incorrectly requires this header.
Fixes #1661.
 2021-06-25 18:36:40 +09:00
Takeshi Nakatani
5b6684ca19 Fixed a bug utimens is calling before flush 2021-06-21 17:20:24 +09:00
Carsten Grohmann
c4ac923b4c
Ensure NUL-terminated result after strncpy (#1694) 
Long symlinks may cause that the result buffer is filled and not proper
terminated with a null byte.
 2021-06-21 08:08:56 +09:00
Takeshi Nakatani
cc022a68f4 Fixed a bug in s3fs_write 2021-06-20 23:13:44 +09:00
Takeshi Nakatani
858562ed53 Fixed comparison warning for size_t variable 2021-06-20 16:53:25 +09:00
Andrew Gaul
b8724425d3
Require explicit base parameter in cvt_strtoofft (#1683) 
Also convert most callers of cvt_strtoofft to base 10 which avoid the
magical behavior of interpreting a leading 0 as octal.
References #1682.
 2021-06-20 11:00:15 +09:00
Dmitrii Vasilev
b9ce0faee2 Set Content-MD5 header even for empty objects 
The Content-MD5 header is required for any request to upload an object
with a retention period configured using Amazon S3 Object Lock
 2021-06-19 16:10:58 +09:00
Carsten Grohmann
1e0e2752bf Improve warning for terminated threads 
with non-zero return code.
 2021-06-18 07:54:38 +09:00
Carsten Grohmann
7a488b93d0 Set decimal base for converting subseconds to int 
The leading 0 triggers an automatic conversion as an octal value. This
fails because it is a decimal value.

Setting the base to 10 prevents this automatism and treats the value
as a decimal value.

Fixes: [WAN] string_util.cpp:cvt_strtoofft(96): something error is occurred in convert std::string(017080564) to off_t, thus return 0 as default.

Related to #1676
 2021-06-17 06:27:02 +09:00
Carsten Grohmann
d67b83e671 Allow configuration for temporary files directory 2021-06-16 21:29:58 +09:00
Andrew Gaul
a100be9dce
Prefer std::string::empty over length == 0 (#1679) 
This is more concise.
 2021-06-13 20:03:10 +09:00
Andrew Gaul
4d39ea887e
Protect orgmeta with fdent_lock (#1678) 
Fixes a crash seen with the CentOS 7 builder and when running with
AddressSanitizer.  Regression introduced by
ac578d188e.  Fixes #1677.
 2021-06-13 16:14:24 +09:00
Andrew Gaul
7638b5b3e3
Prefer std::string::rbegin over operator[] (#1673) 
This is more concise and safer due to not repeating the variable name.
We cannot use std::string::back since it is not available in C++03.
 2021-06-13 13:26:38 +09:00
Andrew Gaul
600cee118d
Fix implicit narrowing conversions (#1672) 
These do not appear to be problematic but rather just clean up warnings.
Found via clang -Wshorten-64-to-32.
 2021-06-13 12:50:07 +09:00
Takeshi Nakatani
c2c56d0263 Added info object about multipart uploading for each pseudo fd 
(and fixed typo about method name)
 2021-06-04 22:42:58 +09:00
Takeshi Nakatani
ac578d188e Introduced pseudo fd and separated fd for each file opening 2021-06-04 22:42:58 +09:00
Andrew Gaul
53dfd48f59
Correct usage of istringstream (#1670) 
Previously this looped one more time than necessary due to the eof
check:

https://isocpp.org/wiki/faq/input-output#istream-and-eof

Remove now redundant empty check.
 2021-06-02 07:14:32 +09:00
Andrew Gaul
84174c560d
Fix data races caused by incorrect locking (#1668) 
Found via Threadsanitizer.  Fixes #1471.
 2021-05-29 00:11:55 +09:00
Carsten Grohmann
9bf525ee7a Ensuring multipart size even when storage is low 
When the temporary storage was full, the old implementation started an
upload even if there was not enough data to completely fill the minimum
multipart size or fill the user-selected multipart size.

The new implementation ensures the minimum multipart size by forcing
the user-selected multipart size.

Fixes #1591
 2021-05-27 22:55:52 +09:00
Takeshi Nakatani
4b69d4b1bb Fixed a bug when the disk capacity was insufficient in RowFlush 2021-05-22 23:11:46 +09:00
Andrew Gaul
bbcccd6e98
Make storage class a string (#1663) 
This allows non-standard storage classes like Google Cloud Storage
Nearline.  Fixes #1613.
 2021-05-21 23:34:31 +09:00
Takeshi Nakatani
b5fef788da Additional bug fixing for not creating zero-byte object 2021-05-09 17:33:53 +09:00
Andrew Gaul
42f5965d8a
Fix off_t to int narrowing in PageList::GetSparseFilePages (#1654) 
Found via clang-tidy.
 2021-05-09 14:35:00 +09:00
Andrew Gaul
d904d91252
Prefer std::map::insert over find and operator[] (#1653) 
This avoids a duplicate lookups.
 2021-05-09 14:11:35 +09:00
Andrew Gaul
9abe3fa662
Use same time when creating directory objects (#1652) 
This avoids mismatched times and is slightly faster.
 2021-05-09 13:44:12 +09:00
Andrew Gaul
bb6d2b1b74
Replace snprintf with string and ostringstream (#1649) 
These uses are probably safe from a buffer overflow perspective but
can cause data race issues in logging due to static buffers.
 2021-05-08 02:48:47 +09:00
Andrew Gaul
096a230b70
Allow arbitrary size AWS secret keys (#1648) 
Previously s3fs limited these to 123 characters.  Fixes #1626.
 2021-05-06 22:24:38 +09:00
Andrew Gaul
8ef01d37a9
Fix a few nits (#1645) 
Make some strings more const, initialize members, and abort if lock
initialization fails.  Partially found via clang-tidy.
 2021-05-06 19:40:35 +09:00
Andrew Gaul
cb9148f6cd
Update for clang-tidy 12 (#1644) 2021-05-03 09:43:18 +09:00
Andrew Gaul
771bbfeac5
Do not create zero-byte object when creating file (#1640) 
Previously s3fs created this object to store metadata and overwrote it
when flushing.  This prevented use with object stores which do not
allow overwrites like HDS.  Instead only create an in-memory
representation which reduces the time to create small files.
Fixes #1013.
 2021-04-30 19:56:33 +09:00
Andrew Gaul
3694786112
Propagate errno instead of EIO (#1638) 
This improves error fidelity.  Follows on to
b70f8db037.  References #1523.
 2021-04-30 07:09:00 +09:00
Andrew Gaul
fbf3c83019
Consider S3 errors in HTTP PUT 200 responses (#1635) 
S3 can emit these in unusual situations.  Fixes #1317.
 2021-04-25 13:18:11 +09:00
Andrew Gaul
8a51a26819 Allow truncation of open and modified files 
Regression introduced in f5bf41cf11.
Fixes #1575.
 2021-04-25 12:35:35 +09:00
Takeshi Nakatani
1838f52e19 Declare undefined symbols for fallocate function 2021-04-25 10:12:25 +09:00
Andrew Gaul
e9eb248f2f
Fail CheckBucket when S3 returns PermanentRedirect (#1630) 
Previously s3fs allowed mounting but all operations returned EIO.
References #693.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-04-24 18:46:24 +09:00
Ambika Nair
77581eda59 Change in ibm iam endpoint 2021-04-21 21:42:55 +09:00
Andrew Gaul
7f3e423bbe
Preserve sub-second time precision (#1624) 
Found via pjdfstests.  References #897.  References #1589.
 2021-04-18 13:11:12 +09:00
Andrew Gaul
706e3cbebd
Update ctime and mtime when opening with O_TRUNC (#1623) 
Found via pjdfstests.  References #1589.
 2021-04-15 07:08:49 +09:00
Takeshi Nakatani
2effffd8e2 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Takeshi Nakatani
2908878988 Fixed codes for cppcheck 2.4.1 2021-04-13 21:12:35 +09:00
Andrew Gaul
01a26a9011
Do not allow open with O_RDONLY and O_TRUNC (#1620) 
Found via pjdfstests.  References #1589.
 2021-04-13 07:28:10 +09:00
Andrew Gaul
23e1fbf7b9
Update ctime when truncating file (#1619) 
Found via pjdfstests.  References #1589.
 2021-04-13 07:08:06 +09:00
Andrew Gaul
ed2e877bb6
Return EACESS when lacking required executable bit (#1617) 
This makes the check consistent with read and write.  Found via
pjdfstests.  References #1551.  References #1589.
 2021-04-11 22:39:27 +09:00
Andrew Gaul
3663082a01
Do not allow renaming to a non-empty directory (#1604) 
Found via pjdfstests.  References #1589.
 2021-03-28 19:10:52 +09:00
Andrew Gaul
9645d57c05
Translate KeyTooLongError to ENAMETOOLONG (#1602) 
AWS does not support keys longer than 1024 characters.  Add special
handling for HeadObject which does not have a response body.
Found via pjdfstests.  References #1589.
 2021-03-28 13:17:41 +09:00
Andrew Gaul
f6fbd75320
Return EFBIG when file exceeds multipart limit (#1600) 
Found via pjdfstests.  References #1589.
 2021-03-28 12:33:01 +09:00
Carsten Grohmann
ef079f4e94 Don't ignore nomultipart when storage is low 
When the temporary storage filled up, the old implementation uploaded
all data with multipart uploads, even if "nomultipart" was set.

The new implementation emits a warning and returns -ENOSPC instead.

Fixes #1595
 2021-03-10 08:23:54 +09:00
Takeshi Nakatani
b589ebec23 Output version and command parameters at startup 2021-03-06 19:11:57 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583) 
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
 2021-02-23 09:45:13 +09:00
Takeshi Nakatani
5b5bc3114a Fixed a bug that padded null bytes when changing xattr 2021-02-21 12:23:36 +09:00
Takeshi Nakatani
81e267d421 Fixed the same bug in chmod/chown/utimens and added tests 2021-02-20 23:30:50 +09:00
Takeshi Nakatani
4fc92d59f3 Fixed a bug that filling NULL bytes when changing the attribute after renaming 2021-02-20 23:30:50 +09:00
Takeshi Nakatani
a0f347b10f Added no_time_stamp_ms option mainly for testing 2021-02-20 20:24:26 +09:00
Takeshi Nakatani
4da56acdcc Improved processing when HEAD response is 400 2021-02-13 17:00:13 +09:00
Takeshi Nakatani
493802a605 Fixed osx mistype declaration and typo warnings 2021-02-13 12:04:35 +09:00
林千里
22b0ae9d51 set IsExpireTime to true by default 
fixes #1563
 2021-02-12 12:20:47 +09:00
Andrew Gaul
134a54b32f
Allow configuring the multipart threshold (#1562) 
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 23:35:46 +09:00
Andrew Gaul
7f6fbb0021
Correct singlepart_copy_limit unit (#1561) 
This is documented to be in MB not KB.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 22:53:50 +09:00
Andrew Gaul
ec8bae9827
Do not periodically flush with nomultipartupload (#1559) 
s3fs periodically flushes data to S3 and hole-punches the written
data, marking it as clean.  Without multipart copy, s3fs would write
the now-zeroed data along with any new data.  s3fs should not attempt
to periodically flush dirty data at all without multipart copy
support.  Fixes #1542.
 2021-02-11 22:24:39 +09:00
Andrew Gaul
a4d916af13
Allow configuring multipart copy size (#1555) 
Also align configuration with mixupload.
 2021-02-08 20:32:12 +09:00
Takeshi Nakatani
32f096fa3f Use clock_gettime instead of gettimeofday 2021-02-08 10:19:28 +09:00
Andrew Gaul
c692093921
Copy at most 5 GB per multipart copy part request (#1553) 
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes.  This symptom was seen more frequently due
to periodic flushing of dirty data.  Fixes #1547.
 2021-02-07 23:10:07 +09:00
Andrew Gaul
1a6d0826b5
Allow -1 value to disable max_dirty_data (#1552) 
This matches the man page.
 2021-02-07 22:14:31 +09:00
Andrew Gaul
bf33fe7f55
Increase buffer size to avoid warning (#1550) 
This can be up to 54 bytes.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-07 12:26:58 +09:00
Andrew Gaul
ff9d6a75c7
Parse more specific AWS error codes (#1549) 
Fixes #1545.
 2021-02-07 11:29:08 +09:00
Carsten Grohmann
4c6690f5f0
Add timestamps to debug log and cache integrity check (#1540) 
* Add UTC timestamp to debug output

for easier correlation with external events

Old output:
$ s3fs mybucket /bucket/  -o curldbg,dbglevel=debug -d -d -f
[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from...
[INF]     s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=0, ...

New output:
$ s3fs mybucket /bucket/  -o curldbg,dbglevel=debug -d -d -f
2021-01-28T21:09:16.264Z [CRT] s3fs_logger.cpp:LowSetLogLevel(202):...
2021-01-28T21:09:16.264Z [INF]     s3fs.cpp:set_mountpoint_attrib...

* Add UTC timestamp to cache integrity check

for easier correlation with external events.

$ s3fs mybucket /mybucket  -oset_check_cache_sigusr1=/tmp/check.cache

Old output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
------------------------------------------------------------
Check cache file and its stats file consistency
------------------------------------------------------------
------------------------------------------------------------
Summary - Total files:                0
          Detected error files:       0
          Detected error directories: 0
------------------------------------------------------------

New output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
---------------------------------------------------------------------------
Check cache file and its stats file consistency at 2021-01-30T13:04:14.111Z
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Summary - Total files:                0
          Detected error files:       0
          Detected error directories: 0
---------------------------------------------------------------------------

* Fix indentation of S3fsLog::GetCurrentTime()
 2021-02-04 10:41:29 +09:00
Andrew Gaul
67b9381825 Remove limit for single-part objects 
AWS enforces a 5 GB limit for single-part objects but other S3
implementations like GCS support larger objects.  Fixes #1542.
 2021-02-04 09:19:37 +09:00
Andrew Gaul
4b53d4bf6b
Suppress spurious SHA256 error message (#1544) 2021-02-04 07:28:51 +09:00
Carsten Grohmann
ed85b72bf5 Add warning for failing metadata updates of large files 
Metadata updates fail for large files if "-o nocopyapi" or
"-o nomultipart" is set.

This change is related to #1528
 2021-01-28 10:20:32 +09:00
Andrew Gaul
17fda89ae9
Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors (#1534) 
Follows on to 4d833a4fb9.
 2021-01-25 19:08:14 +09:00
Andrew Gaul
d019dda4f7
Simplify substr manipulations with erase (#1532) 
This avoids creating a new std::string.
 2021-01-25 18:02:32 +09:00
Andrew Gaul
dc9255bc5f
Prefer static_cast where possible (#1531) 2021-01-25 08:15:17 +09:00
Andrew Gaul
b0e8758b63
Use result instead of res for consistency (#1530) 2021-01-25 07:56:10 +09:00
Andrew Gaul
b70f8db037
Propagate errno from get_local_fent (#1525) 
Follows on to 4d833a4fb9.
References #1523.
 2021-01-20 23:21:14 +09:00
Andrew Gaul
4d833a4fb9
Return more specific errno when available (#1520) 
Previously s3fs threw away some function return values and returned
EIO instead.  This was due to not trusting the mix of -1 and errno
return codes.  Correct the obviously incorrect ones via visual
inspection.  Stronger typing may find more occurrences.  Fixes #1519.
 2021-01-18 18:50:49 +09:00
fly3366
168e588ac7
fix: Add reset offset (#1503) 2021-01-04 22:57:56 +09:00
Andrew Gaul
bd0fadbe5f
Remove authorization header when remaking handle (#1505) 
This avoids including Authorization in SignedHeaders.  s3fs will
recreate the Authorization header before sending the request.
 2021-01-04 21:37:34 +09:00
kontrollanten
d1c638ab7a fix 404 error message 2020-12-30 14:10:17 +09:00
fly3366
51f65d7b14 fix: miss header when retry 2020-12-23 22:16:07 +09:00
Andrew Gaul
a16d00d673
Simply curl_slist_sort_insert (#1494) 
Some good taste from Linus:
https://github.com/mkirchner/linked-list-good-taste
Also avoid an allocation when replacing a value and tighten up tests.
 2020-12-23 20:29:33 +09:00
Andrew Gaul
d0363b118e
Add tests for curl_util (#1481) 2020-11-24 21:37:09 +09:00
Takeshi Nakatani
533322859d Added comment for nullPointerRedundantCheck/cppcheck2.2 2020-11-21 20:06:14 +09:00
Andrew Gaul
f8d5b76edb
Simplify handling of returned ETag (#1479) 
This works around lifetime warnings uncovered by cppcheck.
References #1478.
 2020-11-21 06:56:05 +09:00
Takeshi Nakatani
d9f6469b7b Fixed flushing dirty data and compressed the cache size 2020-11-14 16:45:37 +09:00
Andrew Gaul
543231c9f2
Address warnings from clang-tidy 11 (#1470) 2020-11-09 21:15:20 +09:00
Noah Meyerhans
d96a08d4ad fixup! Implement AWS IMDSv2 support 2020-11-07 14:48:52 +09:00
Noah Meyerhans
f2f930300a Implement AWS IMDSv2 support 
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2.  It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.

With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials.  In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode.  It can also be selected explicitly via
the imdsv1only mount option.

More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
 2020-11-07 14:48:52 +09:00
yoshihiko
81ad3ce0ae Fix directory authority 2020-11-07 14:47:53 +09:00
Takeshi Nakatani
38e1eaa8a3 Added logfile option for non-syslog logging 2020-10-19 20:10:19 +09:00
Andrew Gaul
6aa786b886
Flush dirty data after a writing a number of bytes (#1448) 
This allows s3fs to write large files without consuming a large amount
of temporary local storage but can slow uploads due to server-side
copies.  References #617.  Fixed #1056.  Fixes #1257.
 2020-10-13 22:30:42 +09:00
Takeshi Nakatani
58750cc441 Fixed a bug that symlink could not be read after restarting s3fs 2020-10-11 22:14:36 +09:00
Andrew Gaul
910255745e Allow relative path with use_path 
Fixes #1161.
 2020-10-10 18:13:23 +09:00
Takeshi Nakatani
cf86fa51b0
Merge pull request #1441 from gaul/coverity 
Suppress false positive from Coverity
 2020-10-04 23:44:51 +09:00
Takeshi Nakatani
0492f75197
Merge pull request #1439 from gaul/run-once 
Protect curl_warnings_once with a mutex
 2020-10-04 22:43:56 +09:00
Takeshi Nakatani
493cf20f95
Merge pull request #1435 from gaul/no-exceptions 
Remove exceptions from s3fs_strtoofft
 2020-10-04 22:19:25 +09:00
Andrew Gaul
3553fb65a0 Suppress false positive from Coverity 2020-10-04 14:58:13 +09:00
Takeshi Nakatani
059cc57ba6 Added atime and Corrected atime/mtime/ctime operations 2020-10-04 13:54:01 +09:00
Andrew Gaul
4df4ffe06f Protect curl_warnings_once with a mutex 
This allows ThreadSanitizer to run without warnings.
 2020-10-03 21:09:35 +09:00
Takeshi Nakatani
462347256d Changed the message level in the AutoFdEntity::GetFdEntity 2020-10-03 20:41:17 +09:00
Takeshi Nakatani
133feb67c3
Merge pull request #1434 from gaul/const-string 
Use const std::string& where possible
 2020-10-03 10:02:03 +09:00
Takeshi Nakatani
2cf195741c Fixed signature error due to case of hex string 2020-10-02 18:09:13 +00:00
Takeshi Nakatani
f61baada46
Merge pull request #1433 from gaul/hex 
Further simplify and centralize hex conversion
 2020-10-03 00:34:02 +09:00
Takeshi Nakatani
6e1e0d1d31
Merge pull request #1432 from gaul/v2-or-v4 
Add sigv4 flag and fix v4 regression
 2020-10-03 00:09:55 +09:00
Takeshi Nakatani
66419e7292
Merge pull request #1431 from gaul/string/find_first_of 
Prefer string::find_first_of(char) where possible
 2020-10-02 23:42:17 +09:00
Takeshi Nakatani
9e998877e9
Merge pull request #1430 from gaul/ostringstream 
Simplify use of ostringstream
 2020-10-02 23:26:41 +09:00
Andrew Gaul
3628b9d1e2 Remove exceptions from s3fs_strtoofft 
Explicit return codes are simpler and safer.
 2020-10-02 22:53:12 +09:00
Andrew Gaul
8d2bd874d7 Use const std::string& where possible 
This hints to callers on usage.
 2020-10-02 08:48:43 +09:00
Andrew Gaul
7e27c6cf7d Further simplify and centralize hex conversion 2020-10-01 23:31:06 +09:00
Andrew Gaul
b8ff4ede49 Sign GET requests with V4 signature correctly 
This bug was previously masked by v2 fallback.  Fixes regression from
81805715bd.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
081d6c1245 Allow setting signature V4-only 
Default to allowing V2 or V4.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
eb8004c355 Prefer string::find_first_of(char) where possible 2020-10-01 18:24:45 +09:00
Andrew Gaul
b3bf9f8f54 Simplify use of ostringstream 
Resetting this does not re-use the underlying buffer so creating a new
ostringstream has similar efficiency.
 2020-10-01 17:55:34 +09:00
Andrew Gaul
503c86bb8a Call is_prefix instead of compare and substr 2020-09-27 22:19:54 +09:00
Takeshi Nakatani
757f4caee8
Merge pull request #1425 from gaul/create-bucket/sse 
Do not send SSE headers during bucket creation
 2020-09-27 11:34:15 +09:00
Takeshi Nakatani
95fabd1f3a
Merge pull request #1423 from gaul/nettle 
Fix MD5 when using gnutls with nettle
 2020-09-27 11:17:55 +09:00
Takeshi Nakatani
0b42e08636
Merge pull request #1422 from gaul/c++/append-assign-at 
Remove calls to append, assign, and at
 2020-09-27 10:59:48 +09:00
Takeshi Nakatani
d5e4f99e72
Merge pull request #1413 from liuyongqing/master 
not call put headers if not exist pending meta
 2020-09-27 10:37:22 +09:00
Andrew Gaul
f35fe850c0 Do not send SSE headers during bucket creation 
This allows integration tests to pass with use_sse.
 2020-09-26 13:25:18 +09:00
Andrew Gaul
9a55c9fd9f Fix MD5 when using gnutls with nettle 
This matches the non-nettle definition of s3fs_md5_fd.
 2020-09-25 23:07:50 +09:00
Neeraj Kumar Kashyap
041b4ec05c enforce exact key match in IBM IAM response 2020-09-25 22:41:19 +09:00
Andrew Gaul
2438066d52 Remove calls to append, assign, and at 
operator+, operator=, and operator[] are more idiomatic and consistent
with the code base.
 2020-09-25 12:19:51 +09:00
yongqingliu
e98827ec6f not call put headers if not exist pending meta 2020-09-21 14:12:07 +08:00
Andrew Gaul
05863a3178 Fix clang-tidy warnings 2020-09-21 07:03:42 +09:00
Takeshi Nakatani
075d161bb1
Merge pull request #1408 from gaul/env 
Ensure environment variable is set when using ECS
 2020-09-20 13:01:17 +09:00
Takeshi Nakatani
6f40503328
Merge pull request #1406 from gaul/strcmp 
Remove unnecessary calls to strcmp(str.c_str())
 2020-09-20 11:28:41 +09:00