Andrew Gaul
a4d916af13
Allow configuring multipart copy size ( #1555 )
...
Also align configuration with mixupload.
2021-02-08 20:32:12 +09:00
Takeshi Nakatani
32f096fa3f
Use clock_gettime instead of gettimeofday
2021-02-08 10:19:28 +09:00
Andrew Gaul
c692093921
Copy at most 5 GB per multipart copy part request ( #1553 )
...
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes. This symptom was seen more frequently due
to periodic flushing of dirty data. Fixes #1547 .
2021-02-07 23:10:07 +09:00
Andrew Gaul
1a6d0826b5
Allow -1 value to disable max_dirty_data ( #1552 )
...
This matches the man page.
2021-02-07 22:14:31 +09:00
Andrew Gaul
f2c5e38724
Allow integration tests to use larger files ( #1548 )
...
Previously these failed due to dd only copying 32 MB from /dev/urandom
and exhausting the limited space in /tmp. References #1543 .
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-07 12:50:02 +09:00
Andrew Gaul
bf33fe7f55
Increase buffer size to avoid warning ( #1550 )
...
This can be up to 54 bytes.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-07 12:26:58 +09:00
Andrew Gaul
ff9d6a75c7
Parse more specific AWS error codes ( #1549 )
...
Fixes #1545 .
2021-02-07 11:29:08 +09:00
Carsten Grohmann
4c6690f5f0
Add timestamps to debug log and cache integrity check ( #1540 )
...
* Add UTC timestamp to debug output
for easier correlation with external events
Old output:
$ s3fs mybucket /bucket/ -o curldbg,dbglevel=debug -d -d -f
[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from...
[INF] s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=0, ...
New output:
$ s3fs mybucket /bucket/ -o curldbg,dbglevel=debug -d -d -f
2021-01-28T21:09:16.264Z [CRT] s3fs_logger.cpp:LowSetLogLevel(202):...
2021-01-28T21:09:16.264Z [INF] s3fs.cpp:set_mountpoint_attrib...
* Add UTC timestamp to cache integrity check
for easier correlation with external events.
$ s3fs mybucket /mybucket -oset_check_cache_sigusr1=/tmp/check.cache
Old output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
------------------------------------------------------------
Check cache file and its stats file consistency
------------------------------------------------------------
------------------------------------------------------------
Summary - Total files: 0
Detected error files: 0
Detected error directories: 0
------------------------------------------------------------
New output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
---------------------------------------------------------------------------
Check cache file and its stats file consistency at 2021-01-30T13:04:14.111Z
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Summary - Total files: 0
Detected error files: 0
Detected error directories: 0
---------------------------------------------------------------------------
* Fix indentation of S3fsLog::GetCurrentTime()
2021-02-04 10:41:29 +09:00
Andrew Gaul
67b9381825
Remove limit for single-part objects
...
AWS enforces a 5 GB limit for single-part objects but other S3
implementations like GCS support larger objects. Fixes #1542 .
2021-02-04 09:19:37 +09:00
Andrew Gaul
4b53d4bf6b
Suppress spurious SHA256 error message ( #1544 )
2021-02-04 07:28:51 +09:00
Carsten Grohmann
ed85b72bf5
Add warning for failing metadata updates of large files
...
Metadata updates fail for large files if "-o nocopyapi" or
"-o nomultipart" is set.
This change is related to #1528
2021-01-28 10:20:32 +09:00
Andrew Gaul
17fda89ae9
Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors ( #1534 )
...
Follows on to 4d833a4fb9
.
2021-01-25 19:08:14 +09:00
Andrew Gaul
1987bcbea3
Re-fix retry logic ( #1533 )
...
This logic has always been broken but happened to try one time due to
&& operator precedence. 7158e50ee2
broke this further when quoting && since the command was not
evaluated.
2021-01-25 18:32:40 +09:00
Andrew Gaul
d019dda4f7
Simplify substr manipulations with erase ( #1532 )
...
This avoids creating a new std::string.
2021-01-25 18:02:32 +09:00
Andrew Gaul
dc9255bc5f
Prefer static_cast where possible ( #1531 )
2021-01-25 08:15:17 +09:00
Andrew Gaul
b0e8758b63
Use result instead of res for consistency ( #1530 )
2021-01-25 07:56:10 +09:00
Takeshi Nakatani
6d65e30dd5
Fixed brew install with cask in github actions workflow
2021-01-21 17:34:03 +09:00
Andrew Gaul
b70f8db037
Propagate errno from get_local_fent ( #1525 )
...
Follows on to 4d833a4fb9
.
References #1523 .
2021-01-20 23:21:14 +09:00
Andrew Gaul
4d833a4fb9
Return more specific errno when available ( #1520 )
...
Previously s3fs threw away some function return values and returned
EIO instead. This was due to not trusting the mix of -1 and errno
return codes. Correct the obviously incorrect ones via visual
inspection. Stronger typing may find more occurrences. Fixes #1519 .
2021-01-18 18:50:49 +09:00
Andrew Gaul
a6563211af
Update ChangeLog and configure.ac for 1.88
2021-01-11 17:59:28 +09:00
Andrew Gaul
8d66b0e4a8
Updated ChangeLog for release 1.88 ( #1514 )
...
References #1511 .
2021-01-05 16:46:41 +09:00
Takeshi Nakatani
555c1dde3d
Use Gtihub Actions instead of Travis CI
2021-01-04 23:46:54 +09:00
Andrew Gaul
b04bca37a5
Add configuration for Chaos HTTP Proxy ( #1508 )
...
This can find errors in retry logic. Chaos HTTP Proxy does not
support SSL bouncestorage/chaos-http-proxy#1 so users must set
s3proxy.endpoint and run via:
CHAOS_HTTP_PROXY=1 S3_URL=http://127.0.0.1:8080 make check -C test
It can also be helpful to increase retries and reduce sleep times.
References #1504 .
2021-01-04 23:32:04 +09:00
fly3366
168e588ac7
fix: Add reset offset ( #1503 )
2021-01-04 22:57:56 +09:00
Andrew Gaul
7158e50ee2
Quote | and && to allow passing it to retry ( #1506 )
2021-01-04 22:28:40 +09:00
Andrew Gaul
bd0fadbe5f
Remove authorization header when remaking handle ( #1505 )
...
This avoids including Authorization in SignedHeaders. s3fs will
recreate the Authorization header before sending the request.
2021-01-04 21:37:34 +09:00
kontrollanten
d1c638ab7a
fix 404 error message
2020-12-30 14:10:17 +09:00
fly3366
51f65d7b14
fix: miss header when retry
2020-12-23 22:16:07 +09:00
Andrew Gaul
a16d00d673
Simply curl_slist_sort_insert ( #1494 )
...
Some good taste from Linus:
https://github.com/mkirchner/linked-list-good-taste
Also avoid an allocation when replacing a value and tighten up tests.
2020-12-23 20:29:33 +09:00
Takeshi Nakatani
4d0daddad4
Fixed about Github Actions failure(centos8 and macos) ( #1498 )
2020-12-23 19:34:04 +09:00
Brian Myers
d246b9e8bf
Update README for change in brew arguments
...
* Homebrew changed the arguments required to install casks. Previously,
`brew cask install <package>` was the correct syntax, but now it's
`brew install --cask <package>`. This small commit updates the
instructions for installing `osxfuse` to reflect this change
2020-12-22 11:14:24 +09:00
Adam Johnson
ef3d4e506d
Update README note on eventual consistency
...
AWS promise read-after-write consistency now, but presumably non-AWS S3-compatible services are still eventually consistent.
2020-12-15 08:50:53 +09:00
Takeshi Nakatani
462b37b0bb
Added Github Actions workflow
2020-12-05 22:57:40 +09:00
Takeshi Nakatani
4341291cc2
Changed a condition of the atime test( fixed #1477 ) ( #1487 )
2020-12-03 02:45:05 +09:00
Takeshi Nakatani
c589886ba5
Added a condition to the atime test ( #1477 )
2020-12-02 23:21:25 +09:00
Andrew Gaul
d0363b118e
Add tests for curl_util ( #1481 )
2020-11-24 21:37:09 +09:00
Takeshi Nakatani
533322859d
Added comment for nullPointerRedundantCheck/cppcheck2.2
2020-11-21 20:06:14 +09:00
Andrew Gaul
f8d5b76edb
Simplify handling of returned ETag ( #1479 )
...
This works around lifetime warnings uncovered by cppcheck.
References #1478 .
2020-11-21 06:56:05 +09:00
Andrew Gaul
834862f8a4
Add S3_ENDPOINT test variable ( #1476 )
2020-11-15 22:10:26 +09:00
Takeshi Nakatani
d9f6469b7b
Fixed flushing dirty data and compressed the cache size
2020-11-14 16:45:37 +09:00
Takeshi Nakatani
852e6ee4c6
Fixed .gitignore for src/test_string_util.cpp
2020-11-14 16:22:30 +09:00
Andrew Gaul
ecb24c9c26
Disable preprocessor warnings w/o optimizations ( #1473 )
...
This removes some fortify source warnings. Also disable failing
memory sanitizer.
2020-11-09 21:48:07 +09:00
Andrew Gaul
543231c9f2
Address warnings from clang-tidy 11 ( #1470 )
2020-11-09 21:15:20 +09:00
Noah Meyerhans
d96a08d4ad
fixup! Implement AWS IMDSv2 support
2020-11-07 14:48:52 +09:00
Noah Meyerhans
f2f930300a
Implement AWS IMDSv2 support
...
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2. It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.
With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials. In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode. It can also be selected explicitly via
the imdsv1only mount option.
More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/
Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
2020-11-07 14:48:52 +09:00
yoshihiko
81ad3ce0ae
Fix directory authority
2020-11-07 14:47:53 +09:00
Takeshi Nakatani
38e1eaa8a3
Added logfile option for non-syslog logging
2020-10-19 20:10:19 +09:00
Andrew Gaul
6aa786b886
Flush dirty data after a writing a number of bytes ( #1448 )
...
This allows s3fs to write large files without consuming a large amount
of temporary local storage but can slow uploads due to server-side
copies. References #617 . Fixed #1056 . Fixes #1257 .
2020-10-13 22:30:42 +09:00
Takeshi Nakatani
58750cc441
Fixed a bug that symlink could not be read after restarting s3fs
2020-10-11 22:14:36 +09:00
Takeshi Nakatani
2188fb067e
Merge pull request #1446 from gaul/use-cache/relative-path
...
Allow relative path with use_path
2020-10-10 23:13:07 +09:00