Takeshi Nakatani
b589ebec23
Output version and command parameters at startup
2021-03-06 19:11:57 +09:00
Andrew Gaul
8c58ba8ac0
Update ChangeLog and configure.ac for 1.89 ( #1584 )
...
Also make formatting more consistent. Fixes #1588 .
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-24 08:03:55 +09:00
Takeshi Nakatani
a19d223434
Bypass an OSX test(no different from other OS) which takes very time consuming
2021-02-23 20:47:51 +09:00
Takeshi Nakatani
fc06419549
Fixed aws command error by rsa on ubuntu16.04 ( #1585 )
2021-02-23 11:41:52 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 ( #1583 )
...
This allows use of s3fs on object stores that do not implement
the V1 API. Fixes #1573 .
2021-02-23 09:45:13 +09:00
Takeshi Nakatani
5b5bc3114a
Fixed a bug that padded null bytes when changing xattr
2021-02-21 12:23:36 +09:00
Takeshi Nakatani
81e267d421
Fixed the same bug in chmod/chown/utimens and added tests
2021-02-20 23:30:50 +09:00
Takeshi Nakatani
4fc92d59f3
Fixed a bug that filling NULL bytes when changing the attribute after renaming
2021-02-20 23:30:50 +09:00
Takeshi Nakatani
a0f347b10f
Added no_time_stamp_ms option mainly for testing
2021-02-20 20:24:26 +09:00
Carsten Grohmann
5debf523b0
Allow changing temporary directory used for tests
...
Set temporary directory to $TMPDIR if set or use default /var/tmp
2021-02-19 13:59:23 +09:00
Carsten Grohmann
1c8aadafd1
Fix typo in an error message of the test suite
2021-02-19 13:59:23 +09:00
Takeshi Nakatani
d33f252404
Fixed forgetting to clean up test files
2021-02-14 00:07:46 +09:00
Takeshi Nakatani
4da56acdcc
Improved processing when HEAD response is 400
2021-02-13 17:00:13 +09:00
Takeshi Nakatani
493802a605
Fixed osx mistype declaration and typo warnings
2021-02-13 12:04:35 +09:00
林千里
22b0ae9d51
set IsExpireTime to true by default
...
fixes #1563
2021-02-12 12:20:47 +09:00
Andrew Gaul
134a54b32f
Allow configuring the multipart threshold ( #1562 )
...
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-11 23:35:46 +09:00
Andrew Gaul
7f6fbb0021
Correct singlepart_copy_limit unit ( #1561 )
...
This is documented to be in MB not KB.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-11 22:53:50 +09:00
Andrew Gaul
ec8bae9827
Do not periodically flush with nomultipartupload ( #1559 )
...
s3fs periodically flushes data to S3 and hole-punches the written
data, marking it as clean. Without multipart copy, s3fs would write
the now-zeroed data along with any new data. s3fs should not attempt
to periodically flush dirty data at all without multipart copy
support. Fixes #1542 .
2021-02-11 22:24:39 +09:00
Andrew Gaul
a4d916af13
Allow configuring multipart copy size ( #1555 )
...
Also align configuration with mixupload.
2021-02-08 20:32:12 +09:00
Takeshi Nakatani
32f096fa3f
Use clock_gettime instead of gettimeofday
2021-02-08 10:19:28 +09:00
Andrew Gaul
c692093921
Copy at most 5 GB per multipart copy part request ( #1553 )
...
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes. This symptom was seen more frequently due
to periodic flushing of dirty data. Fixes #1547 .
2021-02-07 23:10:07 +09:00
Andrew Gaul
1a6d0826b5
Allow -1 value to disable max_dirty_data ( #1552 )
...
This matches the man page.
2021-02-07 22:14:31 +09:00
Andrew Gaul
f2c5e38724
Allow integration tests to use larger files ( #1548 )
...
Previously these failed due to dd only copying 32 MB from /dev/urandom
and exhausting the limited space in /tmp. References #1543 .
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-07 12:50:02 +09:00
Andrew Gaul
bf33fe7f55
Increase buffer size to avoid warning ( #1550 )
...
This can be up to 54 bytes.
Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
2021-02-07 12:26:58 +09:00
Andrew Gaul
ff9d6a75c7
Parse more specific AWS error codes ( #1549 )
...
Fixes #1545 .
2021-02-07 11:29:08 +09:00
Carsten Grohmann
4c6690f5f0
Add timestamps to debug log and cache integrity check ( #1540 )
...
* Add UTC timestamp to debug output
for easier correlation with external events
Old output:
$ s3fs mybucket /bucket/ -o curldbg,dbglevel=debug -d -d -f
[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from...
[INF] s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=0, ...
New output:
$ s3fs mybucket /bucket/ -o curldbg,dbglevel=debug -d -d -f
2021-01-28T21:09:16.264Z [CRT] s3fs_logger.cpp:LowSetLogLevel(202):...
2021-01-28T21:09:16.264Z [INF] s3fs.cpp:set_mountpoint_attrib...
* Add UTC timestamp to cache integrity check
for easier correlation with external events.
$ s3fs mybucket /mybucket -oset_check_cache_sigusr1=/tmp/check.cache
Old output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
------------------------------------------------------------
Check cache file and its stats file consistency
------------------------------------------------------------
------------------------------------------------------------
Summary - Total files: 0
Detected error files: 0
Detected error directories: 0
------------------------------------------------------------
New output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
---------------------------------------------------------------------------
Check cache file and its stats file consistency at 2021-01-30T13:04:14.111Z
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Summary - Total files: 0
Detected error files: 0
Detected error directories: 0
---------------------------------------------------------------------------
* Fix indentation of S3fsLog::GetCurrentTime()
2021-02-04 10:41:29 +09:00
Andrew Gaul
67b9381825
Remove limit for single-part objects
...
AWS enforces a 5 GB limit for single-part objects but other S3
implementations like GCS support larger objects. Fixes #1542 .
2021-02-04 09:19:37 +09:00
Andrew Gaul
4b53d4bf6b
Suppress spurious SHA256 error message ( #1544 )
2021-02-04 07:28:51 +09:00
Carsten Grohmann
ed85b72bf5
Add warning for failing metadata updates of large files
...
Metadata updates fail for large files if "-o nocopyapi" or
"-o nomultipart" is set.
This change is related to #1528
2021-01-28 10:20:32 +09:00
Andrew Gaul
17fda89ae9
Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors ( #1534 )
...
Follows on to 4d833a4fb9
.
2021-01-25 19:08:14 +09:00
Andrew Gaul
1987bcbea3
Re-fix retry logic ( #1533 )
...
This logic has always been broken but happened to try one time due to
&& operator precedence. 7158e50ee2
broke this further when quoting && since the command was not
evaluated.
2021-01-25 18:32:40 +09:00
Andrew Gaul
d019dda4f7
Simplify substr manipulations with erase ( #1532 )
...
This avoids creating a new std::string.
2021-01-25 18:02:32 +09:00
Andrew Gaul
dc9255bc5f
Prefer static_cast where possible ( #1531 )
2021-01-25 08:15:17 +09:00
Andrew Gaul
b0e8758b63
Use result instead of res for consistency ( #1530 )
2021-01-25 07:56:10 +09:00
Takeshi Nakatani
6d65e30dd5
Fixed brew install with cask in github actions workflow
2021-01-21 17:34:03 +09:00
Andrew Gaul
b70f8db037
Propagate errno from get_local_fent ( #1525 )
...
Follows on to 4d833a4fb9
.
References #1523 .
2021-01-20 23:21:14 +09:00
Andrew Gaul
4d833a4fb9
Return more specific errno when available ( #1520 )
...
Previously s3fs threw away some function return values and returned
EIO instead. This was due to not trusting the mix of -1 and errno
return codes. Correct the obviously incorrect ones via visual
inspection. Stronger typing may find more occurrences. Fixes #1519 .
2021-01-18 18:50:49 +09:00
Andrew Gaul
a6563211af
Update ChangeLog and configure.ac for 1.88
2021-01-11 17:59:28 +09:00
Andrew Gaul
8d66b0e4a8
Updated ChangeLog for release 1.88 ( #1514 )
...
References #1511 .
2021-01-05 16:46:41 +09:00
Takeshi Nakatani
555c1dde3d
Use Gtihub Actions instead of Travis CI
2021-01-04 23:46:54 +09:00
Andrew Gaul
b04bca37a5
Add configuration for Chaos HTTP Proxy ( #1508 )
...
This can find errors in retry logic. Chaos HTTP Proxy does not
support SSL bouncestorage/chaos-http-proxy#1 so users must set
s3proxy.endpoint and run via:
CHAOS_HTTP_PROXY=1 S3_URL=http://127.0.0.1:8080 make check -C test
It can also be helpful to increase retries and reduce sleep times.
References #1504 .
2021-01-04 23:32:04 +09:00
fly3366
168e588ac7
fix: Add reset offset ( #1503 )
2021-01-04 22:57:56 +09:00
Andrew Gaul
7158e50ee2
Quote | and && to allow passing it to retry ( #1506 )
2021-01-04 22:28:40 +09:00
Andrew Gaul
bd0fadbe5f
Remove authorization header when remaking handle ( #1505 )
...
This avoids including Authorization in SignedHeaders. s3fs will
recreate the Authorization header before sending the request.
2021-01-04 21:37:34 +09:00
kontrollanten
d1c638ab7a
fix 404 error message
2020-12-30 14:10:17 +09:00
fly3366
51f65d7b14
fix: miss header when retry
2020-12-23 22:16:07 +09:00
Andrew Gaul
a16d00d673
Simply curl_slist_sort_insert ( #1494 )
...
Some good taste from Linus:
https://github.com/mkirchner/linked-list-good-taste
Also avoid an allocation when replacing a value and tighten up tests.
2020-12-23 20:29:33 +09:00
Takeshi Nakatani
4d0daddad4
Fixed about Github Actions failure(centos8 and macos) ( #1498 )
2020-12-23 19:34:04 +09:00
Brian Myers
d246b9e8bf
Update README for change in brew arguments
...
* Homebrew changed the arguments required to install casks. Previously,
`brew cask install <package>` was the correct syntax, but now it's
`brew install --cask <package>`. This small commit updates the
instructions for installing `osxfuse` to reflect this change
2020-12-22 11:14:24 +09:00
Adam Johnson
ef3d4e506d
Update README note on eventual consistency
...
AWS promise read-after-write consistency now, but presumably non-AWS S3-compatible services are still eventually consistent.
2020-12-15 08:50:53 +09:00