octoleo
/
s3fs-fuse
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,701 Commits 1 Branch 22 Tags 11 MiB
Commit Graph

1701 Commits

Author SHA1 Message Date
Andrew Gaul 8c58ba8ac0
Update ChangeLog and configure.ac for 1.89 (#1584) 
Also make formatting more consistent.  Fixes #1588.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-24 08:03:55 +09:00
Takeshi Nakatani a19d223434 Bypass an OSX test(no different from other OS) which takes very time consuming 2021-02-23 20:47:51 +09:00
Takeshi Nakatani fc06419549
Fixed aws command error by rsa on ubuntu16.04 (#1585) 2021-02-23 11:41:52 +09:00
Andrew Gaul 032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583) 
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
 2021-02-23 09:45:13 +09:00
Takeshi Nakatani 5b5bc3114a Fixed a bug that padded null bytes when changing xattr 2021-02-21 12:23:36 +09:00
Takeshi Nakatani 81e267d421 Fixed the same bug in chmod/chown/utimens and added tests 2021-02-20 23:30:50 +09:00
Takeshi Nakatani 4fc92d59f3 Fixed a bug that filling NULL bytes when changing the attribute after renaming 2021-02-20 23:30:50 +09:00
Takeshi Nakatani a0f347b10f Added no_time_stamp_ms option mainly for testing 2021-02-20 20:24:26 +09:00
Carsten Grohmann 5debf523b0 Allow changing temporary directory used for tests 
Set temporary directory to $TMPDIR if set or use default /var/tmp
 2021-02-19 13:59:23 +09:00
Carsten Grohmann 1c8aadafd1 Fix typo in an error message of the test suite 2021-02-19 13:59:23 +09:00
Takeshi Nakatani d33f252404 Fixed forgetting to clean up test files 2021-02-14 00:07:46 +09:00
Takeshi Nakatani 4da56acdcc Improved processing when HEAD response is 400 2021-02-13 17:00:13 +09:00
Takeshi Nakatani 493802a605 Fixed osx mistype declaration and typo warnings 2021-02-13 12:04:35 +09:00
林千里 22b0ae9d51 set IsExpireTime to true by default 
fixes #1563
 2021-02-12 12:20:47 +09:00
Andrew Gaul 134a54b32f
Allow configuring the multipart threshold (#1562) 
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 23:35:46 +09:00
Andrew Gaul 7f6fbb0021
Correct singlepart_copy_limit unit (#1561) 
This is documented to be in MB not KB.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 22:53:50 +09:00
Andrew Gaul ec8bae9827
Do not periodically flush with nomultipartupload (#1559) 
s3fs periodically flushes data to S3 and hole-punches the written
data, marking it as clean.  Without multipart copy, s3fs would write
the now-zeroed data along with any new data.  s3fs should not attempt
to periodically flush dirty data at all without multipart copy
support.  Fixes #1542.
 2021-02-11 22:24:39 +09:00
Andrew Gaul a4d916af13
Allow configuring multipart copy size (#1555) 
Also align configuration with mixupload.
 2021-02-08 20:32:12 +09:00
Takeshi Nakatani 32f096fa3f Use clock_gettime instead of gettimeofday 2021-02-08 10:19:28 +09:00
Andrew Gaul c692093921
Copy at most 5 GB per multipart copy part request (#1553) 
Previously mixupload failed with larger objects due to the AWS
constraint on part sizes.  This symptom was seen more frequently due
to periodic flushing of dirty data.  Fixes #1547.
 2021-02-07 23:10:07 +09:00
Andrew Gaul 1a6d0826b5
Allow -1 value to disable max_dirty_data (#1552) 
This matches the man page.
 2021-02-07 22:14:31 +09:00
Andrew Gaul f2c5e38724
Allow integration tests to use larger files (#1548) 
Previously these failed due to dd only copying 32 MB from /dev/urandom
and exhausting the limited space in /tmp.  References #1543.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-07 12:50:02 +09:00
Andrew Gaul bf33fe7f55
Increase buffer size to avoid warning (#1550) 
This can be up to 54 bytes.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-07 12:26:58 +09:00
Andrew Gaul ff9d6a75c7
Parse more specific AWS error codes (#1549) 
Fixes #1545.
 2021-02-07 11:29:08 +09:00
Carsten Grohmann 4c6690f5f0
Add timestamps to debug log and cache integrity check (#1540) 
* Add UTC timestamp to debug output

for easier correlation with external events

Old output:
$ s3fs mybucket /bucket/  -o curldbg,dbglevel=debug -d -d -f
[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from...
[INF]     s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=0, ...

New output:
$ s3fs mybucket /bucket/  -o curldbg,dbglevel=debug -d -d -f
2021-01-28T21:09:16.264Z [CRT] s3fs_logger.cpp:LowSetLogLevel(202):...
2021-01-28T21:09:16.264Z [INF]     s3fs.cpp:set_mountpoint_attrib...

* Add UTC timestamp to cache integrity check

for easier correlation with external events.

$ s3fs mybucket /mybucket  -oset_check_cache_sigusr1=/tmp/check.cache

Old output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
------------------------------------------------------------
Check cache file and its stats file consistency
------------------------------------------------------------
------------------------------------------------------------
Summary - Total files:                0
          Detected error files:       0
          Detected error directories: 0
------------------------------------------------------------

New output:
$ kill -s SIGUSR1 $(pgrep s3fs)
$ cat /tmp/check.cache
---------------------------------------------------------------------------
Check cache file and its stats file consistency at 2021-01-30T13:04:14.111Z
---------------------------------------------------------------------------
---------------------------------------------------------------------------
Summary - Total files:                0
          Detected error files:       0
          Detected error directories: 0
---------------------------------------------------------------------------

* Fix indentation of S3fsLog::GetCurrentTime()
 2021-02-04 10:41:29 +09:00
Andrew Gaul 67b9381825 Remove limit for single-part objects 
AWS enforces a 5 GB limit for single-part objects but other S3
implementations like GCS support larger objects.  Fixes #1542.
 2021-02-04 09:19:37 +09:00
Andrew Gaul 4b53d4bf6b
Suppress spurious SHA256 error message (#1544) 2021-02-04 07:28:51 +09:00
Carsten Grohmann ed85b72bf5 Add warning for failing metadata updates of large files 
Metadata updates fail for large files if "-o nocopyapi" or
"-o nomultipart" is set.

This change is related to #1528
 2021-01-28 10:20:32 +09:00
Andrew Gaul 17fda89ae9
Handle s3fs_get_content_md5 and s3fs_sha256_hex_fd errors (#1534) 
Follows on to 4d833a4fb9.
 2021-01-25 19:08:14 +09:00
Andrew Gaul 1987bcbea3
Re-fix retry logic (#1533) 
This logic has always been broken but happened to try one time due to
&& operator precedence.  7158e50ee2
broke this further when quoting && since the command was not
evaluated.
 2021-01-25 18:32:40 +09:00
Andrew Gaul d019dda4f7
Simplify substr manipulations with erase (#1532) 
This avoids creating a new std::string.
 2021-01-25 18:02:32 +09:00
Andrew Gaul dc9255bc5f
Prefer static_cast where possible (#1531) 2021-01-25 08:15:17 +09:00
Andrew Gaul b0e8758b63
Use result instead of res for consistency (#1530) 2021-01-25 07:56:10 +09:00
Takeshi Nakatani 6d65e30dd5 Fixed brew install with cask in github actions workflow 2021-01-21 17:34:03 +09:00
Andrew Gaul b70f8db037
Propagate errno from get_local_fent (#1525) 
Follows on to 4d833a4fb9.
References #1523.
 2021-01-20 23:21:14 +09:00
Andrew Gaul 4d833a4fb9
Return more specific errno when available (#1520) 
Previously s3fs threw away some function return values and returned
EIO instead.  This was due to not trusting the mix of -1 and errno
return codes.  Correct the obviously incorrect ones via visual
inspection.  Stronger typing may find more occurrences.  Fixes #1519.
 2021-01-18 18:50:49 +09:00
Andrew Gaul a6563211af Update ChangeLog and configure.ac for 1.88 2021-01-11 17:59:28 +09:00
Andrew Gaul 8d66b0e4a8
Updated ChangeLog for release 1.88 (#1514) 
References #1511.
 2021-01-05 16:46:41 +09:00
Takeshi Nakatani 555c1dde3d Use Gtihub Actions instead of Travis CI 2021-01-04 23:46:54 +09:00
Andrew Gaul b04bca37a5
Add configuration for Chaos HTTP Proxy (#1508) 
This can find errors in retry logic.  Chaos HTTP Proxy does not
support SSL bouncestorage/chaos-http-proxy#1 so users must set
s3proxy.endpoint and run via:

CHAOS_HTTP_PROXY=1 S3_URL=http://127.0.0.1:8080 make check -C test

It can also be helpful to increase retries and reduce sleep times.
References #1504.
 2021-01-04 23:32:04 +09:00
fly3366 168e588ac7
fix: Add reset offset (#1503) 2021-01-04 22:57:56 +09:00
Andrew Gaul 7158e50ee2
Quote | and && to allow passing it to retry (#1506) 2021-01-04 22:28:40 +09:00
Andrew Gaul bd0fadbe5f
Remove authorization header when remaking handle (#1505) 
This avoids including Authorization in SignedHeaders.  s3fs will
recreate the Authorization header before sending the request.
 2021-01-04 21:37:34 +09:00
kontrollanten d1c638ab7a fix 404 error message 2020-12-30 14:10:17 +09:00
fly3366 51f65d7b14 fix: miss header when retry 2020-12-23 22:16:07 +09:00
Andrew Gaul a16d00d673
Simply curl_slist_sort_insert (#1494) 
Some good taste from Linus:
https://github.com/mkirchner/linked-list-good-taste
Also avoid an allocation when replacing a value and tighten up tests.
 2020-12-23 20:29:33 +09:00
Takeshi Nakatani 4d0daddad4
Fixed about Github Actions failure(centos8 and macos) (#1498) 2020-12-23 19:34:04 +09:00
Brian Myers d246b9e8bf Update README for change in brew arguments 
* Homebrew changed the arguments required to install casks. Previously,
  `brew cask install <package>` was the correct syntax, but now it's
  `brew install --cask <package>`. This small commit updates the
  instructions for installing `osxfuse` to reflect this change
 2020-12-22 11:14:24 +09:00
Adam Johnson ef3d4e506d Update README note on eventual consistency 
AWS promise read-after-write consistency now, but presumably non-AWS S3-compatible services are still eventually consistent.
 2020-12-15 08:50:53 +09:00
Takeshi Nakatani 462b37b0bb Added Github Actions workflow 2020-12-05 22:57:40 +09:00