octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-12-23 09:18:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,729 Commits 1 Branch 23 Tags 13 MiB
096a230b70
Commit Graph

191 Commits

Author SHA1 Message Date
Andrew Gaul
a44fc1103d Avoid setting bogus group id 
Addresses a symptom on newer macOS in Travis.
 2020-04-12 22:37:22 +09:00
Andrew Gaul
f373df9682 Test cache eviction 2020-04-11 19:00:38 +09:00
Takeshi Nakatani
7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
Takeshi Nakatani
0665d78550 Temporary measure: reduce test time on MacOS 2020-02-04 15:38:19 +00:00
Takeshi Nakatani
55d670f22f Change content-types test for MacOS 2020-02-04 14:51:50 +00:00
Takeshi Nakatani
32ae0d2c79 Bypassed MacOS ensure_diskfree test 2020-02-04 14:00:47 +00:00
Andrew Gaul
bc9126d774 Set directory MIME type to application/x-directory 
Previously s3fs auto-detected the MIME type of directories like
"TOYOTA TRUCK 8.2.2" as application/x-troff-man.  This caused get_mode
to not set S_IFDIR which failed directory creation.  Instead force all
object names ending in / to application/x-directory.  Fixes #1183.
 2020-02-04 20:03:21 +09:00
Andrew Gaul
4df50e7f85 Test that s3fs automatically detects Content-Type 
References #1217.
 2020-02-04 19:47:09 +09:00
Takeshi Nakatani
018ccb9a11 Suppressed a lot of message output about cache cleanup 2020-01-30 14:42:28 +00:00
Takeshi Nakatani
ee1d3a9057
Merge pull request #1151 from liuyongqing/master 
fix deadlock in clean up cache
 2020-01-30 22:17:50 +09:00
yongqingliu
b762a0a85b fix deadlock due to fdmanager_lock and fdent_data_lock 2020-01-29 12:03:53 +08:00
Takeshi Nakatani
010a6b83ef Not trap chown command errors directly 2020-01-28 10:32:55 +00:00
Takeshi Nakatani
9e77650e8c Fixed removing test file in integration-test-main.sh 2020-01-26 13:32:41 +00:00
Takeshi Nakatani
9789ca1a4d
Merge pull request #1188 from gaul/s3proxy 
Upgrade to S3Proxy 1.7.0
 2019-10-31 23:00:35 +09:00
Andrew Gaul
dc62953040 Upgrade to S3Proxy 1.7.0 
Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.7.0
 2019-10-28 20:11:01 -07:00
Takeshi Nakatani
cf3e82d10a Fixed a rename bug when enable_noobj_cache 2019-10-23 12:39:42 +00:00
Takeshi Nakatani
27a5536749 Fixed test_write_multiple_offsets error on osx 2019-10-14 17:29:39 +00:00
Andrew Gaul
80162c126b Add test for writing to multiple offsets 
References #1098.
 2019-09-25 19:31:08 -07:00
Takeshi Nakatani
1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00
liuyongqing
ae4bcd405c use correct content-type when overwriting and copying files 2019-08-28 22:25:09 -07:00
Andrew Gaul
2daa1d53d9 Verify file size via metadata as well as data 2019-08-20 18:36:44 -07:00
Takeshi Nakatani
a7d83df3eb Fixed typo in #1129 2019-08-14 12:39:38 +00:00
Takeshi Nakatani
ca7266fb76 Modified the test script a little 2019-08-13 14:21:42 +00:00
Takeshi Nakatani
f5bf41cf11 Fixed simple copy with nocpoyapi and modify flag in fdcache 2019-08-11 07:42:48 +00:00
Takeshi Nakatani
d8f1aef7be Added stats cache expire option for test 2019-08-06 10:55:38 +00:00
Takeshi Nakatani
daba563a1c Added auto_cache option for test on osx 2019-08-06 02:44:18 +00:00
Takeshi Nakatani
b79b0b1a92
Merge pull request #1119 from ggtakec/osx_direct_io 
Added direct_io option for test on osx
 2019-08-06 11:14:51 +09:00
Takeshi Nakatani
dc85eda188 Added direct_io option for test on osx 2019-08-05 17:12:49 +00:00
Andrew Gaul
d22acae9a3 Increase test startup retries on Linux 
This matches the number of retries on macOS and allows Valgrind to
work.
 2019-08-03 16:10:42 -07:00
Takeshi Nakatani
8cc008c501
Merge pull request #1111 from gaul/test/read-external-object 
Add test to read an externally created object
 2019-08-03 22:36:34 +09:00
Andrew Gaul
c321c8c23f Add test to read an externally created object 
References #890.
 2019-08-01 16:12:51 -07:00
Andrew Gaul
aba8e6ccfa Test enable_noobj_cache 2019-08-01 12:39:11 -07:00
Takeshi Nakatani
5c3c6bff2f Fixed filter-suite-log.sh for on osx 2019-07-21 15:32:10 +00:00
Takeshi Nakatani
2c4c78fd65
Merge pull request #1102 from gaul/test/concurrent-write 
Add a test for concurrent writes
 2019-07-17 22:24:45 +09:00
Takeshi Nakatani
80f598f439
Merge pull request #1099 from ggtakec/filter_log 
Add log output filter script at test failure
 2019-07-17 22:01:28 +09:00
Andrew Gaul
5cb7a31c09 Add a test for concurrent writes 2019-07-16 21:57:43 -07:00
Takeshi Nakatani
c7f8f61d09 Change test_concurrency test 2019-07-15 15:58:54 -07:00
Takeshi Nakatani
159cd2c682 Add log output filter script at test failure 2019-07-15 06:37:43 +00:00
Andrew Gaul
4a192ffdf9 Run xattr tests on macOS 2019-07-07 00:39:45 -07:00
Takeshi Nakatani
6177d7b096
Merge pull request #1073 from gaul/deps/s3proxy 
Upgrade to S3Proxy 1.6.2
 2019-07-07 15:19:00 +09:00
Takeshi Nakatani
3161bf4608
Merge pull request #1070 from gaul/bash/nounset 
Prohibit pipeline failures
 2019-07-07 15:10:53 +09:00
Takeshi Nakatani
2349dafb98
Merge pull request #1069 from gaul/macos/sed 
Use system sed on macOS
 2019-07-07 15:00:48 +09:00
Andrew Gaul
4f23f38583 Individually test multiple s3fs flags 
Remove unneeded comments; single part limits ensure that the tests
exercise multipart code paths even with smaller files.
References #971.
 2019-07-03 21:09:40 -07:00
Andrew Gaul
f439c6382f Upgrade to S3Proxy 1.6.2 
Notably this includes support for List Objects v2 which AWS CLI uses.
Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.6.2
 2019-07-03 10:37:26 -07:00
Andrew Gaul
9472ee4a01 Prohibit pipeline failures 2019-07-03 02:57:40 -07:00
Andrew Gaul
1f1f824da7 Use system sed on macOS 
This requires fewer developer customizations to work on macOS.
Requires some GNU workarounds.
 2019-07-02 22:17:02 -07:00
Andrew Gaul
c596441f58 Flush file when opening second fd 
Previously when s3fs had dirty local data and an application opened a
second fd it would remove the stat cache entry, query S3 for the size
which was still zero, then reinitialize FdEntry with this incorrect
size.  Instead flush local data to S3 so this query works.  It is
possible that a more involved patch could do this with a less
heavyweight approach but this requires changing open.  This does not
completely fix git clone but allows it to proceed further.  Also make
some cache methods const-correct.  References #839.
 2019-07-02 01:12:09 -07:00
Andrew Gaul
048aea1151 Remove litter from test_concurrency 2019-06-26 20:29:27 -07:00
Takeshi Nakatani
a78d8d1da4
Merge pull request #1049 from gaul/external-modication 
Remove cache file when object time differs
 2019-06-24 00:51:54 +09:00
Andrew Gaul
9e5eaad79b Remove cache file when object time differs 
Check the modification times to determine whether an object has
updated.  This relies on low clock skew between s3fs and the S3
server; a more robust approach could use the ETag.  Fixes #1047.
 2019-06-22 19:09:00 -07:00
Andrew Gaul
1cf3d2452e Correct macOS integration test umount 
Previously errexit could prevent this from succeeding.
 2019-06-19 11:01:32 -07:00
Andrew Gaul
0176fc712b Address shellcheck errors 2019-04-09 21:11:31 +09:00
Andrew Gaul
e5785d4690 Prefer HTTPS links in documentation where possible 
This also corrects a test URL which was HTTPS in practice.
 2019-04-05 17:31:06 +09:00
Jay Caines-Gooby
ddbcec5c96 Work with filenames that include spaces 
• Simplified the stat generation line (single exec using only stat)
• Quote variables so that the cache directory can also include spaces
• while/read loop to handle spaces in cached files
  (IFS was causing problems when all the files were saved into a single variable)
 2019-04-02 10:35:25 +01:00
Andrew Gaul
f53503438c Increase FdEntity reference count when returning 
Previously s3fs had a race condition where one client could delete
FdEntity that another client was using.  Add a simple concurrent test
which previously failed but now succeeds.  Fixes #964.
 2019-03-09 15:55:04 +09:00
Andrew Gaul
e9297f39ea Upgrade to S3Proxy 1.6.1 
Notably, this improve performance when listing objects using the
filesystem backend, allowing new tests like expanding large tar files.
Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.6.1
 2019-02-06 17:06:43 -08:00
Takeshi Nakatani
c0cf90cf8b Checked and corrected all typo 2019-01-27 12:04:29 +00:00
Takeshi Nakatani
d487348d21
Merge pull request #919 from gaul/test/mv-nonempty-dir 
Add test for mv non-empty directory
 2019-01-23 19:21:12 +09:00
Andrew Gaul
79bd3441eb Add test for mv non-empty directory 2019-01-22 22:13:17 -08:00
Andrew Gaul
5f5da4b2cb Load tail range during overwrite 
Previously s3fs experienced data loss when writing to the middle of a
file.  Corrupt files would have the expected data from 0..offset+size
but unexpected NUL bytes from offset+size..EOF.  References #808.
 2019-01-22 22:02:40 -08:00
Andrew Gaul
6401b4ae92 Flush file before renaming 
Previously s3fs could copy the zero-byte stub object without including
any pending writes.  Fixes #145.
 2019-01-17 12:05:10 -08:00
Andrew Gaul
2c43b1e12b Store and retrieve file change time 
This introduces a new header with the change time; existing objects
will report modification time.  Fixes #771.
 2019-01-14 10:05:11 -08:00
Andrew Gaul
523fe1e309 Repair xattr tests 
These did not run due to missing use_xattr flag and Travis
misconfiguration.
 2019-01-13 18:54:16 -08:00
Andrew Gaul
1b86e4d414 Upgrade to S3Proxy 1.6.0 
Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.6.0
https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.5.5
https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.5.4
 2018-05-16 16:38:17 -07:00
Takeshi Nakatani
f4515b5cfa Merge pull request #646 from andrewgaul/s3proxy-pid 
Simplify S3Proxy PID handling
 2017-09-20 05:17:00 +09:00
Andrew Gaul
5014c1827b Simplify S3Proxy PID handling 
Also remove log prefixing since newer S3Proxy versions do this
already.  Finally remove unnecessary wait.
 2017-09-17 20:19:56 -07:00
Andrew Gaul
f531e6aff2 Configure S3Proxy for SSL 
This also demonstrates that SSL certificate checking occurs and the
tests must disable it for S3Proxy's self-signed certificate.
References #640.
 2017-09-17 16:16:18 -07:00
Takeshi Nakatani
96d8e6d823 Merge remote-tracking branch 'upstream/macosx' into macosx 2017-08-11 14:20:12 +00:00
Takeshi Nakatani
62b8084300 Added travis test on osx for #601 2017-08-11 14:09:43 +00:00
Andrew Gaul
98b724391f Upgrade to S3Proxy 1.5.3 
Release notes:

https://github.com/andrewgaul/s3proxy/releases/tag/s3proxy-1.5.3
 2017-07-09 22:41:39 -07:00
Andrew Gaul
25a03c370a Upgrade to S3Proxy 1.5.2 
Release notes:

https://github.com/andrewgaul/s3proxy/releases/tag/s3proxy-1.5.2
 2017-03-02 10:55:24 -08:00
Takeshi Nakatani
8a11d7bc2f Merge pull request #505 from andrewgaul/spelling 
Correct typos
 2016-11-20 09:10:27 +09:00
Takeshi Nakatani
7aae4782d9 Merge pull request #504 from andrewgaul/test 
Use describe helper function
 2016-11-20 09:10:16 +09:00
Andrew Gaul
d375bca0d0 Correct typos 2016-11-19 15:57:41 -08:00
Andrew Gaul
cd0c8599cc Use describe helper function 2016-11-19 15:36:02 -08:00
Andrew Gaul
20878a1618 Add missing call to mtime test 2016-11-19 15:10:41 -08:00
Andrew Gaul
cafe6015e3 Upgrade to S3Proxy 1.5.1 
Enabled previously broken tests and test with default v4 signer.
Release notes:

https://github.com/andrewgaul/s3proxy/releases/tag/s3proxy-1.5.0
https://github.com/andrewgaul/s3proxy/releases/tag/s3proxy-1.5.1
 2016-11-15 23:09:48 -08:00
Takeshi Nakatani
ec110bb0f3 Added small logic in test script for test_chown 2016-09-11 13:41:50 +00:00
Takeshi Nakatani
232befb52a Added small logic in test script for test_chown 2016-09-11 13:37:53 +00:00
Takeshi Nakatani
f363c21ff5 Added comments in failure message for test_chown 2016-09-11 13:28:40 +00:00
Tianlong Wu
cf23dc78ab Use 'return' instead of 'exit' in test 2016-04-22 16:24:26 +08:00
Tianlong Wu
115bd51f3f Fix a bug of truncating empty file 2016-04-22 14:49:37 +08:00
Robb Kistler
4e583583cd Test for writing after an lseek past end of file 
This is a test to demonstrate Issue #375
 2016-03-23 16:03:38 -07:00
Takeshi Nakatani
6472eedddc Supported regex type for additional header format. 2016-02-07 05:08:52 +00:00
Robb Kistler
7bfaa24d25 Integration test summary, continue on error 
Details in README.md and s3fs-integration-test-common.sh

Factor out s3fs-fuse and s3proxy start/stop.  The plan is to make it easier to
add test suites besides small-integration-test.sh that can test with various
s3fs options.

Each test run starts in a uniquely named at the top of the bucket.  This allows
multiple runs against persistent storage without worrying about cleaning
up in error conditions that leave artifiacts behind.

Tests continues if a test case fails.

Results are summarized at the end of the test run

Environment variable to control debug level of s3fs-fuse

Environment variable to enable public bucket (makes it easier to poke
around with tools like curl)

Environment variable to start s3fs-fuse under valgrind

Environment variable that casues script sets up s3fs-fuse and then wait
indefinitely, making it easy to experiment manually with the mount
point.

Additional test case
 2016-02-05 05:40:28 -08:00
Robb Kistler
1927ccfe0a Don't loop on fusermount if mountpoint is gone 2015-12-15 15:07:00 -08:00
Takeshi Nakatani
4bd5ffb0fa Update integration-test-main.sh as additional change for #300 2015-12-03 13:44:43 +00:00
bazeli
b85bd53336 Update integration-test-main.sh 
new test for mtime preservation copying file with `cp -p`
 2015-11-24 17:29:54 +09:00
Robb Kistler
a3ef5c820d Add file truncate test 
This test creates a file with contents, truncates it to
zero and verifies that it is zero length.
 2015-11-03 21:47:15 -08:00
Takeshi Nakatani
497b108109 Merge pull request #285 from andrewgaul/symlink-test 
Add test for symlink
 2015-11-01 18:46:06 +09:00
Andrew Gaul
86f95b05bf Add test for symlink 2015-10-24 14:20:26 -07:00
Takeshi Nakatani
1b323a6252 Changed debug option to dbglevel in test script. 2015-10-18 17:31:31 +00:00
Robb Kistler
3d5b8a7672 Use 127.0.0.1 not localhost in s3proxy wait loop 
localhost doesn't always resolve to 127.0.0.1
 2015-09-16 00:06:41 -07:00
Takeshi Nakatani
489f9edec7 Merge pull request #266 from RobbKistler/fix-integration-test 
Cleanup from PR #265
 2015-09-13 16:44:41 +09:00
Robb Kistler
718db57ade Code review changes 
Missed some cleanup from the code review
 2015-09-13 00:31:56 -07:00
Takeshi Nakatani
639dcf19b0 Merge pull request #265 from RobbKistler/fix-integration-test 
Fix integration tests
 2015-09-13 15:59:36 +09:00
Robb Kistler
6b21d9d424 Code review changes 2015-09-11 16:09:00 -07:00
Robb Kistler
dac9844765 Fix remove_nonempty_directory test bug 
Wrap the attempt to rmdir in an if statement, otherwise the entire
test process exists (errexit is set). This test expects the rmdir
to fail.
 2015-09-11 15:24:17 -07:00
Robb Kistler
849e66f6a1 Change test_append_file to avoid object read-after-after-overwrite 
Open the test file once outside of the tests for loop.  This helps avoid
object consistency problems when running against S3 providers without
strong consistency (like Amazon).  See Issue #263.
 2015-09-11 15:24:17 -07:00
Robb Kistler
6a8a2e4800 Allow integration testing against Amazon S3 
Example command line:
S3FS_CREDENTIALS_FILE=keyfile \
TEST_BUCKET_1=somebucket \
S3PROXY_BINARY="" \
URL="http://s3.amazonaws.com" ./small-integration-test.sh
 2015-09-11 14:35:12 -07:00