octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-16 17:25:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,878 Commits 1 Branch 23 Tags 13 MiB
4dfe2bfdd7
Commit Graph

229 Commits

Author SHA1 Message Date
Andrew Gaul
1987bcbea3
Re-fix retry logic (#1533) 
This logic has always been broken but happened to try one time due to
&& operator precedence.  7158e50ee2
broke this further when quoting && since the command was not
evaluated.
 2021-01-25 18:32:40 +09:00
Andrew Gaul
b04bca37a5
Add configuration for Chaos HTTP Proxy (#1508) 
This can find errors in retry logic.  Chaos HTTP Proxy does not
support SSL bouncestorage/chaos-http-proxy#1 so users must set
s3proxy.endpoint and run via:

CHAOS_HTTP_PROXY=1 S3_URL=http://127.0.0.1:8080 make check -C test

It can also be helpful to increase retries and reduce sleep times.
References #1504.
 2021-01-04 23:32:04 +09:00
Andrew Gaul
7158e50ee2
Quote | and && to allow passing it to retry (#1506) 2021-01-04 22:28:40 +09:00
Takeshi Nakatani
4d0daddad4
Fixed about Github Actions failure(centos8 and macos) (#1498) 2020-12-23 19:34:04 +09:00
Takeshi Nakatani
4341291cc2
Changed a condition of the atime test(fixed #1477) (#1487) 2020-12-03 02:45:05 +09:00
Takeshi Nakatani
c589886ba5
Added a condition to the atime test (#1477) 2020-12-02 23:21:25 +09:00
Andrew Gaul
834862f8a4
Add S3_ENDPOINT test variable (#1476) 2020-11-15 22:10:26 +09:00
Andrew Gaul
ecb24c9c26
Disable preprocessor warnings w/o optimizations (#1473) 
This removes some fortify source warnings.  Also disable failing
memory sanitizer.
 2020-11-09 21:48:07 +09:00
Takeshi Nakatani
a4e4ce8aea
Merge pull request #1440 from gaul/test/undefined-behavior-sanitizer 
Add memory and undefined behavior sanitizer
 2020-10-04 23:19:36 +09:00
Andrew Gaul
7f43b7fa53 Add memory and undefined behavior sanitizer 
Remove Valgrind which duplicates the sanitizers and is much slower.
 2020-10-04 22:55:27 +09:00
Takeshi Nakatani
059cc57ba6 Added atime and Corrected atime/mtime/ctime operations 2020-10-04 13:54:01 +09:00
Andrew Gaul
081d6c1245 Allow setting signature V4-only 
Default to allowing V2 or V4.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
89b1c32b24 Rework s3fs pid handling 
Previously S3FS_PID was not set in the correct shell and thus
ps u $S3FS_PID showed all programs.  This caused the flag detection to
fail when users ran other instances of s3fs with different flags.
References #1402.
 2020-09-18 22:05:37 +09:00
Takeshi Nakatani
76d88f2291
Merge pull request #1403 from gaul/vim/expandtab 
Fix expandtab settings for tests
 2020-09-17 21:35:58 +09:00
Takeshi Nakatani
72340cfbd9
Merge pull request #1401 from gaul/test/hardlink 
Add test for hardlink
 2020-09-17 20:04:07 +09:00
Andrew Gaul
8fb70c5e4a Fix expandtab settings for tests 
This matches the main source.
 2020-09-15 22:11:14 +09:00
Andrew Gaul
78e2345c19 Add test for hardlink 2020-09-15 22:07:25 +09:00
Andrew Gaul
3bc565b986 Add test for concurrent reads 2020-09-15 21:25:51 +09:00
Takeshi Nakatani
b5ffd419d8 Source file division and set 4 spaces and cleanup 2020-08-26 17:43:50 +09:00
Takeshi Nakatani
2feefeec47 Added mv file test when disk space is insufficient 2020-08-16 12:47:29 +00:00
Takeshi Nakatani
f28e3bd89e
Merge pull request #1362 from gaul/test/sanitizers 
Add script to run tests using sanitizers
 2020-08-16 19:15:05 +09:00
Takeshi Nakatani
2bb745cdd7 Fixed a bug about move file over limit of ensure space 2020-08-16 17:42:20 +09:00
Andrew Gaul
c8e13300e1 Add script to run tests using sanitizers 
These currently show several kinds of errors.
 2020-08-15 17:28:35 +09:00
Andrew Gaul
0c1bc0f803 Add portability wrapper for stat(1) 
Fixes #1344.
 2020-08-02 23:14:58 +09:00
Andrew Gaul
4154e539ea Test creating a directory with external program 
References #1264.
 2020-07-25 21:22:53 +09:00
Takeshi Nakatani
62c8be85d6 Interim fix for Travis macOS Build 2020-07-10 09:47:18 +09:00
Andrew Gaul
4d1f5c899f Use Python 2 for write_multiple_offsets 
This aligns with ut_test.py .  Using an older Python also allows
compatibility with the older macOS 10.12 Travis CI.  References #1323.
 2020-07-07 21:12:52 +09:00
Takeshi Nakatani
1f796d432d Fixed upload error about mixuploading sparse file and truncating file 2020-06-27 22:44:19 +09:00
Takeshi Nakatani
35006e318f Fixed about ParallelMixMultipartUpload 2020-06-24 12:48:55 +09:00
Andrew Gaul
8a7548a9d4 Import ossfs tests 
This turns up a failure in test_rename_file when calling seek(0) on a
renamed file.
 2020-06-01 09:08:25 +09:00
Takeshi Nakatani
eb58460175
Merge pull request #1294 from gaul/test/profile 
Allow overriding test credentials with a profile
 2020-05-24 16:14:31 +09:00
Andrew Gaul
73098220bf Allow overriding test credentials with a profile 2020-05-24 12:05:16 +09:00
Andrew Gaul
ca7756fa77 Update to S3Proxy 1.7.1 
Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.7.1
 2020-05-23 10:06:44 +09:00
Andrew Gaul
da70cb92a8 Provide AWS CLI with correct test credentials 
This allows tests to pass against real S3 implementations like
Backblaze.  References #272.
 2020-05-22 19:27:18 +09:00
Andrew Gaul
8945e98d8b Support Google Cloud Storage headers 
This allows s3fs to interpret objects created by gsutil.
 2020-05-03 18:33:13 +09:00
Takeshi Nakatani
ce1221c867 Add test for cache file stat content 2020-04-22 15:53:00 +00:00
Andrew Gaul
3af17c3019 Add test for out-of-order writes 
References #1220.  References #1277.
 2020-04-22 21:48:55 +09:00
Takeshi Nakatani
538fbed302
Merge pull request #1266 from gaul/test/cache-eviction 
Test cache eviction
 2020-04-14 22:30:04 +09:00
Andrew Gaul
a44fc1103d Avoid setting bogus group id 
Addresses a symptom on newer macOS in Travis.
 2020-04-12 22:37:22 +09:00
Andrew Gaul
f373df9682 Test cache eviction 2020-04-11 19:00:38 +09:00
Takeshi Nakatani
7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
Takeshi Nakatani
0665d78550 Temporary measure: reduce test time on MacOS 2020-02-04 15:38:19 +00:00
Takeshi Nakatani
55d670f22f Change content-types test for MacOS 2020-02-04 14:51:50 +00:00
Takeshi Nakatani
32ae0d2c79 Bypassed MacOS ensure_diskfree test 2020-02-04 14:00:47 +00:00
Andrew Gaul
bc9126d774 Set directory MIME type to application/x-directory 
Previously s3fs auto-detected the MIME type of directories like
"TOYOTA TRUCK 8.2.2" as application/x-troff-man.  This caused get_mode
to not set S_IFDIR which failed directory creation.  Instead force all
object names ending in / to application/x-directory.  Fixes #1183.
 2020-02-04 20:03:21 +09:00
Andrew Gaul
4df50e7f85 Test that s3fs automatically detects Content-Type 
References #1217.
 2020-02-04 19:47:09 +09:00
Takeshi Nakatani
018ccb9a11 Suppressed a lot of message output about cache cleanup 2020-01-30 14:42:28 +00:00
Takeshi Nakatani
ee1d3a9057
Merge pull request #1151 from liuyongqing/master 
fix deadlock in clean up cache
 2020-01-30 22:17:50 +09:00
yongqingliu
b762a0a85b fix deadlock due to fdmanager_lock and fdent_data_lock 2020-01-29 12:03:53 +08:00
Takeshi Nakatani
010a6b83ef Not trap chown command errors directly 2020-01-28 10:32:55 +00:00
Takeshi Nakatani
9e77650e8c Fixed removing test file in integration-test-main.sh 2020-01-26 13:32:41 +00:00
Takeshi Nakatani
9789ca1a4d
Merge pull request #1188 from gaul/s3proxy 
Upgrade to S3Proxy 1.7.0
 2019-10-31 23:00:35 +09:00
Andrew Gaul
dc62953040 Upgrade to S3Proxy 1.7.0 
Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.7.0
 2019-10-28 20:11:01 -07:00
Takeshi Nakatani
cf3e82d10a Fixed a rename bug when enable_noobj_cache 2019-10-23 12:39:42 +00:00
Takeshi Nakatani
27a5536749 Fixed test_write_multiple_offsets error on osx 2019-10-14 17:29:39 +00:00
Andrew Gaul
80162c126b Add test for writing to multiple offsets 
References #1098.
 2019-09-25 19:31:08 -07:00
Takeshi Nakatani
1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00
liuyongqing
ae4bcd405c use correct content-type when overwriting and copying files 2019-08-28 22:25:09 -07:00
Andrew Gaul
2daa1d53d9 Verify file size via metadata as well as data 2019-08-20 18:36:44 -07:00
Takeshi Nakatani
a7d83df3eb Fixed typo in #1129 2019-08-14 12:39:38 +00:00
Takeshi Nakatani
ca7266fb76 Modified the test script a little 2019-08-13 14:21:42 +00:00
Takeshi Nakatani
f5bf41cf11 Fixed simple copy with nocpoyapi and modify flag in fdcache 2019-08-11 07:42:48 +00:00
Takeshi Nakatani
d8f1aef7be Added stats cache expire option for test 2019-08-06 10:55:38 +00:00
Takeshi Nakatani
daba563a1c Added auto_cache option for test on osx 2019-08-06 02:44:18 +00:00
Takeshi Nakatani
b79b0b1a92
Merge pull request #1119 from ggtakec/osx_direct_io 
Added direct_io option for test on osx
 2019-08-06 11:14:51 +09:00
Takeshi Nakatani
dc85eda188 Added direct_io option for test on osx 2019-08-05 17:12:49 +00:00
Andrew Gaul
d22acae9a3 Increase test startup retries on Linux 
This matches the number of retries on macOS and allows Valgrind to
work.
 2019-08-03 16:10:42 -07:00
Takeshi Nakatani
8cc008c501
Merge pull request #1111 from gaul/test/read-external-object 
Add test to read an externally created object
 2019-08-03 22:36:34 +09:00
Andrew Gaul
c321c8c23f Add test to read an externally created object 
References #890.
 2019-08-01 16:12:51 -07:00
Andrew Gaul
aba8e6ccfa Test enable_noobj_cache 2019-08-01 12:39:11 -07:00
Takeshi Nakatani
5c3c6bff2f Fixed filter-suite-log.sh for on osx 2019-07-21 15:32:10 +00:00
Takeshi Nakatani
2c4c78fd65
Merge pull request #1102 from gaul/test/concurrent-write 
Add a test for concurrent writes
 2019-07-17 22:24:45 +09:00
Takeshi Nakatani
80f598f439
Merge pull request #1099 from ggtakec/filter_log 
Add log output filter script at test failure
 2019-07-17 22:01:28 +09:00
Andrew Gaul
5cb7a31c09 Add a test for concurrent writes 2019-07-16 21:57:43 -07:00
Takeshi Nakatani
c7f8f61d09 Change test_concurrency test 2019-07-15 15:58:54 -07:00
Takeshi Nakatani
159cd2c682 Add log output filter script at test failure 2019-07-15 06:37:43 +00:00
Andrew Gaul
4a192ffdf9 Run xattr tests on macOS 2019-07-07 00:39:45 -07:00
Takeshi Nakatani
6177d7b096
Merge pull request #1073 from gaul/deps/s3proxy 
Upgrade to S3Proxy 1.6.2
 2019-07-07 15:19:00 +09:00
Takeshi Nakatani
3161bf4608
Merge pull request #1070 from gaul/bash/nounset 
Prohibit pipeline failures
 2019-07-07 15:10:53 +09:00
Takeshi Nakatani
2349dafb98
Merge pull request #1069 from gaul/macos/sed 
Use system sed on macOS
 2019-07-07 15:00:48 +09:00
Andrew Gaul
4f23f38583 Individually test multiple s3fs flags 
Remove unneeded comments; single part limits ensure that the tests
exercise multipart code paths even with smaller files.
References #971.
 2019-07-03 21:09:40 -07:00
Andrew Gaul
f439c6382f Upgrade to S3Proxy 1.6.2 
Notably this includes support for List Objects v2 which AWS CLI uses.
Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.6.2
 2019-07-03 10:37:26 -07:00
Andrew Gaul
9472ee4a01 Prohibit pipeline failures 2019-07-03 02:57:40 -07:00
Andrew Gaul
1f1f824da7 Use system sed on macOS 
This requires fewer developer customizations to work on macOS.
Requires some GNU workarounds.
 2019-07-02 22:17:02 -07:00
Andrew Gaul
c596441f58 Flush file when opening second fd 
Previously when s3fs had dirty local data and an application opened a
second fd it would remove the stat cache entry, query S3 for the size
which was still zero, then reinitialize FdEntry with this incorrect
size.  Instead flush local data to S3 so this query works.  It is
possible that a more involved patch could do this with a less
heavyweight approach but this requires changing open.  This does not
completely fix git clone but allows it to proceed further.  Also make
some cache methods const-correct.  References #839.
 2019-07-02 01:12:09 -07:00
Andrew Gaul
048aea1151 Remove litter from test_concurrency 2019-06-26 20:29:27 -07:00
Takeshi Nakatani
a78d8d1da4
Merge pull request #1049 from gaul/external-modication 
Remove cache file when object time differs
 2019-06-24 00:51:54 +09:00
Andrew Gaul
9e5eaad79b Remove cache file when object time differs 
Check the modification times to determine whether an object has
updated.  This relies on low clock skew between s3fs and the S3
server; a more robust approach could use the ETag.  Fixes #1047.
 2019-06-22 19:09:00 -07:00
Andrew Gaul
1cf3d2452e Correct macOS integration test umount 
Previously errexit could prevent this from succeeding.
 2019-06-19 11:01:32 -07:00
Andrew Gaul
0176fc712b Address shellcheck errors 2019-04-09 21:11:31 +09:00
Andrew Gaul
e5785d4690 Prefer HTTPS links in documentation where possible 
This also corrects a test URL which was HTTPS in practice.
 2019-04-05 17:31:06 +09:00
Jay Caines-Gooby
ddbcec5c96 Work with filenames that include spaces 
• Simplified the stat generation line (single exec using only stat)
• Quote variables so that the cache directory can also include spaces
• while/read loop to handle spaces in cached files
  (IFS was causing problems when all the files were saved into a single variable)
 2019-04-02 10:35:25 +01:00
Andrew Gaul
f53503438c Increase FdEntity reference count when returning 
Previously s3fs had a race condition where one client could delete
FdEntity that another client was using.  Add a simple concurrent test
which previously failed but now succeeds.  Fixes #964.
 2019-03-09 15:55:04 +09:00
Andrew Gaul
e9297f39ea Upgrade to S3Proxy 1.6.1 
Notably, this improve performance when listing objects using the
filesystem backend, allowing new tests like expanding large tar files.
Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.6.1
 2019-02-06 17:06:43 -08:00
Takeshi Nakatani
c0cf90cf8b Checked and corrected all typo 2019-01-27 12:04:29 +00:00
Takeshi Nakatani
d487348d21
Merge pull request #919 from gaul/test/mv-nonempty-dir 
Add test for mv non-empty directory
 2019-01-23 19:21:12 +09:00
Andrew Gaul
79bd3441eb Add test for mv non-empty directory 2019-01-22 22:13:17 -08:00
Andrew Gaul
5f5da4b2cb Load tail range during overwrite 
Previously s3fs experienced data loss when writing to the middle of a
file.  Corrupt files would have the expected data from 0..offset+size
but unexpected NUL bytes from offset+size..EOF.  References #808.
 2019-01-22 22:02:40 -08:00
Andrew Gaul
6401b4ae92 Flush file before renaming 
Previously s3fs could copy the zero-byte stub object without including
any pending writes.  Fixes #145.
 2019-01-17 12:05:10 -08:00
Andrew Gaul
2c43b1e12b Store and retrieve file change time 
This introduces a new header with the change time; existing objects
will report modification time.  Fixes #771.
 2019-01-14 10:05:11 -08:00