octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-12-22 16:58:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,065 Commits 1 Branch 23 Tags 13 MiB
ea42911530
Commit Graph

280 Commits

Author SHA1 Message Date
Andrew Gaul
3fa03d4e1e
Pass explicit -p option to ps (#1904) 
This ensures that a pid follows.
 2022-02-20 20:40:29 +09:00
Takeshi Nakatani
94e8e23eef
Fixed test_external_directory_creation test when cache enabled (#1885) 2022-02-13 13:32:19 +09:00
Andrew Gaul
dbf93c0152
Propagate return code properly (#1884) 
Previously this did not propagate test failures.  A bad rebase
introduced this logic in 495d51113c.
 2022-02-06 22:45:20 +09:00
Takeshi Nakatani
30cf7a50bb Added stat check for subdir created with awscli 2022-01-30 18:31:36 +09:00
Andrew Gaul
74c11ef226
Check bucket before trying to create it (#1874) 
This makes it easier to run tests against S3 other than S3Proxy.
 2022-01-26 23:42:12 +09:00
Takeshi Nakatani
46014397d8 Added test by a shell script static analysis tool(ShellCheck) 2022-01-22 22:23:08 +09:00
Andrew Gaul
6300859c80
Prefer = over == for older shell compatibility (#1857) 2022-01-14 12:40:55 +09:00
Andrew Gaul
2892d3b755
Call curl --fail to propagate exit code (#1856) 2022-01-14 11:52:52 +09:00
Andrew Gaul
577e2bc987
Generate S3Proxy SSL certificate during tests (#1845) 
Also provide CA bundle to AWS CLI to work around CI failures instead
of ignoring errors.  Fixes #1812.
 2022-01-09 15:13:36 +09:00
Andrew Gaul
adb58af17b
Annotate local variables (#1844) 
This prevents collisions with other globals.  Fixes #1843.
 2022-01-09 13:03:36 +09:00
Andrew Gaul
fc7543fa25
Make ut_test compatible with Python 3 (#1838) 
This may allow removing Python 2 on newer distros.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2022-01-09 11:08:36 +09:00
Andrew Gaul
a44ea7c12a
Replace write_multiple_offsets.py with write_multiblock (#1837) 
This reduces the dependency on Python 2.
 2022-01-09 10:51:17 +09:00
Andrew Gaul
e734763002
Remove createbucket option (#1841) 
AWS CLI can do this.  Fixes #1840.
 2022-01-05 01:59:31 +09:00
Andrew Gaul
616db5bf1c
Prefer curl over wget for fetching dependencies (#1836) 
The former is lighter-weight and libcurl is already a dependency for
s3fs.
 2022-01-05 00:43:36 +09:00
Andrew Gaul
07e2e3f72a
Remove sleep 1 from test_update_directory_time (#1803) 
Reduces per-flag test run-time by 5 seconds.
 2021-11-04 08:16:40 +09:00
Andrew Gaul
3cf00626a2
Add option to allow unsigned payloads (#1801) 
This reduces CPU usage of sigv4.  This reduces test run-time by 7
seconds per flag.
 2021-11-01 23:33:55 +09:00
Andrew Gaul
e289915dcb
Remove require-root script (#1800) 
Tests do not require this.
 2021-10-31 10:48:15 +09:00
Andrew Gaul
06dec32965
Use AWS CLI to create explicit times in the past (#1797) 
s3fs can also do this via utimensat but tests should not trust this.
Also break tests into individual functions.  This further reduces test
run-time 8 seconds per flag.
 2021-10-30 10:54:18 +09:00
Andrew Gaul
86317dd185
Replace dd if=/dev/urandom with junk data generator (#1786) 
This reduces test run time for a single flag from 73 to 60 seconds.
 2021-10-28 22:54:25 +09:00
Andrew Gaul
40d2e0d1ad
Reduce sleep time to 1 (#1793) 
This reduces test run-time 15 seconds per flag or 2.5 minutes when
testing all flags.
 2021-10-27 23:47:08 +09:00
Takeshi Nakatani
b6c5069ef7 Fixed the test is multi-block writing by one flush 2021-10-27 08:19:05 +09:00
Andrew Gaul
72a9f40f3f
Update to S3Proxy 1.9.0 (#1788) 
Notably this fixes an issue with the transient provider reading parts
of large files.

Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.9.0
 2021-10-26 23:20:52 +09:00
Andrew Gaul
495d51113c
Remove unneeded sleeps from tests (#1784) 
Also use a unique file name for every test.  This ensures that tests
like test_external_directory_creation and test_external_modification
do not collide.
 2021-10-26 23:19:14 +09:00
Carsten Grohmann
0abeec9cae Simpily errexit modifications 2021-10-26 21:47:36 +09:00
Andrew Gaul
ea3c21f270
Reduce errexit modifications (#1785) 
This is less error prone but requires some magic && ||.
 2021-10-25 23:53:45 +09:00
Takeshi Nakatani
23fe6f4dee Fixed parse_string function in write_multiblock.cc 2021-10-25 17:56:49 +09:00
Takeshi Nakatani
34ea2acd75 Add a test that is multi-block writing by one flush 2021-10-25 17:56:49 +09:00
Takeshi Nakatani
ea64886469 Fixed a bug in test_(zero_)cache_file_stat test function 2021-10-24 18:24:12 +09:00
Takeshi Nakatani
023aaf7dff Fixed wrong stat of cache after new creation file 
And added a test for stat of cache after new creation file
 2021-10-17 16:10:14 +09:00
Takeshi Nakatani
34f89e5936 Added fake_diskfree option to deceive free disk space for test 2021-08-02 11:04:42 +09:00
Andrew Gaul
9cfa177af0
Add flag to run all tests (#1710) 
This allows CI to run the full set of tests while making local
development faster.
 2021-07-02 08:41:47 +09:00
Andrew Gaul
2154e898bc
Fix typos (#1700) 2021-06-27 11:22:33 +09:00
Takeshi Nakatani
f9e80f995d Fixed a bug about rename existing file of different sizes by mpcopy 2021-06-26 00:04:58 +09:00
Andrew Gaul
944c1fd148
Clean up litter in ut_test (#1697) 2021-06-25 19:15:03 +09:00
Takeshi Nakatani
0555c4216e Added test case for utimens is calling during multipart upload 2021-06-21 17:20:24 +09:00
Andrew Gaul
1571379304
Set LC_ALL=en_US.UTF-8 in tests (#1691) 
This allows tests to pass using the S3Proxy filesystem provider.
References #1665.
 2021-06-20 19:27:25 +09:00
Andrew Gaul
f5701fa9ad
Correct typo in undefinedsanitizer config (#1669) 2021-05-29 22:07:33 +09:00
Carsten Grohmann
c0bcb41175
Increase test robustness by adding LC_ALL=C (#1660) 
Use C locale, because some tests check for English expressions

Fixes #1658
 2021-05-20 21:19:50 +09:00
Takeshi Nakatani
9d8f1b00f7 Fixed make_random_string test code being slow on macos 2021-05-08 13:55:39 +09:00
Andrew Gaul
771bbfeac5
Do not create zero-byte object when creating file (#1640) 
Previously s3fs created this object to store metadata and overwrote it
when flushing.  This prevented use with object stores which do not
allow overwrites like HDS.  Instead only create an in-memory
representation which reduces the time to create small files.
Fixes #1013.
 2021-04-30 19:56:33 +09:00
Andrew Gaul
8a51a26819 Allow truncation of open and modified files 
Regression introduced in f5bf41cf11.
Fixes #1575.
 2021-04-25 12:35:35 +09:00
Andrew Gaul
20a6c9d35d
Update to S3Proxy 1.8.0 (#1598) 
Notably this fixes an issue with the filesystem backend which returned
the incorrect Content-Type for some directories.  It also reduces
memory use with GetObject using range requests with large objects.
References #1543.

Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.8.0
 2021-03-28 11:56:10 +09:00
Takeshi Nakatani
a19d223434 Bypass an OSX test(no different from other OS) which takes very time consuming 2021-02-23 20:47:51 +09:00
Takeshi Nakatani
5b5bc3114a Fixed a bug that padded null bytes when changing xattr 2021-02-21 12:23:36 +09:00
Takeshi Nakatani
81e267d421 Fixed the same bug in chmod/chown/utimens and added tests 2021-02-20 23:30:50 +09:00
Takeshi Nakatani
a0f347b10f Added no_time_stamp_ms option mainly for testing 2021-02-20 20:24:26 +09:00
Carsten Grohmann
5debf523b0 Allow changing temporary directory used for tests 
Set temporary directory to $TMPDIR if set or use default /var/tmp
 2021-02-19 13:59:23 +09:00
Carsten Grohmann
1c8aadafd1 Fix typo in an error message of the test suite 2021-02-19 13:59:23 +09:00
Takeshi Nakatani
d33f252404 Fixed forgetting to clean up test files 2021-02-14 00:07:46 +09:00
Andrew Gaul
7f6fbb0021
Correct singlepart_copy_limit unit (#1561) 
This is documented to be in MB not KB.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 22:53:50 +09:00
Andrew Gaul
f2c5e38724
Allow integration tests to use larger files (#1548) 
Previously these failed due to dd only copying 32 MB from /dev/urandom
and exhausting the limited space in /tmp.  References #1543.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-07 12:50:02 +09:00
Andrew Gaul
1987bcbea3
Re-fix retry logic (#1533) 
This logic has always been broken but happened to try one time due to
&& operator precedence.  7158e50ee2
broke this further when quoting && since the command was not
evaluated.
 2021-01-25 18:32:40 +09:00
Andrew Gaul
b04bca37a5
Add configuration for Chaos HTTP Proxy (#1508) 
This can find errors in retry logic.  Chaos HTTP Proxy does not
support SSL bouncestorage/chaos-http-proxy#1 so users must set
s3proxy.endpoint and run via:

CHAOS_HTTP_PROXY=1 S3_URL=http://127.0.0.1:8080 make check -C test

It can also be helpful to increase retries and reduce sleep times.
References #1504.
 2021-01-04 23:32:04 +09:00
Andrew Gaul
7158e50ee2
Quote | and && to allow passing it to retry (#1506) 2021-01-04 22:28:40 +09:00
Takeshi Nakatani
4d0daddad4
Fixed about Github Actions failure(centos8 and macos) (#1498) 2020-12-23 19:34:04 +09:00
Takeshi Nakatani
4341291cc2
Changed a condition of the atime test(fixed #1477) (#1487) 2020-12-03 02:45:05 +09:00
Takeshi Nakatani
c589886ba5
Added a condition to the atime test (#1477) 2020-12-02 23:21:25 +09:00
Andrew Gaul
834862f8a4
Add S3_ENDPOINT test variable (#1476) 2020-11-15 22:10:26 +09:00
Andrew Gaul
ecb24c9c26
Disable preprocessor warnings w/o optimizations (#1473) 
This removes some fortify source warnings.  Also disable failing
memory sanitizer.
 2020-11-09 21:48:07 +09:00
Takeshi Nakatani
a4e4ce8aea
Merge pull request #1440 from gaul/test/undefined-behavior-sanitizer 
Add memory and undefined behavior sanitizer
 2020-10-04 23:19:36 +09:00
Andrew Gaul
7f43b7fa53 Add memory and undefined behavior sanitizer 
Remove Valgrind which duplicates the sanitizers and is much slower.
 2020-10-04 22:55:27 +09:00
Takeshi Nakatani
059cc57ba6 Added atime and Corrected atime/mtime/ctime operations 2020-10-04 13:54:01 +09:00
Andrew Gaul
081d6c1245 Allow setting signature V4-only 
Default to allowing V2 or V4.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
89b1c32b24 Rework s3fs pid handling 
Previously S3FS_PID was not set in the correct shell and thus
ps u $S3FS_PID showed all programs.  This caused the flag detection to
fail when users ran other instances of s3fs with different flags.
References #1402.
 2020-09-18 22:05:37 +09:00
Takeshi Nakatani
76d88f2291
Merge pull request #1403 from gaul/vim/expandtab 
Fix expandtab settings for tests
 2020-09-17 21:35:58 +09:00
Takeshi Nakatani
72340cfbd9
Merge pull request #1401 from gaul/test/hardlink 
Add test for hardlink
 2020-09-17 20:04:07 +09:00
Andrew Gaul
8fb70c5e4a Fix expandtab settings for tests 
This matches the main source.
 2020-09-15 22:11:14 +09:00
Andrew Gaul
78e2345c19 Add test for hardlink 2020-09-15 22:07:25 +09:00
Andrew Gaul
3bc565b986 Add test for concurrent reads 2020-09-15 21:25:51 +09:00
Takeshi Nakatani
b5ffd419d8 Source file division and set 4 spaces and cleanup 2020-08-26 17:43:50 +09:00
Takeshi Nakatani
2feefeec47 Added mv file test when disk space is insufficient 2020-08-16 12:47:29 +00:00
Takeshi Nakatani
f28e3bd89e
Merge pull request #1362 from gaul/test/sanitizers 
Add script to run tests using sanitizers
 2020-08-16 19:15:05 +09:00
Takeshi Nakatani
2bb745cdd7 Fixed a bug about move file over limit of ensure space 2020-08-16 17:42:20 +09:00
Andrew Gaul
c8e13300e1 Add script to run tests using sanitizers 
These currently show several kinds of errors.
 2020-08-15 17:28:35 +09:00
Andrew Gaul
0c1bc0f803 Add portability wrapper for stat(1) 
Fixes #1344.
 2020-08-02 23:14:58 +09:00
Andrew Gaul
4154e539ea Test creating a directory with external program 
References #1264.
 2020-07-25 21:22:53 +09:00
Takeshi Nakatani
62c8be85d6 Interim fix for Travis macOS Build 2020-07-10 09:47:18 +09:00
Andrew Gaul
4d1f5c899f Use Python 2 for write_multiple_offsets 
This aligns with ut_test.py .  Using an older Python also allows
compatibility with the older macOS 10.12 Travis CI.  References #1323.
 2020-07-07 21:12:52 +09:00
Takeshi Nakatani
1f796d432d Fixed upload error about mixuploading sparse file and truncating file 2020-06-27 22:44:19 +09:00
Takeshi Nakatani
35006e318f Fixed about ParallelMixMultipartUpload 2020-06-24 12:48:55 +09:00
Andrew Gaul
8a7548a9d4 Import ossfs tests 
This turns up a failure in test_rename_file when calling seek(0) on a
renamed file.
 2020-06-01 09:08:25 +09:00
Takeshi Nakatani
eb58460175
Merge pull request #1294 from gaul/test/profile 
Allow overriding test credentials with a profile
 2020-05-24 16:14:31 +09:00
Andrew Gaul
73098220bf Allow overriding test credentials with a profile 2020-05-24 12:05:16 +09:00
Andrew Gaul
ca7756fa77 Update to S3Proxy 1.7.1 
Release notes:

https://github.com/gaul/s3proxy/releases/tag/s3proxy-1.7.1
 2020-05-23 10:06:44 +09:00
Andrew Gaul
da70cb92a8 Provide AWS CLI with correct test credentials 
This allows tests to pass against real S3 implementations like
Backblaze.  References #272.
 2020-05-22 19:27:18 +09:00
Andrew Gaul
8945e98d8b Support Google Cloud Storage headers 
This allows s3fs to interpret objects created by gsutil.
 2020-05-03 18:33:13 +09:00
Takeshi Nakatani
ce1221c867 Add test for cache file stat content 2020-04-22 15:53:00 +00:00
Andrew Gaul
3af17c3019 Add test for out-of-order writes 
References #1220.  References #1277.
 2020-04-22 21:48:55 +09:00
Takeshi Nakatani
538fbed302
Merge pull request #1266 from gaul/test/cache-eviction 
Test cache eviction
 2020-04-14 22:30:04 +09:00
Andrew Gaul
a44fc1103d Avoid setting bogus group id 
Addresses a symptom on newer macOS in Travis.
 2020-04-12 22:37:22 +09:00
Andrew Gaul
f373df9682 Test cache eviction 2020-04-11 19:00:38 +09:00
Takeshi Nakatani
7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
Takeshi Nakatani
0665d78550 Temporary measure: reduce test time on MacOS 2020-02-04 15:38:19 +00:00
Takeshi Nakatani
55d670f22f Change content-types test for MacOS 2020-02-04 14:51:50 +00:00
Takeshi Nakatani
32ae0d2c79 Bypassed MacOS ensure_diskfree test 2020-02-04 14:00:47 +00:00
Andrew Gaul
bc9126d774 Set directory MIME type to application/x-directory 
Previously s3fs auto-detected the MIME type of directories like
"TOYOTA TRUCK 8.2.2" as application/x-troff-man.  This caused get_mode
to not set S_IFDIR which failed directory creation.  Instead force all
object names ending in / to application/x-directory.  Fixes #1183.
 2020-02-04 20:03:21 +09:00
Andrew Gaul
4df50e7f85 Test that s3fs automatically detects Content-Type 
References #1217.
 2020-02-04 19:47:09 +09:00
Takeshi Nakatani
018ccb9a11 Suppressed a lot of message output about cache cleanup 2020-01-30 14:42:28 +00:00
Takeshi Nakatani
ee1d3a9057
Merge pull request #1151 from liuyongqing/master 
fix deadlock in clean up cache
 2020-01-30 22:17:50 +09:00
yongqingliu
b762a0a85b fix deadlock due to fdmanager_lock and fdent_data_lock 2020-01-29 12:03:53 +08:00