octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-16 17:25:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,405 Commits 1 Branch 23 Tags 13 MiB
80162c126b
Commit Graph

734 Commits

Author SHA1 Message Date
Takeshi Nakatani
d39e4e4b1f
Merge pull request #1087 from gaul/clang-tidy/deprecated-headers 
Prefer modern C headers
 2019-07-15 03:50:15 +09:00
Lars Christensen
58037da061 Fix multipart upload not setting curl options 2019-07-14 14:09:37 +02:00
Andrew Gaul
1eb266588e Add a missing lock to SetCtime 
Also lock before log statements which touch member variables.
 2019-07-13 17:53:38 -07:00
root
4e351c59e3 fix coredump caused by fd_manager_lock locking 2019-07-13 12:03:27 +08:00
Andrew Gaul
eb597289cb Avoid null dereference 
Found via clang-tidy.
 2019-07-12 18:40:24 -07:00
Andrew Gaul
6fd42d9fe4 Prefer modern C headers 
Found and fixed via clang-tidy.
 2019-07-12 03:50:59 -07:00
Andrew Gaul
efff9c01a6 Avoid misplaced const via removing unneeded typedef 
Found via clang-tidy.
 2019-07-12 03:41:19 -07:00
Andrew Gaul
a83d5baa90 Remove recursive locking 
Recursive locking is frowned upon and is incompatible with
PTHREAD_MUTEX_ERRORCHECK.  Also clean up pthread_mutex_lock error
checking.
 2019-07-10 12:39:00 -07:00
Takeshi Nakatani
50d13255e4
Merge pull request #1081 from gaul/write-blocks-readdir 
Break FdEntity locks into data and metadata
 2019-07-11 03:40:26 +09:00
Takeshi Nakatani
e5e63d6ac3
Merge pull request #1067 from gaul/mismatched-free 
Fix mismatched new[]/free
 2019-07-07 16:55:07 +09:00
Takeshi Nakatani
7a65a414c3
Merge pull request #1080 from gaul/xml/simplification 
Add simple XML parsing wrapper
 2019-07-07 16:44:18 +09:00
Andrew Gaul
944d21cabb Fix mismatched new[]/free 
Found via Valgrind.
 2019-07-07 00:36:48 -07:00
Takeshi Nakatani
d267212289
Merge pull request #1079 from gaul/cppcheck/shadowing 
Avoid shadowing variable in FdEntity::Open
 2019-07-07 16:29:42 +09:00
Takeshi Nakatani
58d8e5586a
Merge pull request #1078 from gaul/multipart/2x 
Issue multipart when object size exceeds part size
 2019-07-07 16:21:22 +09:00
Takeshi Nakatani
ce803daf4a
Merge pull request #1077 from gaul/openssl/locking 
Annotate OpenSSL locking functions as unused
 2019-07-07 16:10:14 +09:00
Takeshi Nakatani
9bf34e2fda
Merge pull request #1076 from gaul/doc/man-help 
Add documentation for use_session_token and use_rrs
 2019-07-07 15:45:42 +09:00
Takeshi Nakatani
6bd1a7eac0
Merge pull request #1074 from gaul/clang-tidy 
Configure clang-tidy target
 2019-07-07 15:28:08 +09:00
Andrew Gaul
8aa06d621a Add documentation for use_session_token and use_rrs 
Fixes #929.
 2019-07-06 10:04:43 -07:00
Andrew Gaul
ecf13a8cb9 Break FdEntity locks into data and metadata 
Previously long-running data operations like RowFlush would block
metadata operations like GetStats and thus user readdir.  Fixes #928.
 2019-07-05 23:12:24 -07:00
Andrew Gaul
b8ff6a647e Add simple XML parsing wrapper 
Also simplify check_region_error.
 2019-07-05 17:39:09 -07:00
Andrew Gaul
49110c671d Avoid shadowing variable in FdEntity::Open 
Found via cppcheck 1.88.
 2019-07-05 15:06:18 -07:00
Andrew Gaul
febaf6849f Issue multipart when object size exceeds part size 
Previously s3fs issued multipart uploads when the object size was
twice the part size.  Conjoining this with the part size was confusing
and s3fs should add a separate tunable for this if needed, similar to
singlepart_copy_limit.  Fixes #1058.
 2019-07-05 12:14:56 -07:00
Andrew Gaul
4893174652 Annotate OpenSSL locking functions as unused 
OpenSSL 1.1.0 removed these and generates warnings with newer
versions.
 2019-07-05 10:45:57 -07:00
Andrew Gaul
4f23f38583 Individually test multiple s3fs flags 
Remove unneeded comments; single part limits ensure that the tests
exercise multipart code paths even with smaller files.
References #971.
 2019-07-03 21:09:40 -07:00
Andrew Gaul
bbfa91141a Configure clang-tidy target 
Also fix nits.
 2019-07-03 14:04:11 -07:00
Takeshi Nakatani
21321a9d96
Merge pull request #1065 from gaul/doc/ssl-verify-hostname 
Document ssl_verify_hostname
 2019-07-03 22:15:33 +09:00
Takeshi Nakatani
f03b50fd13
Merge pull request #1063 from gaul/truncate-2nd-open-fd 
Flush file when opening second fd
 2019-07-03 21:14:03 +09:00
Andrew Gaul
f02105c346 Document ssl_verify_hostname 
References #1064.
 2019-07-02 10:26:29 -07:00
Andrew Gaul
c596441f58 Flush file when opening second fd 
Previously when s3fs had dirty local data and an application opened a
second fd it would remove the stat cache entry, query S3 for the size
which was still zero, then reinitialize FdEntry with this incorrect
size.  Instead flush local data to S3 so this query works.  It is
possible that a more involved patch could do this with a less
heavyweight approach but this requires changing open.  This does not
completely fix git clone but allows it to proceed further.  Also make
some cache methods const-correct.  References #839.
 2019-07-02 01:12:09 -07:00
Andrew Gaul
455e29cbea Make fdpage a value type in fdpage_list_t 
This simplifies memory management.
 2019-07-01 09:30:48 -07:00
Takeshi Nakatani
511d223468
Merge pull request #1059 from gaul/casting 
Fix casting warning on 32-bit
 2019-07-01 22:26:14 +09:00
Andrew Gaul
d7f77a6282 Fix casting warning on 32-bit 2019-06-29 20:57:42 -07:00
Andrew Gaul
f1ad626b46 Log flags as hexadecimal 2019-06-24 16:13:07 -07:00
Takeshi Nakatani
a78d8d1da4
Merge pull request #1049 from gaul/external-modication 
Remove cache file when object time differs
 2019-06-24 00:51:54 +09:00
Takeshi Nakatani
fbebc6fa57
Merge pull request #1048 from gaul/log/curl 
Emit older curl warnings only once
 2019-06-24 00:42:44 +09:00
Andrew Gaul
9e5eaad79b Remove cache file when object time differs 
Check the modification times to determine whether an object has
updated.  This relies on low clock skew between s3fs and the S3
server; a more robust approach could use the ETag.  Fixes #1047.
 2019-06-22 19:09:00 -07:00
Andrew Gaul
738eaadcbf Emit older curl warnings only once 
This makes Travis logs actually readable.
 2019-06-22 10:01:20 -07:00
Andrew Gaul
15b7450713 Convert file offsets from size_t to off_t 
The latter is 64-bits on 32-bit platforms when specifying
-D_FILE_OFFSET_BITS=64.  This allows early Raspberry Pis to use files
larger than 2 GB.  It also cleans up some ugly casting.  Fixes #620.
Fixes #656.
 2019-06-15 17:05:37 -07:00
Andrew Gaul
059ab1f0f4 Update ctime during nocopyapi operations 
Follows on to 2c43b1e12b.  Fixes #971.
 2019-04-30 18:59:51 +09:00
Michal Lula
0d4847596e fix issue with aws session token provided inside ~/.aws/credentials file 2019-04-25 14:29:35 +02:00
Takeshi Nakatani
8e86ef1634
Merge pull request #1015 from gaul/c++03 
Remove uses of std::map::at
 2019-04-22 23:01:29 +09:00
Andrew Gaul
9e4f9d4bdc Remove uses of std::map::at 
This provides compatibility with C++03 and removes duplicate lookups.
 2019-04-18 16:06:59 +09:00
Andrew Gaul
0677a096a9 Prefer simple over compound statements in macros 
This prohibits missing semicolons.
 2019-04-17 21:46:24 +09:00
Michal Lula
f9cd43b684 add session token support 2019-04-16 16:53:05 +02:00
Andrew Gaul
8a18806a57 Make man page and --help more consistent 
Mostly small fixes but also some reordering.  References #929.  Found
via:

diff -u <(man --no-hyphenation doc/man/s3fs.1 | tr -s ' ' '\n' | sed '/^-o$/d' ) <(src/s3fs --help | tr -s ' ' '\n' | sed '/^-$/d')
 2019-04-14 14:34:24 +09:00
Takeshi Nakatani
0a99470369
Merge pull request #1002 from gaul/sse-c 
Correctly calculate MD5 hash for SSE-C keys
 2019-04-09 00:17:12 +09:00
Takeshi Nakatani
cd280d8702
Merge pull request #1001 from gaul/missing-braces 
Add missing braces
 2019-04-08 23:55:02 +09:00
Andrew Gaul
6f8ecb28c5 Correctly calculate MD5 hash for SSE-C keys 
Previously s3fs calculated the strlen of a binary input instead of
using the stored length.  This yielded IO errors when interacting with
SSE-encrypted objects.  Fixes #696.
 2019-04-08 21:49:42 +09:00
Andrew Gaul
042332bcec Add missing braces 
Found via clang-tidy.  Also fix errant indentation.
 2019-04-07 23:12:27 +09:00
Takeshi Nakatani
071cd0f849 Modified to discard _netdev option etc 2019-04-07 11:51:55 +00:00
Andrew Gaul
6f6a67807b Prefer new over malloc 
The former cannot return NULL and will allow use of scoped pointers.
 2019-04-07 16:54:24 +09:00
Takeshi Nakatani
ea517c80a4 Fixed a bug about overwrite metadata at updating directory stats 2019-03-29 15:30:30 +00:00
Takeshi Nakatani
71766039ff Support undefined CURLoption in libcurl library used in build 2019-03-22 10:47:42 +00:00
Andrew Gaul
058706014b Simplify string comparison 
Found via newer cppcheck.
 2019-03-14 22:19:33 +09:00
Takeshi Nakatani
62dcda6a56 Fixed ref-count when error occurred. 2019-03-10 06:04:19 +00:00
Takeshi Nakatani
cbf072bc55
Merge pull request #967 from gaul/reference-count 
Increase FdEntity reference count when returning
 2019-03-10 14:27:10 +09:00
Takeshi Nakatani
b5ca400500
Merge pull request #970 from gaul/nocopyapi 
Remove from file from stat cache during rename
 2019-03-10 12:56:04 +09:00
Andrew Gaul
08b132ddb9 Remove from file from stat cache during rename 
This addresses failures with test_rename_before_close when using
nocopyapi.  Note that test_update_time still fails.
 2019-03-09 17:53:35 +09:00
Andrew Gaul
1e86cc643d Fix lazy typo 2019-03-09 17:46:02 +09:00
Andrew Gaul
f53503438c Increase FdEntity reference count when returning 
Previously s3fs had a race condition where one client could delete
FdEntity that another client was using.  Add a simple concurrent test
which previously failed but now succeeds.  Fixes #964.
 2019-03-09 15:55:04 +09:00
Takeshi Nakatani
0d43d070cc
Merge pull request #965 from ggtakec/fix_multi_x 
Improvement of curl session pool for multipart
 2019-02-27 22:44:25 +09:00
Takeshi Nakatani
0791fdca2a
Merge pull request #960 from kristjanvalur/wtf8 
Wtf8
 2019-02-27 21:21:38 +09:00
Kristján Valur Jónsson
6e8678d5e3 remove lines that wer accidentally duplicated 2019-02-25 16:02:01 +00:00
Takeshi Nakatani
10d9f75366 Improvement of curl session pool for multipart 
Improvement of curl session pool for multipart(2)

Improvement of curl session pool for multipart(3)
 2019-02-25 14:46:24 +00:00
Andrew Gaul
74d8671e54 Work around cppcheck warnings 
Follows on to eeb839242b.
 2019-02-20 21:55:41 +09:00
Kristján Valur Jónsson
4c41eac29c fix documentation and man page 2019-02-20 11:24:29 +00:00
Kristján Valur Jónsson
3c97c1b251 merged main 2019-02-19 10:53:00 +00:00
Kristján Valur Jónsson
84c671a81a fix indentation 2019-02-19 10:37:43 +00:00
Kristján Valur Jónsson
f336bdebcc add command line flag and documentation 2019-02-19 10:32:37 +00:00
Kristján Valur Jónsson
e5b8377202 fix comments and code 2019-02-19 10:32:10 +00:00
Kristján Valur Jónsson
4f42f4ab0c Enable s3fs encoding and decoding in the fuse interface 2019-02-18 13:36:48 +00:00
Kristján Valur Jónsson
11b385820d more robust wtf8 encoding 2019-02-18 12:27:44 +00:00
Takeshi Nakatani
eeb839242b Fixed code for latest cppcheck error on OSX 2019-02-17 13:59:11 +00:00
Kristján Valur Jónsson
ca2d1d873d Adding utility functions to convert invalid utf8 to wtf8 encoding 
This is to deal with windows clients who pass in cp1252 as if it
were utf8
 2019-02-15 15:57:03 +00:00
Takeshi Nakatani
231fd001d9 Added a non-interactive option to utility mode 
Rebase for resolving conflicts

Deleted interactive mode on utility mode

Fixed minor mistakes
 2019-02-10 17:36:18 +00:00
Takeshi Nakatani
314dc5a398
Merge pull request #955 from gaul/clang-tidy/misc 
Address miscellaneous clang-tidy warnings
 2019-02-06 21:21:54 +09:00
Takeshi Nakatani
e07cb020cc
Merge pull request #954 from gaul/clear-iter 
Clear containers instead of individual erases
 2019-02-06 21:12:36 +09:00
Andrew Gaul
e87e40b3b4 Address miscellaneous clang-tidy warnings 
Also add configuration with suppressions.
 2019-02-04 22:58:40 -08:00
Takeshi Nakatani
bd66b57ad3
Merge pull request #952 from gaul/multipart/failed 
Automatically abort failed multipart requests
 2019-02-04 22:19:27 +09:00
Andrew Gaul
a1d3ff9766 Automatically abort failed multipart requests 
This can avoid dangling parts.  However, many transfers fail due to
network errors so we still need other mechanisms to handle these
parts.
 2019-02-03 10:29:20 -08:00
Andrew Gaul
4d0bef1e90 Clear containers instead of individual erases 
This has O(n) runtime instead of O(n log n).
 2019-02-02 23:58:43 -08:00
Takeshi Nakatani
960823fb40 Added S3FS_MALLOC_TRIM build switch 2019-02-03 07:36:17 +00:00
Takeshi Nakatani
c04e8e7a9d
Merge pull request #949 from gaul/503-slow-down 
Implement exponential backoff for 503
 2019-02-03 15:19:28 +09:00
Takeshi Nakatani
fb6debd986
Merge pull request #948 from gaul/too-many-parts 
Add logging for too many parts
 2019-02-03 14:55:54 +09:00
Takeshi Nakatani
d8185a25aa
Merge pull request #946 from gaul/async/completed-tids 
Simplify async request completion code
 2019-02-03 14:08:49 +09:00
Takeshi Nakatani
53337a0a28
Merge pull request #944 from gaul/utility-mode 
Repair utility mode
 2019-02-03 13:59:56 +09:00
Takeshi Nakatani
ae51556d04
Merge pull request #943 from gaul/hard-link 
Return not supported when hard linking
 2019-02-03 12:45:21 +09:00
Takeshi Nakatani
055ecf6ea7
Merge pull request #940 from gaul/parallel-multipart-copy 
Copy parts in parallel
 2019-02-03 11:44:15 +09:00
Takeshi Nakatani
c603680e02
Merge pull request #939 from gaul/stringstream-specific 
Prefer specific [io]stringstream where possible
 2019-02-03 11:06:57 +09:00
Takeshi Nakatani
814aadd7e3
Merge pull request #938 from gaul/clang-tidy/redundant-void 
Remove unneeded void parameter
 2019-02-03 10:57:56 +09:00
Andrew Gaul
8ff05d8e38 Implement exponential backoff for 503 
Amazon returns SlowDown when overloaded.  Also return ENOTSUP for 501
and immediately return EIO for 500 instead of retrying.  Fixes #603.
 2019-02-02 00:03:50 -08:00
Andrew Gaul
dfa84b82a8 Add logging for too many parts 
References #610.
 2019-02-01 19:34:26 -08:00
Andrew Gaul
8c527c3616 Simplify async request completion code 
Workers now notify the master thread when they complete, unifying the
Linux and macOS code paths.  This also avoids excessive
pthread_tryjoin_np calls.  Follows on to
88cd8feb05.
 2019-01-31 22:55:23 -08:00
Andrew Gaul
54a074647e Repair utility mode 
This deinitialized S3fsCurl twice and incorrectly calculated V4
signatures.
 2019-01-31 18:45:39 -08:00
Andrew Gaul
c5ebf5d328 Copy parts in parallel 
S3 can copy multipart much faster than single part due to IO
parallelization.  Renaming a 4 GB file reduces from 72 to 20 seconds
with bigger gains with larger files.
 2019-01-31 10:21:39 -08:00
Andrew Gaul
43c6ef560e Return not supported when hard linking 
This is more correct than permission denied.
 2019-01-30 16:43:04 -08:00
Andrew Gaul
3076abc744 Disable malloc_trim 
This avoids walking the entire heap multiple times for complex
operations like readdir.  This does not entirely eliminate the
observed performance regression but does dramatically reduce s3fs CPU
usage.  References #935.  Fixes #936.
 2019-01-29 15:29:07 -08:00
Andrew Gaul
07636c8a8d Prefer specific [io]stringstream where possible 
These better communicate intent and are slightly more efficient.
 2019-01-29 10:44:33 -08:00
Andrew Gaul
35d55ee513 Remove unneeded void parameter 
This is implicit in C++.  Found and fixed via clang-tidy.
 2019-01-28 23:22:27 -08:00
Takeshi Nakatani
c0cf90cf8b Checked and corrected all typo 2019-01-27 12:04:29 +00:00