octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-12-25 09:59:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,624 Commits 1 Branch 23 Tags 13 MiB
462347256d
Commit Graph

851 Commits

Author SHA1 Message Date
Takeshi Nakatani
a78d8d1da4
Merge pull request #1049 from gaul/external-modication 
Remove cache file when object time differs
 2019-06-24 00:51:54 +09:00
Takeshi Nakatani
fbebc6fa57
Merge pull request #1048 from gaul/log/curl 
Emit older curl warnings only once
 2019-06-24 00:42:44 +09:00
Andrew Gaul
9e5eaad79b Remove cache file when object time differs 
Check the modification times to determine whether an object has
updated.  This relies on low clock skew between s3fs and the S3
server; a more robust approach could use the ETag.  Fixes #1047.
 2019-06-22 19:09:00 -07:00
Andrew Gaul
738eaadcbf Emit older curl warnings only once 
This makes Travis logs actually readable.
 2019-06-22 10:01:20 -07:00
Andrew Gaul
15b7450713 Convert file offsets from size_t to off_t 
The latter is 64-bits on 32-bit platforms when specifying
-D_FILE_OFFSET_BITS=64.  This allows early Raspberry Pis to use files
larger than 2 GB.  It also cleans up some ugly casting.  Fixes #620.
Fixes #656.
 2019-06-15 17:05:37 -07:00
Andrew Gaul
059ab1f0f4 Update ctime during nocopyapi operations 
Follows on to 2c43b1e12b.  Fixes #971.
 2019-04-30 18:59:51 +09:00
Michal Lula
0d4847596e fix issue with aws session token provided inside ~/.aws/credentials file 2019-04-25 14:29:35 +02:00
Takeshi Nakatani
8e86ef1634
Merge pull request #1015 from gaul/c++03 
Remove uses of std::map::at
 2019-04-22 23:01:29 +09:00
Andrew Gaul
9e4f9d4bdc Remove uses of std::map::at 
This provides compatibility with C++03 and removes duplicate lookups.
 2019-04-18 16:06:59 +09:00
Andrew Gaul
0677a096a9 Prefer simple over compound statements in macros 
This prohibits missing semicolons.
 2019-04-17 21:46:24 +09:00
Michal Lula
f9cd43b684 add session token support 2019-04-16 16:53:05 +02:00
Andrew Gaul
8a18806a57 Make man page and --help more consistent 
Mostly small fixes but also some reordering.  References #929.  Found
via:

diff -u <(man --no-hyphenation doc/man/s3fs.1 | tr -s ' ' '\n' | sed '/^-o$/d' ) <(src/s3fs --help | tr -s ' ' '\n' | sed '/^-$/d')
 2019-04-14 14:34:24 +09:00
Takeshi Nakatani
0a99470369
Merge pull request #1002 from gaul/sse-c 
Correctly calculate MD5 hash for SSE-C keys
 2019-04-09 00:17:12 +09:00
Takeshi Nakatani
cd280d8702
Merge pull request #1001 from gaul/missing-braces 
Add missing braces
 2019-04-08 23:55:02 +09:00
Andrew Gaul
6f8ecb28c5 Correctly calculate MD5 hash for SSE-C keys 
Previously s3fs calculated the strlen of a binary input instead of
using the stored length.  This yielded IO errors when interacting with
SSE-encrypted objects.  Fixes #696.
 2019-04-08 21:49:42 +09:00
Andrew Gaul
042332bcec Add missing braces 
Found via clang-tidy.  Also fix errant indentation.
 2019-04-07 23:12:27 +09:00
Takeshi Nakatani
071cd0f849 Modified to discard _netdev option etc 2019-04-07 11:51:55 +00:00
Andrew Gaul
6f6a67807b Prefer new over malloc 
The former cannot return NULL and will allow use of scoped pointers.
 2019-04-07 16:54:24 +09:00
Takeshi Nakatani
ea517c80a4 Fixed a bug about overwrite metadata at updating directory stats 2019-03-29 15:30:30 +00:00
Takeshi Nakatani
71766039ff Support undefined CURLoption in libcurl library used in build 2019-03-22 10:47:42 +00:00
Andrew Gaul
058706014b Simplify string comparison 
Found via newer cppcheck.
 2019-03-14 22:19:33 +09:00
Takeshi Nakatani
62dcda6a56 Fixed ref-count when error occurred. 2019-03-10 06:04:19 +00:00
Takeshi Nakatani
cbf072bc55
Merge pull request #967 from gaul/reference-count 
Increase FdEntity reference count when returning
 2019-03-10 14:27:10 +09:00
Takeshi Nakatani
b5ca400500
Merge pull request #970 from gaul/nocopyapi 
Remove from file from stat cache during rename
 2019-03-10 12:56:04 +09:00
Andrew Gaul
08b132ddb9 Remove from file from stat cache during rename 
This addresses failures with test_rename_before_close when using
nocopyapi.  Note that test_update_time still fails.
 2019-03-09 17:53:35 +09:00
Andrew Gaul
1e86cc643d Fix lazy typo 2019-03-09 17:46:02 +09:00
Andrew Gaul
f53503438c Increase FdEntity reference count when returning 
Previously s3fs had a race condition where one client could delete
FdEntity that another client was using.  Add a simple concurrent test
which previously failed but now succeeds.  Fixes #964.
 2019-03-09 15:55:04 +09:00
Takeshi Nakatani
0d43d070cc
Merge pull request #965 from ggtakec/fix_multi_x 
Improvement of curl session pool for multipart
 2019-02-27 22:44:25 +09:00
Takeshi Nakatani
0791fdca2a
Merge pull request #960 from kristjanvalur/wtf8 
Wtf8
 2019-02-27 21:21:38 +09:00
Kristján Valur Jónsson
6e8678d5e3 remove lines that wer accidentally duplicated 2019-02-25 16:02:01 +00:00
Takeshi Nakatani
10d9f75366 Improvement of curl session pool for multipart 
Improvement of curl session pool for multipart(2)

Improvement of curl session pool for multipart(3)
 2019-02-25 14:46:24 +00:00
Andrew Gaul
74d8671e54 Work around cppcheck warnings 
Follows on to eeb839242b.
 2019-02-20 21:55:41 +09:00
Kristján Valur Jónsson
4c41eac29c fix documentation and man page 2019-02-20 11:24:29 +00:00
Kristján Valur Jónsson
3c97c1b251 merged main 2019-02-19 10:53:00 +00:00
Kristján Valur Jónsson
84c671a81a fix indentation 2019-02-19 10:37:43 +00:00
Kristján Valur Jónsson
f336bdebcc add command line flag and documentation 2019-02-19 10:32:37 +00:00
Kristján Valur Jónsson
e5b8377202 fix comments and code 2019-02-19 10:32:10 +00:00
Kristján Valur Jónsson
4f42f4ab0c Enable s3fs encoding and decoding in the fuse interface 2019-02-18 13:36:48 +00:00
Kristján Valur Jónsson
11b385820d more robust wtf8 encoding 2019-02-18 12:27:44 +00:00
Takeshi Nakatani
eeb839242b Fixed code for latest cppcheck error on OSX 2019-02-17 13:59:11 +00:00
Kristján Valur Jónsson
ca2d1d873d Adding utility functions to convert invalid utf8 to wtf8 encoding 
This is to deal with windows clients who pass in cp1252 as if it
were utf8
 2019-02-15 15:57:03 +00:00
Takeshi Nakatani
231fd001d9 Added a non-interactive option to utility mode 
Rebase for resolving conflicts

Deleted interactive mode on utility mode

Fixed minor mistakes
 2019-02-10 17:36:18 +00:00
Takeshi Nakatani
314dc5a398
Merge pull request #955 from gaul/clang-tidy/misc 
Address miscellaneous clang-tidy warnings
 2019-02-06 21:21:54 +09:00
Takeshi Nakatani
e07cb020cc
Merge pull request #954 from gaul/clear-iter 
Clear containers instead of individual erases
 2019-02-06 21:12:36 +09:00
Andrew Gaul
e87e40b3b4 Address miscellaneous clang-tidy warnings 
Also add configuration with suppressions.
 2019-02-04 22:58:40 -08:00
Takeshi Nakatani
bd66b57ad3
Merge pull request #952 from gaul/multipart/failed 
Automatically abort failed multipart requests
 2019-02-04 22:19:27 +09:00
Andrew Gaul
a1d3ff9766 Automatically abort failed multipart requests 
This can avoid dangling parts.  However, many transfers fail due to
network errors so we still need other mechanisms to handle these
parts.
 2019-02-03 10:29:20 -08:00
Andrew Gaul
4d0bef1e90 Clear containers instead of individual erases 
This has O(n) runtime instead of O(n log n).
 2019-02-02 23:58:43 -08:00
Takeshi Nakatani
960823fb40 Added S3FS_MALLOC_TRIM build switch 2019-02-03 07:36:17 +00:00
Takeshi Nakatani
c04e8e7a9d
Merge pull request #949 from gaul/503-slow-down 
Implement exponential backoff for 503
 2019-02-03 15:19:28 +09:00
Takeshi Nakatani
fb6debd986
Merge pull request #948 from gaul/too-many-parts 
Add logging for too many parts
 2019-02-03 14:55:54 +09:00
Takeshi Nakatani
d8185a25aa
Merge pull request #946 from gaul/async/completed-tids 
Simplify async request completion code
 2019-02-03 14:08:49 +09:00
Takeshi Nakatani
53337a0a28
Merge pull request #944 from gaul/utility-mode 
Repair utility mode
 2019-02-03 13:59:56 +09:00
Takeshi Nakatani
ae51556d04
Merge pull request #943 from gaul/hard-link 
Return not supported when hard linking
 2019-02-03 12:45:21 +09:00
Takeshi Nakatani
055ecf6ea7
Merge pull request #940 from gaul/parallel-multipart-copy 
Copy parts in parallel
 2019-02-03 11:44:15 +09:00
Takeshi Nakatani
c603680e02
Merge pull request #939 from gaul/stringstream-specific 
Prefer specific [io]stringstream where possible
 2019-02-03 11:06:57 +09:00
Takeshi Nakatani
814aadd7e3
Merge pull request #938 from gaul/clang-tidy/redundant-void 
Remove unneeded void parameter
 2019-02-03 10:57:56 +09:00
Andrew Gaul
8ff05d8e38 Implement exponential backoff for 503 
Amazon returns SlowDown when overloaded.  Also return ENOTSUP for 501
and immediately return EIO for 500 instead of retrying.  Fixes #603.
 2019-02-02 00:03:50 -08:00
Andrew Gaul
dfa84b82a8 Add logging for too many parts 
References #610.
 2019-02-01 19:34:26 -08:00
Andrew Gaul
8c527c3616 Simplify async request completion code 
Workers now notify the master thread when they complete, unifying the
Linux and macOS code paths.  This also avoids excessive
pthread_tryjoin_np calls.  Follows on to
88cd8feb05.
 2019-01-31 22:55:23 -08:00
Andrew Gaul
54a074647e Repair utility mode 
This deinitialized S3fsCurl twice and incorrectly calculated V4
signatures.
 2019-01-31 18:45:39 -08:00
Andrew Gaul
c5ebf5d328 Copy parts in parallel 
S3 can copy multipart much faster than single part due to IO
parallelization.  Renaming a 4 GB file reduces from 72 to 20 seconds
with bigger gains with larger files.
 2019-01-31 10:21:39 -08:00
Andrew Gaul
43c6ef560e Return not supported when hard linking 
This is more correct than permission denied.
 2019-01-30 16:43:04 -08:00
Andrew Gaul
3076abc744 Disable malloc_trim 
This avoids walking the entire heap multiple times for complex
operations like readdir.  This does not entirely eliminate the
observed performance regression but does dramatically reduce s3fs CPU
usage.  References #935.  Fixes #936.
 2019-01-29 15:29:07 -08:00
Andrew Gaul
07636c8a8d Prefer specific [io]stringstream where possible 
These better communicate intent and are slightly more efficient.
 2019-01-29 10:44:33 -08:00
Andrew Gaul
35d55ee513 Remove unneeded void parameter 
This is implicit in C++.  Found and fixed via clang-tidy.
 2019-01-28 23:22:27 -08:00
Takeshi Nakatani
c0cf90cf8b Checked and corrected all typo 2019-01-27 12:04:29 +00:00
Takeshi Nakatani
3b1cc3b197
Merge pull request #933 from gaul/cache/remove-mirror-path 
Remove mirror path when deleting cache
 2019-01-27 16:15:49 +09:00
Takeshi Nakatani
a0c1f30ae7
Merge pull request #932 from gaul/autolock 
Prefer AutoLock for synchronization
 2019-01-27 15:59:18 +09:00
Takeshi Nakatani
8822a86709
Merge pull request #931 from gaul/doc/typo 
Correct sigv2 typo
 2019-01-27 15:48:24 +09:00
Takeshi Nakatani
98f397de0e
Merge pull request #930 from gaul/doc/md5-multipart 
Correct enable_content_md5 docs
 2019-01-27 15:36:56 +09:00
Takeshi Nakatani
fd4d23f8f7
Merge pull request #926 from kzidane/master 
Accept paths with : in them
 2019-01-27 15:23:24 +09:00
Takeshi Nakatani
4820f0a42b
Merge pull request #925 from gaul/clang-tidy/delete-null 
Remove redundant null checks before delete
 2019-01-27 15:15:31 +09:00
Andrew Gaul
a93e500b44 Remove mirror path when deleting cache 
Fixes #827.
 2019-01-25 18:10:03 -08:00
Andrew Gaul
92d3114584 Prefer AutoLock for synchronization 
This simplifies the code and fixes an issue with unlocked access.
Also use a recursive lock for StatCache to avoid races between
lock..unlock..lock sequences.
 2019-01-25 15:28:41 -08:00
Andrew Gaul
5062d6fbd9 Correct sigv2 typo 2019-01-25 14:30:30 -08:00
Andrew Gaul
7d14ebaf09 Correct enable_content_md5 docs 
Both S3fsCurl::PutRequest and S3fsCurl::UploadMultipartPostSetup can
calculate and send Content-MD5 to the server.  Remove spurious comment
about large files and make man page and help consistent.
References #929.
 2019-01-25 14:27:52 -08:00
Kareem Zidane
cd794a6985 Accept paths with : in them 2019-01-23 14:56:25 -05:00
Andrew Gaul
84b421d6ef Prefer empty over size checks 
Found and fixed via clang-tidy.
 2019-01-23 11:30:28 -08:00
Andrew Gaul
8316da5bbe Remove redundant null checks before delete 
Found by clang-tidy.
 2019-01-23 11:25:25 -08:00
Takeshi Nakatani
caaf4cac55 Reverted #912(Automatic region change) and added message 2019-01-23 13:23:03 +00:00
Takeshi Nakatani
010276ceab
Merge pull request #921 from gaul/clang-tidy/redundant-string-init 
Remove redundant string initializations
 2019-01-23 19:44:59 +09:00
Takeshi Nakatani
f219817eb3
Merge pull request #920 from gaul/clang-tidy/string-copy 
Remove unnecessary string copies
 2019-01-23 19:30:55 +09:00
Andrew Gaul
1fc25e8c3f Remove redundant string initializations 
Found and fixed via clang-tidy.
 2019-01-22 23:16:37 -08:00
Andrew Gaul
61ecafd426 Remove unnecessary string copies 
Found via clang-tidy.
 2019-01-22 23:09:37 -08:00
Andrew Gaul
5f5da4b2cb Load tail range during overwrite 
Previously s3fs experienced data loss when writing to the middle of a
file.  Corrupt files would have the expected data from 0..offset+size
but unexpected NUL bytes from offset+size..EOF.  References #808.
 2019-01-22 22:02:40 -08:00
Takeshi Nakatani
fada95f58e
Merge pull request #914 from gaul/readdir/head-of-line 
Issue readdir HEAD requests without batching
 2019-01-21 22:10:50 +09:00
Takeshi Nakatani
014b8c5982
Merge pull request #913 from gaul/assert 
Prefer abort over assert(false)
 2019-01-21 21:56:41 +09:00
Andrew Gaul
46d79c5bc2 Issue readdir HEAD requests without batching 
Previously s3fs would issue a batch of HEAD requests and wait for all
to succeed before issuing the next batch.  Now it issues the first
batch and only waits for a single call to succeed before issuing the
next call.  This can improve performance when one call lags due to
network errors.  I measured 25% improvement with the same level of
parallelism.  This commit also reparents parallelism knobs for
consistency.  Follows on to 88cd8feb05.
Fixes #223.
 2019-01-20 18:07:22 -08:00
Andrew Gaul
40ba3b44a1 Prefer abort over assert(false) 
The compiler can remove the latter when compiled with NDEBUG which may
cause unintended control flow.
 2019-01-20 12:30:27 -08:00
Takeshi Nakatani
2887f8916b Automatic region change made possible other than us-east-1(default) 2019-01-20 10:51:49 +00:00
Takeshi Nakatani
ac72431195 Added detail error message when HTTP 301/307 status 2019-01-20 10:07:58 +00:00
Takeshi Nakatani
f2184e34dd Ignore after period character of floating point in x-amz-meta-mtime 2019-01-20 08:28:06 +00:00
Takeshi Nakatani
36a4903843 Added an error message when HTTP 301 status 2019-01-20 07:17:40 +00:00
Takeshi Nakatani
c83a3e67c9
Merge pull request #885 from LutzFinsterle2019/master 
Update s3fs_util.cpp for correspondence of Nextcloud contype
 2019-01-20 15:32:58 +09:00
Takeshi Nakatani
05014c49c8
Merge pull request #906 from gaul/doc/https 
Prefer HTTPS links where possible
 2019-01-20 15:10:51 +09:00
Takeshi Nakatani
aa69107165
Merge pull request #905 from gaul/clang-tidy/redundant 
Fix comparison in s3fs_strtoofft
 2019-01-20 15:01:40 +09:00
Takeshi Nakatani
d373b0eca3
Merge pull request #904 from gaul/clang-tidy/c-str 
Remove unnecessary calls to std::string::c_str
 2019-01-20 14:13:46 +09:00
Takeshi Nakatani
6aa40b2747
Merge pull request #903 from gaul/clang-tidy/find_char 
Prefer find(char) over find(const char *)
 2019-01-20 14:05:06 +09:00
Takeshi Nakatani
34c3bfe408
Merge pull request #902 from gaul/clang-tidy/pass-by-value 
Avoid pass-by-value when not necessary
 2019-01-20 13:45:44 +09:00
Takeshi Nakatani
6ac56e722d
Merge pull request #901 from gaul/clang-tidy/leaks 
Plug memory leaks
 2019-01-20 12:41:08 +09:00
Takeshi Nakatani
61dc7f0a70
Merge pull request #900 from gaul/leak 
Plug memory leak
 2019-01-20 12:30:31 +09:00
Takeshi Nakatani
9f000957dd
Merge pull request #899 from gaul/response-code 
Tighten up HTTP response code check
 2019-01-20 11:16:23 +09:00
Andrew Gaul
aa9bd1fa3c Prefer HTTPS links where possible 
Fix a few stale links as well.
 2019-01-18 11:09:08 -08:00
Andrew Gaul
5a2dc03a1c Fix comparison in s3fs_strtoofft 
Also backfill unit tests.  Document limitations.  Found via
clang-tidy.
 2019-01-17 22:59:25 -08:00
Andrew Gaul
508fafbe62 Remove unnecessary calls to std::string::c_str 
Found via clang-tidy.
 2019-01-17 22:05:16 -08:00
Andrew Gaul
e29548178b Prefer find(char) over find(const char *) 
The former can be faster.  Found via clang-tidy.
 2019-01-17 20:24:24 -08:00
Andrew Gaul
ab2f36f202 Plug memory leaks 
Found via clang-tidy.
 2019-01-17 18:54:34 -08:00
Andrew Gaul
b8c9fcfd70 Avoid pass-by-value when not necessary 
This requires unnecessary memcpy.  Found via clang-tidy.
 2019-01-17 18:22:11 -08:00
Andrew Gaul
58ce544e83 Plug memory leak 
Previously this appended to an empty list, zeroed the list, then
appended to the list.  Instead zero the list first and then append.
This also enables sending Content-MD5 which can eagerly detect
transmission errors.  Found via Valgrind.
 2019-01-17 16:24:34 -08:00
Andrew Gaul
e98ce36301 Tighten up HTTP response code check 
Previously s3fs considered 3xx codes as successful.  When writing an
object to a misconfigured region, s3fs did not propagate the 301 as a
write error to the client.  References #693.
 2019-01-17 15:04:59 -08:00
Andrew Gaul
6401b4ae92 Flush file before renaming 
Previously s3fs could copy the zero-byte stub object without including
any pending writes.  Fixes #145.
 2019-01-17 12:05:10 -08:00
Takeshi Nakatani
25b49e1a2e
Merge pull request #894 from gaul/default-mode 
Default uid/gid/mode when object lacks permissions
 2019-01-16 18:48:54 +09:00
Takeshi Nakatani
c7def35b54
Merge pull request #895 from gaul/bucket-name-dot 
Emit more friendly error for buckets with dots
 2019-01-16 18:39:47 +09:00
Andrew Gaul
c512516e14 Emit more friendly error for buckets with dots 
These fail SSL certificate checks due to the *.s3.amazon.com wildcard.
Fixes #284.
 2019-01-14 18:47:36 -08:00
Andrew Gaul
2c43b1e12b Store and retrieve file change time 
This introduces a new header with the change time; existing objects
will report modification time.  Fixes #771.
 2019-01-14 10:05:11 -08:00
Andrew Gaul
e2d5641d99 Default uid/gid/mode when object lacks permissions 
This addresses a common use case when interacting with objects from
both s3fs and other S3 tools.  Fixes #890.
 2019-01-13 21:57:23 -08:00
LutzFinsterle2019
c985b5e4d0
Corrected Comment to C++ style 2019-01-12 10:19:48 +01:00
LutzFinsterle2019
18cb2e2662
Update s3fs_util.cpp 
Sorry for answering late, have been busy lately.
The comment is: "Nextcloud stores Directory objects with this mime type when mounting a Bucket as external Storage"
 2019-01-10 07:44:36 +01:00
LutzFinsterle2019
743c706b0a
Update s3fs_util.cpp 
Nextcloud Compatibility in directory mime-types
 2019-01-07 07:41:27 +01:00
Takeshi Nakatani
4ed0e5f35a
Merge pull request #882 from earlchew/issue-817 
[curl] Assume long encryption keys are base64 encoded
 2019-01-06 17:29:04 +09:00
Takeshi Nakatani
fd6b37d3da
Merge pull request #877 from gaul/aws/credentials 
Check arguments and environment before .aws/creds
 2019-01-06 17:03:11 +09:00
Takeshi Nakatani
56e24de0d4
Merge pull request #870 from gaul/typos 
Correct typos in command-line parsing
 2019-01-06 16:21:46 +09:00
Earl Chew
ed5795eead [curl] Assume long encryption keys are base64 encoded 
Correct tabs and whitespace.

Signed-off-by: Earl Chew <earl_chew@yahoo.com>
 2019-01-05 21:08:41 +00:00
Andrew Gaul
3d225163f8 Correct typos in command-line parsing 
Also fix stray macOS references.  References #869.
 2019-01-02 14:40:57 -08:00
Andrew Gaul
0569cec3ea Check arguments and environment before .aws/creds 
Fixes #857.  Fixes #864.
 2018-12-20 15:51:32 -08:00
Andrew Gaul
a2f8ac535e Address cppcheck 1.86 errors 
Lifetime, shadowing, and unused variables.  Found via the Travis macOS
builder.
 2018-12-20 14:56:31 -08:00
Takeshi Nakatani
d9e89deef6
Merge pull request #865 from orozery/multihead_warning_check 
fix multihead warning check
 2018-11-29 20:53:47 +09:00
Or Ozeri
b91fc5409e fix multihead warning check 2018-11-28 09:54:02 +02:00
Michael Chen
3c970646d1
Add 'profile' option to command line help. 2018-11-19 08:26:23 -08:00
Andrew Gaul
88cd8feb05 Upload S3 parts without batching 
Previously s3fs would issue a batch of uploads and wait for all to
succeed before issuing the next batch.  Now it issues the first batch
and only waits for a single part to succeed before uploading the next
part.  This can improve performance when one part lags due to network
errors.  Fixes #183.
 2018-11-16 18:32:38 -08:00
Takeshi Nakatani
5fba542a29
Merge pull request #852 from gaul/aws-credentials-file 
Allow credentials from ${HOME}/.aws/credentials
 2018-11-11 11:37:38 +09:00
Andrew Gaul
9e530c86ae Allow credentials from ${HOME}/.aws/credentials 
This matches the configuration from popular tools like AWS CLI and
allows multiple profile names via -o profile=name.  The existing
credential mechanisms continue to work.  Fixes #822.
 2018-11-04 17:47:07 -08:00
Andrew Gaul
664f910083 Correctly compare list_object_max_keys 
Previously this did not allow an argument.  Fixes #843.
References #793.
 2018-11-04 10:49:35 -08:00
Takeshi Nakatani
6fe92d5ed6
Merge pull request #832 from gaul/hex 
Simplify hex conversion
 2018-10-08 23:03:42 +09:00
Andrew Gaul
c703fa15c0 Simplify hex conversion 
Addresses GCC 8 warning:

common_auth.cpp: In function ‘std::__cxx11::string s3fs_sha256sum(int, off_t, ssize_t)’:
common_auth.cpp:84:12: warning: ‘char* strncat(char*, const char*, size_t)’ output may be truncated copying 2 bytes from a string of length 2 [-Wstringop-truncation]
    strncat(sha256, hexbuf, 2);
 2018-10-01 19:08:27 -07:00
Jonatan Nilsson
d9c106cfde Add support for storage class ONEZONE_IA. 2018-09-30 14:27:31 +02:00
Takeshi Nakatani
c5af62b023
Merge pull request #820 from gaul/big-writes 
Enable big writes if capable
 2018-09-17 17:24:06 +09:00
Earl Chew
41c23adb0e [curl] Assume long encryption keys are base64 encoded 
Amazon SSE-C https://tinyurl.com/ychug4cg writes:

> Use this header to provide the 256-bit, base64-encoded encryption key
> for Amazon S3 to use to encrypt or decrypt your data.

It seems likely that future keys would be as long, or longer, so
this change assumes that text longer than 32 characters are keys
that are base64 encoded.

Signed-off-by: Earl Chew <earl_chew@yahoo.com>
 2018-09-16 21:23:22 +00:00
Andrew Gaul
a85183d42c Enable big writes if capable 
Fixes #813.
 2018-09-10 22:21:51 -07:00
Mattia Rizzolo
c376efdd28
Fix typo s/mutliple/multiple/ 
Signed-off-by: Mattia Rizzolo <mattia@mapreri.org>
 2018-08-19 00:14:57 +02:00
Takeshi Nakatani
7e0c53dfe9 Added list_object_max_keys option based on #783 PR 2018-07-08 03:49:10 +00:00
root
35d3fce7a0 Review comment: Include the error code being returned 2018-07-06 05:14:32 -04:00
root
4177d8bd3b Review comment: Include the error code being returned 2018-07-06 03:03:57 -04:00
root
ad5349a488 Changes as per review comments 2018-07-05 05:02:04 -04:00
root
92a4034c5e Log messages for 5xx and 4xx HTTP response code 2018-07-04 03:50:45 -04:00
Takeshi Nakatani
3e4002df0d
Merge pull request #780 from wjt/initialize-libgcry 
gnutls_auth: initialize libgcrypt
 2018-06-24 12:48:08 +09:00
Takeshi Nakatani
1b9ec7f4fc
Merge pull request #774 from nkkashyap/master 
Option for IAM authentication endpoint
 2018-06-24 12:36:23 +09:00
Takeshi Nakatani
0d3fb0658a Fixed a error by cppcheck on OSX 2018-06-24 02:38:59 +00:00
Will Thompson
73cf2ba95d
gnutls_auth: initialize libgcrypt 
Without this change, the following warning appears in the syslog/journal
during startup:

  Libgcrypt warning: missing initialization - please fix the application

From the [documentation][0]:

> The function `gcry_check_version` initializes some subsystems used by
> Libgcrypt and must be invoked before any other function in the
> library.

Fixes #524, which says:

> gnutls is initialized by gnutls_global_init() function and
> gcry_check_version() function for initializing libgcry is called from
> this gnutls_global_init().

I checked the gnutls source and it hasn't contained a call to
gcry_check_version() since the libgcrypt backend was removed in 2011
(commit 8116cdc8f131edd586dad3128ae35dd744cfc32f). In any case, the
gcry_check_version() documentation continues:

> It is important that these initialization steps are not done by a
> library but by the actual application.

so it would be incorrect for a library used by s3fs to initialize
libgcrypt.

[0]: https://www.gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
 2018-06-21 20:55:00 +01:00
Neeraj Kumar Kashyap
5a481e6a01 Option for IBM IAM auth endpoint added return 2018-06-04 16:44:14 +05:30
Neeraj Kumar Kashyap
d8e12839af Option for IBM IAM auth endpoint 2018-05-31 16:02:48 +05:30
Or Ozeri
d4e86a17d1 Revert "enable FUSE read_sync by default" 
This reverts commit 86b0921ac4.

Conflicts:
	src/s3fs.cpp
 2018-05-28 13:49:54 +03:00
Takeshi Nakatani
ae9d8eb734 Fixed memory leak 2018-05-27 10:48:03 +00:00
Takeshi Nakatani
57b2a60172
Merge pull request #764 from orozery/remove_false_multihead_warnings 
Remove false multihead warnings
 2018-05-23 22:38:35 +09:00
Takeshi Nakatani
212bbbbdf0
Merge pull request #763 from orozery/cleanup_share_after_handles 
cleanup curl handles before curl share
 2018-05-23 22:30:36 +09:00
Takeshi Nakatani
da95afba8a
Merge pull request #756 from orozery/optimize_defaults 
Optimize defaults
 2018-05-23 22:05:00 +09:00
Or Ozeri
0bd875eb9e remove false readdir_multi_head warnings 2018-05-22 17:10:50 +03:00
Or Ozeri
af63a42773 cleanup curl handles before curl share 2018-05-21 13:20:09 +03:00
Or Ozeri
86b0921ac4 enable FUSE read_sync by default 2018-05-06 16:10:36 +03:00
Or Ozeri
4a72b60707 increase default stat cache size from 1000 to 100000 2018-05-06 15:31:07 +03:00
Takeshi Nakatani
e3de6ea458 Added reset curl handle when returning to handle pool 2018-05-06 12:11:53 +00:00
Neeraj Kumar Kashyap
25375a6b48 Validate the URL fixed inefficient usage of find 2018-05-04 11:24:32 +05:30
Neeraj Kumar Kashyap
ca87df7d44 Validate the URL format for http/https 2018-05-03 22:08:28 +05:30
Takeshi Nakatani
d052dc0b9d
Merge pull request #753 from cfz/master 
fix xpath selector in bucket listing
 2018-05-02 12:04:12 +09:00
陈方舟
04493de767
fix xpath selector in bucket listing 
the original implementation in get_base_exp() depends on the order of xml return from the server.
patriotically, when listing a directory with sub directory(s), the xml document response contains more than 2 <Prefix> nodes(some of them are in <CommonPrefixes> node).
the source code arbitrarily select the first one in the documents (nodes->nodeTab[0]->xmlChildrenNode).
some s3 compatible service return the list-bucket result in different result, leading the s3fs to a wrong behavior
 2018-04-23 15:11:29 +08:00
Or Ozeri
4fdab46617 don't fail mkdir when directory exists 2018-04-08 11:13:47 +03:00
Takeshi Nakatani
1a23b880d5
Merge pull request #739 from orozery/cleanup_failing_curl_handles 
cleanup curl handle state on retries
 2018-04-01 22:45:04 +09:00
Or Ozeri
adcf5754ae cleanup failing curl handles on retries 2018-03-29 13:56:08 +03:00
Tristen Horton
0863672e27
add a more helpful error message for when there are multiple entries for the same bucket in the passwd file 2018-03-13 14:37:34 -07:00
Takeshi Nakatani
0f503ced25
Merge pull request #729 from dmgk/master 
FreeBSD build fixes
 2018-03-04 16:36:31 +09:00
Takeshi Nakatani
987a166bf4
Merge pull request #726 from orozery/instance_name_logging 
add an instance_name option for logging
 2018-03-04 15:41:12 +09:00
Takeshi Nakatani
57b6f0eeaf
Merge pull request #724 from orozery/dont_fail_multirequest 
don't fail multirequest on single thread error
 2018-03-04 15:35:29 +09:00
Takeshi Nakatani
f71a28f9b9
Merge pull request #714 from orozery/reduce_lock_contention 
reduce lock contention on file open
 2018-03-04 13:36:08 +09:00
Dmitri Goutnik
c9f4312588 FreeBSD build fixes 2018-03-02 15:58:52 -05:00
Or Ozeri
8b657eee41 add disk space reservation 2018-02-28 19:20:23 +02:00
Takeshi Nakatani
be9d407fa0 Fixed cppcheck error on osx 2018-02-28 12:29:58 +00:00
Takeshi Nakatani
c494e54320 Fixed cppcheck error on osx 2018-02-28 12:06:06 +00:00
Or Ozeri
b52b6f3fc5 add an instance_name option for logging 2018-02-28 09:51:35 +02:00
Or Ozeri
82c9733101 don't fail multirequest on single thread error 2018-02-26 12:06:08 +02:00
Takeshi Nakatani
a45ff6cdaa Fixed cppcheck error and clean ^M code 2018-02-25 13:08:41 +00:00
Takeshi Nakatani
960d45c853 Fixed cppcheck error on osx 2018-02-25 08:51:19 +00:00
Or Ozeri
0edf056e95 reduce lock contention on file open 2018-02-04 17:13:58 +02:00
Takeshi Nakatani
6b58220009
Merge pull request #697 from pwulff/master 
Fixing race condition in FdEntity::GetStats
 2017-12-17 15:46:48 +09:00
Paul Wulff
ee6abea956 Race condition in FdManager::Rename because no mutex is used. 2017-12-15 15:27:51 +01:00
Paul Wulff
cea7d44717 Fixing race condition in FdEntity::GetStats 2017-12-13 10:49:00 +01:00
Or Ozeri
0da87e75fe fix condition for parallel download 2017-12-04 16:07:33 +02:00
Or Ozeri
bd4bc0e7f1 add support for IBM IAM authentication 2017-11-23 12:01:52 +02:00
Takeshi Nakatani
026260e7a1 Improved use of temporary files 2017-11-23 09:18:11 +00:00
Takeshi Nakatani
99fe93b7f1
Merge pull request #684 from gaul/signedness 
Correct signedness warning
 2017-11-23 17:33:46 +09:00
Or Ozeri
11bd7128d2 remove use of jsoncpp 2017-11-22 13:36:27 +02:00
Andrew Gaul
7cda32664b Correct signedness warning 
Fixes regression from 0418e53b3c.
 2017-11-19 11:14:37 -08:00
Takeshi Nakatani
97fc845a6a Changed functions about reading passwd file. 2017-11-19 11:49:11 +00:00
Takeshi Nakatani
7d9ac0163b Changed functions about reading passwd file. 2017-11-19 11:38:12 +00:00
Takeshi Nakatani
e1928288fe
Merge pull request #676 from gaul/sstream 
Move str definition from header to implementation
 2017-11-19 16:34:15 +09:00
Andrew Gaul
0418e53b3c Reduce use of preprocessor 
This provides type-safety and avoids token expansion side effects.
 2017-11-18 22:40:06 -08:00
Takeshi Nakatani
bad48ab59a
Merge pull request #671 from psyvision/master 
Add support for ECS metadata endpoint
 2017-11-19 14:43:06 +09:00
Andrew Gaul
bbad76bb71 Move str definition from header to implementation 
Also clean up some char * to std::string conversions.
 2017-11-18 11:34:34 -08:00
Richard Caunt
b95e4acaeb
Remove debug logging statements 2017-11-08 15:52:35 +00:00
Richard Caunt
c238701d09
Corrected ECS headers 2017-11-08 15:21:49 +00:00
Richard Caunt
60d2ac3c7a
Adding x-amz-security-token header 2017-11-08 15:09:59 +00:00
Richard Caunt
967ef4d56b
Corrected fat finger mistakes 2017-11-08 13:14:49 +00:00
Richard Caunt
ad57bdda6c
Corrected keycount check 2017-11-08 13:06:22 +00:00
Richard Caunt
a0b69d1d3d
Corrected keyval[].c_str() 2017-11-08 13:01:52 +00:00
Richard Caunt
5df94d7e33
Add debug messages 2017-11-08 09:50:39 +00:00
Richard Caunt
1cbe9fb7a3 Gotta pass that cppcheckgit add . 2017-11-07 21:41:51 +00:00
Richard Caunt
8660abaea2 Use jsoncpp to parse AWS JSON 2017-11-07 21:20:02 +00:00
Richard Caunt
366f0705a0 ECS credentials bug fixes 2017-11-06 21:45:58 +00:00
Richard Caunt
ccea87ca68
Added check for is_ecs during get_access_keys 
We need to presume that if `is_ecs` we are deferring access key loading
 2017-11-06 11:02:27 +00:00
Richard Caunt
5d54883e2f Remove commented out code 2017-11-05 19:25:34 +00:00
Richard Caunt
662f65c3c8 Add support for ECS metadata endpoint 2017-11-05 19:24:02 +00:00
Takeshi Nakatani
5db550a298 Fixed a bug in S3fsCurl::LocateBundle 2017-11-05 11:26:05 +00:00
Takeshi Nakatani
e3c77d2906
Merge pull request #664 from orozery/auth_refactor 
auth headers insertion refactoring
 2017-11-05 15:14:40 +09:00
Or Ozeri
384b4cbafa auth headers insertion refactoring 2017-10-30 11:52:58 +02:00
Andrew Gaul
40501a7a73 Lock FdEntity when mutating orgmeta 
References #654.
 2017-10-23 22:41:42 -07:00
Takeshi Nakatani
ab89b4cd4a Merge pull request #659 from ggtakec/master 
Do not fail updating directory when removing old-style object(ref #658)
 2017-10-15 21:56:35 +09:00
Takeshi Nakatani
1eba27a50a Refixed s3fs_init message(ref #652) 2017-10-15 06:45:19 +00:00
Takeshi Nakatani
41206fa0e2 Do not fail updating directory when removing old-style object(ref #658) 2017-10-15 05:03:44 +00:00
jurafxp
ae91b6f673 Fix s3fs_init message 2017-10-01 00:08:00 +02:00
Takeshi Nakatani
5957d9ead0 Fixed with unnecessary equal in POST uploads url argment - #643 2017-09-17 10:52:28 +00:00
Takeshi Nakatani
00bc9142c4 Fixed potential atomic violation in S3fsCurl::AddUserAgent - #633 2017-09-17 09:16:05 +00:00
Scott Talbert
20da0e4dd3 Fix intermittent upload failures on macOS 
There were multiple problems with the FdManager::GetFreeDiskSpace() function
on macOS:
1) When calling statvfs(), f_frsize should be used instead of f_bsize when
converting available blocks to bytes.  This was causing the free space
calculation to be incorrect.
2) On macOS, fsblkcnt_t is a 32-bit integer.  Thus, when calculating available
disk space, there were frequently overflows.  This caused s3fs to incorrectly
determine that the cache location was out of space in the middle of a transfer
which caused uploads to fail.  Changing this to a uint64_t resolves the
problem.
 2017-09-08 15:23:10 -04:00
Takeshi Nakatani
96d8e6d823 Merge remote-tracking branch 'upstream/macosx' into macosx 2017-08-11 14:20:12 +00:00
Takeshi Nakatani
62b8084300 Added travis test on osx for #601 2017-08-11 14:09:43 +00:00
Takeshi Nakatani
0c6a3882a2 Fixed clock_gettime build failure on macOS 10.12 Sierra - #600 2017-05-28 10:04:25 +00:00
Takeshi Nakatani
a08880ae15 Merge pull request #608 from tlevi/chown_nocopy 
Fix chown_nocopy losing existing uid/gid if unspecified
 2017-05-28 18:36:08 +09:00
Tony Levi
cc94e1da26 Fix chown_nocopy losing existing uid/gid if unspecified 2017-05-25 16:53:08 +09:30
Tony Levi
2b7ea5813c Expand buffer for group information if too small and retry 2017-05-23 10:42:43 +09:30
Takeshi Nakatani
337da59368 Not fallback to HTTP - #596 2017-05-13 15:47:39 +00:00
Takeshi Nakatani
d2ae14d8b7 Check bucket at public bucket and add nocopyapi option automatically 2017-05-13 07:48:50 +00:00
Takeshi Nakatani
7115835834 Check bucket at public bucket and add nocopyapi option automatically 2017-05-13 07:35:55 +00:00
Takeshi Nakatani
24df69f688 Backward compatible for changing default transport to HTTPS 2017-05-13 06:47:51 +00:00
Takeshi Nakatani
465c15ef40 Merge pull request #588 from andrewgaul/https 
Default transport to HTTPS
 2017-05-09 23:04:57 +09:00
Takeshi Nakatani
a22675bafd Merge pull request #567 from andrewgaul/default-acl 
Do not send ACL unless overridden
 2017-05-09 23:03:27 +09:00
Andrew Gaul
0e0ae38f6d Default transport to HTTPS 
This protects private data when used over the public Internet.  Users
can opt-in to unencrypted HTTP if they need additional performance on
a local network.  Fixes #282.
 2017-05-07 10:59:54 -07:00
Andrew Gaul
7b30d5d15b Do not send canned ACL header when empty string 
Some providers such as StorageGRID do not support canned ACLs.
Setting to empty allows callers to omit the header.  References #125.
 2017-05-07 10:52:31 -07:00
Takeshi Nakatani
9d10a5aa70 Changed copyright year format for debian pkg 2017-05-07 11:24:17 +00:00
Takeshi Nakatani
a12e0d5ec4 Fixed failure to upload/copy with SSE_C and SSE_KMS 2017-05-07 09:29:08 +00:00
Takeshi Nakatani
d07c3f38b7 Check errors returned in 200 OK responses for put header request 2017-05-06 02:15:53 +00:00
Takeshi Nakatani
1fe0334c08 Enhanced bucket/path parameter check 2017-05-05 19:55:24 +00:00
Takeshi Nakatani
3ac39d61f8 Added notsup_compat_dir option 2017-05-05 17:28:29 +00:00
Takeshi Nakatani
864e20e1f2 Refactored the get_object_attribute function 2017-05-05 10:02:21 +00:00
Takeshi Nakatani
51b3183cba Refactored the check_object_access function 2017-05-05 09:51:30 +00:00
Takeshi Nakatani
758b92e823 Added option for complementing lack of stat mode 2017-05-04 03:41:24 +00:00
Takeshi Nakatani
df0ff3a2fd Merge pull request #556 from orozery/fix_nocache_multipart_upload 
fix multipart upload handling without cache
 2017-04-16 19:22:15 +09:00
Takeshi Nakatani
edcf4c6218 Merge pull request #555 from orozery/dont_sign_empty_headers 
don't sign empty headers (as they are discarded by libcurl)
 2017-04-16 19:16:47 +09:00
Takeshi Nakatani
28efff5986 Merge pull request #554 from orozery/cache_cleanup 
cleanup cache directory when running out of disk space
 2017-04-16 19:13:11 +09:00
Takeshi Nakatani
efba9bcbc1 Merge pull request #553 from orozery/custom_cipher_suite 
add TLS cipher suites customization
 2017-04-16 19:09:27 +09:00
Takeshi Nakatani
6bd179c92b Merge pull request #552 from orozery/foreground_threads 
switch S3fsMultiCurl to use foreground threads
 2017-04-16 19:05:16 +09:00
Or Ozeri
96764b7410 switch S3fsMultiCurl to use foreground threads 2017-04-09 16:56:49 +03:00