octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-15 16:57:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,697 Commits 1 Branch 23 Tags 13 MiB
5b5bc3114a
Commit Graph

901 Commits

Author SHA1 Message Date
Takeshi Nakatani
6ac56e722d
Merge pull request #901 from gaul/clang-tidy/leaks 
Plug memory leaks
 2019-01-20 12:41:08 +09:00
Takeshi Nakatani
61dc7f0a70
Merge pull request #900 from gaul/leak 
Plug memory leak
 2019-01-20 12:30:31 +09:00
Takeshi Nakatani
9f000957dd
Merge pull request #899 from gaul/response-code 
Tighten up HTTP response code check
 2019-01-20 11:16:23 +09:00
Andrew Gaul
aa9bd1fa3c Prefer HTTPS links where possible 
Fix a few stale links as well.
 2019-01-18 11:09:08 -08:00
Andrew Gaul
5a2dc03a1c Fix comparison in s3fs_strtoofft 
Also backfill unit tests.  Document limitations.  Found via
clang-tidy.
 2019-01-17 22:59:25 -08:00
Andrew Gaul
508fafbe62 Remove unnecessary calls to std::string::c_str 
Found via clang-tidy.
 2019-01-17 22:05:16 -08:00
Andrew Gaul
e29548178b Prefer find(char) over find(const char *) 
The former can be faster.  Found via clang-tidy.
 2019-01-17 20:24:24 -08:00
Andrew Gaul
ab2f36f202 Plug memory leaks 
Found via clang-tidy.
 2019-01-17 18:54:34 -08:00
Andrew Gaul
b8c9fcfd70 Avoid pass-by-value when not necessary 
This requires unnecessary memcpy.  Found via clang-tidy.
 2019-01-17 18:22:11 -08:00
Andrew Gaul
58ce544e83 Plug memory leak 
Previously this appended to an empty list, zeroed the list, then
appended to the list.  Instead zero the list first and then append.
This also enables sending Content-MD5 which can eagerly detect
transmission errors.  Found via Valgrind.
 2019-01-17 16:24:34 -08:00
Andrew Gaul
e98ce36301 Tighten up HTTP response code check 
Previously s3fs considered 3xx codes as successful.  When writing an
object to a misconfigured region, s3fs did not propagate the 301 as a
write error to the client.  References #693.
 2019-01-17 15:04:59 -08:00
Andrew Gaul
6401b4ae92 Flush file before renaming 
Previously s3fs could copy the zero-byte stub object without including
any pending writes.  Fixes #145.
 2019-01-17 12:05:10 -08:00
Takeshi Nakatani
25b49e1a2e
Merge pull request #894 from gaul/default-mode 
Default uid/gid/mode when object lacks permissions
 2019-01-16 18:48:54 +09:00
Takeshi Nakatani
c7def35b54
Merge pull request #895 from gaul/bucket-name-dot 
Emit more friendly error for buckets with dots
 2019-01-16 18:39:47 +09:00
Andrew Gaul
c512516e14 Emit more friendly error for buckets with dots 
These fail SSL certificate checks due to the *.s3.amazon.com wildcard.
Fixes #284.
 2019-01-14 18:47:36 -08:00
Andrew Gaul
2c43b1e12b Store and retrieve file change time 
This introduces a new header with the change time; existing objects
will report modification time.  Fixes #771.
 2019-01-14 10:05:11 -08:00
Andrew Gaul
e2d5641d99 Default uid/gid/mode when object lacks permissions 
This addresses a common use case when interacting with objects from
both s3fs and other S3 tools.  Fixes #890.
 2019-01-13 21:57:23 -08:00
LutzFinsterle2019
c985b5e4d0
Corrected Comment to C++ style 2019-01-12 10:19:48 +01:00
LutzFinsterle2019
18cb2e2662
Update s3fs_util.cpp 
Sorry for answering late, have been busy lately.
The comment is: "Nextcloud stores Directory objects with this mime type when mounting a Bucket as external Storage"
 2019-01-10 07:44:36 +01:00
LutzFinsterle2019
743c706b0a
Update s3fs_util.cpp 
Nextcloud Compatibility in directory mime-types
 2019-01-07 07:41:27 +01:00
Takeshi Nakatani
4ed0e5f35a
Merge pull request #882 from earlchew/issue-817 
[curl] Assume long encryption keys are base64 encoded
 2019-01-06 17:29:04 +09:00
Takeshi Nakatani
fd6b37d3da
Merge pull request #877 from gaul/aws/credentials 
Check arguments and environment before .aws/creds
 2019-01-06 17:03:11 +09:00
Takeshi Nakatani
56e24de0d4
Merge pull request #870 from gaul/typos 
Correct typos in command-line parsing
 2019-01-06 16:21:46 +09:00
Earl Chew
ed5795eead [curl] Assume long encryption keys are base64 encoded 
Correct tabs and whitespace.

Signed-off-by: Earl Chew <earl_chew@yahoo.com>
 2019-01-05 21:08:41 +00:00
Andrew Gaul
3d225163f8 Correct typos in command-line parsing 
Also fix stray macOS references.  References #869.
 2019-01-02 14:40:57 -08:00
Andrew Gaul
0569cec3ea Check arguments and environment before .aws/creds 
Fixes #857.  Fixes #864.
 2018-12-20 15:51:32 -08:00
Andrew Gaul
a2f8ac535e Address cppcheck 1.86 errors 
Lifetime, shadowing, and unused variables.  Found via the Travis macOS
builder.
 2018-12-20 14:56:31 -08:00
Takeshi Nakatani
d9e89deef6
Merge pull request #865 from orozery/multihead_warning_check 
fix multihead warning check
 2018-11-29 20:53:47 +09:00
Or Ozeri
b91fc5409e fix multihead warning check 2018-11-28 09:54:02 +02:00
Michael Chen
3c970646d1
Add 'profile' option to command line help. 2018-11-19 08:26:23 -08:00
Andrew Gaul
88cd8feb05 Upload S3 parts without batching 
Previously s3fs would issue a batch of uploads and wait for all to
succeed before issuing the next batch.  Now it issues the first batch
and only waits for a single part to succeed before uploading the next
part.  This can improve performance when one part lags due to network
errors.  Fixes #183.
 2018-11-16 18:32:38 -08:00
Takeshi Nakatani
5fba542a29
Merge pull request #852 from gaul/aws-credentials-file 
Allow credentials from ${HOME}/.aws/credentials
 2018-11-11 11:37:38 +09:00
Andrew Gaul
9e530c86ae Allow credentials from ${HOME}/.aws/credentials 
This matches the configuration from popular tools like AWS CLI and
allows multiple profile names via -o profile=name.  The existing
credential mechanisms continue to work.  Fixes #822.
 2018-11-04 17:47:07 -08:00
Andrew Gaul
664f910083 Correctly compare list_object_max_keys 
Previously this did not allow an argument.  Fixes #843.
References #793.
 2018-11-04 10:49:35 -08:00
Takeshi Nakatani
6fe92d5ed6
Merge pull request #832 from gaul/hex 
Simplify hex conversion
 2018-10-08 23:03:42 +09:00
Andrew Gaul
c703fa15c0 Simplify hex conversion 
Addresses GCC 8 warning:

common_auth.cpp: In function ‘std::__cxx11::string s3fs_sha256sum(int, off_t, ssize_t)’:
common_auth.cpp:84:12: warning: ‘char* strncat(char*, const char*, size_t)’ output may be truncated copying 2 bytes from a string of length 2 [-Wstringop-truncation]
    strncat(sha256, hexbuf, 2);
 2018-10-01 19:08:27 -07:00
Jonatan Nilsson
d9c106cfde Add support for storage class ONEZONE_IA. 2018-09-30 14:27:31 +02:00
Takeshi Nakatani
c5af62b023
Merge pull request #820 from gaul/big-writes 
Enable big writes if capable
 2018-09-17 17:24:06 +09:00
Earl Chew
41c23adb0e [curl] Assume long encryption keys are base64 encoded 
Amazon SSE-C https://tinyurl.com/ychug4cg writes:

> Use this header to provide the 256-bit, base64-encoded encryption key
> for Amazon S3 to use to encrypt or decrypt your data.

It seems likely that future keys would be as long, or longer, so
this change assumes that text longer than 32 characters are keys
that are base64 encoded.

Signed-off-by: Earl Chew <earl_chew@yahoo.com>
 2018-09-16 21:23:22 +00:00
Andrew Gaul
a85183d42c Enable big writes if capable 
Fixes #813.
 2018-09-10 22:21:51 -07:00
Mattia Rizzolo
c376efdd28
Fix typo s/mutliple/multiple/ 
Signed-off-by: Mattia Rizzolo <mattia@mapreri.org>
 2018-08-19 00:14:57 +02:00
Takeshi Nakatani
7e0c53dfe9 Added list_object_max_keys option based on #783 PR 2018-07-08 03:49:10 +00:00
root
35d3fce7a0 Review comment: Include the error code being returned 2018-07-06 05:14:32 -04:00
root
4177d8bd3b Review comment: Include the error code being returned 2018-07-06 03:03:57 -04:00
root
ad5349a488 Changes as per review comments 2018-07-05 05:02:04 -04:00
root
92a4034c5e Log messages for 5xx and 4xx HTTP response code 2018-07-04 03:50:45 -04:00
Takeshi Nakatani
3e4002df0d
Merge pull request #780 from wjt/initialize-libgcry 
gnutls_auth: initialize libgcrypt
 2018-06-24 12:48:08 +09:00
Takeshi Nakatani
1b9ec7f4fc
Merge pull request #774 from nkkashyap/master 
Option for IAM authentication endpoint
 2018-06-24 12:36:23 +09:00
Takeshi Nakatani
0d3fb0658a Fixed a error by cppcheck on OSX 2018-06-24 02:38:59 +00:00
Will Thompson
73cf2ba95d
gnutls_auth: initialize libgcrypt 
Without this change, the following warning appears in the syslog/journal
during startup:

  Libgcrypt warning: missing initialization - please fix the application

From the [documentation][0]:

> The function `gcry_check_version` initializes some subsystems used by
> Libgcrypt and must be invoked before any other function in the
> library.

Fixes #524, which says:

> gnutls is initialized by gnutls_global_init() function and
> gcry_check_version() function for initializing libgcry is called from
> this gnutls_global_init().

I checked the gnutls source and it hasn't contained a call to
gcry_check_version() since the libgcrypt backend was removed in 2011
(commit 8116cdc8f131edd586dad3128ae35dd744cfc32f). In any case, the
gcry_check_version() documentation continues:

> It is important that these initialization steps are not done by a
> library but by the actual application.

so it would be incorrect for a library used by s3fs to initialize
libgcrypt.

[0]: https://www.gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
 2018-06-21 20:55:00 +01:00
Neeraj Kumar Kashyap
5a481e6a01 Option for IBM IAM auth endpoint added return 2018-06-04 16:44:14 +05:30
Neeraj Kumar Kashyap
d8e12839af Option for IBM IAM auth endpoint 2018-05-31 16:02:48 +05:30
Or Ozeri
d4e86a17d1 Revert "enable FUSE read_sync by default" 
This reverts commit 86b0921ac4.

Conflicts:
	src/s3fs.cpp
 2018-05-28 13:49:54 +03:00
Takeshi Nakatani
ae9d8eb734 Fixed memory leak 2018-05-27 10:48:03 +00:00
Takeshi Nakatani
57b2a60172
Merge pull request #764 from orozery/remove_false_multihead_warnings 
Remove false multihead warnings
 2018-05-23 22:38:35 +09:00
Takeshi Nakatani
212bbbbdf0
Merge pull request #763 from orozery/cleanup_share_after_handles 
cleanup curl handles before curl share
 2018-05-23 22:30:36 +09:00
Takeshi Nakatani
da95afba8a
Merge pull request #756 from orozery/optimize_defaults 
Optimize defaults
 2018-05-23 22:05:00 +09:00
Or Ozeri
0bd875eb9e remove false readdir_multi_head warnings 2018-05-22 17:10:50 +03:00
Or Ozeri
af63a42773 cleanup curl handles before curl share 2018-05-21 13:20:09 +03:00
Or Ozeri
86b0921ac4 enable FUSE read_sync by default 2018-05-06 16:10:36 +03:00
Or Ozeri
4a72b60707 increase default stat cache size from 1000 to 100000 2018-05-06 15:31:07 +03:00
Takeshi Nakatani
e3de6ea458 Added reset curl handle when returning to handle pool 2018-05-06 12:11:53 +00:00
Neeraj Kumar Kashyap
25375a6b48 Validate the URL fixed inefficient usage of find 2018-05-04 11:24:32 +05:30
Neeraj Kumar Kashyap
ca87df7d44 Validate the URL format for http/https 2018-05-03 22:08:28 +05:30
Takeshi Nakatani
d052dc0b9d
Merge pull request #753 from cfz/master 
fix xpath selector in bucket listing
 2018-05-02 12:04:12 +09:00
陈方舟
04493de767
fix xpath selector in bucket listing 
the original implementation in get_base_exp() depends on the order of xml return from the server.
patriotically, when listing a directory with sub directory(s), the xml document response contains more than 2 <Prefix> nodes(some of them are in <CommonPrefixes> node).
the source code arbitrarily select the first one in the documents (nodes->nodeTab[0]->xmlChildrenNode).
some s3 compatible service return the list-bucket result in different result, leading the s3fs to a wrong behavior
 2018-04-23 15:11:29 +08:00
Or Ozeri
4fdab46617 don't fail mkdir when directory exists 2018-04-08 11:13:47 +03:00
Takeshi Nakatani
1a23b880d5
Merge pull request #739 from orozery/cleanup_failing_curl_handles 
cleanup curl handle state on retries
 2018-04-01 22:45:04 +09:00
Or Ozeri
adcf5754ae cleanup failing curl handles on retries 2018-03-29 13:56:08 +03:00
Tristen Horton
0863672e27
add a more helpful error message for when there are multiple entries for the same bucket in the passwd file 2018-03-13 14:37:34 -07:00
Takeshi Nakatani
0f503ced25
Merge pull request #729 from dmgk/master 
FreeBSD build fixes
 2018-03-04 16:36:31 +09:00
Takeshi Nakatani
987a166bf4
Merge pull request #726 from orozery/instance_name_logging 
add an instance_name option for logging
 2018-03-04 15:41:12 +09:00
Takeshi Nakatani
57b6f0eeaf
Merge pull request #724 from orozery/dont_fail_multirequest 
don't fail multirequest on single thread error
 2018-03-04 15:35:29 +09:00
Takeshi Nakatani
f71a28f9b9
Merge pull request #714 from orozery/reduce_lock_contention 
reduce lock contention on file open
 2018-03-04 13:36:08 +09:00
Dmitri Goutnik
c9f4312588 FreeBSD build fixes 2018-03-02 15:58:52 -05:00
Or Ozeri
8b657eee41 add disk space reservation 2018-02-28 19:20:23 +02:00
Takeshi Nakatani
be9d407fa0 Fixed cppcheck error on osx 2018-02-28 12:29:58 +00:00
Takeshi Nakatani
c494e54320 Fixed cppcheck error on osx 2018-02-28 12:06:06 +00:00
Or Ozeri
b52b6f3fc5 add an instance_name option for logging 2018-02-28 09:51:35 +02:00
Or Ozeri
82c9733101 don't fail multirequest on single thread error 2018-02-26 12:06:08 +02:00
Takeshi Nakatani
a45ff6cdaa Fixed cppcheck error and clean ^M code 2018-02-25 13:08:41 +00:00
Takeshi Nakatani
960d45c853 Fixed cppcheck error on osx 2018-02-25 08:51:19 +00:00
Or Ozeri
0edf056e95 reduce lock contention on file open 2018-02-04 17:13:58 +02:00
Takeshi Nakatani
6b58220009
Merge pull request #697 from pwulff/master 
Fixing race condition in FdEntity::GetStats
 2017-12-17 15:46:48 +09:00
Paul Wulff
ee6abea956 Race condition in FdManager::Rename because no mutex is used. 2017-12-15 15:27:51 +01:00
Paul Wulff
cea7d44717 Fixing race condition in FdEntity::GetStats 2017-12-13 10:49:00 +01:00
Or Ozeri
0da87e75fe fix condition for parallel download 2017-12-04 16:07:33 +02:00
Or Ozeri
bd4bc0e7f1 add support for IBM IAM authentication 2017-11-23 12:01:52 +02:00
Takeshi Nakatani
026260e7a1 Improved use of temporary files 2017-11-23 09:18:11 +00:00
Takeshi Nakatani
99fe93b7f1
Merge pull request #684 from gaul/signedness 
Correct signedness warning
 2017-11-23 17:33:46 +09:00
Or Ozeri
11bd7128d2 remove use of jsoncpp 2017-11-22 13:36:27 +02:00
Andrew Gaul
7cda32664b Correct signedness warning 
Fixes regression from 0418e53b3c.
 2017-11-19 11:14:37 -08:00
Takeshi Nakatani
97fc845a6a Changed functions about reading passwd file. 2017-11-19 11:49:11 +00:00
Takeshi Nakatani
7d9ac0163b Changed functions about reading passwd file. 2017-11-19 11:38:12 +00:00
Takeshi Nakatani
e1928288fe
Merge pull request #676 from gaul/sstream 
Move str definition from header to implementation
 2017-11-19 16:34:15 +09:00
Andrew Gaul
0418e53b3c Reduce use of preprocessor 
This provides type-safety and avoids token expansion side effects.
 2017-11-18 22:40:06 -08:00
Takeshi Nakatani
bad48ab59a
Merge pull request #671 from psyvision/master 
Add support for ECS metadata endpoint
 2017-11-19 14:43:06 +09:00
Andrew Gaul
bbad76bb71 Move str definition from header to implementation 
Also clean up some char * to std::string conversions.
 2017-11-18 11:34:34 -08:00
Richard Caunt
b95e4acaeb
Remove debug logging statements 2017-11-08 15:52:35 +00:00
Richard Caunt
c238701d09
Corrected ECS headers 2017-11-08 15:21:49 +00:00
Richard Caunt
60d2ac3c7a
Adding x-amz-security-token header 2017-11-08 15:09:59 +00:00
Richard Caunt
967ef4d56b
Corrected fat finger mistakes 2017-11-08 13:14:49 +00:00
Richard Caunt
ad57bdda6c
Corrected keycount check 2017-11-08 13:06:22 +00:00
Richard Caunt
a0b69d1d3d
Corrected keyval[].c_str() 2017-11-08 13:01:52 +00:00
Richard Caunt
5df94d7e33
Add debug messages 2017-11-08 09:50:39 +00:00
Richard Caunt
1cbe9fb7a3 Gotta pass that cppcheckgit add . 2017-11-07 21:41:51 +00:00
Richard Caunt
8660abaea2 Use jsoncpp to parse AWS JSON 2017-11-07 21:20:02 +00:00
Richard Caunt
366f0705a0 ECS credentials bug fixes 2017-11-06 21:45:58 +00:00
Richard Caunt
ccea87ca68
Added check for is_ecs during get_access_keys 
We need to presume that if `is_ecs` we are deferring access key loading
 2017-11-06 11:02:27 +00:00
Richard Caunt
5d54883e2f Remove commented out code 2017-11-05 19:25:34 +00:00
Richard Caunt
662f65c3c8 Add support for ECS metadata endpoint 2017-11-05 19:24:02 +00:00
Takeshi Nakatani
5db550a298 Fixed a bug in S3fsCurl::LocateBundle 2017-11-05 11:26:05 +00:00
Takeshi Nakatani
e3c77d2906
Merge pull request #664 from orozery/auth_refactor 
auth headers insertion refactoring
 2017-11-05 15:14:40 +09:00
Or Ozeri
384b4cbafa auth headers insertion refactoring 2017-10-30 11:52:58 +02:00
Andrew Gaul
40501a7a73 Lock FdEntity when mutating orgmeta 
References #654.
 2017-10-23 22:41:42 -07:00
Takeshi Nakatani
ab89b4cd4a Merge pull request #659 from ggtakec/master 
Do not fail updating directory when removing old-style object(ref #658)
 2017-10-15 21:56:35 +09:00
Takeshi Nakatani
1eba27a50a Refixed s3fs_init message(ref #652) 2017-10-15 06:45:19 +00:00
Takeshi Nakatani
41206fa0e2 Do not fail updating directory when removing old-style object(ref #658) 2017-10-15 05:03:44 +00:00
jurafxp
ae91b6f673 Fix s3fs_init message 2017-10-01 00:08:00 +02:00
Takeshi Nakatani
5957d9ead0 Fixed with unnecessary equal in POST uploads url argment - #643 2017-09-17 10:52:28 +00:00
Takeshi Nakatani
00bc9142c4 Fixed potential atomic violation in S3fsCurl::AddUserAgent - #633 2017-09-17 09:16:05 +00:00
Scott Talbert
20da0e4dd3 Fix intermittent upload failures on macOS 
There were multiple problems with the FdManager::GetFreeDiskSpace() function
on macOS:
1) When calling statvfs(), f_frsize should be used instead of f_bsize when
converting available blocks to bytes.  This was causing the free space
calculation to be incorrect.
2) On macOS, fsblkcnt_t is a 32-bit integer.  Thus, when calculating available
disk space, there were frequently overflows.  This caused s3fs to incorrectly
determine that the cache location was out of space in the middle of a transfer
which caused uploads to fail.  Changing this to a uint64_t resolves the
problem.
 2017-09-08 15:23:10 -04:00
Takeshi Nakatani
96d8e6d823 Merge remote-tracking branch 'upstream/macosx' into macosx 2017-08-11 14:20:12 +00:00
Takeshi Nakatani
62b8084300 Added travis test on osx for #601 2017-08-11 14:09:43 +00:00
Takeshi Nakatani
0c6a3882a2 Fixed clock_gettime build failure on macOS 10.12 Sierra - #600 2017-05-28 10:04:25 +00:00
Takeshi Nakatani
a08880ae15 Merge pull request #608 from tlevi/chown_nocopy 
Fix chown_nocopy losing existing uid/gid if unspecified
 2017-05-28 18:36:08 +09:00
Tony Levi
cc94e1da26 Fix chown_nocopy losing existing uid/gid if unspecified 2017-05-25 16:53:08 +09:30
Tony Levi
2b7ea5813c Expand buffer for group information if too small and retry 2017-05-23 10:42:43 +09:30
Takeshi Nakatani
337da59368 Not fallback to HTTP - #596 2017-05-13 15:47:39 +00:00
Takeshi Nakatani
d2ae14d8b7 Check bucket at public bucket and add nocopyapi option automatically 2017-05-13 07:48:50 +00:00
Takeshi Nakatani
7115835834 Check bucket at public bucket and add nocopyapi option automatically 2017-05-13 07:35:55 +00:00
Takeshi Nakatani
24df69f688 Backward compatible for changing default transport to HTTPS 2017-05-13 06:47:51 +00:00
Takeshi Nakatani
465c15ef40 Merge pull request #588 from andrewgaul/https 
Default transport to HTTPS
 2017-05-09 23:04:57 +09:00
Takeshi Nakatani
a22675bafd Merge pull request #567 from andrewgaul/default-acl 
Do not send ACL unless overridden
 2017-05-09 23:03:27 +09:00
Andrew Gaul
0e0ae38f6d Default transport to HTTPS 
This protects private data when used over the public Internet.  Users
can opt-in to unencrypted HTTP if they need additional performance on
a local network.  Fixes #282.
 2017-05-07 10:59:54 -07:00
Andrew Gaul
7b30d5d15b Do not send canned ACL header when empty string 
Some providers such as StorageGRID do not support canned ACLs.
Setting to empty allows callers to omit the header.  References #125.
 2017-05-07 10:52:31 -07:00
Takeshi Nakatani
9d10a5aa70 Changed copyright year format for debian pkg 2017-05-07 11:24:17 +00:00
Takeshi Nakatani
a12e0d5ec4 Fixed failure to upload/copy with SSE_C and SSE_KMS 2017-05-07 09:29:08 +00:00
Takeshi Nakatani
d07c3f38b7 Check errors returned in 200 OK responses for put header request 2017-05-06 02:15:53 +00:00
Takeshi Nakatani
1fe0334c08 Enhanced bucket/path parameter check 2017-05-05 19:55:24 +00:00
Takeshi Nakatani
3ac39d61f8 Added notsup_compat_dir option 2017-05-05 17:28:29 +00:00
Takeshi Nakatani
864e20e1f2 Refactored the get_object_attribute function 2017-05-05 10:02:21 +00:00
Takeshi Nakatani
51b3183cba Refactored the check_object_access function 2017-05-05 09:51:30 +00:00
Takeshi Nakatani
758b92e823 Added option for complementing lack of stat mode 2017-05-04 03:41:24 +00:00
Takeshi Nakatani
df0ff3a2fd Merge pull request #556 from orozery/fix_nocache_multipart_upload 
fix multipart upload handling without cache
 2017-04-16 19:22:15 +09:00
Takeshi Nakatani
edcf4c6218 Merge pull request #555 from orozery/dont_sign_empty_headers 
don't sign empty headers (as they are discarded by libcurl)
 2017-04-16 19:16:47 +09:00
Takeshi Nakatani
28efff5986 Merge pull request #554 from orozery/cache_cleanup 
cleanup cache directory when running out of disk space
 2017-04-16 19:13:11 +09:00
Takeshi Nakatani
efba9bcbc1 Merge pull request #553 from orozery/custom_cipher_suite 
add TLS cipher suites customization
 2017-04-16 19:09:27 +09:00
Takeshi Nakatani
6bd179c92b Merge pull request #552 from orozery/foreground_threads 
switch S3fsMultiCurl to use foreground threads
 2017-04-16 19:05:16 +09:00
Or Ozeri
96764b7410 switch S3fsMultiCurl to use foreground threads 2017-04-09 16:56:49 +03:00
Takeshi Nakatani
ff3eb1971f Merge branch 'master' into fix_nocache_multipart_upload 2017-04-09 22:13:33 +09:00
Takeshi Nakatani
b4c90d6957 Fixed a bug about multipart uploading at no disk free space related to #509 2017-04-09 04:37:20 +00:00
Or Ozeri
75b59a7c16 switch S3fsMultiCurl to use foreground threads 2017-04-04 15:32:53 +03:00
Or Ozeri
3bcca75a88 don't sign empty headers (as they are discarded by libcurl) 2017-04-04 15:24:20 +03:00
Takeshi Nakatani
f0f61b3b55 Fixed a bug in logic about truncating stat cache 2017-04-02 11:51:58 +00:00
Takeshi Nakatani
fef3fbc225 Added check_cache_dir_exist option(refixed #347) - #538 2017-04-02 08:10:16 +00:00
Or Ozeri
8ee95ff7ab fix multipart upload handling without cache 2017-04-02 10:27:43 +03:00
Or Ozeri
95578cad43 cleanup cache directory when running out of disk space 2017-04-02 10:22:12 +03:00
Takeshi Nakatani
465bbd3729 Updated stat_cache_expire option description - #545 2017-04-02 07:19:16 +00:00
Takeshi Nakatani
980ba398bc Fixed double initialization of SSL library - #524 2017-03-19 17:11:18 +00:00
Takeshi Nakatani
523043a2aa Changed base cached time of stat_cache_expire option - #523 2017-03-19 15:19:04 +00:00
Andrew Gaul
03217baa99 Address cppcheck 1.77 warnings 2017-03-06 12:41:08 -08:00
Takeshi Nakatani
6affefff5b Merge pull request #509 from andrewgaul/mpu 
Use server-provided ETag during complete upload
 2017-03-06 21:54:51 +09:00
Takeshi Nakatani
6fc972972f Changed clock_gettime func to s3fs_clock_gettime for homebrew - #468 2016-12-04 10:31:41 +00:00
Takeshi Nakatani
7b307601b5 Merge pull request #511 from s3fs-fuse/issue#435 
Fixed a bug about uploading NULL to some part of the file contents
 2016-12-04 17:20:50 +09:00
Andrew Gaul
915a1321c7 Use server-provided ETag during complete upload 
This avoids calculating the MD5 locally and enables use with object
stores which do not use MD5 as ETag.
 2016-11-23 18:48:57 -08:00
Andrew Gaul
d375bca0d0 Correct typos 2016-11-19 15:57:41 -08:00
Takeshi Nakatani
edd0a11fb5 Merge pull request #494 from mapreri/typo 
Fix typo s/destroied/destroyed/
 2016-11-20 05:53:23 +09:00
Takeshi Nakatani
67a836223a Merge pull request #495 from driskell/fix_sse_copy 
Fix invalid V4 signature on multipart copy requests
 2016-11-20 05:43:10 +09:00
Takeshi Nakatani
81e209bdd1 Fixed issue#435 codes 2016-11-19 20:09:35 +00:00
Jason Woods
6f688770fd Fix invalid V4 signature on multipart copy requests 2016-11-13 13:22:00 +00:00
Mattia Rizzolo
8c0b1d9c5b
Fix typo s/destroied/destroyed/ 2016-11-11 23:27:17 +00:00
Takeshi Nakatani
efde0ec9de Merge pull request #489 from ggtakec/master 
Changed headers_t map using nocase compare function - #488
 2016-10-23 23:23:31 +09:00
Takeshi Nakatani
632495374b Chnaged headers_t map using nocase compare function - #488 2016-10-23 14:14:19 +00:00
Takeshi Nakatani
15b797f3ee Merge pull request #488 from ggtakec/master 
Fixed searching Content-Length without case sensitive - #480
 2016-10-23 22:42:52 +09:00
Takeshi Nakatani
a7a64d954a Fixed searching Content-Length without case sensitive - #480 2016-10-23 13:27:01 +00:00
Takeshi Nakatani
cca217f613 Merge pull request #487 from driskell/debugging 
Split header debugging onto multiple lines for easier reading
 2016-10-23 21:51:38 +09:00
Takeshi Nakatani
1a9cf6f66d Fixed searching Content-Length without case sensitive - #480 2016-10-23 12:40:51 +00:00
Jason Woods
02d7296210 Split header debugging onto multiple lines for easier reading 2016-10-22 15:11:18 +01:00
Takeshi Nakatani
a688df813e Fixed a bug at read symlink 2016-10-11 13:32:08 +00:00
Takeshi Nakatani
164424bc89 Merge branch 'master' into issue#435 2016-10-11 12:13:03 +00:00
Takeshi Nakatani
7fabd18b1f Trim symbolic link original path in file. 2016-10-11 10:22:30 +00:00
Takeshi Nakatani
5db369d67e Trim symbolic link original path in file. 2016-10-11 10:17:46 +00:00
Takeshi Nakatani
dba32fdf78 Trim symbolic link original path in file. 2016-10-11 10:06:21 +00:00
Takeshi Nakatani
716baada22 Testing patch codes for issue#435 2016-10-10 12:16:09 +00:00
Takeshi Nakatani
9fd1368611 OS-specific correspondence of the extended attribute header 2016-10-02 07:23:19 +00:00
Takeshi Nakatani
65d52506c4 Added use_xattr option for #467 and #460 2016-09-19 04:28:01 +00:00
Takeshi Nakatani
1a96f40a10 Fixed a bug about could not copy file mode from org file 2016-09-11 13:09:23 +00:00
Takeshi Nakatani
1ddc14d59d Changed for accepting mount options compatible with mtab - #449 2016-07-24 08:17:58 +00:00
Peter Watkins
a19206cf0f Accept mount arguments compatible with mtab 
Using "mount -a" fails for already-mounted s3fs directories,
because s3fs mount arguments don't match the form in /etc/mtab.
Calling "mount -a" should quietly succeed when a directory is
already mounted.

To fix this, accept mount commands of the form:

s3fs s3fs /srv/object-store -o bucket=mybucket

or in /etc/fstab form:

s3fs  /srv/object-store  fuse.s3fs  bucket=mybucket 0 0

This matches the form in /etc/mtab and allows "mount -a" to
work properly.
 2016-07-13 17:23:33 -04:00
Takeshi Nakatani
0f9428ad5a Add mirror file logic for removing cache file 2016-07-13 17:23:32 -04:00
Tony Levi
ab4b92074c Fix cppcheck complaint 2016-06-30 11:51:25 +09:30
Tony Levi
d57c12d3c3 Possible fix for leaks during entry expiry / truncation (#340) 2016-06-28 15:01:14 +09:30
Takeshi Nakatani
6005929a96 Handled all curl error without exiting process - #437 2016-06-27 10:38:49 +00:00
Takeshi Nakatani
49ffaa1d94 Merge pull request #432 from andrewgaul/create-bucket-location 
Include location constraint when creating bucket
 2016-06-26 09:39:47 +09:00
Andrew Gaul
320b8e1171 Include location constraint when creating bucket 
This allows creating buckets in non-default regions.  Also improve
do_create_bucket error handling.
 2016-06-13 10:35:37 -07:00
Andrew Gaul
95cb5d201f Correct search and replace typo 2016-06-13 10:25:33 -07:00
Nathaniel W. Turner
584ea488bf Use role name instead of profile name when iam_role=auto 
When using an instance with an IAM Role, transient credentials can be
found in http://169.254.169.254/latest/meta-data/ at
iam/security-credentials/role-name and s3fs tries to do this. However,
it is using the profile-name where role-name is needed. In many cases
the role and profile name are the same, but they are not always.

The simplest way to find the role name appears to be to GET
http://169.254.169.254/latest/meta-data/iam/security-credentials/
itself, which returns a listing of the role names for which temporary
credentials exist. (I think there will probably only be one, but we
probably want to split on newlines and take the first one here in case
that assumption is not valid). This is the approach the AWS SDK appears
to use (based on WireShark analysis).

Bug: https://github.com/s3fs-fuse/s3fs-fuse/issues/421
Signed-off-by: Nathaniel W. Turner <nate@houseofnate.net>
 2016-05-24 13:34:19 -04:00
Nathaniel W. Turner
594c9ca7d2 Skip early credential checks when iam_role=auto 
If user specifies iam_role=auto (or just iam_role), credentials will not
be loaded during early phase, so skip credential checks there.

Signed-off-by: Nathaniel W. Turner <nate@houseofnate.net>
 2016-05-20 12:49:02 -04:00
Takeshi Nakatani
34b604cdfe Fixed a bug about stat_cache_expire - #382 2016-05-14 09:03:52 +00:00
Takeshi Nakatani
50f1ad51c8 loading IAM role name automatically(iam_role option) - #387 2016-05-06 04:37:32 +00:00
Takeshi Nakatani
6cc30eea44 Allow duplicate key in ahbe_conf - #386 2016-05-06 01:08:39 +00:00
Takeshi Nakatani
1ddbd4d6bb Fixed 'load_sse_c' option not working - #388 2016-05-06 00:36:54 +00:00
Takeshi Nakatani
845fdb43f2 Merge pull request #404 from rockuw/keep-alive 
Add curl handler pool to reuse connections
 2016-04-26 23:40:45 +09:00
Tianlong Wu
b78adb4bb0 Add curl handler pool to reuse connections 2016-04-22 14:57:31 +08:00
Tianlong Wu
115bd51f3f Fix a bug of truncating empty file 2016-04-22 14:49:37 +08:00
Takeshi Nakatani
10589a9497 Supported User-Agent header - #383 2016-04-17 07:44:03 +00:00
Takeshi Nakatani
090c37a1c1 Fixed writing sparsed file - #375,#379,#394 2016-04-12 18:24:36 +00:00
Takeshi Nakatani
fff40bbff3 Revert "Fixed a bug about writing sparsed file - #375" 2016-04-13 01:24:24 +09:00
Mattia Rizzolo
136ec654c2
fix typo in curl.cpp: s/returing/returning/ 2016-04-02 15:19:06 +00:00
Takeshi Nakatani
ded4faf2e4 Fixed a bug about writing sparsed file - #375 2016-03-22 05:44:14 +00:00
Takeshi Nakatani
98d55582eb Chnaged about constructor(destructor) in cache.h 2016-03-13 09:47:37 +00:00
Takeshi Nakatani
84bdd51021 Fixed a bug about etag comparison in stats cache. 2016-03-13 09:29:06 +00:00
Takeshi Nakatani
67efc11d94 Always set stats cache for opened file 2016-03-13 05:43:28 +00:00
Takeshi Nakatani
4c65c09f4d Checked content-type by no case-sensitivity - #363 2016-02-13 05:58:59 +00:00
yurykats
230991782b Update s3fs_util.cpp 2016-02-08 16:39:56 -05:00
Takeshi Nakatani
f81e6103cb Fixed codes about clock_gettime for osx(3) 2016-02-07 08:27:02 +00:00
Takeshi Nakatani
0755c6f60c Fixed codes about clock_gettime for osx(2) 2016-02-07 08:10:23 +00:00
Takeshi Nakatani
e01ded9e27 Fixed codes about clock_gettime for osx 2016-02-07 07:40:55 +00:00
Takeshi Nakatani
1af7aaeccb Fixed addhead.cpp for cppcheck 2016-02-07 05:53:56 +00:00
Takeshi Nakatani
c7cf86c2ef Sepalated AdditionalHeader class from curl.* 2016-02-07 05:41:56 +00:00
Takeshi Nakatani
6472eedddc Supported regex type for additional header format. 2016-02-07 05:08:52 +00:00
Takeshi Nakatani
150b83f61e Remove stat file cache dir if specified del_cache - #337 2016-02-06 18:59:13 +00:00
Takeshi Nakatani
c5a94cfc0c Check cache dirctory path and attributes - #347 2016-02-06 13:38:48 +00:00
Takeshi Nakatani
203df6b58a Merge pull request #346 from RobbKistler/empty-dir 
Fix empty directory check against AWS S3
 2016-02-06 18:59:45 +09:00
Takeshi Nakatani
b90b51f2c5 Changed cache out logic for stat - #340 2016-02-06 09:09:17 +00:00
Takeshi Nakatani
8b457133da Merge pull request #341 from hryang/master 
Fix the memory leak issue in fdcache.
 2016-02-06 14:41:28 +09:00
Robb Kistler
4eff6b4dd1 Fix empty directory check against AWS S3 
For ListBucketResult on an empty directory, AWS S3 and S3Proxy 1.4
differ.  AWS will match the directory name, S3Proxy does not.

Changing max-keys=1 to max-keys-2 works for both implementations.
append_objects_from_xml() will swallow the directory key.  The log
level of this message is changed from ERROR to DBG.

Fixes #345
 2016-02-04 23:13:00 -08:00
haoran.yanghr
e3765ad497 Tune the code indent. 2016-01-28 11:16:06 +08:00
haoran.yanghr
dd9f3aed36 Fix the memory leak issue in fdcache. See issue #340 2016-01-28 11:11:53 +08:00
Takeshi Nakatani
83937700dd Fixed a bug about IAMCRED type could not be retried. 2016-01-24 05:01:50 +00:00
Takeshi Nakatani
e932583309 Merge pull request #334 from andrewgaul/bucket-host 
Bucket host should include port and not path
 2016-01-17 14:46:40 +09:00
Andrew Gaul
88a4f04217 Bucket host should include port and not path 
This resolves issues when using v4 signing with path-style requests.
 2016-01-16 15:58:54 -08:00
Andrew Gaul
ff607e1a2d Correct multiple issues with ListBucketRequest 
* provide correct path
* sign query string
* URL encode query string
 2016-01-16 10:17:20 -08:00
Takeshi Nakatani
4bfbfa3621 Merge pull request #331 from andrewgaul/clang 
Address various clang warnings
 2016-01-16 16:14:50 +09:00
Takeshi Nakatani
43b91d3235 Merge pull request #330 from andrewgaul/pass-by-reference 
Pass by const reference where possible
 2016-01-16 16:14:31 +09:00
Andrew Gaul
e003732f18 Address various clang warnings 
Found with:

-Wc++11-extensions
-Wc++11-extra-semi
-Wmissing-variable-declarations
-Wundef
 2016-01-11 00:52:24 -08:00
Andrew Gaul
b946b59522 Pass by const reference where possible 2016-01-10 16:58:24 -08:00
Andrew Gaul
ea6b287d1a Fix v4 signature with use_path_request_style 
Previously s3fs omitted the bucket name when using path request style
causing SignatureDoesNotMatch with v4 signatures.
 2016-01-10 13:41:56 -08:00
Takeshi Nakatani
ea151a70c4 Merge pull request #321 from mcellis33/320 
320: delete stat cache entry in s3fs_fsync so st_size is refreshed
 2015-12-20 15:05:28 +09:00
Mark Ellis
163daa5de1 320: delete stat cache entry in s3fs_fsync so st_size is refreshed 2015-12-18 15:39:25 -08:00
Robb Kistler
b581290c30 Cleanly exit fuse loop on error in s3fs_init 
This allows FUSE to clean the mount point up, preventing
"Transport endpoint not connected" errors on subsequent
access to the mount.
 2015-12-15 15:25:56 -08:00
Takeshi Nakatani
2b3ece467b Merge pull request #311 from RobbKistler/dbg-message 
Change error log to debug log in s3fs_read()
 2015-12-08 00:03:48 +09:00
Robb Kistler
8e688816d4 Change error log to debug log in s3fs_read() 2015-12-03 21:25:27 -08:00
Takeshi Nakatani
759b44135a Check pthread prtability in configure as additional change for #307 2015-12-03 07:47:17 +00:00
Takeshi Nakatani
8b53e0d931 Merge pull request #307 from rockuw/master 
Fix pthread portability problem
 2015-12-03 16:35:30 +09:00
Tianlong Wu
3e655bad3b PTHREAD_MUTEX_RECURSIVE_NP is a enum not macro 2015-12-03 13:44:11 +08:00
Takeshi Nakatani
5e97cb0f48 Changed ensure free disk space as additional change for #306 2015-12-03 05:40:26 +00:00
Tianlong Wu
f44b61c403 Fix pthread portability problem 2015-12-03 10:44:38 +08:00
Guy
6067af6ef1 Fix read concurrency to work in parallel count 
When the prefetch size is limited to the multipart size, the entire parallel logic of the read flow does not have an opportunity to use parallel get.
This fix increases the read performance significantly over our own s3 on-premise solution.
 2015-11-30 18:38:15 +02:00
Takeshi Nakatani
7b62de80f6 Fixed a bug about mtime - #299 2015-11-29 15:53:53 +00:00
Robb Kistler
fff2952d5f Fix syslog level used by S3FS_PRN_EXIT() 
Without converting from s3fs log levels to syslog levels, the syslog
ends up being LOG_EMERG which can cause a broadcast message to all
users.
 2015-11-25 13:53:08 -08:00
Takeshi Nakatani
c673d9d935 File opened with O_TRUNC is not flushed - changed #291 2015-11-08 04:55:17 +00:00
Takeshi Nakatani
0fdda61fb5 Merge pull request #293 from SnakeHunt2012/master 
Fix a small spelling issue.
 2015-11-08 13:26:01 +09:00
Takeshi Nakatani
331b8456a0 Merge pull request #291 from RobbKistler/truncate 
Issue #290: File opened with O_TRUNC is not flushed
 2015-11-08 13:23:06 +09:00
SnakeHunt2012
c04bcce206 Fix a small spelling issue. 2015-11-06 16:49:37 +08:00
Robb Kistler
dd7d9268f2 Force flush in s3fs_open() if file is truncated. 2015-11-03 22:06:25 -08:00
Robb Kistler
e4da5c59b6 Print source file in log messages 2015-11-03 08:34:02 -08:00
Takeshi Nakatani
001206f7c1 Fixed a bug about head request(copy) for SSE - issue#286 2015-11-01 14:05:47 +00:00
Takeshi Nakatani
2ef7f497f6 Fixed a bug about head request(copy) for SSE - issue#286 2015-11-01 13:54:47 +00:00
Takeshi Nakatani
8dd234dd8f Fixed bugs about cppcheck error 2015-10-20 15:47:07 +00:00
Takeshi Nakatani
83d46ef8c6 Fixed bugs about a object larger than free disk space 2015-10-20 15:19:04 +00:00
Takeshi Nakatani
d102eb752d Supported a object which is larger than free disk space 2015-10-18 17:03:41 +00:00
Takeshi Nakatani
f51ad1f33e Supported for SSE KMS 2015-10-06 14:46:14 +00:00
Takeshi Nakatani
92e52dadd4 Changed and cleaned the logic for debug message. 2015-09-30 19:41:27 +00:00
Andrew Gaul
785ed642ba Add support for standard_ia storage class 
This enables storage with lower at-rest prices, higher request prices,
and lower availability.  Also rework existing reduced redundancy
parsing into a more generic storage class.  More background on
standard_ia:

https://aws.amazon.com/blogs/aws/aws-storage-update-new-lower-cost-s3-storage-option-glacier-price-reduction/
 2015-09-17 13:35:25 -07:00
Takeshi Nakatani
ead346c6d3 Merge pull request #261 from andrewgaul/help-timeouts 
Correct help timeouts
 2015-09-13 15:49:15 +09:00
Andrew Gaul
0358908910 Correct help timeouts 
Follow-on to #167.
 2015-09-10 11:45:05 -07:00
Andrew Gaul
32ce1a7267 Wrap help text at 80 characters 2015-09-10 11:43:09 -07:00
Takeshi Nakatani
97b8b34aab Added chacking cache dir perms at starting(2). 2015-08-23 04:14:57 +00:00
Takeshi Nakatani
ce66430fac Added chacking cache dir perms at starting. 2015-08-23 03:57:34 +00:00
Oliver Friedmann
d7d96907cf This fixes an issue with caching when the creation of a subdirectory within the cache is aborted because a common cached parent directory already exists. 2015-08-21 19:30:04 -04:00
Takeshi Nakatani
751c868769 s3fs can print version with short commit hash - #228 2015-08-21 16:19:31 +00:00
Takeshi Nakatani
632578f328 Merge pull request #247 from andrewgaul/base64 
Base64 cleanup
 2015-08-22 00:45:26 +09:00
Andrew Gaul
bcfadbe1a8 Unlock during early return in TruncateCache 
Found via Coverity.  Regression from
dfa63345ed.
 2015-08-19 13:54:14 -07:00
Andrew Gaul
b5c027f15d Add unit tests for base64 encoding and decoding 2015-08-19 13:49:10 -07:00
Andrew Gaul
15db80b459 NUL terminate decoded base64 string 
For consistency with encoded strings.
 2015-08-19 13:48:07 -07:00
Andrew Gaul
76c0ef86e4 Move base64 and hex functions to string_util 2015-08-19 13:47:26 -07:00
Takeshi Nakatani
a3e820e733 Merge pull request #245 from andrewgaul/map-duplicate-lookups 
Elide duplicate lookups of std::map via iterators
 2015-08-20 01:22:06 +09:00
Takeshi Nakatani
4ad57bdea5 Merge pull request #240 from andrewgaul/md5 
Enable Content-MD5 during multipart upload part
 2015-08-20 01:19:01 +09:00
Takeshi Nakatani
085733d7c9 Merge pull request #239 from andrewgaul/google-code 
Update stale Google Code reference in --help
 2015-08-20 01:08:00 +09:00
Takeshi Nakatani
fcb58aec3c Merge pull request #238 from andrewgaul/cppcheck 
Enable all cppcheck rules
 2015-08-20 01:06:50 +09:00
Takeshi Nakatani
026a9f2bdc Merge pull request #235 from andrewgaul/complete-mpu-leak 
Plug leak during complete multipart upload
 2015-08-20 00:40:00 +09:00
Takeshi Nakatani
fd04b9a437 Merge pull request #233 from andrewgaul/remove-inttostr 
Remove IntToStr
 2015-08-20 00:34:21 +09:00
Takeshi Nakatani
ea99603b58 Merge pull request #232 from andrewgaul/stat-cache-locking 
Always hold stat_cache_lock when using stat_cache
 2015-08-20 00:28:57 +09:00
Andrew Gaul
67d1576dfb Elide duplicate lookups of std::map via iterators 
Also remove use of C++11 std::map::at.
 2015-08-18 14:00:42 -07:00
Andrew Gaul
a157ac59ca Enable Content-MD5 during multipart upload part 
This allows retries of multi-part uploads instead of discovering a
fatal error during complete multipart upload.  Also enable Content-MD5
for integration tests and refactor hexadecimal code.
 2015-08-18 02:54:00 -07:00
Andrew Gaul
32520fd1fb Update stale Google Code reference in --help 2015-08-16 23:30:41 -07:00
Andrew Gaul
c0b21d8808 Enable all cppcheck rules 2015-08-16 17:13:24 -07:00
Andrew Gaul
9c5bf0bb66 Plug leak during complete multipart upload 2015-08-15 22:38:24 -07:00
Andrew Gaul
dfa63345ed Always hold stat_cache_lock when using stat_cache 
We could further improve this code by holding stat_cache_lock before
calls to DelStat instead of unlocking then relocking it.
 2015-08-14 20:14:12 -07:00
Andrew Gaul
3f59b8da01 Rewrite AutoLock 
Previously AutoLock::Lock allowed subsequent callers to proceed
without the lock.  Further is_locked was not always protected by
auto_mutex.  Finally AutoLock eagerly released auto_mutex when
recursively unlocking.  s3fs does not need recursive locks so we
rewrite and simplify AutoLock.  Partially surfaced by Coverity.
 2015-08-14 20:00:56 -07:00
Andrew Gaul
0ea88a73c7 Remove IntToStr 
str duplicates this functionality.  Also add unit test.
 2015-08-12 08:25:09 -07:00
Takeshi Nakatani
756d1e5e81 Configure cppcheck #224 2015-08-12 15:04:16 +00:00
Takeshi Nakatani
2482aada43 Merge pull request #222 from andrewgaul/explicit 
Annotate constructors as explicit
 2015-08-12 23:42:03 +09:00
Takeshi Nakatani
64146f69a4 Merge pull request #221 from andrewgaul/compare 
Compare idiomatically
 2015-08-12 23:41:24 +09:00
Takeshi Nakatani
49e32967ec Merge pull request #219 from andrewgaul/coverity 
Address Coverity errors
 2015-08-12 23:40:47 +09:00
Takeshi Nakatani
09dff484e1 Merge pull request #215 from RobbKistler/memleak 
Fix mem leak in openssl_auth.cpp:s3fs_sha256hexsum
 2015-08-12 23:38:51 +09:00
Andrew Gaul
ff8a0c2eea Parse ETag from copy multipart correctly 
Previously s3fs misparsed this, preventing renames of files larger
than 5 GB.  Integration test disabled until S3Proxy 1.5.0 is released.
 2015-08-11 14:43:35 -07:00
Robb Kistler
fcb55c2109 Fix mem leaks in openssl_auth.cpp, nss_auth.cpp 
Fix memory leaks in openssl_auth.cpp:s3fs_sha256hexsum and
nss_auth.cpp:s3fs_sha256hexsum.  Leaks occur every time a file
is created.
 2015-08-06 12:45:40 -07:00
Andrew Gaul
b6fa2deb9f Annotate constructors as explicit 
This prevents implicit conversions.
 2015-08-05 23:41:53 -07:00
Andrew Gaul
801ca0c2d3 Compare idiomatically 2015-08-05 23:35:08 -07:00
Andrew Gaul
8ee71caabb Address Coverity errors 
Fixed an uninitialized member, misordered NULL check, resource leak,
and unconsumed return value.
 2015-08-05 23:28:06 -07:00
Jamie Alessio
912bc58df0 Fixed a few small spelling issues. 2015-07-10 11:50:40 -07:00
Nate Rosenblum
9b3c87ec97 Specialize {set,get}xattr for OS X 
These system calls take an extra 'position' parameter on OS X. A
non-zero position value is only valid for resource forks (the Darwin
VFS layer will reject anything else with EINVAL); this patch simply
adds and ignores the parameter on Apple platforms.

Allows building against OSXFUSE.
 2015-06-25 12:56:15 -07:00
Takeshi Nakatani
966d229787 fixed fallback to sigv2 for bucket create and GCS 2015-06-20 04:34:32 +00:00
Takeshi Nakatani
4d49ace06b Merge pull request #192 from andrewgaul/special-characters 
Simplify URL encoding
 2015-06-20 11:47:22 +09:00
Takeshi Nakatani
ad8c64104e Merge pull request #199 from s3fs-fuse/xattr 
Supported extended attributes(retry)
 2015-06-20 11:46:47 +09:00
Takeshi Nakatani
fe3abed9f0 Chaged codes about iterator etc 2015-06-13 03:27:07 +00:00
Takeshi Nakatani
0ecf4aa6b4 Chaged codes about iterator 2015-06-13 03:08:56 +00:00
Andrew Gaul
4e03acf17a Simplify URL encoding 
This also encodes asterisk and tilde correctly when listing a file
with a V4 auth endpoint.  Also add tests for special characters
although s3proxy does not yet support V4 auth.
Fixes #188.  Fixes #194.
 2015-06-10 13:15:58 -07:00
Takeshi Nakatani
84fb3d83d8 Fixed xattr for binary value 2015-06-06 16:39:39 +00:00
Bartlomiej Palmowski
3522e5eda3 Add no_check_certificate option which allows to ignore issues with self signed certs. 2015-05-20 17:32:36 +02:00
Takeshi Nakatani
3056644969 Merge pull request #185 from andrewgaul/typos 
Correct obvious typos in usage and README
 2015-05-06 22:37:22 +09:00
Andrew Gaul
8a73d9fff0 Correct obvious typos in usage and README 2015-05-04 16:25:05 -07:00
Andrew Gaul
28ee9f27b9 Add usage information for multipart_size 
Also improve error message.
 2015-05-04 16:21:58 -07:00
Dan Moore
5b11ac0f4c Moved __APPLE__ #endif to correct position 2015-04-27 12:14:09 -07:00
Dan Moore
adb5a35097 configure.ac: detect target, if target is darwin (OSX), then 
change the minimum version of fuse required.  Change the
checkers to use a variable for the minimum fuse version
instead of it being hardcoded in four different places.

src/s3fs.cpp: Use __APPLE__ define around fuse code that
is offensive to osxfuse. Not including the code doesn't
seem to matter.
 2015-04-25 17:13:20 -07:00
Takeshi Nakatani
aac92bd6c0 Fixed wrong owner checking and return codes 2015-04-21 16:18:05 +00:00
Takeshi Nakatani
f258a14070 Supported extended attributes, initial commit 2015-04-20 17:24:57 +00:00
Peter A. Bigot
92fcee824b curl: use pathrequeststyle option when constructing Host endpoint 
Buckets with mixed-case names can't be accessed with the virtual-hosted
style API due to DNS limitations.  S3FS has an option for
pathrequeststyle which is used for the URL, but it was not applied when
building the endpoint passed through the Host header.  Fix this, and
relax the validation on bucket names when using this style.

See: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro

Signed-off-by: Peter A. Bigot <pab@pabigot.com>
 2015-04-19 08:31:40 -05:00
Takeshi Nakatani
490ed8f689 Reviewed and fixed response codes print in curl.cpp - #157 2015-04-18 13:32:04 +00:00
Ka-Hing Cheung
70097709b2 switch to use region specific endpoints to compute correct v4 signature 
fix #133
 2015-04-14 16:25:17 -07:00
Takeshi Nakatani
bd27294ab0 Increased default connecting/reading/writing timeout value 2015-04-12 02:04:13 +00:00
Ka-Hing Cheung
6e0a302f7d refactor sigv4 to reduce code duplication 2015-04-09 15:11:59 -07:00
Ka-Hing Cheung
98af055d8b send the correct Host header when using -o url 
fixes #161
 2015-04-09 13:53:50 -07:00
Takeshi Nakatani
0f13c8fe97 Fixed a bug about ssl session sharing with libcurl older 7.23.0 - issue/#126 2015-03-21 07:04:20 +00:00
Takeshi Nakatani
2fc3a4e91e Fixed a bug: unable to mount bucket subdirectory 2015-03-21 04:31:59 +00:00
Takeshi Nakatani
a04bec85b2 Fixed url-encoding for ampersand etc on sigv4 - Improvement/#149 2015-03-21 02:11:55 +00:00
Takeshi Nakatani
26453c4874 Fixed a bug not handling fsync. 2015-03-10 16:18:03 +00:00
Takeshi Nakatani
d06b6d7d41 Fixed a bug no use_cache case about fixed #138 - issue#141 2015-03-08 16:41:14 +00:00
Takeshi Nakatani
114966e7c0 Fixed bugs, not turn use_cache off and ty to load to end - issue#97 2015-03-04 08:48:37 +00:00
Takeshi Nakatani
2f8ad7ace8 Merge pull request #135 from andrewgaul/mpu-v4 
Correct V4 signature for initiate multipart upload
 2015-03-01 22:57:10 +09:00
Andrew Gaul
a07e804f57 Include Content-Type in complete MPU V2 signature 
Previously this failed with SignatureDoesNotMatch since the headers
included it but the signature did not.  Fixes #125.
 2015-02-28 18:03:21 -08:00
Andrew Gaul
e9656810e3 Correct V4 signature for initiate multipart upload 
Query parameters need a trailing = for V4 signatures.  Send correct
content-sha256 although Amazon does not seem to enforce this for
zero-length bodies.  Finally remove a stale comment.  Fixes #133.
 2015-02-28 17:50:06 -08:00
Takeshi Nakatani
53083202ba Merge pull request #132 from andrewgaul/s3proxy-integration-test 
Use S3Proxy to run integration tests
 2015-02-27 00:17:46 +09:00
Andrew Gaul
e811ae1104 Use s3proxy to run integration tests 
References #129.
 2015-02-24 12:08:22 -08:00
Timur Alperovich
be5735edb8 Fallback to v2 signatures correctly. 
Missing parameter to SetSignatureV4() call in the fallback code path
results in not actually falling back.
 2015-02-16 17:35:09 -08:00
Takeshi Nakatani
cf2b0cca22 Added new mp_umask option about issue#107, pr#110 2015-02-07 17:16:45 +00:00
Takeshi Nakatani
1424f87754 Supported signature version 4 for GnuTLS/NSS and automatically set endpoint/sigv2 2015-02-02 16:36:08 +00:00
Takeshi Nakatani
4f953f9bd7 Clean codes for signature v4 and added new sigv2 option 2015-01-28 17:13:11 +00:00
Takeshi Nakatani
0d2f3e2dc4 Fixed bugs, segfault and signature error at listing. 2015-01-24 16:36:30 +00:00
Takeshi Nakatani
bb1f1d3faa Merged manually from caxapniy/s3fs-fuse/tree/1.77v4merge for signature v4 - #102 2015-01-20 16:31:36 +00:00
Takeshi Nakatani
98daf16681 Merge pull request #104 from kahing/rename_before_close 
fix rename before close
 2015-01-14 00:40:41 +09:00
Takeshi Nakatani
939ba2b4b3 Merge pull request #101 from adobos/directory_empty_optimization 
Optimized function "bool directory_empty()"
 2015-01-14 00:21:47 +09:00
Takeshi Nakatani
d0b82428d5 Merge pull request #100 from adobos/dns_ssl_switch_bugfix 
CURL handles not properly initialized to use DNS or SSL session caching.
 2015-01-14 00:11:46 +09:00
Takeshi Nakatani
902911765e Merge pull request #93 from andrewgaul/unit-test 
Add simple unit tests for trim functions
 2015-01-14 00:07:01 +09:00
Ka-Hing Cheung
03d84a07d1 fix rename before close 
nautilus does this when you drag and drop to overwrite a file:

1) create .goutputstream-XXXXXX to write to
2) fsync the fd for .goutputstream-XXXXXX
3) rename .goutputstream-XXXXXX to target file
4) close the fd for .goutputstream-XXXXXX

previously, doing this on s3fs would result in an empty target file
because after the rename, s3fs would not flush the content of
.goutputstream-XXXXXX to target file.

this change moves the FdEntity from the old path to the new path
whenever rename happens. On flush s3fs would now flush the correct
content to the rename target.
 2015-01-12 15:05:54 -08:00
Andrej Dobos
045f1e7906 CURL handles were not properly initialized to use DNS caching, or SSL session caching. 2014-12-23 22:31:54 -08:00
Andrej Dobos
69ef7fbefb Optimized function directory_empty: check for at most one entry when evaluating whether a directory is empty or not (as opposed to doing full directory listing) 2014-12-23 22:29:13 -08:00
Andrew Gaul
a56b8db410 Add simple unit tests for trim functions 
Subsequent commits will use this infrastructure.  Also reparent
prepare_url which relies on unrelated bucket, foreground2, and
pathrequeststyle symbols.
 2014-12-06 18:07:14 -08:00
Takeshi Nakatani
082eb24c12 Merge pull request #83 from tmwong2003/develop 
Changed option processing to use strtol() to get a umask
 2014-11-16 23:49:24 +09:00
Theodore Wong
f04b659f5e Changed option processing to use strtol() to get a umask 
get_mode()/s3fs_strtoofft() does not handle octal umask values, which
results in unexpected behavior when trying to set a world-readable umask
value.
 2014-11-12 23:29:41 +00:00
bupt_tengteng
b31ec5c4af Update curl.cpp 
The space causes signature mismatch when using "ahbe_conf" file to add additional headers.When s3 use the" x-amaz" header to calculates the signature, the format is as follow:
PUT

application/octet-stream
Wed, 05 Nov 2014 03:05:08 GMT
x-amz-acl:private
x-amz-meta-gid:0
x-amz-meta-mode:33188
x-amz-meta-mtime:1415156708
x-amz-meta-uid:0
There is no space after colon.
 2014-11-05 11:28:33 +08:00
Takeshi Nakatani
651e8c3158 Merge pull request #64 from andrewgaul/failed-read-eio 
Return EIO on failed read
 2014-11-03 01:03:32 +09:00
Vincent Bernat
7212072ff0 url: handle scheme omission 
When the scheme is omitted in URL overriding (for example `example.com`
instead of `https://example.com`), s3fs is modifying the URL by
inserting `s3.` in the middle of the name  (`examples3..com`).

This can be a bit difficult to troubleshoot and curl seems to handle
schema-less requests just fine. So, just handle this case correctly.
 2014-10-23 10:25:17 +02:00
Takeshi Nakatani
28d82c9ccd Fixed #68(FreeBSD issue) 2014-10-22 14:21:01 +00:00
Andrew Gaul
50137fe026 Address clang always true warnings 2014-10-16 23:34:12 -07:00
Takeshi Nakatani
9237d07226 Merge pull request #63 from jollyroger/spelling 
Fix spelling errors
 2014-10-13 11:38:13 +09:00
Takeshi Nakatani
8c2be4aa85 Merge pull request #62 from jollyroger/fix-stray-chars 
Remove stray chars from source files
 2014-10-13 11:34:52 +09:00
Andrew Gaul
a1ca8b7124 Return EIO on failed read 
Previously S3fsMultiCurl::MultiRead did not report read errors since
it did not treat failed callback setup as a fatal operation error.
Failed callback setups usually result from exceeding the number of
allowed retries.  Previously cp did not report an error during a
network outage but now does:

$ cp ~/s3-path/s3-file .
cp: error reading ‘/home/gaul/s3-path/s3-file’: Input/output error
cp: failed to extend ‘./s3-file’: Input/output error
 2014-10-03 21:30:11 -07:00
Andriy Senkovych
6633366218 Fix spelling errors 2014-10-01 13:42:39 +03:00
Andriy Senkovych
22ea65f02c Remove stray chars from source files 2014-10-01 13:20:29 +03:00
Andrew Gaul
3d69ee0c30 Emit response on failed CheckBucket requests 
This allows callers to diagnose errors like InvalidAccessKeyId and
RequestTimeTooSkewed.
 2014-09-28 16:12:53 -07:00
Andrew Gaul
c88a5f38be Disable CURLOPT_FAILONERROR for CheckBucket 
curl will not consume the body of a response when CURLOPT_FAILONERROR
is set.  This prevents logging of responses for failed requests.
 2014-09-28 16:12:43 -07:00
Takeshi Nakatani
7a7c7572ea Cleaned up codes for next packaging. 2014-09-07 15:08:27 +00:00
Takeshi Nakatani
ae4ae88b6d Merge pull request #53 from s3fs-fuse/sse-c 
Support for SSE-C #39
 2014-09-07 22:31:33 +09:00
Takeshi Nakatani
f0c33f8ef2 clean codes 2014-08-27 00:59:49 +00:00
Takeshi Nakatani
20b1c207be fixed issue #39 2014-08-26 17:11:10 +00:00
YAMASAKI Masahide
f1ca5d0340 :set nobomb 2014-08-25 19:18:34 +09:00
Takeshi Nakatani
cbec8da9a3 fixed a bug issue #49 2014-08-14 15:58:06 +00:00
Takeshi Nakatani
7a55eab399 Support for SSE-C, issue #39 2014-07-19 19:02:55 +00:00
Takeshi Nakatani
c1a6d76fc3 Fixed a bug issue #40 2014-06-28 17:36:35 +00:00
Takeshi Nakatani
ba34ba181a Fixed a bug issue #41 2014-06-28 17:24:25 +00:00
Andrew Dunn
601482eff5 Added support for path API request style. 
Rather than using virtual host style requests, path style requests can be used
instead.

i.e. rather than bucketname.s3.amazon.com/... the s3fs will be able to request
from s3.amazon.com/bucketname/...

This is useful for S3 compatible APIs which don't support the virtual host style
request.

It is enabled with the new option, `use_path_style_request`.

Example:

    /usr/bin/s3fs data ~/netcdf -o url="https://swift.rc.nectar.org.au:8888/" -o use_path_request_style -o allow_other -o uid=500 -o gid=500
 2014-06-04 00:03:49 +10:00
Takeshi Nakatani
61020370d5 Changed codes for CR code in passwd file(googlecode issue#417). 2014-06-02 16:12:55 +00:00
Takeshi Nakatani
160196798b Changed initializing logic for nss lib/openssl lib/s3fs own. 2014-06-01 03:54:02 +00:00
Takeshi Nakatani
cd27f0aa54 Supported another crypt libraries as GnuTLS and NSS, and added configure options 2014-05-06 14:23:05 +00:00
Takeshi Nakatani
8bba566774 Retry to send request at CURLE_SSL_CONNECT_ERROR 2014-04-04 16:23:56 +00:00
Takeshi Nakatani
157612e7e7 Merge pull request #28 from andrewgaul/fdcache-init-signedness-warning 
Address signedness warning in FdCache::Init
 2014-04-05 01:02:13 +09:00
Andrew Gaul
d475e22774 Address signedness warning in FdCache::Init 
This commit allows GCC 4.8 to compile s3fs withing warnings.
 2014-04-01 23:50:08 -07:00
Takeshi Nakatani
4762e53b5d Added multipart_size option for #16 2014-03-30 07:53:41 +00:00
Takeshi Nakatani
d7563309a2 Fixed a bug #18(losing check retry error) 2014-03-30 06:40:49 +00:00
Mason Malone
74fb29d9fb Fix typos in docs for max_stat_cache_size (the default is actually 1000) 2014-03-06 12:44:10 -05:00
Takeshi Nakatani
16487c4e26 Fixed a bug: issue #14, s3fs -u should return 0 if there are no lost multiparts 2014-03-03 17:00:20 +00:00
Takeshi Nakatani
52d56d15e4 Fixed a bug(googlecode issue 405), enable_content_md5 Input/output error 2014-03-03 16:19:08 +00:00
Takeshi Nakatani
5fd33405af Fixed a bug - issue#5 2014-01-16 16:04:41 +00:00
Pascal Deschênes
db3bd7c366 Fix compilation error on MacOSX with missing const 2014-01-04 21:24:27 -05:00
worpet
33ec3739e2 Update s3fs_util.cpp 
Fixed local time was incorrectly being applied to IAM and Last-Modified dates.
 2013-12-30 09:35:39 -06:00
root
d5626fe595 Changed url encoding of space character to use %20 instead of the plus (+) sign. 2013-12-17 15:12:03 +01:00
ggtakec@gmail.com
74db6748dd Changes codes 
1) Changed condition for retrying multipart error
   Changes condition that 404 is not retrying, other case is retrying,
   when multipart request failed.

2) file type wrong
   fdcache.h file type is wrong, so that fixed it.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@498 df820570-a93a-0410-bd06-b72b767a4274
 2013-11-19 01:48:53 +00:00
ggtakec@gmail.com
8acbaf7199 Fixed a bug. 
* Fixed a bug
  Fixes a bug that is dead loop when s3fs listed in a directory with
  directory objects which was not object(no information).
  This bug made by r493, and reported by issue 389.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@497 df820570-a93a-0410-bd06-b72b767a4274
 2013-11-18 02:29:41 +00:00
ggtakec@gmail.com
40b9f0a408 Changes codes 
1) Changed buffer size for file size
   Changes a internal buffer size from size_t to offt_t.
   It is a bug for 32 bit OS enviroment.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@496 df820570-a93a-0410-bd06-b72b767a4274
 2013-11-17 08:50:41 +00:00
ggtakec@gmail.com
882f13020e Fixed bugs(overflow) 
1) Overflow
   About over 4GB file, when st_size which is member stat structure,
   the value is overflow.
   Fixed this bug and fixed like this bug in all sources. 

2) Changed retrying request
   If s3fs gets 500 HTTP status for multipart request, s3fs retry
   to send same request.





git-svn-id: http://s3fs.googlecode.com/svn/trunk@495 df820570-a93a-0410-bd06-b72b767a4274
 2013-11-13 16:26:50 +00:00
ggtakec@gmail.com
c785be917f Changed a code 
* Cut a #ifdef
  Cuts a code for clearing r493.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@494 df820570-a93a-0410-bd06-b72b767a4274
 2013-11-11 15:03:04 +00:00
ggtakec@gmail.com
09fc2593e3 Fixed bugs and Changed utility mode 
1) Fixed bugs
 * Rename objects
   Fixes s3fs specifies wrong part number of multipart rename.
   And s3fs adds x-amz-acl and x-amz-server-side-encryption
   header when rename objects.

2) Changed retry logic for multipart uploading(and renaming)
   Sometimes, s3fs gets 400 HTTP response for one of part from
   S3 when s3fs uploads a large object by multipart.
   New logic retries uploading failed part until "retries"
   option count.

3) Added action on utility mode.
   s3fs have had utility mode for displaying the result of REST
   listing multipart uploading.
   Changed this row result(xml) to list, after that, s3fs starts
   conversation for removing it.
   Then you can remove the object which is failed uploading by
   multipart, and do not need to pay for that ever.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@493 df820570-a93a-0410-bd06-b72b767a4274
 2013-11-11 13:45:35 +00:00
ggtakec@gmail.com
1bae39e21f Fixed a bug. 
* Fixed a bug
   Fixes a bug that the retrying multipart post request is not 
   complete, then it is fixed.
   This is reported by Issue 371#32.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@492 df820570-a93a-0410-bd06-b72b767a4274
 2013-10-09 01:44:56 +00:00
ggtakec@gmail.com
33431dec46 Fixed a bug. 
* Fixed a bug
   Fixes a bug that is mis-coding about retrying Multipart Post.
   This is reported by Issue 371#28.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@491 df820570-a93a-0410-bd06-b72b767a4274
 2013-10-08 08:19:10 +00:00
ggtakec@gmail.com
99db6d13af Changes codes 
1) Supported IAM role
   Supports IAM role by option, that is instead of AccessKeyID/
   SecretAccessKey.
   Adds new option "iam_role" which is specified as IAM role
   name.(like s3fs-c)



git-svn-id: http://s3fs.googlecode.com/svn/trunk@490 df820570-a93a-0410-bd06-b72b767a4274
 2013-10-06 13:45:32 +00:00
ggtakec@gmail.com
44468ba00f Fixed a bug(public_bucket) 
1) Fixed a bug(about public_bucket)
   Fixes a bug that public_bucket option does not work.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@489 df820570-a93a-0410-bd06-b72b767a4274
 2013-09-29 13:24:30 +00:00