octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-15 16:57:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,427 Commits 1 Branch 23 Tags 13 MiB
23945a0130
Commit Graph

363 Commits

Author SHA1 Message Date
Takeshi Nakatani
d8185a25aa
Merge pull request #946 from gaul/async/completed-tids 
Simplify async request completion code
 2019-02-03 14:08:49 +09:00
Takeshi Nakatani
53337a0a28
Merge pull request #944 from gaul/utility-mode 
Repair utility mode
 2019-02-03 13:59:56 +09:00
Takeshi Nakatani
ae51556d04
Merge pull request #943 from gaul/hard-link 
Return not supported when hard linking
 2019-02-03 12:45:21 +09:00
Takeshi Nakatani
055ecf6ea7
Merge pull request #940 from gaul/parallel-multipart-copy 
Copy parts in parallel
 2019-02-03 11:44:15 +09:00
Andrew Gaul
8c527c3616 Simplify async request completion code 
Workers now notify the master thread when they complete, unifying the
Linux and macOS code paths.  This also avoids excessive
pthread_tryjoin_np calls.  Follows on to
88cd8feb05.
 2019-01-31 22:55:23 -08:00
Andrew Gaul
54a074647e Repair utility mode 
This deinitialized S3fsCurl twice and incorrectly calculated V4
signatures.
 2019-01-31 18:45:39 -08:00
Andrew Gaul
c5ebf5d328 Copy parts in parallel 
S3 can copy multipart much faster than single part due to IO
parallelization.  Renaming a 4 GB file reduces from 72 to 20 seconds
with bigger gains with larger files.
 2019-01-31 10:21:39 -08:00
Andrew Gaul
43c6ef560e Return not supported when hard linking 
This is more correct than permission denied.
 2019-01-30 16:43:04 -08:00
Andrew Gaul
35d55ee513 Remove unneeded void parameter 
This is implicit in C++.  Found and fixed via clang-tidy.
 2019-01-28 23:22:27 -08:00
Takeshi Nakatani
fd4d23f8f7
Merge pull request #926 from kzidane/master 
Accept paths with : in them
 2019-01-27 15:23:24 +09:00
Takeshi Nakatani
4820f0a42b
Merge pull request #925 from gaul/clang-tidy/delete-null 
Remove redundant null checks before delete
 2019-01-27 15:15:31 +09:00
Kareem Zidane
cd794a6985 Accept paths with : in them 2019-01-23 14:56:25 -05:00
Andrew Gaul
84b421d6ef Prefer empty over size checks 
Found and fixed via clang-tidy.
 2019-01-23 11:30:28 -08:00
Andrew Gaul
8316da5bbe Remove redundant null checks before delete 
Found by clang-tidy.
 2019-01-23 11:25:25 -08:00
Takeshi Nakatani
caaf4cac55 Reverted #912(Automatic region change) and added message 2019-01-23 13:23:03 +00:00
Andrew Gaul
1fc25e8c3f Remove redundant string initializations 
Found and fixed via clang-tidy.
 2019-01-22 23:16:37 -08:00
Andrew Gaul
46d79c5bc2 Issue readdir HEAD requests without batching 
Previously s3fs would issue a batch of HEAD requests and wait for all
to succeed before issuing the next batch.  Now it issues the first
batch and only waits for a single call to succeed before issuing the
next call.  This can improve performance when one call lags due to
network errors.  I measured 25% improvement with the same level of
parallelism.  This commit also reparents parallelism knobs for
consistency.  Follows on to 88cd8feb05.
Fixes #223.
 2019-01-20 18:07:22 -08:00
Takeshi Nakatani
2887f8916b Automatic region change made possible other than us-east-1(default) 2019-01-20 10:51:49 +00:00
Takeshi Nakatani
05014c49c8
Merge pull request #906 from gaul/doc/https 
Prefer HTTPS links where possible
 2019-01-20 15:10:51 +09:00
Takeshi Nakatani
6aa40b2747
Merge pull request #903 from gaul/clang-tidy/find_char 
Prefer find(char) over find(const char *)
 2019-01-20 14:05:06 +09:00
Andrew Gaul
aa9bd1fa3c Prefer HTTPS links where possible 
Fix a few stale links as well.
 2019-01-18 11:09:08 -08:00
Andrew Gaul
e29548178b Prefer find(char) over find(const char *) 
The former can be faster.  Found via clang-tidy.
 2019-01-17 20:24:24 -08:00
Andrew Gaul
6401b4ae92 Flush file before renaming 
Previously s3fs could copy the zero-byte stub object without including
any pending writes.  Fixes #145.
 2019-01-17 12:05:10 -08:00
Takeshi Nakatani
c7def35b54
Merge pull request #895 from gaul/bucket-name-dot 
Emit more friendly error for buckets with dots
 2019-01-16 18:39:47 +09:00
Andrew Gaul
c512516e14 Emit more friendly error for buckets with dots 
These fail SSL certificate checks due to the *.s3.amazon.com wildcard.
Fixes #284.
 2019-01-14 18:47:36 -08:00
Andrew Gaul
2c43b1e12b Store and retrieve file change time 
This introduces a new header with the change time; existing objects
will report modification time.  Fixes #771.
 2019-01-14 10:05:11 -08:00
Takeshi Nakatani
fd6b37d3da
Merge pull request #877 from gaul/aws/credentials 
Check arguments and environment before .aws/creds
 2019-01-06 17:03:11 +09:00
Takeshi Nakatani
56e24de0d4
Merge pull request #870 from gaul/typos 
Correct typos in command-line parsing
 2019-01-06 16:21:46 +09:00
Andrew Gaul
3d225163f8 Correct typos in command-line parsing 
Also fix stray macOS references.  References #869.
 2019-01-02 14:40:57 -08:00
Andrew Gaul
0569cec3ea Check arguments and environment before .aws/creds 
Fixes #857.  Fixes #864.
 2018-12-20 15:51:32 -08:00
Andrew Gaul
a2f8ac535e Address cppcheck 1.86 errors 
Lifetime, shadowing, and unused variables.  Found via the Travis macOS
builder.
 2018-12-20 14:56:31 -08:00
Takeshi Nakatani
5fba542a29
Merge pull request #852 from gaul/aws-credentials-file 
Allow credentials from ${HOME}/.aws/credentials
 2018-11-11 11:37:38 +09:00
Andrew Gaul
9e530c86ae Allow credentials from ${HOME}/.aws/credentials 
This matches the configuration from popular tools like AWS CLI and
allows multiple profile names via -o profile=name.  The existing
credential mechanisms continue to work.  Fixes #822.
 2018-11-04 17:47:07 -08:00
Andrew Gaul
664f910083 Correctly compare list_object_max_keys 
Previously this did not allow an argument.  Fixes #843.
References #793.
 2018-11-04 10:49:35 -08:00
Jonatan Nilsson
d9c106cfde Add support for storage class ONEZONE_IA. 2018-09-30 14:27:31 +02:00
Takeshi Nakatani
c5af62b023
Merge pull request #820 from gaul/big-writes 
Enable big writes if capable
 2018-09-17 17:24:06 +09:00
Andrew Gaul
a85183d42c Enable big writes if capable 
Fixes #813.
 2018-09-10 22:21:51 -07:00
Mattia Rizzolo
c376efdd28
Fix typo s/mutliple/multiple/ 
Signed-off-by: Mattia Rizzolo <mattia@mapreri.org>
 2018-08-19 00:14:57 +02:00
Takeshi Nakatani
7e0c53dfe9 Added list_object_max_keys option based on #783 PR 2018-07-08 03:49:10 +00:00
Neeraj Kumar Kashyap
5a481e6a01 Option for IBM IAM auth endpoint added return 2018-06-04 16:44:14 +05:30
Neeraj Kumar Kashyap
d8e12839af Option for IBM IAM auth endpoint 2018-05-31 16:02:48 +05:30
Or Ozeri
d4e86a17d1 Revert "enable FUSE read_sync by default" 
This reverts commit 86b0921ac4.

Conflicts:
	src/s3fs.cpp
 2018-05-28 13:49:54 +03:00
Takeshi Nakatani
ae9d8eb734 Fixed memory leak 2018-05-27 10:48:03 +00:00
Takeshi Nakatani
57b2a60172
Merge pull request #764 from orozery/remove_false_multihead_warnings 
Remove false multihead warnings
 2018-05-23 22:38:35 +09:00
Or Ozeri
0bd875eb9e remove false readdir_multi_head warnings 2018-05-22 17:10:50 +03:00
Or Ozeri
86b0921ac4 enable FUSE read_sync by default 2018-05-06 16:10:36 +03:00
Neeraj Kumar Kashyap
25375a6b48 Validate the URL fixed inefficient usage of find 2018-05-04 11:24:32 +05:30
Neeraj Kumar Kashyap
ca87df7d44 Validate the URL format for http/https 2018-05-03 22:08:28 +05:30
陈方舟
04493de767
fix xpath selector in bucket listing 
the original implementation in get_base_exp() depends on the order of xml return from the server.
patriotically, when listing a directory with sub directory(s), the xml document response contains more than 2 <Prefix> nodes(some of them are in <CommonPrefixes> node).
the source code arbitrarily select the first one in the documents (nodes->nodeTab[0]->xmlChildrenNode).
some s3 compatible service return the list-bucket result in different result, leading the s3fs to a wrong behavior
 2018-04-23 15:11:29 +08:00
Tristen Horton
0863672e27
add a more helpful error message for when there are multiple entries for the same bucket in the passwd file 2018-03-13 14:37:34 -07:00
Takeshi Nakatani
0f503ced25
Merge pull request #729 from dmgk/master 
FreeBSD build fixes
 2018-03-04 16:36:31 +09:00
Takeshi Nakatani
987a166bf4
Merge pull request #726 from orozery/instance_name_logging 
add an instance_name option for logging
 2018-03-04 15:41:12 +09:00
Dmitri Goutnik
c9f4312588 FreeBSD build fixes 2018-03-02 15:58:52 -05:00
Or Ozeri
8b657eee41 add disk space reservation 2018-02-28 19:20:23 +02:00
Takeshi Nakatani
c494e54320 Fixed cppcheck error on osx 2018-02-28 12:06:06 +00:00
Or Ozeri
b52b6f3fc5 add an instance_name option for logging 2018-02-28 09:51:35 +02:00
Takeshi Nakatani
a45ff6cdaa Fixed cppcheck error and clean ^M code 2018-02-25 13:08:41 +00:00
Or Ozeri
bd4bc0e7f1 add support for IBM IAM authentication 2017-11-23 12:01:52 +02:00
Takeshi Nakatani
97fc845a6a Changed functions about reading passwd file. 2017-11-19 11:49:11 +00:00
Takeshi Nakatani
7d9ac0163b Changed functions about reading passwd file. 2017-11-19 11:38:12 +00:00
Andrew Gaul
0418e53b3c Reduce use of preprocessor 
This provides type-safety and avoids token expansion side effects.
 2017-11-18 22:40:06 -08:00
Richard Caunt
366f0705a0 ECS credentials bug fixes 2017-11-06 21:45:58 +00:00
Richard Caunt
ccea87ca68
Added check for is_ecs during get_access_keys 
We need to presume that if `is_ecs` we are deferring access key loading
 2017-11-06 11:02:27 +00:00
Richard Caunt
662f65c3c8 Add support for ECS metadata endpoint 2017-11-05 19:24:02 +00:00
Takeshi Nakatani
ab89b4cd4a Merge pull request #659 from ggtakec/master 
Do not fail updating directory when removing old-style object(ref #658)
 2017-10-15 21:56:35 +09:00
Takeshi Nakatani
1eba27a50a Refixed s3fs_init message(ref #652) 2017-10-15 06:45:19 +00:00
Takeshi Nakatani
41206fa0e2 Do not fail updating directory when removing old-style object(ref #658) 2017-10-15 05:03:44 +00:00
jurafxp
ae91b6f673 Fix s3fs_init message 2017-10-01 00:08:00 +02:00
Takeshi Nakatani
00bc9142c4 Fixed potential atomic violation in S3fsCurl::AddUserAgent - #633 2017-09-17 09:16:05 +00:00
Tony Levi
cc94e1da26 Fix chown_nocopy losing existing uid/gid if unspecified 2017-05-25 16:53:08 +09:30
Takeshi Nakatani
337da59368 Not fallback to HTTP - #596 2017-05-13 15:47:39 +00:00
Takeshi Nakatani
d2ae14d8b7 Check bucket at public bucket and add nocopyapi option automatically 2017-05-13 07:48:50 +00:00
Takeshi Nakatani
7115835834 Check bucket at public bucket and add nocopyapi option automatically 2017-05-13 07:35:55 +00:00
Takeshi Nakatani
24df69f688 Backward compatible for changing default transport to HTTPS 2017-05-13 06:47:51 +00:00
Andrew Gaul
0e0ae38f6d Default transport to HTTPS 
This protects private data when used over the public Internet.  Users
can opt-in to unencrypted HTTP if they need additional performance on
a local network.  Fixes #282.
 2017-05-07 10:59:54 -07:00
Takeshi Nakatani
9d10a5aa70 Changed copyright year format for debian pkg 2017-05-07 11:24:17 +00:00
Takeshi Nakatani
1fe0334c08 Enhanced bucket/path parameter check 2017-05-05 19:55:24 +00:00
Takeshi Nakatani
3ac39d61f8 Added notsup_compat_dir option 2017-05-05 17:28:29 +00:00
Takeshi Nakatani
864e20e1f2 Refactored the get_object_attribute function 2017-05-05 10:02:21 +00:00
Takeshi Nakatani
51b3183cba Refactored the check_object_access function 2017-05-05 09:51:30 +00:00
Takeshi Nakatani
758b92e823 Added option for complementing lack of stat mode 2017-05-04 03:41:24 +00:00
Takeshi Nakatani
efba9bcbc1 Merge pull request #553 from orozery/custom_cipher_suite 
add TLS cipher suites customization
 2017-04-16 19:09:27 +09:00
Or Ozeri
75b59a7c16 switch S3fsMultiCurl to use foreground threads 2017-04-04 15:32:53 +03:00
Takeshi Nakatani
fef3fbc225 Added check_cache_dir_exist option(refixed #347) - #538 2017-04-02 08:10:16 +00:00
Takeshi Nakatani
980ba398bc Fixed double initialization of SSL library - #524 2017-03-19 17:11:18 +00:00
Takeshi Nakatani
523043a2aa Changed base cached time of stat_cache_expire option - #523 2017-03-19 15:19:04 +00:00
Andrew Gaul
03217baa99 Address cppcheck 1.77 warnings 2017-03-06 12:41:08 -08:00
Andrew Gaul
d375bca0d0 Correct typos 2016-11-19 15:57:41 -08:00
Takeshi Nakatani
7fabd18b1f Trim symbolic link original path in file. 2016-10-11 10:22:30 +00:00
Takeshi Nakatani
5db369d67e Trim symbolic link original path in file. 2016-10-11 10:17:46 +00:00
Takeshi Nakatani
dba32fdf78 Trim symbolic link original path in file. 2016-10-11 10:06:21 +00:00
Takeshi Nakatani
9fd1368611 OS-specific correspondence of the extended attribute header 2016-10-02 07:23:19 +00:00
Takeshi Nakatani
65d52506c4 Added use_xattr option for #467 and #460 2016-09-19 04:28:01 +00:00
Takeshi Nakatani
1a96f40a10 Fixed a bug about could not copy file mode from org file 2016-09-11 13:09:23 +00:00
Takeshi Nakatani
1ddc14d59d Changed for accepting mount options compatible with mtab - #449 2016-07-24 08:17:58 +00:00
Peter Watkins
a19206cf0f Accept mount arguments compatible with mtab 
Using "mount -a" fails for already-mounted s3fs directories,
because s3fs mount arguments don't match the form in /etc/mtab.
Calling "mount -a" should quietly succeed when a directory is
already mounted.

To fix this, accept mount commands of the form:

s3fs s3fs /srv/object-store -o bucket=mybucket

or in /etc/fstab form:

s3fs  /srv/object-store  fuse.s3fs  bucket=mybucket 0 0

This matches the form in /etc/mtab and allows "mount -a" to
work properly.
 2016-07-13 17:23:33 -04:00
Andrew Gaul
320b8e1171 Include location constraint when creating bucket 
This allows creating buckets in non-default regions.  Also improve
do_create_bucket error handling.
 2016-06-13 10:35:37 -07:00
Nathaniel W. Turner
584ea488bf Use role name instead of profile name when iam_role=auto 
When using an instance with an IAM Role, transient credentials can be
found in http://169.254.169.254/latest/meta-data/ at
iam/security-credentials/role-name and s3fs tries to do this. However,
it is using the profile-name where role-name is needed. In many cases
the role and profile name are the same, but they are not always.

The simplest way to find the role name appears to be to GET
http://169.254.169.254/latest/meta-data/iam/security-credentials/
itself, which returns a listing of the role names for which temporary
credentials exist. (I think there will probably only be one, but we
probably want to split on newlines and take the first one here in case
that assumption is not valid). This is the approach the AWS SDK appears
to use (based on WireShark analysis).

Bug: https://github.com/s3fs-fuse/s3fs-fuse/issues/421
Signed-off-by: Nathaniel W. Turner <nate@houseofnate.net>
 2016-05-24 13:34:19 -04:00
Nathaniel W. Turner
594c9ca7d2 Skip early credential checks when iam_role=auto 
If user specifies iam_role=auto (or just iam_role), credentials will not
be loaded during early phase, so skip credential checks there.

Signed-off-by: Nathaniel W. Turner <nate@houseofnate.net>
 2016-05-20 12:49:02 -04:00
Takeshi Nakatani
50f1ad51c8 loading IAM role name automatically(iam_role option) - #387 2016-05-06 04:37:32 +00:00