Commit Graph

552 Commits

Author SHA1 Message Date
Takeshi Nakatani
5c4a0a862a Fixed multipart copy and its retry handler 2019-08-11 16:21:26 +00:00
Takeshi Nakatani
f74c7407db
Merge pull request #1118 from gaul/atoi
Prefer s3fs_strtoofft over atoi and strtol
2019-08-06 21:23:37 +09:00
Andrew Gaul
ccf3e7bfa2 Prefer s3fs_strtoofft over atoi and strtol
The former propagates errors consistently.
2019-08-03 16:13:48 -07:00
Andrew Gaul
43d1439420 Consume return value from FdEntity::Flush
Found via Coverity.
2019-08-02 10:01:06 -07:00
Andrew Gaul
fb937635f5 Eagerly initialize sysconf variables
Previously s3fs had races updating these shared variables.  Found via
ThreadSanitizer.
2019-07-17 09:08:13 -07:00
Andrew Gaul
ffac4c8417 Avoid narrowing time_t and off_t args in logging
This displays correct results on 32-bit platforms.
2019-07-14 17:02:36 -07:00
Takeshi Nakatani
5bbcd3b981
Merge pull request #1089 from gaul/null-dereference
Avoid null dereference
2019-07-15 04:17:10 +09:00
Andrew Gaul
eb597289cb Avoid null dereference
Found via clang-tidy.
2019-07-12 18:40:24 -07:00
Andrew Gaul
6fd42d9fe4 Prefer modern C headers
Found and fixed via clang-tidy.
2019-07-12 03:50:59 -07:00
Takeshi Nakatani
7a65a414c3
Merge pull request #1080 from gaul/xml/simplification
Add simple XML parsing wrapper
2019-07-07 16:44:18 +09:00
Takeshi Nakatani
6bd1a7eac0
Merge pull request #1074 from gaul/clang-tidy
Configure clang-tidy target
2019-07-07 15:28:08 +09:00
Andrew Gaul
b8ff6a647e Add simple XML parsing wrapper
Also simplify check_region_error.
2019-07-05 17:39:09 -07:00
Andrew Gaul
4f23f38583 Individually test multiple s3fs flags
Remove unneeded comments; single part limits ensure that the tests
exercise multipart code paths even with smaller files.
References #971.
2019-07-03 21:09:40 -07:00
Andrew Gaul
bbfa91141a Configure clang-tidy target
Also fix nits.
2019-07-03 14:04:11 -07:00
Andrew Gaul
c596441f58 Flush file when opening second fd
Previously when s3fs had dirty local data and an application opened a
second fd it would remove the stat cache entry, query S3 for the size
which was still zero, then reinitialize FdEntry with this incorrect
size.  Instead flush local data to S3 so this query works.  It is
possible that a more involved patch could do this with a less
heavyweight approach but this requires changing open.  This does not
completely fix git clone but allows it to proceed further.  Also make
some cache methods const-correct.  References #839.
2019-07-02 01:12:09 -07:00
Andrew Gaul
f1ad626b46 Log flags as hexadecimal 2019-06-24 16:13:07 -07:00
Andrew Gaul
15b7450713 Convert file offsets from size_t to off_t
The latter is 64-bits on 32-bit platforms when specifying
-D_FILE_OFFSET_BITS=64.  This allows early Raspberry Pis to use files
larger than 2 GB.  It also cleans up some ugly casting.  Fixes #620.
Fixes #656.
2019-06-15 17:05:37 -07:00
Andrew Gaul
059ab1f0f4 Update ctime during nocopyapi operations
Follows on to 2c43b1e12b.  Fixes #971.
2019-04-30 18:59:51 +09:00
Michal Lula
0d4847596e fix issue with aws session token provided inside ~/.aws/credentials file 2019-04-25 14:29:35 +02:00
Andrew Gaul
9e4f9d4bdc Remove uses of std::map::at
This provides compatibility with C++03 and removes duplicate lookups.
2019-04-18 16:06:59 +09:00
Michal Lula
f9cd43b684 add session token support 2019-04-16 16:53:05 +02:00
Takeshi Nakatani
cd280d8702
Merge pull request #1001 from gaul/missing-braces
Add missing braces
2019-04-08 23:55:02 +09:00
Andrew Gaul
042332bcec Add missing braces
Found via clang-tidy.  Also fix errant indentation.
2019-04-07 23:12:27 +09:00
Takeshi Nakatani
071cd0f849 Modified to discard _netdev option etc 2019-04-07 11:51:55 +00:00
Andrew Gaul
6f6a67807b Prefer new over malloc
The former cannot return NULL and will allow use of scoped pointers.
2019-04-07 16:54:24 +09:00
Takeshi Nakatani
ea517c80a4 Fixed a bug about overwrite metadata at updating directory stats 2019-03-29 15:30:30 +00:00
Andrew Gaul
058706014b Simplify string comparison
Found via newer cppcheck.
2019-03-14 22:19:33 +09:00
Takeshi Nakatani
cbf072bc55
Merge pull request #967 from gaul/reference-count
Increase FdEntity reference count when returning
2019-03-10 14:27:10 +09:00
Andrew Gaul
08b132ddb9 Remove from file from stat cache during rename
This addresses failures with test_rename_before_close when using
nocopyapi.  Note that test_update_time still fails.
2019-03-09 17:53:35 +09:00
Andrew Gaul
f53503438c Increase FdEntity reference count when returning
Previously s3fs had a race condition where one client could delete
FdEntity that another client was using.  Add a simple concurrent test
which previously failed but now succeeds.  Fixes #964.
2019-03-09 15:55:04 +09:00
Kristján Valur Jónsson
3c97c1b251 merged main 2019-02-19 10:53:00 +00:00
Kristján Valur Jónsson
84c671a81a fix indentation 2019-02-19 10:37:43 +00:00
Kristján Valur Jónsson
f336bdebcc add command line flag and documentation 2019-02-19 10:32:37 +00:00
Kristján Valur Jónsson
4f42f4ab0c Enable s3fs encoding and decoding in the fuse interface 2019-02-18 13:36:48 +00:00
Takeshi Nakatani
eeb839242b Fixed code for latest cppcheck error on OSX 2019-02-17 13:59:11 +00:00
Takeshi Nakatani
231fd001d9 Added a non-interactive option to utility mode
Rebase for resolving conflicts

Deleted interactive mode on utility mode

Fixed minor mistakes
2019-02-10 17:36:18 +00:00
Takeshi Nakatani
314dc5a398
Merge pull request #955 from gaul/clang-tidy/misc
Address miscellaneous clang-tidy warnings
2019-02-06 21:21:54 +09:00
Andrew Gaul
e87e40b3b4 Address miscellaneous clang-tidy warnings
Also add configuration with suppressions.
2019-02-04 22:58:40 -08:00
Andrew Gaul
4d0bef1e90 Clear containers instead of individual erases
This has O(n) runtime instead of O(n log n).
2019-02-02 23:58:43 -08:00
Takeshi Nakatani
d8185a25aa
Merge pull request #946 from gaul/async/completed-tids
Simplify async request completion code
2019-02-03 14:08:49 +09:00
Takeshi Nakatani
53337a0a28
Merge pull request #944 from gaul/utility-mode
Repair utility mode
2019-02-03 13:59:56 +09:00
Takeshi Nakatani
ae51556d04
Merge pull request #943 from gaul/hard-link
Return not supported when hard linking
2019-02-03 12:45:21 +09:00
Takeshi Nakatani
055ecf6ea7
Merge pull request #940 from gaul/parallel-multipart-copy
Copy parts in parallel
2019-02-03 11:44:15 +09:00
Andrew Gaul
8c527c3616 Simplify async request completion code
Workers now notify the master thread when they complete, unifying the
Linux and macOS code paths.  This also avoids excessive
pthread_tryjoin_np calls.  Follows on to
88cd8feb05.
2019-01-31 22:55:23 -08:00
Andrew Gaul
54a074647e Repair utility mode
This deinitialized S3fsCurl twice and incorrectly calculated V4
signatures.
2019-01-31 18:45:39 -08:00
Andrew Gaul
c5ebf5d328 Copy parts in parallel
S3 can copy multipart much faster than single part due to IO
parallelization.  Renaming a 4 GB file reduces from 72 to 20 seconds
with bigger gains with larger files.
2019-01-31 10:21:39 -08:00
Andrew Gaul
43c6ef560e Return not supported when hard linking
This is more correct than permission denied.
2019-01-30 16:43:04 -08:00
Andrew Gaul
35d55ee513 Remove unneeded void parameter
This is implicit in C++.  Found and fixed via clang-tidy.
2019-01-28 23:22:27 -08:00
Takeshi Nakatani
fd4d23f8f7
Merge pull request #926 from kzidane/master
Accept paths with : in them
2019-01-27 15:23:24 +09:00
Takeshi Nakatani
4820f0a42b
Merge pull request #925 from gaul/clang-tidy/delete-null
Remove redundant null checks before delete
2019-01-27 15:15:31 +09:00
Kareem Zidane
cd794a6985 Accept paths with : in them 2019-01-23 14:56:25 -05:00
Andrew Gaul
84b421d6ef Prefer empty over size checks
Found and fixed via clang-tidy.
2019-01-23 11:30:28 -08:00
Andrew Gaul
8316da5bbe Remove redundant null checks before delete
Found by clang-tidy.
2019-01-23 11:25:25 -08:00
Takeshi Nakatani
caaf4cac55 Reverted #912(Automatic region change) and added message 2019-01-23 13:23:03 +00:00
Andrew Gaul
1fc25e8c3f Remove redundant string initializations
Found and fixed via clang-tidy.
2019-01-22 23:16:37 -08:00
Andrew Gaul
46d79c5bc2 Issue readdir HEAD requests without batching
Previously s3fs would issue a batch of HEAD requests and wait for all
to succeed before issuing the next batch.  Now it issues the first
batch and only waits for a single call to succeed before issuing the
next call.  This can improve performance when one call lags due to
network errors.  I measured 25% improvement with the same level of
parallelism.  This commit also reparents parallelism knobs for
consistency.  Follows on to 88cd8feb05.
Fixes #223.
2019-01-20 18:07:22 -08:00
Takeshi Nakatani
2887f8916b Automatic region change made possible other than us-east-1(default) 2019-01-20 10:51:49 +00:00
Takeshi Nakatani
05014c49c8
Merge pull request #906 from gaul/doc/https
Prefer HTTPS links where possible
2019-01-20 15:10:51 +09:00
Takeshi Nakatani
6aa40b2747
Merge pull request #903 from gaul/clang-tidy/find_char
Prefer find(char) over find(const char *)
2019-01-20 14:05:06 +09:00
Andrew Gaul
aa9bd1fa3c Prefer HTTPS links where possible
Fix a few stale links as well.
2019-01-18 11:09:08 -08:00
Andrew Gaul
e29548178b Prefer find(char) over find(const char *)
The former can be faster.  Found via clang-tidy.
2019-01-17 20:24:24 -08:00
Andrew Gaul
6401b4ae92 Flush file before renaming
Previously s3fs could copy the zero-byte stub object without including
any pending writes.  Fixes #145.
2019-01-17 12:05:10 -08:00
Takeshi Nakatani
c7def35b54
Merge pull request #895 from gaul/bucket-name-dot
Emit more friendly error for buckets with dots
2019-01-16 18:39:47 +09:00
Andrew Gaul
c512516e14 Emit more friendly error for buckets with dots
These fail SSL certificate checks due to the *.s3.amazon.com wildcard.
Fixes #284.
2019-01-14 18:47:36 -08:00
Andrew Gaul
2c43b1e12b Store and retrieve file change time
This introduces a new header with the change time; existing objects
will report modification time.  Fixes #771.
2019-01-14 10:05:11 -08:00
Takeshi Nakatani
fd6b37d3da
Merge pull request #877 from gaul/aws/credentials
Check arguments and environment before .aws/creds
2019-01-06 17:03:11 +09:00
Takeshi Nakatani
56e24de0d4
Merge pull request #870 from gaul/typos
Correct typos in command-line parsing
2019-01-06 16:21:46 +09:00
Andrew Gaul
3d225163f8 Correct typos in command-line parsing
Also fix stray macOS references.  References #869.
2019-01-02 14:40:57 -08:00
Andrew Gaul
0569cec3ea Check arguments and environment before .aws/creds
Fixes #857.  Fixes #864.
2018-12-20 15:51:32 -08:00
Andrew Gaul
a2f8ac535e Address cppcheck 1.86 errors
Lifetime, shadowing, and unused variables.  Found via the Travis macOS
builder.
2018-12-20 14:56:31 -08:00
Takeshi Nakatani
5fba542a29
Merge pull request #852 from gaul/aws-credentials-file
Allow credentials from ${HOME}/.aws/credentials
2018-11-11 11:37:38 +09:00
Andrew Gaul
9e530c86ae Allow credentials from ${HOME}/.aws/credentials
This matches the configuration from popular tools like AWS CLI and
allows multiple profile names via -o profile=name.  The existing
credential mechanisms continue to work.  Fixes #822.
2018-11-04 17:47:07 -08:00
Andrew Gaul
664f910083 Correctly compare list_object_max_keys
Previously this did not allow an argument.  Fixes #843.
References #793.
2018-11-04 10:49:35 -08:00
Jonatan Nilsson
d9c106cfde Add support for storage class ONEZONE_IA. 2018-09-30 14:27:31 +02:00
Takeshi Nakatani
c5af62b023
Merge pull request #820 from gaul/big-writes
Enable big writes if capable
2018-09-17 17:24:06 +09:00
Andrew Gaul
a85183d42c Enable big writes if capable
Fixes #813.
2018-09-10 22:21:51 -07:00
Mattia Rizzolo
c376efdd28
Fix typo s/mutliple/multiple/
Signed-off-by: Mattia Rizzolo <mattia@mapreri.org>
2018-08-19 00:14:57 +02:00
Takeshi Nakatani
7e0c53dfe9 Added list_object_max_keys option based on #783 PR 2018-07-08 03:49:10 +00:00
Neeraj Kumar Kashyap
5a481e6a01 Option for IBM IAM auth endpoint added return 2018-06-04 16:44:14 +05:30
Neeraj Kumar Kashyap
d8e12839af Option for IBM IAM auth endpoint 2018-05-31 16:02:48 +05:30
Or Ozeri
d4e86a17d1 Revert "enable FUSE read_sync by default"
This reverts commit 86b0921ac4.

Conflicts:
	src/s3fs.cpp
2018-05-28 13:49:54 +03:00
Takeshi Nakatani
ae9d8eb734 Fixed memory leak 2018-05-27 10:48:03 +00:00
Takeshi Nakatani
57b2a60172
Merge pull request #764 from orozery/remove_false_multihead_warnings
Remove false multihead warnings
2018-05-23 22:38:35 +09:00
Or Ozeri
0bd875eb9e remove false readdir_multi_head warnings 2018-05-22 17:10:50 +03:00
Or Ozeri
86b0921ac4 enable FUSE read_sync by default 2018-05-06 16:10:36 +03:00
Neeraj Kumar Kashyap
25375a6b48 Validate the URL fixed inefficient usage of find 2018-05-04 11:24:32 +05:30
Neeraj Kumar Kashyap
ca87df7d44 Validate the URL format for http/https 2018-05-03 22:08:28 +05:30
陈方舟
04493de767
fix xpath selector in bucket listing
the original implementation in get_base_exp() depends on the order of xml return from the server.
patriotically, when listing a directory with sub directory(s), the xml document response contains more than 2 <Prefix> nodes(some of them are in <CommonPrefixes> node).
the source code arbitrarily select the first one in the documents (nodes->nodeTab[0]->xmlChildrenNode).
some s3 compatible service return the list-bucket result in different result, leading the s3fs to a wrong behavior
2018-04-23 15:11:29 +08:00
Tristen Horton
0863672e27
add a more helpful error message for when there are multiple entries for the same bucket in the passwd file 2018-03-13 14:37:34 -07:00
Takeshi Nakatani
0f503ced25
Merge pull request #729 from dmgk/master
FreeBSD build fixes
2018-03-04 16:36:31 +09:00
Takeshi Nakatani
987a166bf4
Merge pull request #726 from orozery/instance_name_logging
add an instance_name option for logging
2018-03-04 15:41:12 +09:00
Dmitri Goutnik
c9f4312588 FreeBSD build fixes 2018-03-02 15:58:52 -05:00
Or Ozeri
8b657eee41 add disk space reservation 2018-02-28 19:20:23 +02:00
Takeshi Nakatani
c494e54320 Fixed cppcheck error on osx 2018-02-28 12:06:06 +00:00
Or Ozeri
b52b6f3fc5 add an instance_name option for logging 2018-02-28 09:51:35 +02:00
Takeshi Nakatani
a45ff6cdaa Fixed cppcheck error and clean ^M code 2018-02-25 13:08:41 +00:00
Or Ozeri
bd4bc0e7f1 add support for IBM IAM authentication 2017-11-23 12:01:52 +02:00
Takeshi Nakatani
97fc845a6a Changed functions about reading passwd file. 2017-11-19 11:49:11 +00:00
Takeshi Nakatani
7d9ac0163b Changed functions about reading passwd file. 2017-11-19 11:38:12 +00:00
Andrew Gaul
0418e53b3c Reduce use of preprocessor
This provides type-safety and avoids token expansion side effects.
2017-11-18 22:40:06 -08:00
Richard Caunt
366f0705a0 ECS credentials bug fixes 2017-11-06 21:45:58 +00:00
Richard Caunt
ccea87ca68
Added check for is_ecs during get_access_keys
We need to presume that if `is_ecs` we are deferring access key loading
2017-11-06 11:02:27 +00:00
Richard Caunt
662f65c3c8 Add support for ECS metadata endpoint 2017-11-05 19:24:02 +00:00
Takeshi Nakatani
ab89b4cd4a Merge pull request #659 from ggtakec/master
Do not fail updating directory when removing old-style object(ref #658)
2017-10-15 21:56:35 +09:00
Takeshi Nakatani
1eba27a50a Refixed s3fs_init message(ref #652) 2017-10-15 06:45:19 +00:00
Takeshi Nakatani
41206fa0e2 Do not fail updating directory when removing old-style object(ref #658) 2017-10-15 05:03:44 +00:00
jurafxp
ae91b6f673 Fix s3fs_init message 2017-10-01 00:08:00 +02:00
Takeshi Nakatani
00bc9142c4 Fixed potential atomic violation in S3fsCurl::AddUserAgent - #633 2017-09-17 09:16:05 +00:00
Tony Levi
cc94e1da26 Fix chown_nocopy losing existing uid/gid if unspecified 2017-05-25 16:53:08 +09:30
Takeshi Nakatani
337da59368 Not fallback to HTTP - #596 2017-05-13 15:47:39 +00:00
Takeshi Nakatani
d2ae14d8b7 Check bucket at public bucket and add nocopyapi option automatically 2017-05-13 07:48:50 +00:00
Takeshi Nakatani
7115835834 Check bucket at public bucket and add nocopyapi option automatically 2017-05-13 07:35:55 +00:00
Takeshi Nakatani
24df69f688 Backward compatible for changing default transport to HTTPS 2017-05-13 06:47:51 +00:00
Andrew Gaul
0e0ae38f6d Default transport to HTTPS
This protects private data when used over the public Internet.  Users
can opt-in to unencrypted HTTP if they need additional performance on
a local network.  Fixes #282.
2017-05-07 10:59:54 -07:00
Takeshi Nakatani
9d10a5aa70 Changed copyright year format for debian pkg 2017-05-07 11:24:17 +00:00
Takeshi Nakatani
1fe0334c08 Enhanced bucket/path parameter check 2017-05-05 19:55:24 +00:00
Takeshi Nakatani
3ac39d61f8 Added notsup_compat_dir option 2017-05-05 17:28:29 +00:00
Takeshi Nakatani
864e20e1f2 Refactored the get_object_attribute function 2017-05-05 10:02:21 +00:00
Takeshi Nakatani
51b3183cba Refactored the check_object_access function 2017-05-05 09:51:30 +00:00
Takeshi Nakatani
758b92e823 Added option for complementing lack of stat mode 2017-05-04 03:41:24 +00:00
Takeshi Nakatani
efba9bcbc1 Merge pull request #553 from orozery/custom_cipher_suite
add TLS cipher suites customization
2017-04-16 19:09:27 +09:00
Or Ozeri
75b59a7c16 switch S3fsMultiCurl to use foreground threads 2017-04-04 15:32:53 +03:00
Takeshi Nakatani
fef3fbc225 Added check_cache_dir_exist option(refixed #347) - #538 2017-04-02 08:10:16 +00:00
Takeshi Nakatani
980ba398bc Fixed double initialization of SSL library - #524 2017-03-19 17:11:18 +00:00
Takeshi Nakatani
523043a2aa Changed base cached time of stat_cache_expire option - #523 2017-03-19 15:19:04 +00:00
Andrew Gaul
03217baa99 Address cppcheck 1.77 warnings 2017-03-06 12:41:08 -08:00
Andrew Gaul
d375bca0d0 Correct typos 2016-11-19 15:57:41 -08:00
Takeshi Nakatani
7fabd18b1f Trim symbolic link original path in file. 2016-10-11 10:22:30 +00:00
Takeshi Nakatani
5db369d67e Trim symbolic link original path in file. 2016-10-11 10:17:46 +00:00
Takeshi Nakatani
dba32fdf78 Trim symbolic link original path in file. 2016-10-11 10:06:21 +00:00
Takeshi Nakatani
9fd1368611 OS-specific correspondence of the extended attribute header 2016-10-02 07:23:19 +00:00
Takeshi Nakatani
65d52506c4 Added use_xattr option for #467 and #460 2016-09-19 04:28:01 +00:00
Takeshi Nakatani
1a96f40a10 Fixed a bug about could not copy file mode from org file 2016-09-11 13:09:23 +00:00
Takeshi Nakatani
1ddc14d59d Changed for accepting mount options compatible with mtab - #449 2016-07-24 08:17:58 +00:00
Peter Watkins
a19206cf0f Accept mount arguments compatible with mtab
Using "mount -a" fails for already-mounted s3fs directories,
because s3fs mount arguments don't match the form in /etc/mtab.
Calling "mount -a" should quietly succeed when a directory is
already mounted.

To fix this, accept mount commands of the form:

s3fs s3fs /srv/object-store -o bucket=mybucket

or in /etc/fstab form:

s3fs  /srv/object-store  fuse.s3fs  bucket=mybucket 0 0

This matches the form in /etc/mtab and allows "mount -a" to
work properly.
2016-07-13 17:23:33 -04:00
Andrew Gaul
320b8e1171 Include location constraint when creating bucket
This allows creating buckets in non-default regions.  Also improve
do_create_bucket error handling.
2016-06-13 10:35:37 -07:00
Nathaniel W. Turner
584ea488bf Use role name instead of profile name when iam_role=auto
When using an instance with an IAM Role, transient credentials can be
found in http://169.254.169.254/latest/meta-data/ at
iam/security-credentials/role-name and s3fs tries to do this. However,
it is using the profile-name where role-name is needed. In many cases
the role and profile name are the same, but they are not always.

The simplest way to find the role name appears to be to GET
http://169.254.169.254/latest/meta-data/iam/security-credentials/
itself, which returns a listing of the role names for which temporary
credentials exist. (I think there will probably only be one, but we
probably want to split on newlines and take the first one here in case
that assumption is not valid). This is the approach the AWS SDK appears
to use (based on WireShark analysis).

Bug: https://github.com/s3fs-fuse/s3fs-fuse/issues/421
Signed-off-by: Nathaniel W. Turner <nate@houseofnate.net>
2016-05-24 13:34:19 -04:00
Nathaniel W. Turner
594c9ca7d2 Skip early credential checks when iam_role=auto
If user specifies iam_role=auto (or just iam_role), credentials will not
be loaded during early phase, so skip credential checks there.

Signed-off-by: Nathaniel W. Turner <nate@houseofnate.net>
2016-05-20 12:49:02 -04:00
Takeshi Nakatani
50f1ad51c8 loading IAM role name automatically(iam_role option) - #387 2016-05-06 04:37:32 +00:00
Takeshi Nakatani
1ddbd4d6bb Fixed 'load_sse_c' option not working - #388 2016-05-06 00:36:54 +00:00
Takeshi Nakatani
10589a9497 Supported User-Agent header - #383 2016-04-17 07:44:03 +00:00
Takeshi Nakatani
67efc11d94 Always set stats cache for opened file 2016-03-13 05:43:28 +00:00
Takeshi Nakatani
c7cf86c2ef Sepalated AdditionalHeader class from curl.* 2016-02-07 05:41:56 +00:00
Takeshi Nakatani
150b83f61e Remove stat file cache dir if specified del_cache - #337 2016-02-06 18:59:13 +00:00
Takeshi Nakatani
c5a94cfc0c Check cache dirctory path and attributes - #347 2016-02-06 13:38:48 +00:00
Robb Kistler
4eff6b4dd1 Fix empty directory check against AWS S3
For ListBucketResult on an empty directory, AWS S3 and S3Proxy 1.4
differ.  AWS will match the directory name, S3Proxy does not.

Changing max-keys=1 to max-keys-2 works for both implementations.
append_objects_from_xml() will swallow the directory key.  The log
level of this message is changed from ERROR to DBG.

Fixes #345
2016-02-04 23:13:00 -08:00
Andrew Gaul
e003732f18 Address various clang warnings
Found with:

-Wc++11-extensions
-Wc++11-extra-semi
-Wmissing-variable-declarations
-Wundef
2016-01-11 00:52:24 -08:00
Takeshi Nakatani
ea151a70c4 Merge pull request #321 from mcellis33/320
320: delete stat cache entry in s3fs_fsync so st_size is refreshed
2015-12-20 15:05:28 +09:00
Mark Ellis
163daa5de1 320: delete stat cache entry in s3fs_fsync so st_size is refreshed 2015-12-18 15:39:25 -08:00
Robb Kistler
b581290c30 Cleanly exit fuse loop on error in s3fs_init
This allows FUSE to clean the mount point up, preventing
"Transport endpoint not connected" errors on subsequent
access to the mount.
2015-12-15 15:25:56 -08:00
Robb Kistler
8e688816d4 Change error log to debug log in s3fs_read() 2015-12-03 21:25:27 -08:00
Takeshi Nakatani
7b62de80f6 Fixed a bug about mtime - #299 2015-11-29 15:53:53 +00:00
Takeshi Nakatani
c673d9d935 File opened with O_TRUNC is not flushed - changed #291 2015-11-08 04:55:17 +00:00
Robb Kistler
dd7d9268f2 Force flush in s3fs_open() if file is truncated. 2015-11-03 22:06:25 -08:00
Takeshi Nakatani
8dd234dd8f Fixed bugs about cppcheck error 2015-10-20 15:47:07 +00:00
Takeshi Nakatani
83d46ef8c6 Fixed bugs about a object larger than free disk space 2015-10-20 15:19:04 +00:00
Takeshi Nakatani
d102eb752d Supported a object which is larger than free disk space 2015-10-18 17:03:41 +00:00
Takeshi Nakatani
f51ad1f33e Supported for SSE KMS 2015-10-06 14:46:14 +00:00
Takeshi Nakatani
92e52dadd4 Changed and cleaned the logic for debug message. 2015-09-30 19:41:27 +00:00
Andrew Gaul
785ed642ba Add support for standard_ia storage class
This enables storage with lower at-rest prices, higher request prices,
and lower availability.  Also rework existing reduced redundancy
parsing into a more generic storage class.  More background on
standard_ia:

https://aws.amazon.com/blogs/aws/aws-storage-update-new-lower-cost-s3-storage-option-glacier-price-reduction/
2015-09-17 13:35:25 -07:00
Takeshi Nakatani
ce66430fac Added chacking cache dir perms at starting. 2015-08-23 03:57:34 +00:00
Takeshi Nakatani
751c868769 s3fs can print version with short commit hash - #228 2015-08-21 16:19:31 +00:00
Takeshi Nakatani
a3e820e733 Merge pull request #245 from andrewgaul/map-duplicate-lookups
Elide duplicate lookups of std::map via iterators
2015-08-20 01:22:06 +09:00
Andrew Gaul
67d1576dfb Elide duplicate lookups of std::map via iterators
Also remove use of C++11 std::map::at.
2015-08-18 14:00:42 -07:00
Andrew Gaul
c0b21d8808 Enable all cppcheck rules 2015-08-16 17:13:24 -07:00
Takeshi Nakatani
756d1e5e81 Configure cppcheck #224 2015-08-12 15:04:16 +00:00
Takeshi Nakatani
49e32967ec Merge pull request #219 from andrewgaul/coverity
Address Coverity errors
2015-08-12 23:40:47 +09:00
Andrew Gaul
ff8a0c2eea Parse ETag from copy multipart correctly
Previously s3fs misparsed this, preventing renames of files larger
than 5 GB.  Integration test disabled until S3Proxy 1.5.0 is released.
2015-08-11 14:43:35 -07:00
Andrew Gaul
8ee71caabb Address Coverity errors
Fixed an uninitialized member, misordered NULL check, resource leak,
and unconsumed return value.
2015-08-05 23:28:06 -07:00
Jamie Alessio
912bc58df0 Fixed a few small spelling issues. 2015-07-10 11:50:40 -07:00
Nate Rosenblum
9b3c87ec97 Specialize {set,get}xattr for OS X
These system calls take an extra 'position' parameter on OS X. A
non-zero position value is only valid for resource forks (the Darwin
VFS layer will reject anything else with EINVAL); this patch simply
adds and ignores the parameter on Apple platforms.

Allows building against OSXFUSE.
2015-06-25 12:56:15 -07:00
Takeshi Nakatani
966d229787 fixed fallback to sigv2 for bucket create and GCS 2015-06-20 04:34:32 +00:00
Takeshi Nakatani
ad8c64104e Merge pull request #199 from s3fs-fuse/xattr
Supported extended attributes(retry)
2015-06-20 11:46:47 +09:00
Takeshi Nakatani
fe3abed9f0 Chaged codes about iterator etc 2015-06-13 03:27:07 +00:00
Takeshi Nakatani
0ecf4aa6b4 Chaged codes about iterator 2015-06-13 03:08:56 +00:00
Takeshi Nakatani
84fb3d83d8 Fixed xattr for binary value 2015-06-06 16:39:39 +00:00
Bartlomiej Palmowski
3522e5eda3 Add no_check_certificate option which allows to ignore issues with self signed certs. 2015-05-20 17:32:36 +02:00
Andrew Gaul
28ee9f27b9 Add usage information for multipart_size
Also improve error message.
2015-05-04 16:21:58 -07:00
Dan Moore
5b11ac0f4c Moved __APPLE__ #endif to correct position 2015-04-27 12:14:09 -07:00
Dan Moore
adb5a35097 configure.ac: detect target, if target is darwin (OSX), then
change the minimum version of fuse required.  Change the
checkers to use a variable for the minimum fuse version
instead of it being hardcoded in four different places.

src/s3fs.cpp: Use __APPLE__ define around fuse code that
is offensive to osxfuse. Not including the code doesn't
seem to matter.
2015-04-25 17:13:20 -07:00
Takeshi Nakatani
aac92bd6c0 Fixed wrong owner checking and return codes 2015-04-21 16:18:05 +00:00
Takeshi Nakatani
f258a14070 Supported extended attributes, initial commit 2015-04-20 17:24:57 +00:00
Peter A. Bigot
92fcee824b curl: use pathrequeststyle option when constructing Host endpoint
Buckets with mixed-case names can't be accessed with the virtual-hosted
style API due to DNS limitations.  S3FS has an option for
pathrequeststyle which is used for the URL, but it was not applied when
building the endpoint passed through the Host header.  Fix this, and
relax the validation on bucket names when using this style.

See: http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro

Signed-off-by: Peter A. Bigot <pab@pabigot.com>
2015-04-19 08:31:40 -05:00
Ka-Hing Cheung
70097709b2 switch to use region specific endpoints to compute correct v4 signature
fix #133
2015-04-14 16:25:17 -07:00
Takeshi Nakatani
26453c4874 Fixed a bug not handling fsync. 2015-03-10 16:18:03 +00:00
Takeshi Nakatani
114966e7c0 Fixed bugs, not turn use_cache off and ty to load to end - issue#97 2015-03-04 08:48:37 +00:00
Takeshi Nakatani
53083202ba Merge pull request #132 from andrewgaul/s3proxy-integration-test
Use S3Proxy to run integration tests
2015-02-27 00:17:46 +09:00
Andrew Gaul
e811ae1104 Use s3proxy to run integration tests
References #129.
2015-02-24 12:08:22 -08:00
Timur Alperovich
be5735edb8 Fallback to v2 signatures correctly.
Missing parameter to SetSignatureV4() call in the fallback code path
results in not actually falling back.
2015-02-16 17:35:09 -08:00
Takeshi Nakatani
cf2b0cca22 Added new mp_umask option about issue#107, pr#110 2015-02-07 17:16:45 +00:00
Takeshi Nakatani
1424f87754 Supported signature version 4 for GnuTLS/NSS and automatically set endpoint/sigv2 2015-02-02 16:36:08 +00:00
Takeshi Nakatani
4f953f9bd7 Clean codes for signature v4 and added new sigv2 option 2015-01-28 17:13:11 +00:00
Takeshi Nakatani
0d2f3e2dc4 Fixed bugs, segfault and signature error at listing. 2015-01-24 16:36:30 +00:00
Takeshi Nakatani
bb1f1d3faa Merged manually from caxapniy/s3fs-fuse/tree/1.77v4merge for signature v4 - #102 2015-01-20 16:31:36 +00:00
Takeshi Nakatani
98daf16681 Merge pull request #104 from kahing/rename_before_close
fix rename before close
2015-01-14 00:40:41 +09:00
Ka-Hing Cheung
03d84a07d1 fix rename before close
nautilus does this when you drag and drop to overwrite a file:

1) create .goutputstream-XXXXXX to write to
2) fsync the fd for .goutputstream-XXXXXX
3) rename .goutputstream-XXXXXX to target file
4) close the fd for .goutputstream-XXXXXX

previously, doing this on s3fs would result in an empty target file
because after the rename, s3fs would not flush the content of
.goutputstream-XXXXXX to target file.

this change moves the FdEntity from the old path to the new path
whenever rename happens. On flush s3fs would now flush the correct
content to the rename target.
2015-01-12 15:05:54 -08:00
Andrej Dobos
69ef7fbefb Optimized function directory_empty: check for at most one entry when evaluating whether a directory is empty or not (as opposed to doing full directory listing) 2014-12-23 22:29:13 -08:00
Takeshi Nakatani
082eb24c12 Merge pull request #83 from tmwong2003/develop
Changed option processing to use strtol() to get a umask
2014-11-16 23:49:24 +09:00
Theodore Wong
f04b659f5e Changed option processing to use strtol() to get a umask
get_mode()/s3fs_strtoofft() does not handle octal umask values, which
results in unexpected behavior when trying to set a world-readable umask
value.
2014-11-12 23:29:41 +00:00
Andrew Gaul
50137fe026 Address clang always true warnings 2014-10-16 23:34:12 -07:00
Andriy Senkovych
6633366218 Fix spelling errors 2014-10-01 13:42:39 +03:00
Takeshi Nakatani
7a7c7572ea Cleaned up codes for next packaging. 2014-09-07 15:08:27 +00:00
Takeshi Nakatani
f0c33f8ef2 clean codes 2014-08-27 00:59:49 +00:00
Takeshi Nakatani
20b1c207be fixed issue #39 2014-08-26 17:11:10 +00:00
Takeshi Nakatani
cbec8da9a3 fixed a bug issue #49 2014-08-14 15:58:06 +00:00
Takeshi Nakatani
7a55eab399 Support for SSE-C, issue #39 2014-07-19 19:02:55 +00:00
Takeshi Nakatani
c1a6d76fc3 Fixed a bug issue #40 2014-06-28 17:36:35 +00:00
Andrew Dunn
601482eff5 Added support for path API request style.
Rather than using virtual host style requests, path style requests can be used
instead.

i.e. rather than bucketname.s3.amazon.com/... the s3fs will be able to request
from s3.amazon.com/bucketname/...

This is useful for S3 compatible APIs which don't support the virtual host style
request.

It is enabled with the new option, `use_path_style_request`.

Example:

    /usr/bin/s3fs data ~/netcdf -o url="https://swift.rc.nectar.org.au:8888/" -o use_path_request_style -o allow_other -o uid=500 -o gid=500
2014-06-04 00:03:49 +10:00
Takeshi Nakatani
61020370d5 Changed codes for CR code in passwd file(googlecode issue#417). 2014-06-02 16:12:55 +00:00
Takeshi Nakatani
160196798b Changed initializing logic for nss lib/openssl lib/s3fs own. 2014-06-01 03:54:02 +00:00
Takeshi Nakatani
cd27f0aa54 Supported another crypt libraries as GnuTLS and NSS, and added configure options 2014-05-06 14:23:05 +00:00
Takeshi Nakatani
4762e53b5d Added multipart_size option for #16 2014-03-30 07:53:41 +00:00
Takeshi Nakatani
16487c4e26 Fixed a bug: issue #14, s3fs -u should return 0 if there are no lost multiparts 2014-03-03 17:00:20 +00:00
Takeshi Nakatani
5fd33405af Fixed a bug - issue#5 2014-01-16 16:04:41 +00:00
ggtakec@gmail.com
8acbaf7199 Fixed a bug.
* Fixed a bug
  Fixes a bug that is dead loop when s3fs listed in a directory with
  directory objects which was not object(no information).
  This bug made by r493, and reported by issue 389.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@497 df820570-a93a-0410-bd06-b72b767a4274
2013-11-18 02:29:41 +00:00
ggtakec@gmail.com
40b9f0a408 Changes codes
1) Changed buffer size for file size
   Changes a internal buffer size from size_t to offt_t.
   It is a bug for 32 bit OS enviroment.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@496 df820570-a93a-0410-bd06-b72b767a4274
2013-11-17 08:50:41 +00:00
ggtakec@gmail.com
882f13020e Fixed bugs(overflow)
1) Overflow
   About over 4GB file, when st_size which is member stat structure,
   the value is overflow.
   Fixed this bug and fixed like this bug in all sources. 

2) Changed retrying request
   If s3fs gets 500 HTTP status for multipart request, s3fs retry
   to send same request.





git-svn-id: http://s3fs.googlecode.com/svn/trunk@495 df820570-a93a-0410-bd06-b72b767a4274
2013-11-13 16:26:50 +00:00
ggtakec@gmail.com
09fc2593e3 Fixed bugs and Changed utility mode
1) Fixed bugs
 * Rename objects
   Fixes s3fs specifies wrong part number of multipart rename.
   And s3fs adds x-amz-acl and x-amz-server-side-encryption
   header when rename objects.

2) Changed retry logic for multipart uploading(and renaming)
   Sometimes, s3fs gets 400 HTTP response for one of part from
   S3 when s3fs uploads a large object by multipart.
   New logic retries uploading failed part until "retries"
   option count.

3) Added action on utility mode.
   s3fs have had utility mode for displaying the result of REST
   listing multipart uploading.
   Changed this row result(xml) to list, after that, s3fs starts
   conversation for removing it.
   Then you can remove the object which is failed uploading by
   multipart, and do not need to pay for that ever.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@493 df820570-a93a-0410-bd06-b72b767a4274
2013-11-11 13:45:35 +00:00
ggtakec@gmail.com
99db6d13af Changes codes
1) Supported IAM role
   Supports IAM role by option, that is instead of AccessKeyID/
   SecretAccessKey.
   Adds new option "iam_role" which is specified as IAM role
   name.(like s3fs-c)



git-svn-id: http://s3fs.googlecode.com/svn/trunk@490 df820570-a93a-0410-bd06-b72b767a4274
2013-10-06 13:45:32 +00:00
ggtakec@gmail.com
44468ba00f Fixed a bug(public_bucket)
1) Fixed a bug(about public_bucket)
   Fixes a bug that public_bucket option does not work.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@489 df820570-a93a-0410-bd06-b72b767a4274
2013-09-29 13:24:30 +00:00
ggtakec@gmail.com
3ba40a30cd Fixed a bug(Issue 371)
1) Fixed Issue 371
   Fixes a bug which is wrong return value in s3fs_truncate function(mis-coding).
 
   Issue 371: ftruncate failed



git-svn-id: http://s3fs.googlecode.com/svn/trunk@486 df820570-a93a-0410-bd06-b72b767a4274
2013-09-19 09:22:02 +00:00
ggtakec@gmail.com
41bf0e3558 * Re-Fixed a bug( Issue 368 )
1) Re-Fixed Issue 368
   Changes that s3fs checks always object stat information before opening it.
   Then the object updates other s3fs process or other client, s3fs can know
   it.

   (Issue 368)1.73: Updating existing file on server 'a' does not change length
   of file on server 'b'



git-svn-id: http://s3fs.googlecode.com/svn/trunk@485 df820570-a93a-0410-bd06-b72b767a4274
2013-09-17 05:16:30 +00:00
ggtakec@gmail.com
79152325ca Changes codes
1) fixed a bug
   fixes a code in s3fs.cpp.
   It freed memory twice because of careless.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@484 df820570-a93a-0410-bd06-b72b767a4274
2013-09-17 01:26:26 +00:00
ggtakec@gmail.com
42b74c9d2e Changes codes
1) Changed codes about memory leak
   For memory leak, below codes are changed.
   * calls malloc_trim function
   * calls initializing NSS function, and adds configure
     option "--enable-nss-init".
     If libcurl is with NSS, s3fs initializes NSS manually.
     This initializing NSS is enabled by "--enable-nss-init"
     option at configure. if this option is specified, you
     need "nss-devel" package.
   * calls initializing libxml2(xmlInitParser).
   * BIO functions have memory leak, calls CRYPTO_free_ex_data.
   * changes cache structure.
   * changes cache out logic to LRU.
   * sets alignment for allcated memory in body data structure.
   * adds ssl session into share handle. and adds nosscache option.
   * deletes unused allocated memory.(bug)
   * changes defaule parallel count of head request in readdir
     (500->20)
   * fixes some bugs.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@482 df820570-a93a-0410-bd06-b72b767a4274
2013-09-14 21:50:39 +00:00
ggtakec@gmail.com
3b393fe159 Fixed bugs
1) Fixed Issue 321
   Fixed a bug that a value of umask option is not reflected
   definitely.
   (Issue 321) no write permission for non-root user

2) Fixed a bug about utimens
   Fixed a bug that utimens function cloud not set value to the
   other user's object which does not allowed writable.

3) Strictly option check.
   Fixed checking s3fs options strictly.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@480 df820570-a93a-0410-bd06-b72b767a4274
2013-08-29 06:23:51 +00:00
ggtakec@gmail.com
7ce8135fa9 Changes codes
1) Changed codes about initializing curl and openSSL
   s3fs before this change called curl_global_init() two times
   with curl_global_cleanup(). After reviewing this processing,
   s3fs calls curl_global_init() one time.
   Then s3fs_check_service function which checks user bucket
   existing is called after calling fuse.
   So this new processing does not have a problem, the codes
   updated.

   And about initializing openSSL(CRYPTO), old s3fs called only
   static locking callback function(ex. CRYPTO_set_locking_callback()).
   Added calling dynamic locking callback function for CRYPTO
   (ex. CRYPTO_set_dynlock_lock_callback()).



git-svn-id: http://s3fs.googlecode.com/svn/trunk@479 df820570-a93a-0410-bd06-b72b767a4274
2013-08-27 08:12:01 +00:00
ggtakec@gmail.com
8111edec61 Fixed codes for compiling(32)
1) Fixed codes
   Fixed compiling error on 32bit, which specified wrong dev_t format
   on 32bit.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@478 df820570-a93a-0410-bd06-b72b767a4274
2013-08-26 07:18:05 +00:00
ggtakec@gmail.com
3dda0b20d4 Added debugging message
1) Added debugging message in s3fs_getattr
   If s3fs runs with "f2" option for deep debugging message, s3fs_getattr
   puts debugging message as file's uid/gid/mode.

2) Added curldbg option
   Added new option "curldbg" which is for debugging curl http/https
   information.
   It implements by CURLOPT_VERBOSE on curl_easy_setopt function.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@474 df820570-a93a-0410-bd06-b72b767a4274
2013-08-23 16:28:50 +00:00
ggtakec@gmail.com
b3682f87d2 Fixed bugs(Issue 363)
1) Fixed Issue 363
   Fixed a bug which has below reason.
   Fuse does not wait finishing "release file descriptor" function
   called by fuse, and fuse runs(calls) next processing(commands).
   Then s3fs could not clear stats cache information for that file
   before calling next processing, and s3fs uses old stats cache
   information.
   So that, s3fs clears stats cache in release function at first.

   And found two bad codes(but these codes do not influence normal
   movement) in fdcache.cpp and fixed these.

   Issue 363: make check failing inconsistently



git-svn-id: http://s3fs.googlecode.com/svn/trunk@471 df820570-a93a-0410-bd06-b72b767a4274
2013-08-22 09:36:16 +00:00
ggtakec@gmail.com
07a8caa250 Fixed bugs
1) Fixed Issue 321(#30)
   Fixed a bug(mis-coding).
     Issue 321(#30): no write permission for non-root user

2) Fixed Issue 365
   Fixed a bug(mis-coding).
     Issue 365: there is a logical error in s3fs-1.72 s3fs.cpp:2865




git-svn-id: http://s3fs.googlecode.com/svn/trunk@470 df820570-a93a-0410-bd06-b72b767a4274
2013-08-22 02:39:21 +00:00
ggtakec@gmail.com
ee01c91e02 Fixed bugs for compiling
1) Fixed bugs
   Fixes below bugs( format error and undefined fund ).

   * 1.72 Will not compile on Ubuntu 12.04.2 (precise) i686(Issue 360)
   * complie time error after running #make(Issue 361)

   I'll close these Issue if I can confirm that these problem was solved.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@466 df820570-a93a-0410-bd06-b72b767a4274
2013-08-19 06:29:24 +00:00
ggtakec@gmail.com
d7689151ab Fixed Issue 229 and Changes codes
1) Set metadata "Content-Encoding" automatically(Issue 292)
   For this issue, s3fs is added new option "ahbe_conf".

   New option means the configuration file path, and this file specifies
   additional HTTP header by file(object) extension.
   Thus you can specify any HTTP header for each object by extension.

   * ahbe_conf file format:
     -----------
     line                = [file suffix] HTTP-header [HTTP-header-values]
     file suffix         = file(object) suffix, if this field is empty, 
                           it means "*"(all object).
     HTTP-header         = additional HTTP header name
     HTTP-header-values  = additional HTTP header value
     -----------

   * Example:
     -----------
     .gz      Content-Encoding     gzip
     .Z       Content-Encoding     compress
              X-S3FS-MYHTTPHEAD    myvalue
     -----------
     A sample configuration file is uploaded in "test" directory.

   If ahbe_conf parameter is specified, s3fs loads it's configuration
   and compares extension(suffix) of object(file) when uploading
   (PUT/POST) it. If the extension is same, s3fs adds/sends specified
   HTTP header and value.

   A case of sample configuration file, if a object(it's extension is
   ".gz") which already has Content-Encoding HTTP header is renamed 
   to ".txt" extension, s3fs does not set Content-Encoding. Because
   ".txt" is not match any line in configuration file.
   So, s3fs matches the extension by each PUT/POST action.

   * Please take care about "Content-Encoding".
   This new option allows setting ANY HTTP header by object extension.
   For example, you can specify "Content-Encoding" for ".gz"/etc 
   extension in configuration. But this means that S3 always returns 
   "Content-Encoding: gzip" when a client requests with other 
   "Accept-Encoding:" header. It SHOULD NOT be good.
   Please see RFC 2616.

2) Changes about allow_other/uid/gid option for mount point
   I reviewed about mount point permission and allow_other/uid/gid
   options, and found bugs about these.
   s3fs is fixed bugs and changed to the following specifications.

   * s3fs only allows uid(gid) options as 0(root), when the effective 
     user is zero(root).
   * A mount point(directory) must have a permission to allow
     accessing by effective user/group.
   * If allow_other option is specified, the mount point permission
     is set 0777(all users allow all access).
     In another case, the mount point is set 0700(only allows 
     effective user).
   * When uid/gid option is specified, the mount point owner/group
     is set uid/gid option value.
     If uid/gid is not set, it is set effective user/group id.

   This changes maybe fixes some issue(321, 338).

3) Changes a logic about (Issue 229)
   The chmod command returns -EIO when changing the mount point.
   It is correct, s3fs can not changed owner/group/mtime for the
   mount point, but s3fs sends a request for changing the bucket.
   This revision does not send the request, and returns EIO as
   soon as possible.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@465 df820570-a93a-0410-bd06-b72b767a4274
2013-08-16 19:24:01 +00:00
ggtakec@gmail.com
02c3accb5b Changes codes
1) Changes macros for debugging
   Changed macros for debugging messages.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@461 df820570-a93a-0410-bd06-b72b767a4274
2013-08-10 15:29:39 +00:00
ggtakec@gmail.com
b24c868417 Fixed Issue 355
1) Patch in support for special file and block device types( Issue 355 )
    Patched codes, and s3fs can make special files on S3.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@460 df820570-a93a-0410-bd06-b72b767a4274
2013-07-30 07:27:22 +00:00
ggtakec@gmail.com
3274f58948 Changes codes for performance(part 3)
* Summay
   This revision includes big change about temporary file and local cache file. 
   By this big change, s3fs works with good performance when s3fs opens/
   closes/syncs/reads object.
   I made a big change about the handling about temporary file and local cache
   file to do this implementation.

* Detail
1) About temporary file(local file)
   s3fs uses a temporary file on local file system when s3fs does download/
   upload/open/seek object on S3.
   After this revision, s3fs calls ftruncate() function when s3fs makes the 
   temporary file.
   In this way s3fs can set a file size of precisely length without downloading.
   (Notice - ftruncate function is for XSI-compliant systems, so that possibly
    you have a problem on non-XSI-compliant systems.)

   By this change, s3fs can download a part of a object by requesting with 
   "Range" http header. It seems like downloading by each block unit.
   The default block(part) size is 50MB, it is caused the result which is default 
   parallel requests count(5) by default multipart upload size(10MB).
   If you need to change this block size, you can change by new option 
   "fd_page_size". This option can take from 1MB(1024 * 1024) to any bytes.

   So that, you have to take care about that fdcache.cpp(and fdcache.h) were 
   changed a lot.

2) About local cache
   Local cache files which are in directory specified by "use_cache" option do 
   not have always all of object data.
   This cause is that s3fs uses ftruncate function and reads(writes) each block 
   unit of a temporary file.
   s3fs manages each block unit's status which are "downloaded area" or "not".
   For this status, s3fs makes new temporary file in cache directory which is 
   specified by "use_cache" option. This status files is in a directory which is 
   named "<use_cache sirectory>/.<bucket_name>/".

   When s3fs opens this status file, s3fs locks this file for exclusive control by 
   calling flock function. You need to take care about this, the status files can 
   not be laid on network drive(like NFS).

   This revision changes about file open mode, s3fs always opens a local cache 
   file and each status file with writable mode.
   Last, this revision adds new option "del_cache", this option means that s3fs 
   deletes all local cache file when s3fs starts and exits.

3) Uploading
   When s3fs writes data to file descriptor through FUSE request, old s3fs 
   revision downloads all of the object. But new revision does not download all, 
   it downloads only small percial area(some block units) including writing data 
   area.
   And when s3fs closes or flushes the file descriptor, s3fs downloads other area 
   which is not downloaded from server. After that,  s3fs uploads all of data.
   Already r456 revision has parallel upload function, then this revision with 
   r456 and r457 are very big change for performance.

4) Downloading
   By changing a temporary file and a local cache file, when s3fs downloads a 
   object, it downloads only the required range(some block units). 
   And s3fs downloads units by parallel GET request, it is same as a case of 
   uploading. (Maximum parallel request count and each download size are 
   specified same parameters for uploading.)

   In the new revision, when s3fs opens file, s3fs returns file descriptor soon.
   Because s3fs only opens(makes) the file descriptor with no downloading 
   data. And when s3fs reads a data, s3fs downloads only some block unit 
   including specified area.
   This result is good for performance.

5) Changes option name
   The option "parallel_upload" which added at r456 is changed to new option 
   name as "parallel_count". This reason is this option value is not only used by 
   uploading object, but a uploading object also uses this option. (For a while, 
   you can use old option name "parallel_upload" for compatibility.)



git-svn-id: http://s3fs.googlecode.com/svn/trunk@458 df820570-a93a-0410-bd06-b72b767a4274
2013-07-23 16:01:48 +00:00
ggtakec@gmail.com
1c93dd30c1 Changes codes
1) For uploading performance(part 2)
   Changed a codes about uploading large object(multipart uploading).
   This revision does not make temporary file when s3fs uploads large object by multipart uploading.
   Before this revision, s3fs made temporary file(/tmp/s3fs.XXXXX) for multipart, but it was not good for performance.
   So that, new codes do not use those files, and s3fs reads directly large object from s3fs's cache file.

2) Some value to symbol
   Changed some value to symbol(define).



git-svn-id: http://s3fs.googlecode.com/svn/trunk@457 df820570-a93a-0410-bd06-b72b767a4274
2013-07-12 00:33:36 +00:00
ggtakec@gmail.com
1095b7bc52 Changes codes
1) For uploading performance(part 1)
   Changed a code for large object uploading.
   New codes makes s3fs send parallel requests when s3fs uploads large 
   object(20MB) by multipart post.

   And added new "parallel_upload" option, which limits parallel request 
   count which s3fs requests at once.
   This option's default value is "5", and you can change this value. But it
   is necessary to set this value depending on a CPU and a network band.
   s3fs became to work good performance by this option, please try to set 
   your value for this option.

2) Changes debugging messages
    Changed debugging message in s3fs.cpp.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@456 df820570-a93a-0410-bd06-b72b767a4274
2013-07-10 06:24:06 +00:00
ggtakec@gmail.com
b1e1312cfb Fixed Issue 352(+)
1) Option syntax verbosity in doc (  Issue 352  )
    Mistook fixing codes, then re-check'in correctly code.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@453 df820570-a93a-0410-bd06-b72b767a4274
2013-07-05 05:49:41 +00:00
ggtakec@gmail.com
d1a17cbe3d Fixed Issue 352 and bugs
1) Option syntax verbosity in doc ( Issue 352 )
    Before this revision(version), "use_rrs" option needs to set a parameter like "use_sse" option.
    But this option does not need a parameter, specified "use_rrs" option means enabled RRS.
    (because RRS is desabled by default.)
    After this revision, "use_rrs" option can be specified without a parameter, and "use_sse" too.
    Changed codes, man page and help page.
    Please notice, for old version "use_rrs"(and "use_sse") can be specified with a parameter("1" or "0") yet.

2) Fixes a bug about analizing "use_sse" option.
    Fixed a bug in r451, "use_sse" option is not worked because s3fs mistook to call function for "use_rrs".

3) Fixes a memory leak.
    Fixed a memory leak in r451.
    Fixed that the curl_slist_sort_insert() function forgot to free memory.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@452 df820570-a93a-0410-bd06-b72b767a4274
2013-07-05 05:41:46 +00:00
ggtakec@gmail.com
ad19ffa458 Changes codes
1) Adds new S3fsCurl class
   Added new S3fsCurl class instead of directly calling curl function.
   This class is lapping curl function for s3fs(AWS S3 API).

2) Changes codes about adding S3fsCurl class
    Changed and deleted classes and structures which are related to curl in curl.cpp/curl.h.
    Changed codes which are calling S3 API with curl in s3fs.cpp.

3) Deletes YKIES macro
    Deleted YIKES macro, because this macro is used no more.

4) Changes a code
    s3fs does not get good performance because s3fs copies each byte while downloading.
    So that the codes is changed instead of memcpy, then s3fs performance not a little improves.

5) Fixes a bug
    When s3fs renames a file, s3fs does not use the value which is specified by servicepath option.
    Fixed this bug.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@451 df820570-a93a-0410-bd06-b72b767a4274
2013-07-05 02:28:31 +00:00
ggtakec@gmail.com
45950044f7 Changes codes
1) Changes codes for performance and request's costs
    s3fs gets object's  attributes by using HEAD request.
    Directory objects is following 4 type:
      a) name type is "dir", with meta information
      b) name type is "dir", without meta information(but has files in itself)
      c) name type is "dir/", with(out) meta information
      d) name type is "dir_$folder$", with(out) meta information
    The codes is changed to order checking directory object.
    So that, s3fs decreases requests for checking objects.

    Before version has a bug, that is s3fs can not be recognizable type-b) 
    definitely when s3fs check the object directly.(but s3fs can, when s3fs 
    check the object by listing)
    This change fixes this bug.

2) Adds "multireq_max" option
    Added "multireq_max" option is maximum number of parallel request 
    for listing objects.
    This changes is possible to solve CURLE_COULDNT_CONNECT.
    If this option can not solve it, this option will be useful for tuning 
    performance by each.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@449 df820570-a93a-0410-bd06-b72b767a4274
2013-06-19 14:53:58 +00:00
ggtakec@gmail.com
b65c3e195e Fixed Issue 346
1) Not recognizing group permissions( Issue 346 )
    Fixed umask option which works correctly.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@448 df820570-a93a-0410-bd06-b72b767a4274
2013-06-18 01:17:32 +00:00
ggtakec@gmail.com
b3fb37a87e Changed codes
1) don't use curl_slist directly
    s3fs has auto_curl_slist struct, but some function use curl_slist directly in s3fs.cpp.
    So that, changes codes for using auto_curl_slist.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@444 df820570-a93a-0410-bd06-b72b767a4274
2013-06-15 16:47:17 +00:00
ggtakec@gmail.com
f7e1a2a37f Fixed bugs
1) Fixed a bug(forgot removing temporary files)
    When s3fs gets a error from fwrite in multipart uploading function,
    s3fs does not remove a temporary file.

2) Fixed a bug(wrong prototype of function)
    The prototype of function for CURLSHOPT_UNLOCKFUNC
    is wrong.

3) Changed codes
    - In my_curl_easy_perform function, the codes for debugging messages
      is changed, because it is for not working codes when "-d" option is 
      not specified.
    - Changes struct head_data's member variables, and some codes for this 
      changes.
    - Moving calling function to main for curl_global_init and curl_share_init 
      functions, because these function must call in main thread.

4) Fixed a bug(use uninitialized memory)
    In get_lastmodified function, this function does not initialize value
   (struct tm).

5) Fixed a bug(access freed variable)
    In readdir_multi_head function, access a variable which is already freed.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@442 df820570-a93a-0410-bd06-b72b767a4274
2013-06-15 15:29:08 +00:00
ggtakec@gmail.com
a285716f87 Fixed Issue 343 (Issue 235 , Issue 257 , Issue 265)
1) Fixes "SSL connect error"(curl 35 error)
    Re-fixed "SSL connect error",  so r434 could not fix it completely(mistook fixing).



git-svn-id: http://s3fs.googlecode.com/svn/trunk@441 df820570-a93a-0410-bd06-b72b767a4274
2013-06-06 03:01:21 +00:00
ggtakec@gmail.com
b921844ff8 Fixed Issue 342
1) Segmentation fault on connect on ppc64( Issue 342 )
    The third parameter of curl_easy_getinfo() is wrong.
    It must be "long" but specified "CURLcode".
    Fixes this issue.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@440 df820570-a93a-0410-bd06-b72b767a4274
2013-06-05 02:13:39 +00:00
ggtakec@gmail.com
8ef1c7aa3a Fixed Issue 226(+)
1) Patch adding support for SSE( Issue 226 )
    Forgot changing codes for the error logic which use_sse and 
    use_rrs option is specified.
    (r438 + this fixes = fixed issue 226)




git-svn-id: http://s3fs.googlecode.com/svn/trunk@439 df820570-a93a-0410-bd06-b72b767a4274
2013-06-04 06:12:22 +00:00
ggtakec@gmail.com
9833c7e589 Fixed Issue 226
1) Patch adding support for SSE( Issue 226 )
    Supports SSE(Server-Side Encryption) and adds "use_sse" option.
    * Specifications
       When "use_sse" option is specified as "1", s3fs adds "x-amz-server-side-encryption"
       header as "AES256".
       But it only does when objects upload(writing object). 
       When you do chmod/chown/chgrp/touch/mv commands, s3fs does not 
       add this header and inherit SSE mode from oroginal object. 
    * Notice
       "use_sse" option can not specify with "use_rrs" because we get signature
       error from S3.





git-svn-id: http://s3fs.googlecode.com/svn/trunk@438 df820570-a93a-0410-bd06-b72b767a4274
2013-06-04 06:04:04 +00:00
ggtakec@gmail.com
1758bc59f4 Fixed Issue 235, Issue 257, Issue 265
1) Fixes "SSL connect error"(curl 35 error)
    Fixed "SSL connect error", then s3fs can connect by SSL with no problem.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@434 df820570-a93a-0410-bd06-b72b767a4274
2013-06-01 15:31:31 +00:00
ggtakec@gmail.com
c686a3b2c0 Fixed Issue 241
1) problems with fseek and s3fs(Issue 241)
   This problem is that s3fs returns file stat(size) when client already has opened file discriptor and client modifies file before saving fd.
   So that, client adds bytes into file, but s3fs_getattr() function returns original size before changing.
   Changes codes for that s3fs returns size from opened file discriptor, if client already opens the file.
   * Changes s3fs.cpp
   * Adds fdcache.cpp fdcache.h



git-svn-id: http://s3fs.googlecode.com/svn/trunk@432 df820570-a93a-0410-bd06-b72b767a4274
2013-05-28 05:54:09 +00:00