octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-16 17:25:13 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,085 Commits 1 Branch 23 Tags 13 MiB
ee49ca4abf
Commit Graph

536 Commits

Author SHA1 Message Date
Andrew Gaul
9abe3fa662
Use same time when creating directory objects (#1652) 
This avoids mismatched times and is slightly faster.
 2021-05-09 13:44:12 +09:00
Andrew Gaul
cb9148f6cd
Update for clang-tidy 12 (#1644) 2021-05-03 09:43:18 +09:00
Andrew Gaul
771bbfeac5
Do not create zero-byte object when creating file (#1640) 
Previously s3fs created this object to store metadata and overwrote it
when flushing.  This prevented use with object stores which do not
allow overwrites like HDS.  Instead only create an in-memory
representation which reduces the time to create small files.
Fixes #1013.
 2021-04-30 19:56:33 +09:00
Andrew Gaul
3694786112
Propagate errno instead of EIO (#1638) 
This improves error fidelity.  Follows on to
b70f8db037.  References #1523.
 2021-04-30 07:09:00 +09:00
Andrew Gaul
e9eb248f2f
Fail CheckBucket when S3 returns PermanentRedirect (#1630) 
Previously s3fs allowed mounting but all operations returned EIO.
References #693.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-04-24 18:46:24 +09:00
Ambika Nair
77581eda59 Change in ibm iam endpoint 2021-04-21 21:42:55 +09:00
Andrew Gaul
7f3e423bbe
Preserve sub-second time precision (#1624) 
Found via pjdfstests.  References #897.  References #1589.
 2021-04-18 13:11:12 +09:00
Andrew Gaul
706e3cbebd
Update ctime and mtime when opening with O_TRUNC (#1623) 
Found via pjdfstests.  References #1589.
 2021-04-15 07:08:49 +09:00
Andrew Gaul
01a26a9011
Do not allow open with O_RDONLY and O_TRUNC (#1620) 
Found via pjdfstests.  References #1589.
 2021-04-13 07:28:10 +09:00
Andrew Gaul
23e1fbf7b9
Update ctime when truncating file (#1619) 
Found via pjdfstests.  References #1589.
 2021-04-13 07:08:06 +09:00
Andrew Gaul
ed2e877bb6
Return EACESS when lacking required executable bit (#1617) 
This makes the check consistent with read and write.  Found via
pjdfstests.  References #1551.  References #1589.
 2021-04-11 22:39:27 +09:00
Andrew Gaul
3663082a01
Do not allow renaming to a non-empty directory (#1604) 
Found via pjdfstests.  References #1589.
 2021-03-28 19:10:52 +09:00
Takeshi Nakatani
b589ebec23 Output version and command parameters at startup 2021-03-06 19:11:57 +09:00
Andrew Gaul
032fcf2a47
Allow optional issuing of ListObjectsV2 (#1583) 
This allows use of s3fs on object stores that do not implement
the V1 API.  Fixes #1573.
 2021-02-23 09:45:13 +09:00
Takeshi Nakatani
5b5bc3114a Fixed a bug that padded null bytes when changing xattr 2021-02-21 12:23:36 +09:00
Takeshi Nakatani
81e267d421 Fixed the same bug in chmod/chown/utimens and added tests 2021-02-20 23:30:50 +09:00
Takeshi Nakatani
4fc92d59f3 Fixed a bug that filling NULL bytes when changing the attribute after renaming 2021-02-20 23:30:50 +09:00
Takeshi Nakatani
a0f347b10f Added no_time_stamp_ms option mainly for testing 2021-02-20 20:24:26 +09:00
Takeshi Nakatani
4da56acdcc Improved processing when HEAD response is 400 2021-02-13 17:00:13 +09:00
Andrew Gaul
134a54b32f
Allow configuring the multipart threshold (#1562) 
Also change default which improves write performance for files >= 25
MB and <= 5 GB, particularly over lossy networks.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 23:35:46 +09:00
Andrew Gaul
7f6fbb0021
Correct singlepart_copy_limit unit (#1561) 
This is documented to be in MB not KB.

Co-authored-by: Takeshi Nakatani <ggtakec@gmail.com>
 2021-02-11 22:53:50 +09:00
Andrew Gaul
ec8bae9827
Do not periodically flush with nomultipartupload (#1559) 
s3fs periodically flushes data to S3 and hole-punches the written
data, marking it as clean.  Without multipart copy, s3fs would write
the now-zeroed data along with any new data.  s3fs should not attempt
to periodically flush dirty data at all without multipart copy
support.  Fixes #1542.
 2021-02-11 22:24:39 +09:00
Andrew Gaul
a4d916af13
Allow configuring multipart copy size (#1555) 
Also align configuration with mixupload.
 2021-02-08 20:32:12 +09:00
Andrew Gaul
1a6d0826b5
Allow -1 value to disable max_dirty_data (#1552) 
This matches the man page.
 2021-02-07 22:14:31 +09:00
Andrew Gaul
67b9381825 Remove limit for single-part objects 
AWS enforces a 5 GB limit for single-part objects but other S3
implementations like GCS support larger objects.  Fixes #1542.
 2021-02-04 09:19:37 +09:00
Carsten Grohmann
ed85b72bf5 Add warning for failing metadata updates of large files 
Metadata updates fail for large files if "-o nocopyapi" or
"-o nomultipart" is set.

This change is related to #1528
 2021-01-28 10:20:32 +09:00
Andrew Gaul
d019dda4f7
Simplify substr manipulations with erase (#1532) 
This avoids creating a new std::string.
 2021-01-25 18:02:32 +09:00
Andrew Gaul
b70f8db037
Propagate errno from get_local_fent (#1525) 
Follows on to 4d833a4fb9.
References #1523.
 2021-01-20 23:21:14 +09:00
Andrew Gaul
4d833a4fb9
Return more specific errno when available (#1520) 
Previously s3fs threw away some function return values and returned
EIO instead.  This was due to not trusting the mix of -1 and errno
return codes.  Correct the obviously incorrect ones via visual
inspection.  Stronger typing may find more occurrences.  Fixes #1519.
 2021-01-18 18:50:49 +09:00
kontrollanten
d1c638ab7a fix 404 error message 2020-12-30 14:10:17 +09:00
Takeshi Nakatani
d9f6469b7b Fixed flushing dirty data and compressed the cache size 2020-11-14 16:45:37 +09:00
Noah Meyerhans
f2f930300a Implement AWS IMDSv2 support 
AWS IMDSv2 is a session oriented method for retrieving instance metadata,
including IAM credentials, in Amazon EC2.  It is enabled by default in
non-enforcing mode in AWS (meaning it retains backwards compatibility with
existing IMDSv1 clients), but can be switched to enforcing mode, in which
clients are required to return API tokens with requests.

With this change, we implement support for IMDSv2 and enable it by default when
IAM roles are our source for authentication credentials.  In the event that
s3fs is running in cloud environment offering an IMDSv1-compatible API, we
support graceful fallback to that mode.  It can also be selected explicitly via
the imdsv1only mount option.

More details on IMDSv2 are available at
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
and
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/

Signed-off-by: Noah Meyerhans <nmeyerha@amazon.com>
 2020-11-07 14:48:52 +09:00
yoshihiko
81ad3ce0ae Fix directory authority 2020-11-07 14:47:53 +09:00
Takeshi Nakatani
38e1eaa8a3 Added logfile option for non-syslog logging 2020-10-19 20:10:19 +09:00
Andrew Gaul
6aa786b886
Flush dirty data after a writing a number of bytes (#1448) 
This allows s3fs to write large files without consuming a large amount
of temporary local storage but can slow uploads due to server-side
copies.  References #617.  Fixed #1056.  Fixes #1257.
 2020-10-13 22:30:42 +09:00
Takeshi Nakatani
58750cc441 Fixed a bug that symlink could not be read after restarting s3fs 2020-10-11 22:14:36 +09:00
Takeshi Nakatani
059cc57ba6 Added atime and Corrected atime/mtime/ctime operations 2020-10-04 13:54:01 +09:00
Takeshi Nakatani
6e1e0d1d31
Merge pull request #1432 from gaul/v2-or-v4 
Add sigv4 flag and fix v4 regression
 2020-10-03 00:09:55 +09:00
Takeshi Nakatani
66419e7292
Merge pull request #1431 from gaul/string/find_first_of 
Prefer string::find_first_of(char) where possible
 2020-10-02 23:42:17 +09:00
Andrew Gaul
081d6c1245 Allow setting signature V4-only 
Default to allowing V2 or V4.
 2020-10-01 20:03:14 +09:00
Andrew Gaul
eb8004c355 Prefer string::find_first_of(char) where possible 2020-10-01 18:24:45 +09:00
Andrew Gaul
503c86bb8a Call is_prefix instead of compare and substr 2020-09-27 22:19:54 +09:00
Takeshi Nakatani
0b42e08636
Merge pull request #1422 from gaul/c++/append-assign-at 
Remove calls to append, assign, and at
 2020-09-27 10:59:48 +09:00
Neeraj Kumar Kashyap
041b4ec05c enforce exact key match in IBM IAM response 2020-09-25 22:41:19 +09:00
Andrew Gaul
2438066d52 Remove calls to append, assign, and at 
operator+, operator=, and operator[] are more idiomatic and consistent
with the code base.
 2020-09-25 12:19:51 +09:00
Takeshi Nakatani
3440c3348c
Merge pull request #1405 from gaul/is-prefix 
Rename STR2NCMP to is_prefix for clarity
 2020-09-20 10:03:53 +09:00
Andrew Gaul
ebcbb02d16 Rename STR2NCMP to is_prefix for clarity 
This may prevent confusion seen in
39ec8e242e.
 2020-09-17 19:45:54 +09:00
yongqingliu
211cc0f5f2 fix dead lock in disk insufficient and optimize code 2020-09-16 22:45:28 +08:00
Takeshi Nakatani
3958450c05 Added a class for automating fdentity reference counts 2020-09-14 22:02:28 +09:00
Takeshi Nakatani
8205607716 Fixed incorrect Signal related message at s3fs initialization and termination 2020-09-13 18:54:10 +09:00
Andrew Gaul
1043e08dfa Remove uses of implicit namespace std 
Fixed via:

sed -i '/using namespace std/{N;d}' src/*.cpp
sed -i 's/ string/ std::string/g' src/*.cpp
sed -i 's/(string/(std::string/g' src/*.cpp
sed -i 's/\[string/\[std::string/g' src/*.cpp
sed -i 's/^string/std::string/g' src/*.cpp
sed -i 's/ ifstream/ std::ifstream/g' src/*.cpp
sed -i 's/ istringstream/ std::istringstream/g' src/*.cpp
sed -i 's/ ostringstream/ std::ostringstream/g' src/*.cpp
sed -i 's/ max(/ std::max(/g' src/*.cpp
sed -i 's/ min(/ std::min(/g' src/*.cpp
sed -i 's/ endl/ std::endl/g' src/*.cpp
 2020-09-13 11:57:20 +09:00
Andrew Gaul
39ec8e242e Remove incorrect string prefix comparisons 
Previously flags like -o nouafoobar were parsed as -o noua.  Found
via:

grep STR2NCMP src/s3fs.cpp | grep -v '="'
 2020-09-11 17:38:20 +09:00
Ben Mares
0d4e39ad1c
return 0 after parsing "use_session_token" arg (#1388) 
return 0 after parsing "use_session_token" arg

I hope this might fix #651.

There's clearly an open bug in #651 confirmed by several users related to the parsing of the `-o use_session_token`. Looking at the source, I noticed that there was a return value everywhere except here, so I suspect this may be responsible.
 2020-09-11 08:09:36 +09:00
yongqingliu
986fab8738 close FdEntity to avoid leakage 2020-09-07 20:48:16 +08:00
Takeshi Nakatani
b5ffd419d8 Source file division and set 4 spaces and cleanup 2020-08-26 17:43:50 +09:00
Andrew Gaul
60d456a993 Use scoped enums for acl_t, sse_type_t, and storage_class_t 
This prevents some kinds of implicit conversions.  Also deduplicate
str/from_str logic.  References #1371.
 2020-08-20 18:46:18 +09:00
Andrew Gaul
958ad83a4b Correct vim modeline to 2-space indentation 2020-08-19 21:03:46 +09:00
Takeshi Nakatani
057da86d87
Merge pull request #1356 from gaul/printf-size_t 
Prefer %zd and %zu for ssize_t and size_t
 2020-08-16 18:16:59 +09:00
Takeshi Nakatani
2bb745cdd7 Fixed a bug about move file over limit of ensure space 2020-08-16 17:42:20 +09:00
Andrew Gaul
132a1bebbb Prefer %zd and %zu for ssize_t and size_t 
This removes unnecessary casting.
 2020-08-15 17:30:23 +09:00
Takeshi Nakatani
cbd925c56f Moved the SIGUSR2 handler to S3fsSignals class 2020-07-28 14:54:35 +09:00
Takeshi Nakatani
ad1961417d Added SIGUSR1 option for cache file integrity test 2020-07-26 21:04:11 +09:00
Takeshi Nakatani
1f796d432d Fixed upload error about mixuploading sparse file and truncating file 2020-06-27 22:44:19 +09:00
Eric Vantillard
7d0c66e08a Add support for glacier storage class. 
Just a copy of what have been done in PR #271.
 2020-06-23 11:23:21 +09:00
Andrew Gaul
66597ec5f2 Fix renames of open files with nocopyapi option 
References #1296.
 2020-05-30 15:45:43 +09:00
Takeshi Nakatani
75e72385cc Added a parameter to output body to curldbg option 2020-05-25 08:49:01 +09:00
Andrew Gaul
8b15db6dcb Do not allow zero retries 
Retries actually means tries, e.g., if the user sets zero, s3fs will
never try an operation at all.
 2020-05-23 10:05:23 +09:00
Takeshi Nakatani
97c249d5b9 Not abort process by exception threw from s3fs_strtoofft 2020-05-03 08:08:28 +00:00
Andrew Gaul
2cc88b933f Warn about missing MIME types instead of exiting 
s3fs uses the MIME types file to set Content-Type for uploaded
objects.  Most distribution packages should install this via
recommended (not required) dependencies.  Users compiling from source
may not have this installed and s3fs should not prevent launching
since most users do not care about Content-Type.  Instead warn about
MIME types absence.  Fixes #1270.
 2020-04-29 20:03:50 +09:00
Takeshi Nakatani
7fbda230f5 Added mime option for strict checking of mime types file 2020-03-30 14:41:18 +00:00
Andrew Gaul
924eeb3587 Document host and servicepath 
The various bits of host, url, and servicepath seem to overlap.
References #1203.
 2020-02-04 21:13:29 +09:00
Andrew Gaul
bc9126d774 Set directory MIME type to application/x-directory 
Previously s3fs auto-detected the MIME type of directories like
"TOYOTA TRUCK 8.2.2" as application/x-troff-man.  This caused get_mode
to not set S_IFDIR which failed directory creation.  Instead force all
object names ending in / to application/x-directory.  Fixes #1183.
 2020-02-04 20:03:21 +09:00
Andrei Precup
39102608aa Add intelligent tiering as option for storage class 
Resolves Issue #1219
 2020-01-13 20:24:35 +09:00
Takeshi Nakatani
23945a0130 In memory cache for softlinks with cache out 2020-01-13 20:23:10 +09:00
Alvaro Huarte
a24f78f5a4 Adds requester_pays option to enable requests involving Requester Pays buckets 2019-11-18 12:38:16 +01:00
Takeshi Nakatani
cf3e82d10a Fixed a rename bug when enable_noobj_cache 2019-10-23 12:39:42 +00:00
Takeshi Nakatani
1db94a0b30 Fixed to use copy api in multipart upload 2019-09-25 19:30:58 -07:00
Takeshi Nakatani
bedd648d47 Fixed build error by cppcheck 1.89 2019-09-23 10:49:49 +00:00
Takeshi Nakatani
42fb30852b
Merge pull request #1145 from gaul/bool 
Fix a few bool types
 2019-09-08 20:04:30 +09:00
Andrew Gaul
805cc064af Fix a few bool types 
Found via clang-tidy.
 2019-09-05 10:42:42 -07:00
Andrew Gaul
412876ca33 Expose base parameter in s3fs_strtoofft 
This fixes a regression from ccf3e7bfa2
which caused the misparsing of octal inputs for the mp_umask and umask
flags.  It also allows some callers to be more precise about their
decimal inputs.
 2019-09-04 20:41:47 -07:00
Takeshi Nakatani
1c3507ede1
Merge pull request #1135 from gaul/curl-timeout 
Remove useless CURLE_OPERATION_TIMEDOUT check
 2019-08-25 17:53:02 +09:00
Andrew Gaul
01ac815346 Remove useless CURLE_OPERATION_TIMEDOUT check 
CheckBucket only returns errno and HTTP status codes.
 2019-08-20 18:44:48 -07:00
Andrew Gaul
cc2eed84a5 Promote default ACL to enum 
This sanity checks ACLs during initialization and also omits sending
the header when set to PRIVATE which is the default.
 2019-08-20 15:05:58 -07:00
Takeshi Nakatani
5c4a0a862a Fixed multipart copy and its retry handler 2019-08-11 16:21:26 +00:00
Takeshi Nakatani
f74c7407db
Merge pull request #1118 from gaul/atoi 
Prefer s3fs_strtoofft over atoi and strtol
 2019-08-06 21:23:37 +09:00
Andrew Gaul
ccf3e7bfa2 Prefer s3fs_strtoofft over atoi and strtol 
The former propagates errors consistently.
 2019-08-03 16:13:48 -07:00
Andrew Gaul
43d1439420 Consume return value from FdEntity::Flush 
Found via Coverity.
 2019-08-02 10:01:06 -07:00
Andrew Gaul
fb937635f5 Eagerly initialize sysconf variables 
Previously s3fs had races updating these shared variables.  Found via
ThreadSanitizer.
 2019-07-17 09:08:13 -07:00
Andrew Gaul
ffac4c8417 Avoid narrowing time_t and off_t args in logging 
This displays correct results on 32-bit platforms.
 2019-07-14 17:02:36 -07:00
Takeshi Nakatani
5bbcd3b981
Merge pull request #1089 from gaul/null-dereference 
Avoid null dereference
 2019-07-15 04:17:10 +09:00
Andrew Gaul
eb597289cb Avoid null dereference 
Found via clang-tidy.
 2019-07-12 18:40:24 -07:00
Andrew Gaul
6fd42d9fe4 Prefer modern C headers 
Found and fixed via clang-tidy.
 2019-07-12 03:50:59 -07:00
Takeshi Nakatani
7a65a414c3
Merge pull request #1080 from gaul/xml/simplification 
Add simple XML parsing wrapper
 2019-07-07 16:44:18 +09:00
Takeshi Nakatani
6bd1a7eac0
Merge pull request #1074 from gaul/clang-tidy 
Configure clang-tidy target
 2019-07-07 15:28:08 +09:00
Andrew Gaul
b8ff6a647e Add simple XML parsing wrapper 
Also simplify check_region_error.
 2019-07-05 17:39:09 -07:00
Andrew Gaul
4f23f38583 Individually test multiple s3fs flags 
Remove unneeded comments; single part limits ensure that the tests
exercise multipart code paths even with smaller files.
References #971.
 2019-07-03 21:09:40 -07:00
Andrew Gaul
bbfa91141a Configure clang-tidy target 
Also fix nits.
 2019-07-03 14:04:11 -07:00
Andrew Gaul
c596441f58 Flush file when opening second fd 
Previously when s3fs had dirty local data and an application opened a
second fd it would remove the stat cache entry, query S3 for the size
which was still zero, then reinitialize FdEntry with this incorrect
size.  Instead flush local data to S3 so this query works.  It is
possible that a more involved patch could do this with a less
heavyweight approach but this requires changing open.  This does not
completely fix git clone but allows it to proceed further.  Also make
some cache methods const-correct.  References #839.
 2019-07-02 01:12:09 -07:00
Andrew Gaul
f1ad626b46 Log flags as hexadecimal 2019-06-24 16:13:07 -07:00
Andrew Gaul
15b7450713 Convert file offsets from size_t to off_t 
The latter is 64-bits on 32-bit platforms when specifying
-D_FILE_OFFSET_BITS=64.  This allows early Raspberry Pis to use files
larger than 2 GB.  It also cleans up some ugly casting.  Fixes #620.
Fixes #656.
 2019-06-15 17:05:37 -07:00
Andrew Gaul
059ab1f0f4 Update ctime during nocopyapi operations 
Follows on to 2c43b1e12b.  Fixes #971.
 2019-04-30 18:59:51 +09:00
Michal Lula
0d4847596e fix issue with aws session token provided inside ~/.aws/credentials file 2019-04-25 14:29:35 +02:00
Andrew Gaul
9e4f9d4bdc Remove uses of std::map::at 
This provides compatibility with C++03 and removes duplicate lookups.
 2019-04-18 16:06:59 +09:00
Michal Lula
f9cd43b684 add session token support 2019-04-16 16:53:05 +02:00
Takeshi Nakatani
cd280d8702
Merge pull request #1001 from gaul/missing-braces 
Add missing braces
 2019-04-08 23:55:02 +09:00
Andrew Gaul
042332bcec Add missing braces 
Found via clang-tidy.  Also fix errant indentation.
 2019-04-07 23:12:27 +09:00
Takeshi Nakatani
071cd0f849 Modified to discard _netdev option etc 2019-04-07 11:51:55 +00:00
Andrew Gaul
6f6a67807b Prefer new over malloc 
The former cannot return NULL and will allow use of scoped pointers.
 2019-04-07 16:54:24 +09:00
Takeshi Nakatani
ea517c80a4 Fixed a bug about overwrite metadata at updating directory stats 2019-03-29 15:30:30 +00:00
Andrew Gaul
058706014b Simplify string comparison 
Found via newer cppcheck.
 2019-03-14 22:19:33 +09:00
Takeshi Nakatani
cbf072bc55
Merge pull request #967 from gaul/reference-count 
Increase FdEntity reference count when returning
 2019-03-10 14:27:10 +09:00
Andrew Gaul
08b132ddb9 Remove from file from stat cache during rename 
This addresses failures with test_rename_before_close when using
nocopyapi.  Note that test_update_time still fails.
 2019-03-09 17:53:35 +09:00
Andrew Gaul
f53503438c Increase FdEntity reference count when returning 
Previously s3fs had a race condition where one client could delete
FdEntity that another client was using.  Add a simple concurrent test
which previously failed but now succeeds.  Fixes #964.
 2019-03-09 15:55:04 +09:00
Kristján Valur Jónsson
3c97c1b251 merged main 2019-02-19 10:53:00 +00:00
Kristján Valur Jónsson
84c671a81a fix indentation 2019-02-19 10:37:43 +00:00
Kristján Valur Jónsson
f336bdebcc add command line flag and documentation 2019-02-19 10:32:37 +00:00
Kristján Valur Jónsson
4f42f4ab0c Enable s3fs encoding and decoding in the fuse interface 2019-02-18 13:36:48 +00:00
Takeshi Nakatani
eeb839242b Fixed code for latest cppcheck error on OSX 2019-02-17 13:59:11 +00:00
Takeshi Nakatani
231fd001d9 Added a non-interactive option to utility mode 
Rebase for resolving conflicts

Deleted interactive mode on utility mode

Fixed minor mistakes
 2019-02-10 17:36:18 +00:00
Takeshi Nakatani
314dc5a398
Merge pull request #955 from gaul/clang-tidy/misc 
Address miscellaneous clang-tidy warnings
 2019-02-06 21:21:54 +09:00
Andrew Gaul
e87e40b3b4 Address miscellaneous clang-tidy warnings 
Also add configuration with suppressions.
 2019-02-04 22:58:40 -08:00
Andrew Gaul
4d0bef1e90 Clear containers instead of individual erases 
This has O(n) runtime instead of O(n log n).
 2019-02-02 23:58:43 -08:00
Takeshi Nakatani
d8185a25aa
Merge pull request #946 from gaul/async/completed-tids 
Simplify async request completion code
 2019-02-03 14:08:49 +09:00
Takeshi Nakatani
53337a0a28
Merge pull request #944 from gaul/utility-mode 
Repair utility mode
 2019-02-03 13:59:56 +09:00
Takeshi Nakatani
ae51556d04
Merge pull request #943 from gaul/hard-link 
Return not supported when hard linking
 2019-02-03 12:45:21 +09:00
Takeshi Nakatani
055ecf6ea7
Merge pull request #940 from gaul/parallel-multipart-copy 
Copy parts in parallel
 2019-02-03 11:44:15 +09:00
Andrew Gaul
8c527c3616 Simplify async request completion code 
Workers now notify the master thread when they complete, unifying the
Linux and macOS code paths.  This also avoids excessive
pthread_tryjoin_np calls.  Follows on to
88cd8feb05.
 2019-01-31 22:55:23 -08:00
Andrew Gaul
54a074647e Repair utility mode 
This deinitialized S3fsCurl twice and incorrectly calculated V4
signatures.
 2019-01-31 18:45:39 -08:00
Andrew Gaul
c5ebf5d328 Copy parts in parallel 
S3 can copy multipart much faster than single part due to IO
parallelization.  Renaming a 4 GB file reduces from 72 to 20 seconds
with bigger gains with larger files.
 2019-01-31 10:21:39 -08:00
Andrew Gaul
43c6ef560e Return not supported when hard linking 
This is more correct than permission denied.
 2019-01-30 16:43:04 -08:00
Andrew Gaul
35d55ee513 Remove unneeded void parameter 
This is implicit in C++.  Found and fixed via clang-tidy.
 2019-01-28 23:22:27 -08:00
Takeshi Nakatani
fd4d23f8f7
Merge pull request #926 from kzidane/master 
Accept paths with : in them
 2019-01-27 15:23:24 +09:00
Takeshi Nakatani
4820f0a42b
Merge pull request #925 from gaul/clang-tidy/delete-null 
Remove redundant null checks before delete
 2019-01-27 15:15:31 +09:00
Kareem Zidane
cd794a6985 Accept paths with : in them 2019-01-23 14:56:25 -05:00
Andrew Gaul
84b421d6ef Prefer empty over size checks 
Found and fixed via clang-tidy.
 2019-01-23 11:30:28 -08:00
Andrew Gaul
8316da5bbe Remove redundant null checks before delete 
Found by clang-tidy.
 2019-01-23 11:25:25 -08:00
Takeshi Nakatani
caaf4cac55 Reverted #912(Automatic region change) and added message 2019-01-23 13:23:03 +00:00
Andrew Gaul
1fc25e8c3f Remove redundant string initializations 
Found and fixed via clang-tidy.
 2019-01-22 23:16:37 -08:00
Andrew Gaul
46d79c5bc2 Issue readdir HEAD requests without batching 
Previously s3fs would issue a batch of HEAD requests and wait for all
to succeed before issuing the next batch.  Now it issues the first
batch and only waits for a single call to succeed before issuing the
next call.  This can improve performance when one call lags due to
network errors.  I measured 25% improvement with the same level of
parallelism.  This commit also reparents parallelism knobs for
consistency.  Follows on to 88cd8feb05.
Fixes #223.
 2019-01-20 18:07:22 -08:00
Takeshi Nakatani
2887f8916b Automatic region change made possible other than us-east-1(default) 2019-01-20 10:51:49 +00:00
Takeshi Nakatani
05014c49c8
Merge pull request #906 from gaul/doc/https 
Prefer HTTPS links where possible
 2019-01-20 15:10:51 +09:00
Takeshi Nakatani
6aa40b2747
Merge pull request #903 from gaul/clang-tidy/find_char 
Prefer find(char) over find(const char *)
 2019-01-20 14:05:06 +09:00
Andrew Gaul
aa9bd1fa3c Prefer HTTPS links where possible 
Fix a few stale links as well.
 2019-01-18 11:09:08 -08:00
Andrew Gaul
e29548178b Prefer find(char) over find(const char *) 
The former can be faster.  Found via clang-tidy.
 2019-01-17 20:24:24 -08:00
Andrew Gaul
6401b4ae92 Flush file before renaming 
Previously s3fs could copy the zero-byte stub object without including
any pending writes.  Fixes #145.
 2019-01-17 12:05:10 -08:00
Takeshi Nakatani
c7def35b54
Merge pull request #895 from gaul/bucket-name-dot 
Emit more friendly error for buckets with dots
 2019-01-16 18:39:47 +09:00
Andrew Gaul
c512516e14 Emit more friendly error for buckets with dots 
These fail SSL certificate checks due to the *.s3.amazon.com wildcard.
Fixes #284.
 2019-01-14 18:47:36 -08:00
Andrew Gaul
2c43b1e12b Store and retrieve file change time 
This introduces a new header with the change time; existing objects
will report modification time.  Fixes #771.
 2019-01-14 10:05:11 -08:00
Takeshi Nakatani
fd6b37d3da
Merge pull request #877 from gaul/aws/credentials 
Check arguments and environment before .aws/creds
 2019-01-06 17:03:11 +09:00