octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-10 23:00:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into macosx

Browse Source
This commit is contained in:
Takeshi Nakatani 2015-11-08 06:06:05 +00:00
commit e1de134d94
33 changed files with 3379 additions and 2039 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
.travis.yml
View File

@ -1,17 +1,16 @@
language: cpp language: cpp
sudo: required
dist: trusty dist: trusty
cache: apt cache: apt
before_install: before_install:
- sudo apt-get update -qq - sudo apt-get update -qq
- sudo apt-get install -qq libfuse-dev - sudo apt-get install -qq cppcheck libfuse-dev openjdk-7-jdk
script: script:
- ./autogen.sh - ./autogen.sh
- ./configure - ./configure
- make - make
- make cppcheck
- make check -C src - make check -C src
# Travis granted s3fs access to their upcoming alpha testing stack which may - modprobe fuse
# allow us to use FUSE. - make check -C test
# TODO: Travis changed their infrastructure some time in June 2015 such that - cat test/test-suite.log
# this does not work currently
#- modprobe fuse
#- make check -C test

11
Makefile.am
View File

@ -19,7 +19,7 @@
###################################################################### ######################################################################
SUBDIRS=src test doc SUBDIRS=src test doc
EXTRA_DIST=doc EXTRA_DIST=doc default_commit_hash
dist-hook: dist-hook:
rm -rf `find $(distdir)/doc -type d -name .svn` rm -rf `find $(distdir)/doc -type d -name .svn`
@ -28,3 +28,12 @@ dist-hook:
release : dist ../utils/release.sh release : dist ../utils/release.sh
../utils/release.sh $(DIST_ARCHIVES) ../utils/release.sh $(DIST_ARCHIVES)
cppcheck:
cppcheck --quiet --error-exitcode=1 \
-U CURLE_PEER_FAILED_VERIFICATION \
--enable=all \
--suppress=missingIncludeSystem \
--suppress=unsignedLessThanZero \
--suppress=unusedFunction \
--suppress=variableScope \
src/ test/

0
NEWS
View File

67
README
View File

@ -1,67 +0,0 @@
THIS README CONTAINS OUTDATED INFORMATION - please refer to the wiki or --help
S3FS-Fuse
S3FS is FUSE (File System in User Space) based solution to mount/unmount an Amazon S3 storage buckets and use system commands with S3 just like it was another Hard Disk.
In order to compile s3fs, You'll need the following requirements:
* Kernel-devel packages (or kernel source) installed that is the SAME version of your running kernel
* LibXML2-devel packages
* CURL-devel packages (or compile curl from sources at: curl.haxx.se/ use 7.15.X)
* GCC, GCC-C++
* pkgconfig
* FUSE (>= 2.8.4)
* FUSE Kernel module installed and running (RHEL 4.x/CentOS 4.x users - read below)
* OpenSSL-devel (0.9.8)
GnuTLS(gcrypt and nettle)
NSS
* Git
If you're using YUM or APT to install those packages, then it might require additional packaging, allow it to be installed.
Downloading & Compiling:
------------------------
In order to download s3fs, download from following url:
https://github.com/s3fs-fuse/s3fs-fuse/archive/master.zip
Or clone the following command:
git clone git://github.com/s3fs-fuse/s3fs-fuse.git
Go inside the directory that has been created (s3fs-fuse) and run: ./autogen.sh
This will generate a number of scripts in the project directory, including a configure script which you should run with: ./configure
If configure succeeded, you can now run: make. If it didn't, make sure you meet the dependencies above.
This should compile the code. If everything goes OK, you'll be greeted with "ok!" at the end and you'll have a binary file called "s3fs"
in the src/ directory.
As root (you can use su, su -, sudo) do: "make install" -this will copy the "s3fs" binary to /usr/local/bin.
Congratulations. S3fs is now compiled and installed.
Usage:
------
In order to use s3fs, make sure you have the Access Key and the Secret Key handy. (refer to the wiki)
First, create a directory where to mount the S3 bucket you want to use.
Example (as root): mkdir -p /mnt/s3
Then run: s3fs mybucket[:path] /mnt/s3
This will mount your bucket to /mnt/s3. You can do a simple "ls -l /mnt/s3" to see the content of your bucket.
If you want to allow other people access the same bucket in the same machine, you can add "-o allow_other" to read/write/delete content of the bucket.
You can add a fixed mount point in /etc/fstab, here's an example:
s3fs#mybucket /mnt/s3 fuse allow_other 0 0
This will mount upon reboot (or by launching: mount -a) your bucket on your machine.
If that does not work, probably you should specify with "_netdev" option in fstab.
All other options can be read at: https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon
Known Issues:
-------------
s3fs should be working fine with S3 storage. However, There are couple of limitations:
* Currently s3fs could hang the CPU if you have lots of time-outs. This is *NOT* a fault of s3fs but rather libcurl. This happens when you try to copy thousands of files in 1 session, it doesn't happen when you upload hundreds of files or less.
* CentOS 4.x/RHEL 4.x users - if you use the kernel that shipped with your distribution and didn't upgrade to the latest kernel RedHat/CentOS gives, you might have a problem loading the "fuse" kernel. Please upgrade to the latest kernel (2.6.16 or above) and make sure "fuse" kernel module is compiled and loadable since FUSE requires this kernel module and s3fs requires it as well.
* Moving/renaming/erasing files takes time since the whole file needs to be accessed first. A workaround could be to use s3fs's cache support with the use_cache option.

108
README.md Normal file
View File

@ -0,0 +1,108 @@
s3fs
====
s3fs allows Linux and Mac OS X to mount an S3 bucket via FUSE.
s3fs preserves the native object format for files, allowing use of other tools like [s3cmd](http://s3tools.org/s3cmd).
Features
--------
* large subset of POSIX including reading/writing files, directories, symlinks, mode, uid/gid, and extended attributes
* compatible with Amazon S3, Google Cloud Storage, and other S3-based object stores
* large files via multi-part upload
* renames via server-side copy
* optional server-side encryption
* data integrity via MD5 hashes
* in-memory metadata caching
* local disk data caching
* user-specified regions, including Amazon GovCloud
* authenticate via v2 or v4 signatures
Installation
------------
Ensure you have all the dependencies:
On Ubuntu 14.04:
```
sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
```
On CentOS 7:
```
sudo yum install automake fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
```
Compile from master via the following commands:
```
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make
sudo make install
```
Examples
--------
Enter your S3 identity and credential in a file `/path/to/passwd`:
```
echo MYIDENTITY:MYCREDENTIAL > /path/to/passwd
```
Make sure the file has proper permissions (if you get 'permissions' error when mounting) `/path/to/passwd`:
```
chmod 600 /path/to/passwd
```
Run s3fs with an existing bucket `mybucket` and directory `/path/to/mountpoint`:
```
s3fs mybucket /path/to/mountpoint -o passwd_file=/path/to/passwd
```
If you encounter any errors, enable debug output:
```
s3fs mybucket /path/to/mountpoint -o passwd_file=/path/to/passwd -d -d -f -o f2 -o curldbg
```
You can also mount on boot by entering the following line to `/etc/fstab`:
```
s3fs#mybucket /path/to/mountpoint fuse _netdev,allow_other 0 0
```
Limitations
-----------
Generally S3 cannot offer the same performance or semantics as a local file system. More specifically:
* random writes or appends to files require rewriting the entire file
* metadata operations such as listing directories have poor performance due to network latency
* [eventual consistency](https://en.wikipedia.org/wiki/Eventual_consistency) can temporarily yield stale data
* no atomic renames of files or directories
* no coordination between multiple clients mounting the same bucket
* no hard links
References
----------
* [s3backer](https://github.com/archiecobbs/s3backer) - mount an S3 bucket as a single file
* [s3fs-python](https://fedorahosted.org/s3fs/) - an older and less complete implementation written in Python
* [S3Proxy](https://github.com/andrewgaul/s3proxy) - combine with s3fs to mount EMC Atmos, Microsoft Azure, and OpenStack Swift buckets
* [s3ql](https://bitbucket.org/nikratio/s3ql/) - similar to s3fs but uses its own object format
* [YAS3FS](https://github.com/danilop/yas3fs) - similar to s3fs but uses SNS to allow multiple clients to mount a bucket
License
-------
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
Licensed under the GNU GPL version 2

22
autogen.sh
View File

@ -19,6 +19,28 @@
# #
# See the file ChangeLog for a revision history. # See the file ChangeLog for a revision history.
echo "--- Make commit hash file -------"
SHORTHASH="unknown"
type git > /dev/null 2>&1
if [ $? -eq 0 -a -d .git ]; then
RESULT=`git rev-parse --short HEAD`
if [ $? -eq 0 ]; then
SHORTHASH=${RESULT}
fi
fi
echo ${SHORTHASH} > default_commit_hash
echo "--- Finished commit hash file ---"
echo "--- Start autotools -------------"
aclocal \ aclocal \
&& autoheader \
&& automake --add-missing \ && automake --add-missing \
&& autoconf && autoconf
echo "--- Finished autotools ----------"
exit 0

40
configure.ac
View File

@ -21,15 +21,19 @@ dnl Process this file with autoconf to produce a configure script.
AC_PREREQ(2.59) AC_PREREQ(2.59)
AC_INIT(s3fs, 1.79) AC_INIT(s3fs, 1.79)
AC_CONFIG_HEADER([config.h])
AC_CANONICAL_SYSTEM AC_CANONICAL_SYSTEM
AM_INIT_AUTOMAKE() AM_INIT_AUTOMAKE([foreign])
AC_PROG_CXX AC_PROG_CXX
AC_PROG_CC AC_PROG_CC
CXXFLAGS="$CXXFLAGS -Wall -D_FILE_OFFSET_BITS=64" CXXFLAGS="$CXXFLAGS -Wall -D_FILE_OFFSET_BITS=64"
dnl ----------------------------------------------
dnl For OSX
dnl ----------------------------------------------
case "$target" in case "$target" in
*-darwin* ) *-darwin* )
# Do something specific for mac # Do something specific for mac
@ -219,12 +223,40 @@ AM_CONDITIONAL([USE_GNUTLS_NETTLE], [test "$auth_lib" = nettle])
AM_CONDITIONAL([USE_SSL_NSS], [test "$auth_lib" = nss]) AM_CONDITIONAL([USE_SSL_NSS], [test "$auth_lib" = nss])
dnl ---------------------------------------------- dnl ----------------------------------------------
dnl end of ssl library dnl check functions
dnl ---------------------------------------------- dnl ----------------------------------------------
dnl malloc_trim function dnl malloc_trim function
AC_CHECK_FUNCS(malloc_trim, , ) AC_CHECK_FUNCS([malloc_trim])
dnl ----------------------------------------------
dnl output files
dnl ----------------------------------------------
AC_CONFIG_FILES(Makefile src/Makefile test/Makefile doc/Makefile) AC_CONFIG_FILES(Makefile src/Makefile test/Makefile doc/Makefile)
dnl ----------------------------------------------
dnl short commit hash
dnl ----------------------------------------------
AC_CHECK_PROG([GITCMD], [git —version], [yes], [no])
AC_CHECK_FILE([.git], [DOTGITDIR=yes], [DOTGITDIR=no])
AC_MSG_CHECKING([github short commit hash])
if test “x${GITCMD}” = “xyes” -a “x${DOTGITDIR}” = “xyes”; then
GITCOMMITHASH=`git rev-parse --short HEAD`
elif test -f default_commit_hash; then
GITCOMMITHASH=`cat default_commit_hash`
else
GITCOMMITHASH="unknown"
fi
AC_MSG_RESULT([${GITCOMMITHASH}])
AC_DEFINE_UNQUOTED([COMMIT_HASH_VAL], ["${GITCOMMITHASH}"], [short commit hash value on github])
dnl ----------------------------------------------
dnl put
dnl ----------------------------------------------
AC_OUTPUT AC_OUTPUT
dnl ----------------------------------------------
dnl end configuration
dnl ----------------------------------------------

66
doc/man/s3fs.1
View File

@ -62,19 +62,41 @@ local folder to use for local file cache.
\fB\-o\fR del_cache - delete local file cache \fB\-o\fR del_cache - delete local file cache
delete local file cache when s3fs starts and exits. delete local file cache when s3fs starts and exits.
.TP .TP
\fB\-o\fR storage_class (default is standard)
store object with specified storage class.
this option replaces the old option use_rrs.
Possible values: standard, standard_ia, and reduced_redundancy.
.TP
\fB\-o\fR use_rrs (default is disable) \fB\-o\fR use_rrs (default is disable)
use Amazon's Reduced Redundancy Storage. use Amazon's Reduced Redundancy Storage.
this option can not be specified with use_sse. this option can not be specified with use_sse.
(can specify use_rrs=1 for old version) (can specify use_rrs=1 for old version)
this option has been replaced by new storage_class option.
.TP .TP
\fB\-o\fR use_sse (default is disable) \fB\-o\fR use_sse (default is disable)
use Amazon's Server-Site Encryption or Server-Side Encryption with Customer-Provided Encryption Keys. Specify three type Amazon's Server-Site Encryption: SSE-S3, SSE-C or SSE-KMS. SSE-S3 uses Amazon S3-managed encryption keys, SSE-C uses customer-provided encryption keys, and SSE-KMS uses the master key which you manage in AWS KMS.
this option can not be specified with use_rrs. specifying only "use_sse" or "use_sse=1" enables Server-Side Encryption.(use_sse=1 for old version) You can specify "use_sse" or "use_sse=1" enables SSE-S3 type (use_sse=1 is old type parameter).
specifying this option with file path which has some SSE-C secret key enables Server-Side Encryption with Customer-Provided Encryption Keys.(use_sse=file) Case of setting SSE-C, you can specify "use_sse=custom", "use_sse=custom:<custom key file path>" or "use_sse=<custom key file path>"(only <custom key file path> specified is old type parameter).
the file must be 600 permission. the file can have some lines, each line is one SSE-C key. the first line in file is used as Customer-Provided Encryption Keys for uploading and change headers etc. You can use "c" for short "custom".
if there are some keys after first line, those are used downloading object which are encripted by not first key. The custom key file must be 600 permission. The file can have some lines, each line is one SSE-C key.
so that, you can keep all SSE-C keys in file, that is SSE-C key history. The first line in file is used as Customer-Provided Encryption Keys for uploading and changing headers etc.
if AWSSSECKEYS environment is set, you can set SSE-C key instead of this option. If there are some keys after first line, those are used downloading object which are encrypted by not first key.
So that, you can keep all SSE-C keys in file, that is SSE-C key history.
If you specify "custom"("c") without file path, you need to set custom key by load_sse_c option or AWSSSECKEYS environment.(AWSSSECKEYS environment has some SSE-C keys with ":" separator.)
This option is used to decide the SSE type.
So that if you do not want to encrypt a object at uploading, but you need to decrypt encrypted object at downloaing, you can use load_sse_c option instead of this option.
For setting SSE-KMS, specify "use_sse=kmsid" or "use_sse=kmsid:<kms id>".
You can use "k" for short "kmsid".
If you san specify SSE-KMS type with your <kms id> in AWS KMS, you can set it after "kmsid:"(or "k:").
If you specify only "kmsid"("k"), you need to set AWSSSEKMSID environment which value is <kms id>.
You must be careful about that you can not use the KMS id which is not same EC2 region.
.TP
\fB\-o\fR load_sse_c - specify SSE-C keys
Specify the custom-provided encription keys file path for decrypting at duwnloading.
If you use the custom-provided encription key at uploading, you specify with "use_sse=custom".
The file has many lines, one line means one custom key.
So that you can keep all SSE-C keys in file, that is SSE-C key history.
AWSSSECKEYS environment is as same as this file contents.
.TP .TP
\fB\-o\fR passwd_file (default="") \fB\-o\fR passwd_file (default="")
specify the path to the password file, which which takes precedence over the password in $HOME/.passwd-s3fs and /etc/passwd-s3fs specify the path to the password file, which which takes precedence over the password in $HOME/.passwd-s3fs and /etc/passwd-s3fs
@ -118,7 +140,8 @@ s3fs always has to check whether file(or sub directory) exists under object(path
It increases ListBucket request and makes performance bad. It increases ListBucket request and makes performance bad.
You can specify this option for performance, s3fs memorizes in stat cache that the object(file or directory) does not exist. You can specify this option for performance, s3fs memorizes in stat cache that the object(file or directory) does not exist.
.TP .TP
\fB\-o\fR no_check_certificate (by default this option is disabled) - do not check ssl certificate. \fB\-o\fR no_check_certificate (by default this option is disabled)
do not check ssl certificate.
server certificate won't be checked against the available certificate authorities. server certificate won't be checked against the available certificate authorities.
.TP .TP
\fB\-o\fR nodnscache - disable dns cache. \fB\-o\fR nodnscache - disable dns cache.
@ -135,26 +158,23 @@ number of parallel request for uploading big objects.
s3fs uploads large object(default:over 20MB) by multipart post request, and sends parallel requests. s3fs uploads large object(default:over 20MB) by multipart post request, and sends parallel requests.
This option limits parallel request count which s3fs requests at once. This option limits parallel request count which s3fs requests at once.
It is necessary to set this value depending on a CPU and a network band. It is necessary to set this value depending on a CPU and a network band.
This option is lated to fd_page_size option and affects it.
.TP
\fB\-o\fR fd_page_size(default="52428800"(50MB))
number of internal management page size for each file descriptor.
For delayed reading and writing by s3fs, s3fs manages pages which is separated from object. Each pages has a status that data is already loaded(or not loaded yet).
This option should not be changed when you don't have a trouble with performance.
This value is changed automatically by parallel_count and multipart_size values(fd_page_size value = parallel_count * multipart_size).
.TP .TP
\fB\-o\fR multipart_size(default="10"(10MB)) \fB\-o\fR multipart_size(default="10"(10MB))
number of one part size in multipart uploading request. number of one part size in multipart uploading request.
The default size is 10MB(10485760byte), this value is minimum size. The default size is 10MB(10485760byte), minimum value is 5MB(5242880byte).
Specify number of MB and over 10(MB). Specify number of MB and over 5(MB).
This option is lated to fd_page_size option and affects it. .TP
\fB\-o\fR ensure_diskfree(default the same as multipart_size value)
sets MB to ensure disk free space. This option means the threshold of free space size on disk which is used for the cache file by s3fs.
s3fs makes file for downloading, and uploading and caching files.
If the disk free space is smaller than this value, s3fs do not use diskspace as possible in exchange for the performance.
.TP .TP
\fB\-o\fR url (default="http://s3.amazonaws.com") \fB\-o\fR url (default="http://s3.amazonaws.com")
sets the url to use to access Amazon S3. If you want to use HTTPS, then you can set url=https://s3.amazonaws.com sets the url to use to access Amazon S3. If you want to use HTTPS, then you can set url=https://s3.amazonaws.com
.TP .TP
\fB\-o\fR endpoint (default="us-east-1") \fB\-o\fR endpoint (default="us-east-1")
sets the endpoint to use. sets the endpoint to use.
If this option is not specified, s3fs uses \"us-east-1\" region as the default. If this option is not specified, s3fs uses "us-east-1" region as the default.
If the s3fs could not connect to the region specified by this option, s3fs could not run. If the s3fs could not connect to the region specified by this option, s3fs could not run.
But if you do not specify this option, and if you can not connect with the default region, s3fs will retry to automatically connect to the other region. But if you do not specify this option, and if you can not connect with the default region, s3fs will retry to automatically connect to the other region.
So s3fs can know the correct region name, because s3fs can find it in an error from the S3 server. So s3fs can know the correct region name, because s3fs can find it in an error from the S3 server.
@ -194,6 +214,14 @@ If this option is specified with nocopapi, the s3fs ignores it.
.TP .TP
\fB\-o\fR use_path_request_style (use legacy API calling style) \fB\-o\fR use_path_request_style (use legacy API calling style)
Enble compatibility with S3-like APIs which do not support the virtual-host request style, by using the older path request style. Enble compatibility with S3-like APIs which do not support the virtual-host request style, by using the older path request style.
.TP
\fB\-o\fR dbglevel (default="crit")
Set the debug message level. set value as crit(critical), err(error), warn(warning), info(information) to debug level. default debug level is critical.
If s3fs run with "-d" option, the debug level is set information.
When s3fs catch the signal SIGUSR2, the debug level is bumpup.
.TP
\fB\-o\fR curldbg - put curl debug message
Put the debug message from libcurl when this option is specified.
.SH FUSE/MOUNT OPTIONS .SH FUSE/MOUNT OPTIONS
.TP .TP
Most of the generic mount options described in 'man mount' are supported (ro, rw, suid, nosuid, dev, nodev, exec, noexec, atime, noatime, sync async, dirsync). Filesystems are mounted with '\-onodev,nosuid' by default, which can only be overridden by a privileged user. Most of the generic mount options described in 'man mount' are supported (ro, rw, suid, nosuid, dev, nodev, exec, noexec, atime, noatime, sync async, dirsync). Filesystems are mounted with '\-onodev,nosuid' by default, which can only be overridden by a privileged user.

2
src/Makefile.am
View File

@ -39,6 +39,6 @@ s3fs_LDADD = $(DEPS_LIBS)
noinst_PROGRAMS = test_string_util noinst_PROGRAMS = test_string_util
test_string_util_SOURCES = string_util.cpp test_string_util.cpp test_string_util_SOURCES = string_util.cpp test_string_util.cpp test_util.h
TESTS = test_string_util TESTS = test_string_util

43
src/cache.cpp
View File

@ -164,11 +164,11 @@ bool StatCache::GetStat(string& key, struct stat* pst, headers_t* meta, bool ove
} }
if(is_delete_cache){ if(is_delete_cache){
// not hit by different ETag // not hit by different ETag
DPRNNN("stat cache not hit by ETag[path=%s][time=%jd][hit count=%lu][ETag(%s)!=(%s)]", S3FS_PRN_DBG("stat cache not hit by ETag[path=%s][time=%jd][hit count=%lu][ETag(%s)!=(%s)]",
strpath.c_str(), (intmax_t)(ent->cache_date), ent->hit_count, petag ? petag : "null", ent->meta["ETag"].c_str()); strpath.c_str(), (intmax_t)(ent->cache_date), ent->hit_count, petag ? petag : "null", ent->meta["ETag"].c_str());
}else{ }else{
// hit // hit
DPRNNN("stat cache hit [path=%s][time=%jd][hit count=%lu]", strpath.c_str(), (intmax_t)(ent->cache_date), ent->hit_count); S3FS_PRN_DBG("stat cache hit [path=%s][time=%jd][hit count=%lu]", strpath.c_str(), (intmax_t)(ent->cache_date), ent->hit_count);
if(pst!= NULL){ if(pst!= NULL){
*pst= ent->stbuf; *pst= ent->stbuf;
@ -245,12 +245,19 @@ bool StatCache::AddStat(std::string& key, headers_t& meta, bool forcedir)
if(CacheSize< 1){ if(CacheSize< 1){
return true; return true;
} }
DPRNNN("add stat cache entry[path=%s]", key.c_str()); S3FS_PRN_INFO3("add stat cache entry[path=%s]", key.c_str());
if(stat_cache.end() != stat_cache.find(key)){ pthread_mutex_lock(&StatCache::stat_cache_lock);
bool found = stat_cache.end() != stat_cache.find(key);
bool do_truncate = stat_cache.size() > CacheSize;
pthread_mutex_unlock(&StatCache::stat_cache_lock);
if(found){
DelStat(key.c_str()); DelStat(key.c_str());
}else{ }else{
if(stat_cache.size() > CacheSize){ if(do_truncate){
if(!TruncateCache()){ if(!TruncateCache()){
return false; return false;
} }
@ -300,12 +307,19 @@ bool StatCache::AddNoObjectCache(string& key)
if(CacheSize < 1){ if(CacheSize < 1){
return true; return true;
} }
DPRNNN("add no object cache entry[path=%s]", key.c_str()); S3FS_PRN_INFO3("add no object cache entry[path=%s]", key.c_str());
if(stat_cache.end() != stat_cache.find(key)){ pthread_mutex_lock(&StatCache::stat_cache_lock);
bool found = stat_cache.end() != stat_cache.find(key);
bool do_truncate = stat_cache.size() > CacheSize;
pthread_mutex_unlock(&StatCache::stat_cache_lock);
if(found){
DelStat(key.c_str()); DelStat(key.c_str());
}else{ }else{
if(stat_cache.size() > CacheSize){ if(do_truncate){
if(!TruncateCache()){ if(!TruncateCache()){
return false; return false;
} }
@ -330,17 +344,18 @@ bool StatCache::AddNoObjectCache(string& key)
bool StatCache::TruncateCache(void) bool StatCache::TruncateCache(void)
{ {
if(0 == stat_cache.size()){ pthread_mutex_lock(&StatCache::stat_cache_lock);
if(stat_cache.empty()){
pthread_mutex_unlock(&StatCache::stat_cache_lock);
return true; return true;
} }
pthread_mutex_lock(&StatCache::stat_cache_lock);
time_t lowest_time = time(NULL) + 1; time_t lowest_time = time(NULL) + 1;
stat_cache_t::iterator iter_to_delete = stat_cache.end(); stat_cache_t::iterator iter_to_delete = stat_cache.end();
stat_cache_t::iterator iter; stat_cache_t::iterator iter;
for(iter = stat_cache.begin(); iter != stat_cache.end(); iter++) { for(iter = stat_cache.begin(); iter != stat_cache.end(); ++iter) {
if((*iter).second){ if((*iter).second){
if(lowest_time > (*iter).second->cache_date){ if(lowest_time > (*iter).second->cache_date){
lowest_time = (*iter).second->cache_date; lowest_time = (*iter).second->cache_date;
@ -349,7 +364,7 @@ bool StatCache::TruncateCache(void)
} }
} }
if(stat_cache.end() != iter_to_delete){ if(stat_cache.end() != iter_to_delete){
DPRNNN("truncate stat cache[path=%s]", (*iter_to_delete).first.c_str()); S3FS_PRN_DBG("truncate stat cache[path=%s]", (*iter_to_delete).first.c_str());
if((*iter_to_delete).second){ if((*iter_to_delete).second){
delete (*iter_to_delete).second; delete (*iter_to_delete).second;
} }
@ -367,7 +382,7 @@ bool StatCache::DelStat(const char* key)
if(!key){ if(!key){
return false; return false;
} }
DPRNNN("delete stat cache entry[path=%s]", key); S3FS_PRN_INFO3("delete stat cache entry[path=%s]", key);
pthread_mutex_lock(&StatCache::stat_cache_lock); pthread_mutex_lock(&StatCache::stat_cache_lock);

115
src/common.h
View File

@ -17,64 +17,91 @@
* along with this program; if not, write to the Free Software * along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/ */
#ifndef S3FS_COMMON_H_ #ifndef S3FS_COMMON_H_
#define S3FS_COMMON_H_ #define S3FS_COMMON_H_
#include "../config.h"
// //
// Macro // Macro
// //
#define SAFESTRPTR(strptr) (strptr ? strptr : "") #define SAFESTRPTR(strptr) (strptr ? strptr : "")
// for debug //
#define FPRINT_NEST_SPACE_0 "" // Debug level
#define FPRINT_NEST_SPACE_1 " " //
#define FPRINT_NEST_SPACE_2 " " enum s3fs_log_level{
#define FPRINT_NEST_CHECK(NEST) \ S3FS_LOG_CRIT = 0, // LOG_CRIT
(0 == NEST ? FPRINT_NEST_SPACE_0 : 1 == NEST ? FPRINT_NEST_SPACE_1 : FPRINT_NEST_SPACE_2) S3FS_LOG_ERR = 1, // LOG_ERR
S3FS_LOG_WARN = 3, // LOG_WARNING
S3FS_LOG_INFO = 7, // LOG_INFO
S3FS_LOG_DBG = 15 // LOG_DEBUG
};
#define LOWFPRINT(NEST, ...) \ //
printf("%s%s(%d): ", FPRINT_NEST_CHECK(NEST), __func__, __LINE__); \ // Debug macros
printf(__VA_ARGS__); \ //
printf("\n"); \ #define IS_S3FS_LOG_CRIT() (S3FS_LOG_CRIT == debug_level)
#define IS_S3FS_LOG_ERR() (S3FS_LOG_ERR == (debug_level & S3FS_LOG_DBG))
#define IS_S3FS_LOG_WARN() (S3FS_LOG_WARN == (debug_level & S3FS_LOG_DBG))
#define IS_S3FS_LOG_INFO() (S3FS_LOG_INFO == (debug_level & S3FS_LOG_DBG))
#define IS_S3FS_LOG_DBG() (S3FS_LOG_DBG == (debug_level & S3FS_LOG_DBG))
#define FPRINT(NEST, ...) \ #define S3FS_LOG_LEVEL_TO_SYSLOG(level) \
( S3FS_LOG_DBG == (level & S3FS_LOG_DBG) ? LOG_DEBUG : \
S3FS_LOG_INFO == (level & S3FS_LOG_DBG) ? LOG_INFO : \
S3FS_LOG_WARN == (level & S3FS_LOG_DBG) ? LOG_WARNING : \
S3FS_LOG_ERR == (level & S3FS_LOG_DBG) ? LOG_ERR : LOG_CRIT )
#define S3FS_LOG_LEVEL_STRING(level) \
( S3FS_LOG_DBG == (level & S3FS_LOG_DBG) ? "[DBG] " : \
S3FS_LOG_INFO == (level & S3FS_LOG_DBG) ? "[INF] " : \
S3FS_LOG_WARN == (level & S3FS_LOG_DBG) ? "[WAN] " : \
S3FS_LOG_ERR == (level & S3FS_LOG_DBG) ? "[ERR] " : "[CRT] " )
#define S3FS_LOG_NEST_MAX 4
#define S3FS_LOG_NEST(nest) (nest < S3FS_LOG_NEST_MAX ? s3fs_log_nest[nest] : s3fs_log_nest[S3FS_LOG_NEST_MAX - 1])
#define S3FS_LOW_LOGPRN(level, fmt, ...) \
if(S3FS_LOG_CRIT == level || (S3FS_LOG_CRIT != debug_level && level == (debug_level & level))){ \
if(foreground){ \ if(foreground){ \
LOWFPRINT(NEST, __VA_ARGS__); \ fprintf(stdout, "%s%s:%s(%d): " fmt "%s\n", S3FS_LOG_LEVEL_STRING(level), __FILE__, __func__, __LINE__, __VA_ARGS__); \
}else{ \
syslog(S3FS_LOG_LEVEL_TO_SYSLOG(level), "%s:%s(%d): " fmt "%s", __FILE__, __func__, __LINE__, __VA_ARGS__); \
} \
} }
#define FPRINT2(NEST, ...) \ #define S3FS_LOW_LOGPRN2(level, nest, fmt, ...) \
if(foreground2){ \ if(S3FS_LOG_CRIT == level || (S3FS_LOG_CRIT != debug_level && level == (debug_level & level))){ \
LOWFPRINT(NEST, __VA_ARGS__); \ if(foreground){ \
fprintf(stdout, "%s%s%s:%s(%d): " fmt "%s\n", S3FS_LOG_LEVEL_STRING(level), S3FS_LOG_NEST(nest), __FILE__, __func__, __LINE__, __VA_ARGS__); \
}else{ \
syslog(S3FS_LOG_LEVEL_TO_SYSLOG(level), "%s" fmt "%s", S3FS_LOG_NEST(nest), __VA_ARGS__); \
} \
} }
#define LOWSYSLOGPRINT(LEVEL, ...) \ #define S3FS_LOW_LOGPRN_EXIT(fmt, ...) \
syslog(LEVEL, __VA_ARGS__); if(foreground){ \
fprintf(stderr, "s3fs: " fmt "%s\n", __VA_ARGS__); \
#define SYSLOGPRINT(LEVEL, ...) \ }else{ \
if(LEVEL <= LOG_CRIT || debug){ \ syslog(S3FS_LOG_CRIT, "s3fs: " fmt "%s", __VA_ARGS__); \
LOWSYSLOGPRINT(LEVEL, __VA_ARGS__); \
} }
#define DPRINT(LEVEL, NEST, ...) \ // [NOTE]
FPRINT(NEST, __VA_ARGS__); \ // small trick for VA_ARGS
SYSLOGPRINT(LEVEL, __VA_ARGS__); //
#define S3FS_PRN_EXIT(fmt, ...) S3FS_LOW_LOGPRN_EXIT(fmt, ##__VA_ARGS__, "")
#define DPRINT2(LEVEL, ...) \ #define S3FS_PRN_CRIT(fmt, ...) S3FS_LOW_LOGPRN(S3FS_LOG_CRIT, fmt, ##__VA_ARGS__, "")
FPRINT2(2, __VA_ARGS__); \ #define S3FS_PRN_ERR(fmt, ...) S3FS_LOW_LOGPRN(S3FS_LOG_ERR, fmt, ##__VA_ARGS__, "")
SYSLOGPRINT(LEVEL, __VA_ARGS__); #define S3FS_PRN_WARN(fmt, ...) S3FS_LOW_LOGPRN(S3FS_LOG_WARN, fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_DBG(fmt, ...) S3FS_LOW_LOGPRN(S3FS_LOG_DBG, fmt, ##__VA_ARGS__, "")
// print debug message #define S3FS_PRN_INFO(fmt, ...) S3FS_LOW_LOGPRN2(S3FS_LOG_INFO, 0, fmt, ##__VA_ARGS__, "")
#define FPRN(...) FPRINT(0, __VA_ARGS__) #define S3FS_PRN_INFO0(fmt, ...) S3FS_LOG_INFO(fmt, __VA_ARGS__)
#define FPRNN(...) FPRINT(1, __VA_ARGS__) #define S3FS_PRN_INFO1(fmt, ...) S3FS_LOW_LOGPRN2(S3FS_LOG_INFO, 1, fmt, ##__VA_ARGS__, "")
#define FPRNNN(...) FPRINT(2, __VA_ARGS__) #define S3FS_PRN_INFO2(fmt, ...) S3FS_LOW_LOGPRN2(S3FS_LOG_INFO, 2, fmt, ##__VA_ARGS__, "")
#define FPRNINFO(...) FPRINT2(2, __VA_ARGS__) #define S3FS_PRN_INFO3(fmt, ...) S3FS_LOW_LOGPRN2(S3FS_LOG_INFO, 3, fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_CURL(fmt, ...) S3FS_LOW_LOGPRN2(S3FS_LOG_CRIT, 0, fmt, ##__VA_ARGS__, "")
// print debug message with putting syslog
#define DPRNCRIT(...) DPRINT(LOG_CRIT, 0, __VA_ARGS__)
#define DPRN(...) DPRINT(LOG_ERR, 0, __VA_ARGS__)
#define DPRNN(...) DPRINT(LOG_DEBUG, 1, __VA_ARGS__)
#define DPRNNN(...) DPRINT(LOG_DEBUG, 2, __VA_ARGS__)
#define DPRNINFO(...) DPRINT2(LOG_INFO, __VA_ARGS__)
// //
// Typedef // Typedef
@ -90,7 +117,7 @@ typedef struct xattr_value{
unsigned char* pvalue; unsigned char* pvalue;
size_t length; size_t length;
xattr_value(unsigned char* pval = NULL, size_t len = 0) : pvalue(pval), length(len) {} explicit xattr_value(unsigned char* pval = NULL, size_t len = 0) : pvalue(pval), length(len) {}
~xattr_value() ~xattr_value()
{ {
if(pvalue){ if(pvalue){
@ -104,9 +131,7 @@ typedef std::map<std::string, PXATTRVAL> xattrs_t;
// //
// Global valiables // Global valiables
// //
extern bool debug;
extern bool foreground; extern bool foreground;
extern bool foreground2;
extern bool nomultipart; extern bool nomultipart;
extern bool pathrequeststyle; extern bool pathrequeststyle;
extern std::string program_name; extern std::string program_name;
@ -115,6 +140,8 @@ extern std::string host;
extern std::string bucket; extern std::string bucket;
extern std::string mount_prefix; extern std::string mount_prefix;
extern std::string endpoint; extern std::string endpoint;
extern s3fs_log_level debug_level;
extern const char* s3fs_log_nest[S3FS_LOG_NEST_MAX];
#endif // S3FS_COMMON_H_ #endif // S3FS_COMMON_H_

98
src/common_auth.cpp
View File

@ -18,104 +18,20 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/ */
#include <limits.h>
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
#include <string> #include <string>
#include "s3fs_auth.h" #include "s3fs_auth.h"
#include "string_util.h"
using namespace std; using namespace std;
//------------------------------------------------------------------- //-------------------------------------------------------------------
// Utility Function // Utility Function
//------------------------------------------------------------------- //-------------------------------------------------------------------
char* s3fs_base64(const unsigned char* input, size_t length)
{
static const char* base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
char* result;
if(!input || 0 >= length){
return NULL;
}
if(NULL == (result = (char*)malloc((((length / 3) + 1) * 4 + 1) * sizeof(char)))){
return NULL; // ENOMEM
}
unsigned char parts[4];
size_t rpos;
size_t wpos;
for(rpos = 0, wpos = 0; rpos < length; rpos += 3){
parts[0] = (input[rpos] & 0xfc) >> 2;
parts[1] = ((input[rpos] & 0x03) << 4) | ((((rpos + 1) < length ? input[rpos + 1] : 0x00) & 0xf0) >> 4);
parts[2] = (rpos + 1) < length ? (((input[rpos + 1] & 0x0f) << 2) | ((((rpos + 2) < length ? input[rpos + 2] : 0x00) & 0xc0) >> 6)) : 0x40;
parts[3] = (rpos + 2) < length ? (input[rpos + 2] & 0x3f) : 0x40;
result[wpos++] = base[parts[0]];
result[wpos++] = base[parts[1]];
result[wpos++] = base[parts[2]];
result[wpos++] = base[parts[3]];
}
result[wpos] = '\0';
return result;
}
inline unsigned char char_decode64(const char ch)
{
unsigned char by;
if('A' <= ch && ch <= 'Z'){ // A - Z
by = static_cast<unsigned char>(ch - 'A');
}else if('a' <= ch && ch <= 'z'){ // a - z
by = static_cast<unsigned char>(ch - 'a' + 26);
}else if('0' <= ch && ch <= '9'){ // 0 - 9
by = static_cast<unsigned char>(ch - '0' + 52);
}else if('+' == ch){ // +
by = 62;
}else if('/' == ch){ // /
by = 63;
}else if('=' == ch){ // =
by = 64;
}else{ // something wrong
by = 64;
}
return by;
}
unsigned char* s3fs_decode64(const char* input, size_t* plength)
{
unsigned char* result;
if(!input || 0 == strlen(input) || !plength){
return NULL;
}
if(NULL == (result = (unsigned char*)malloc((strlen(input) + 1)))){
return NULL; // ENOMEM
}
unsigned char parts[4];
size_t input_len = strlen(input);
size_t rpos;
size_t wpos;
for(rpos = 0, wpos = 0; rpos < input_len; rpos += 4){
parts[0] = char_decode64(input[rpos]);
parts[1] = (rpos + 1) < input_len ? char_decode64(input[rpos + 1]) : 64;
parts[2] = (rpos + 2) < input_len ? char_decode64(input[rpos + 2]) : 64;
parts[3] = (rpos + 3) < input_len ? char_decode64(input[rpos + 3]) : 64;
result[wpos++] = ((parts[0] << 2) & 0xfc) | ((parts[1] >> 4) & 0x03);
if(64 == parts[2]){
break;
}
result[wpos++] = ((parts[1] << 4) & 0xf0) | ((parts[2] >> 2) & 0x0f);
if(64 == parts[3]){
break;
}
result[wpos++] = ((parts[2] << 6) & 0xc0) | (parts[3] & 0x3f);
}
*plength = wpos;
return result;
}
string s3fs_get_content_md5(int fd) string s3fs_get_content_md5(int fd)
{ {
unsigned char* md5hex; unsigned char* md5hex;
@ -139,22 +55,16 @@ string s3fs_get_content_md5(int fd)
string s3fs_md5sum(int fd, off_t start, ssize_t size) string s3fs_md5sum(int fd, off_t start, ssize_t size)
{ {
size_t digestlen = get_md5_digest_length(); size_t digestlen = get_md5_digest_length();
char md5[2 * digestlen + 1];
char hexbuf[3];
unsigned char* md5hex; unsigned char* md5hex;
if(NULL == (md5hex = s3fs_md5hexsum(fd, start, size))){ if(NULL == (md5hex = s3fs_md5hexsum(fd, start, size))){
return string(""); return string("");
} }
memset(md5, 0, 2 * digestlen + 1); std::string md5 = s3fs_hex(md5hex, digestlen);
for(size_t pos = 0; pos < digestlen; pos++){
snprintf(hexbuf, 3, "%02x", md5hex[pos]);
strncat(md5, hexbuf, 2);
}
free(md5hex); free(md5hex);
return string(md5); return md5;
} }
string s3fs_sha256sum(int fd, off_t start, ssize_t size) string s3fs_sha256sum(int fd, off_t start, ssize_t size)

770
src/curl.cpp
View File

File diff suppressed because it is too large Load Diff

69
src/curl.h
View File

@ -20,6 +20,11 @@
#ifndef S3FS_CURL_H_ #ifndef S3FS_CURL_H_
#define S3FS_CURL_H_ #define S3FS_CURL_H_
//----------------------------------------------
// Symbols
//----------------------------------------------
#define MIN_MULTIPART_SIZE 5242880 // 5MB
//---------------------------------------------- //----------------------------------------------
// class BodyData // class BodyData
//---------------------------------------------- //----------------------------------------------
@ -122,6 +127,21 @@ typedef std::map<std::string, std::string> iamcredmap_t;
typedef std::map<std::string, std::string> sseckeymap_t; typedef std::map<std::string, std::string> sseckeymap_t;
typedef std::list<sseckeymap_t> sseckeylist_t; typedef std::list<sseckeymap_t> sseckeylist_t;
// strage class(rrs)
enum storage_class_t {
STANDARD,
STANDARD_IA,
REDUCED_REDUNDANCY,
};
// sse type
enum sse_type_t {
SSE_DISABLE = 0, // not use server side encrypting
SSE_S3, // server side encrypting by S3 key
SSE_C, // server side encrypting by custom key
SSE_KMS // server side encrypting by kms id
};
// share // share
#define SHARE_MUTEX_DNS 0 #define SHARE_MUTEX_DNS 0
#define SHARE_MUTEX_SSL_SESSION 1 #define SHARE_MUTEX_SSL_SESSION 1
@ -165,9 +185,10 @@ class S3fsCurl
static int retries; static int retries;
static bool is_public_bucket; static bool is_public_bucket;
static std::string default_acl; // TODO: to enum static std::string default_acl; // TODO: to enum
static bool is_use_rrs; static storage_class_t storage_class;
static sseckeylist_t sseckeys; static sseckeylist_t sseckeys;
static bool is_use_sse; static std::string ssekmsid;
static sse_type_t ssetype;
static bool is_content_md5; static bool is_content_md5;
static bool is_verbose; static bool is_verbose;
static std::string AWSAccessKeyId; static std::string AWSAccessKeyId;
@ -206,12 +227,13 @@ class S3fsCurl
int b_postdata_remaining; // backup for retrying int b_postdata_remaining; // backup for retrying
off_t b_partdata_startpos; // backup for retrying off_t b_partdata_startpos; // backup for retrying
ssize_t b_partdata_size; // backup for retrying ssize_t b_partdata_size; // backup for retrying
bool b_ssekey_pos; // backup for retrying int b_ssekey_pos; // backup for retrying
std::string b_ssekey_md5; // backup for retrying std::string b_ssevalue; // backup for retrying
sse_type_t b_ssetype; // backup for retrying
public: public:
// constructor/destructor // constructor/destructor
S3fsCurl(bool ahbe = false); explicit S3fsCurl(bool ahbe = false);
~S3fsCurl(); ~S3fsCurl();
private: private:
@ -240,8 +262,12 @@ class S3fsCurl
static bool ParseIAMCredentialResponse(const char* response, iamcredmap_t& keyval); static bool ParseIAMCredentialResponse(const char* response, iamcredmap_t& keyval);
static bool SetIAMCredentials(const char* response); static bool SetIAMCredentials(const char* response);
static bool LoadEnvSseCKeys(void);
static bool LoadEnvSseKmsid(void);
static bool PushbackSseKeys(std::string& onekey); static bool PushbackSseKeys(std::string& onekey);
static int CurlDebugFunc(CURL* hcurl, curl_infotype type, char* data, size_t size, void* userptr);
// methods // methods
bool ResetHandle(void); bool ResetHandle(void);
bool RemakeHandle(void); bool RemakeHandle(void);
@ -252,10 +278,7 @@ class S3fsCurl
bool GetUploadId(std::string& upload_id); bool GetUploadId(std::string& upload_id);
int GetIAMCredentials(void); int GetIAMCredentials(void);
int PreMultipartPostRequest(const char* tpath, headers_t& meta, std::string& upload_id, bool is_copy);
int CompleteMultipartPostRequest(const char* tpath, std::string& upload_id, etaglist_t& parts);
int UploadMultipartPostSetup(const char* tpath, int part_num, std::string& upload_id); int UploadMultipartPostSetup(const char* tpath, int part_num, std::string& upload_id);
int UploadMultipartPostRequest(const char* tpath, int part_num, std::string& upload_id);
int CopyMultipartPostRequest(const char* from, const char* to, int part_num, std::string& upload_id, headers_t& meta); int CopyMultipartPostRequest(const char* from, const char* to, int part_num, std::string& upload_id, headers_t& meta);
public: public:
@ -278,16 +301,23 @@ class S3fsCurl
static bool SetPublicBucket(bool flag); static bool SetPublicBucket(bool flag);
static bool IsPublicBucket(void) { return S3fsCurl::is_public_bucket; } static bool IsPublicBucket(void) { return S3fsCurl::is_public_bucket; }
static std::string SetDefaultAcl(const char* acl); static std::string SetDefaultAcl(const char* acl);
static bool SetUseRrs(bool flag); static storage_class_t SetStorageClass(storage_class_t storage_class);
static bool GetUseRrs(void) { return S3fsCurl::is_use_rrs; } static storage_class_t GetStorageClass() { return S3fsCurl::storage_class; }
static bool SetSseKeys(const char* filepath); static bool LoadEnvSse(void) { return (S3fsCurl::LoadEnvSseCKeys() && S3fsCurl::LoadEnvSseKmsid()); }
static bool LoadEnvSseKeys(void); static sse_type_t SetSseType(sse_type_t type);
static sse_type_t GetSseType(void) { return S3fsCurl::ssetype; }
static bool IsSseDisable(void) { return (SSE_DISABLE == S3fsCurl::ssetype); }
static bool IsSseS3Type(void) { return (SSE_S3 == S3fsCurl::ssetype); }
static bool IsSseCType(void) { return (SSE_C == S3fsCurl::ssetype); }
static bool IsSseKmsType(void) { return (SSE_KMS == S3fsCurl::ssetype); }
static bool FinalCheckSse(void);
static bool SetSseCKeys(const char* filepath);
static bool SetSseKmsid(const char* kmsid);
static bool IsSetSseKmsId(void) { return !S3fsCurl::ssekmsid.empty(); }
static const char* GetSseKmsId(void) { return S3fsCurl::ssekmsid.c_str(); }
static bool GetSseKey(std::string& md5, std::string& ssekey); static bool GetSseKey(std::string& md5, std::string& ssekey);
static bool GetSseKeyMd5(int pos, std::string& md5); static bool GetSseKeyMd5(int pos, std::string& md5);
static int GetSseKeyCount(void); static int GetSseKeyCount(void);
static bool IsSseCustomMode(void);
static bool SetUseSse(bool flag);
static bool GetUseSse(void) { return S3fsCurl::is_use_sse; }
static bool SetContentMd5(bool flag); static bool SetContentMd5(bool flag);
static bool SetVerbose(bool flag); static bool SetVerbose(bool flag);
static bool GetVerbose(void) { return S3fsCurl::is_verbose; } static bool GetVerbose(void) { return S3fsCurl::is_verbose; }
@ -310,7 +340,7 @@ class S3fsCurl
bool CreateCurlHandle(bool force = false); bool CreateCurlHandle(bool force = false);
bool DestroyCurlHandle(void); bool DestroyCurlHandle(void);
bool AddSseKeyRequestHead(std::string& md5, bool is_copy); bool AddSseRequestHead(sse_type_t ssetype, std::string& ssevalue, bool is_only_c, bool is_copy);
bool GetResponseCode(long& responseCode); bool GetResponseCode(long& responseCode);
int RequestPerform(void); int RequestPerform(void);
int DeleteRequest(const char* tpath); int DeleteRequest(const char* tpath);
@ -321,14 +351,18 @@ class S3fsCurl
int HeadRequest(const char* tpath, headers_t& meta); int HeadRequest(const char* tpath, headers_t& meta);
int PutHeadRequest(const char* tpath, headers_t& meta, bool is_copy); int PutHeadRequest(const char* tpath, headers_t& meta, bool is_copy);
int PutRequest(const char* tpath, headers_t& meta, int fd); int PutRequest(const char* tpath, headers_t& meta, int fd);
int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size, std::string& ssekeymd5); int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size, sse_type_t ssetype, std::string& ssevalue);
int GetObjectRequest(const char* tpath, int fd, off_t start = -1, ssize_t size = -1); int GetObjectRequest(const char* tpath, int fd, off_t start = -1, ssize_t size = -1);
int CheckBucket(void); int CheckBucket(void);
int ListBucketRequest(const char* tpath, const char* query); int ListBucketRequest(const char* tpath, const char* query);
int PreMultipartPostRequest(const char* tpath, headers_t& meta, std::string& upload_id, bool is_copy);
int CompleteMultipartPostRequest(const char* tpath, std::string& upload_id, etaglist_t& parts);
int UploadMultipartPostRequest(const char* tpath, int part_num, std::string& upload_id);
int MultipartListRequest(std::string& body); int MultipartListRequest(std::string& body);
int AbortMultipartUpload(const char* tpath, std::string& upload_id); int AbortMultipartUpload(const char* tpath, std::string& upload_id);
int MultipartHeadRequest(const char* tpath, off_t size, headers_t& meta, bool is_copy); int MultipartHeadRequest(const char* tpath, off_t size, headers_t& meta, bool is_copy);
int MultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool is_copy); int MultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool is_copy);
int MultipartUploadRequest(std::string upload_id, const char* tpath, int fd, off_t offset, size_t size, etaglist_t& list);
int MultipartRenameRequest(const char* from, const char* to, headers_t& meta, off_t size); int MultipartRenameRequest(const char* from, const char* to, headers_t& meta, off_t size);
// methods(valiables) // methods(valiables)
@ -433,6 +467,7 @@ std::string get_sorted_header_keys(const struct curl_slist* list);
std::string get_canonical_headers(const struct curl_slist* list, bool only_amz = false); std::string get_canonical_headers(const struct curl_slist* list, bool only_amz = false);
bool MakeUrlResource(const char* realpath, std::string& resourcepath, std::string& url); bool MakeUrlResource(const char* realpath, std::string& resourcepath, std::string& url);
std::string prepare_url(const char* url); std::string prepare_url(const char* url);
bool get_object_sse_type(const char* path, sse_type_t& ssetype, std::string& ssevalue); // implement in s3fs.cpp
#endif // S3FS_CURL_H_ #endif // S3FS_CURL_H_

1319
src/fdcache.cpp
View File

File diff suppressed because it is too large Load Diff

105
src/fdcache.h
View File

@ -20,6 +20,9 @@
#ifndef FD_CACHE_H_ #ifndef FD_CACHE_H_
#define FD_CACHE_H_ #define FD_CACHE_H_
#include <sys/statvfs.h>
#include "curl.h"
//------------------------------------------------ //------------------------------------------------
// CacheFileStat // CacheFileStat
//------------------------------------------------ //------------------------------------------------
@ -34,8 +37,9 @@ class CacheFileStat
public: public:
static bool DeleteCacheFileStat(const char* path); static bool DeleteCacheFileStat(const char* path);
static bool CheckCacheFileStatTopDir(void);
CacheFileStat(const char* tpath = NULL); explicit CacheFileStat(const char* tpath = NULL);
~CacheFileStat(); ~CacheFileStat();
bool Open(void); bool Open(void);
@ -52,40 +56,49 @@ struct fdpage
{ {
off_t offset; off_t offset;
size_t bytes; size_t bytes;
bool init; bool loaded;
fdpage(off_t start = 0, size_t size = 0, bool is_init = false) fdpage(off_t start = 0, size_t size = 0, bool is_loaded = false)
: offset(start), bytes(size), init(is_init) {} : offset(start), bytes(size), loaded(is_loaded) {}
off_t next(void) const { return (offset + bytes); } off_t next(void) const { return (offset + bytes); }
off_t end(void) const { return (0 < bytes ? offset + bytes - 1 : 0); } off_t end(void) const { return (0 < bytes ? offset + bytes - 1 : 0); }
}; };
typedef std::list<struct fdpage*> fdpage_list_t; typedef std::list<struct fdpage*> fdpage_list_t;
class FdEntity;
// //
// Management of loading area/modifying // Management of loading area/modifying
// //
class PageList class PageList
{ {
friend class FdEntity; // only one method access directly pages.
private: private:
fdpage_list_t pages; fdpage_list_t pages;
private: private:
void Clear(void); void Clear(void);
bool Compress(void);
bool Parse(off_t new_pos);
public: public:
static void FreeList(fdpage_list_t& list); static void FreeList(fdpage_list_t& list);
PageList(off_t size = 0, bool is_init = false); explicit PageList(size_t size = 0, bool is_loaded = false);
~PageList(); ~PageList();
off_t Size(void) const; bool Init(size_t size, bool is_loaded);
int Resize(off_t size, bool is_init); size_t Size(void) const;
int Init(off_t size, bool is_init); bool Resize(size_t size, bool is_loaded);
bool IsInit(off_t start, off_t size);
bool SetInit(off_t start, off_t size, bool is_init = true); bool IsPageLoaded(off_t start = 0, size_t size = 0) const; // size=0 is checking to end of list
bool FindUninitPage(off_t start, off_t& resstart, size_t& ressize); bool SetPageLoadedStatus(off_t start, size_t size, bool is_loaded = true, bool is_compress = true);
int GetUninitPages(fdpage_list_t& uninit_list, off_t start = 0, off_t size = -1); bool FindUnloadedPage(off_t start, off_t& resstart, size_t& ressize) const;
size_t GetTotalUnloadedPageSize(off_t start = 0, size_t size = 0) const; // size=0 is checking to end of list
int GetUnloadedPages(fdpage_list_t& unloaded_list, off_t start = 0, size_t size = 0) const; // size=0 is checking to end of list
bool Serialize(CacheFileStat& file, bool is_output); bool Serialize(CacheFileStat& file, bool is_output);
void Dump(void); void Dump(void);
}; };
@ -102,36 +115,58 @@ class FdEntity
int refcnt; // reference count int refcnt; // reference count
std::string path; // object path std::string path; // object path
std::string cachepath; // local cache file path std::string cachepath; // local cache file path
// (if this is empty, does not load/save pagelist.)
int fd; // file descriptor(tmp file or cache file) int fd; // file descriptor(tmp file or cache file)
FILE* file; // file pointer(tmp file or cache file) FILE* pfile; // file pointer(tmp file or cache file)
bool is_modify; // if file is changed, this flag is true bool is_modify; // if file is changed, this flag is true
headers_t orgmeta; // original headers at opening
size_t size_orgmeta; // original file size in original headers
std::string upload_id; // for no cached multipart uploading when no disk space
etaglist_t etaglist; // for no cached multipart uploading when no disk space
off_t mp_start; // start position for no cached multipart(write method only)
size_t mp_size; // size for no cached multipart(write method only)
private: private:
static int FillFile(int fd, unsigned char byte, size_t size, off_t start);
void Clear(void); void Clear(void);
int Dup(void); bool SetAllStatus(bool is_loaded); // [NOTE] not locking
bool SetAllStatus(bool is_enable); //bool SetAllStatusLoaded(void) { return SetAllStatus(true); }
bool SetAllStatusUnloaded(void) { return SetAllStatus(false); }
public: public:
FdEntity(const char* tpath = NULL, const char* cpath = NULL); explicit FdEntity(const char* tpath = NULL, const char* cpath = NULL);
~FdEntity(); ~FdEntity();
void Close(void); void Close(void);
bool IsOpen(void) const { return (-1 != fd); } bool IsOpen(void) const { return (-1 != fd); }
int Open(off_t size = -1, time_t time = -1); int Open(headers_t* pmeta = NULL, ssize_t size = -1, time_t time = -1);
bool OpenAndLoadAll(headers_t* pmeta = NULL, size_t* size = NULL, bool force_load = false);
int Dup(void);
const char* GetPath(void) const { return path.c_str(); } const char* GetPath(void) const { return path.c_str(); }
void SetPath(const std::string &newpath) { path = newpath; } void SetPath(const std::string &newpath) { path = newpath; }
int GetFd(void) const { return fd; } int GetFd(void) const { return fd; }
int SetMtime(time_t time);
bool GetSize(off_t& size);
bool GetMtime(time_t& time);
bool GetStats(struct stat& st);
bool SetAllEnable(void) { return SetAllStatus(true); } bool GetStats(struct stat& st);
bool SetAllDisable(void) { return SetAllStatus(false); } int SetMtime(time_t time);
bool LoadFull(off_t* size = NULL, bool force_load = false); bool UpdateMtime(void);
int Load(off_t start, off_t size); bool GetSize(size_t& size);
int RowFlush(const char* tpath, headers_t& meta, bool force_sync = false); bool SetMode(mode_t mode);
int Flush(headers_t& meta, bool force_sync = false) { return RowFlush(NULL, meta, force_sync); } bool SetUId(uid_t uid);
bool SetGId(gid_t gid);
bool SetContentType(const char* path);
int Load(off_t start = 0, size_t size = 0); // size=0 means loading to end
int NoCacheLoadAndPost(off_t start = 0, size_t size = 0); // size=0 means loading to end
int NoCachePreMultipartPost(void);
int NoCacheMultipartPost(int tgfd, off_t start, size_t size);
int NoCacheCompleteMultipartPost(void);
int RowFlush(const char* tpath, bool force_sync = false);
int Flush(bool force_sync = false) { return RowFlush(NULL, force_sync); }
ssize_t Read(char* bytes, off_t start, size_t size, bool force_load = false); ssize_t Read(char* bytes, off_t start, size_t size, bool force_load = false);
ssize_t Write(const char* bytes, off_t start, size_t size); ssize_t Write(const char* bytes, off_t start, size_t size);
}; };
@ -147,10 +182,13 @@ class FdManager
static pthread_mutex_t fd_manager_lock; static pthread_mutex_t fd_manager_lock;
static bool is_lock_init; static bool is_lock_init;
static std::string cache_dir; static std::string cache_dir;
static size_t page_size; static size_t free_disk_space; // limit free disk space
fdent_map_t fent; fdent_map_t fent;
private:
static fsblkcnt_t GetFreeDiskSpace(const char* path);
public: public:
FdManager(); FdManager();
~FdManager(); ~FdManager();
@ -163,16 +201,21 @@ class FdManager
static bool SetCacheDir(const char* dir); static bool SetCacheDir(const char* dir);
static bool IsCacheDir(void) { return (0 < FdManager::cache_dir.size()); } static bool IsCacheDir(void) { return (0 < FdManager::cache_dir.size()); }
static const char* GetCacheDir(void) { return FdManager::cache_dir.c_str(); } static const char* GetCacheDir(void) { return FdManager::cache_dir.c_str(); }
static size_t SetPageSize(size_t size);
static size_t GetPageSize(void) { return FdManager::page_size; }
static bool MakeCachePath(const char* path, std::string& cache_path, bool is_create_dir = true); static bool MakeCachePath(const char* path, std::string& cache_path, bool is_create_dir = true);
static bool CheckCacheTopDir(void);
static bool MakeRandomTempPath(const char* path, std::string& tmppath); static bool MakeRandomTempPath(const char* path, std::string& tmppath);
static size_t GetEnsureFreeDiskSpace(void) { return FdManager::free_disk_space; }
static size_t SetEnsureFreeDiskSpace(size_t size);
static size_t InitEnsureFreeDiskSpace(void) { return SetEnsureFreeDiskSpace(0); }
static bool IsSafeDiskSpace(const char* path, size_t size);
FdEntity* GetFdEntity(const char* path, int existfd = -1); FdEntity* GetFdEntity(const char* path, int existfd = -1);
FdEntity* Open(const char* path, off_t size = -1, time_t time = -1, bool force_tmpfile = false, bool is_create = true); FdEntity* Open(const char* path, headers_t* pmeta = NULL, ssize_t size = -1, time_t time = -1, bool force_tmpfile = false, bool is_create = true);
FdEntity* ExistOpen(const char* path, int existfd = -1); FdEntity* ExistOpen(const char* path, int existfd = -1);
void Rename(const std::string &from, const std::string &to); void Rename(const std::string &from, const std::string &to);
bool Close(FdEntity* ent); bool Close(FdEntity* ent);
bool ChangeEntityToTempPath(FdEntity* ent, const char* path);
}; };
#endif // FD_CACHE_H_ #endif // FD_CACHE_H_

14
src/gnutls_auth.cpp
View File

@ -217,7 +217,7 @@ unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
break; break;
}else if(-1 == bytes){ }else if(-1 == bytes){
// error // error
DPRNNN("file read error(%d)", errno); S3FS_PRN_ERR("file read error(%d)", errno);
return NULL; return NULL;
} }
md5_update(&ctx_md5, bytes, buf); md5_update(&ctx_md5, bytes, buf);
@ -261,7 +261,7 @@ unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
memset(buf, 0, 512); memset(buf, 0, 512);
if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_md5, GCRY_MD_MD5, 0))){ if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_md5, GCRY_MD_MD5, 0))){
DPRNN("MD5 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err)); S3FS_PRN_ERR("MD5 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
return NULL; return NULL;
} }
@ -273,7 +273,7 @@ unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
break; break;
}else if(-1 == bytes){ }else if(-1 == bytes){
// error // error
DPRNNN("file read error(%d)", errno); S3FS_PRN_ERR("file read error(%d)", errno);
return NULL; return NULL;
} }
gcry_md_write(ctx_md5, buf, bytes); gcry_md_write(ctx_md5, buf, bytes);
@ -344,7 +344,7 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
break; break;
}else if(-1 == bytes){ }else if(-1 == bytes){
// error // error
DPRNNN("file read error(%d)", errno); S3FS_PRN_ERR("file read error(%d)", errno);
return NULL; return NULL;
} }
sha256_update(&ctx_sha256, bytes, buf); sha256_update(&ctx_sha256, bytes, buf);
@ -375,7 +375,7 @@ bool s3fs_sha256(const unsigned char* data, unsigned int datalen, unsigned char*
gcry_md_hd_t ctx_sha256; gcry_md_hd_t ctx_sha256;
gcry_error_t err; gcry_error_t err;
if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_sha256, GCRY_MD_SHA256, 0))){ if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_sha256, GCRY_MD_SHA256, 0))){
DPRNN("SHA256 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err)); S3FS_PRN_ERR("SHA256 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
free(*digest); free(*digest);
return false; return false;
} }
@ -409,7 +409,7 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
memset(buf, 0, 512); memset(buf, 0, 512);
if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_sha256, GCRY_MD_SHA256, 0))){ if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_sha256, GCRY_MD_SHA256, 0))){
DPRNN("SHA256 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err)); S3FS_PRN_ERR("SHA256 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
return NULL; return NULL;
} }
@ -421,7 +421,7 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
break; break;
}else if(-1 == bytes){ }else if(-1 == bytes){
// error // error
DPRNNN("file read error(%d)", errno); S3FS_PRN_ERR("file read error(%d)", errno);
return NULL; return NULL;
} }
gcry_md_write(ctx_sha256, buf, bytes); gcry_md_write(ctx_sha256, buf, bytes);

5
src/nss_auth.cpp
View File

@ -182,7 +182,7 @@ unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
break; break;
}else if(-1 == bytes){ }else if(-1 == bytes){
// error // error
DPRNNN("file read error(%d)", errno); S3FS_PRN_ERR("file read error(%d)", errno);
return NULL; return NULL;
} }
PK11_DigestOp(md5ctx, buf, bytes); PK11_DigestOp(md5ctx, buf, bytes);
@ -262,7 +262,8 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
break; break;
}else if(-1 == bytes){ }else if(-1 == bytes){
// error // error
DPRNNN("file read error(%d)", errno); S3FS_PRN_ERR("file read error(%d)", errno);
PK11_DestroyContext(sha256ctx, PR_TRUE);
return NULL; return NULL;
} }
PK11_DigestOp(sha256ctx, buf, bytes); PK11_DigestOp(sha256ctx, buf, bytes);

21
src/openssl_auth.cpp
View File

@ -105,7 +105,7 @@ static struct CRYPTO_dynlock_value* s3fs_dyn_crypt_mutex(const char* file, int l
struct CRYPTO_dynlock_value* dyndata; struct CRYPTO_dynlock_value* dyndata;
if(NULL == (dyndata = static_cast<struct CRYPTO_dynlock_value*>(malloc(sizeof(struct CRYPTO_dynlock_value))))){ if(NULL == (dyndata = static_cast<struct CRYPTO_dynlock_value*>(malloc(sizeof(struct CRYPTO_dynlock_value))))){
DPRNCRIT("Could not allocate memory for CRYPTO_dynlock_value"); S3FS_PRN_CRIT("Could not allocate memory for CRYPTO_dynlock_value");
return NULL; return NULL;
} }
pthread_mutex_init(&(dyndata->dyn_mutex), NULL); pthread_mutex_init(&(dyndata->dyn_mutex), NULL);
@ -134,14 +134,14 @@ static void s3fs_destroy_dyn_crypt_mutex(struct CRYPTO_dynlock_value* dyndata, c
bool s3fs_init_crypt_mutex(void) bool s3fs_init_crypt_mutex(void)
{ {
if(s3fs_crypt_mutex){ if(s3fs_crypt_mutex){
FPRNNN("s3fs_crypt_mutex is not NULL, destroy it."); S3FS_PRN_DBG("s3fs_crypt_mutex is not NULL, destroy it.");
if(!s3fs_destroy_crypt_mutex()){ if(!s3fs_destroy_crypt_mutex()){
DPRN("Failed to s3fs_crypt_mutex"); S3FS_PRN_ERR("Failed to s3fs_crypt_mutex");
return false; return false;
} }
} }
if(NULL == (s3fs_crypt_mutex = static_cast<pthread_mutex_t*>(malloc(CRYPTO_num_locks() * sizeof(pthread_mutex_t))))){ if(NULL == (s3fs_crypt_mutex = static_cast<pthread_mutex_t*>(malloc(CRYPTO_num_locks() * sizeof(pthread_mutex_t))))){
DPRNCRIT("Could not allocate memory for s3fs_crypt_mutex"); S3FS_PRN_CRIT("Could not allocate memory for s3fs_crypt_mutex");
return false; return false;
} }
for(int cnt = 0; cnt < CRYPTO_num_locks(); cnt++){ for(int cnt = 0; cnt < CRYPTO_num_locks(); cnt++){
@ -250,7 +250,7 @@ unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
break; break;
}else if(-1 == bytes){ }else if(-1 == bytes){
// error // error
DPRNNN("file read error(%d)", errno); S3FS_PRN_ERR("file read error(%d)", errno);
return NULL; return NULL;
} }
MD5_Update(&md5ctx, buf, bytes); MD5_Update(&md5ctx, buf, bytes);
@ -298,9 +298,7 @@ bool s3fs_sha256(const unsigned char* data, unsigned int datalen, unsigned char*
unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size) unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
{ {
const EVP_MD* md = EVP_get_digestbyname("sha256"); const EVP_MD* md = EVP_get_digestbyname("sha256");
EVP_MD_CTX* sha256ctx = EVP_MD_CTX_create(); EVP_MD_CTX* sha256ctx;
EVP_DigestInit_ex(sha256ctx, md, NULL);
char buf[512]; char buf[512];
ssize_t bytes; ssize_t bytes;
unsigned char* result; unsigned char* result;
@ -318,6 +316,9 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
return NULL; return NULL;
} }
sha256ctx = EVP_MD_CTX_create();
EVP_DigestInit_ex(sha256ctx, md, NULL);
memset(buf, 0, 512); memset(buf, 0, 512);
for(ssize_t total = 0; total < size; total += bytes){ for(ssize_t total = 0; total < size; total += bytes){
bytes = 512 < (size - total) ? 512 : (size - total); bytes = 512 < (size - total) ? 512 : (size - total);
@ -327,13 +328,15 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
break; break;
}else if(-1 == bytes){ }else if(-1 == bytes){
// error // error
DPRNNN("file read error(%d)", errno); S3FS_PRN_ERR("file read error(%d)", errno);
EVP_MD_CTX_destroy(sha256ctx);
return NULL; return NULL;
} }
EVP_DigestUpdate(sha256ctx, buf, bytes); EVP_DigestUpdate(sha256ctx, buf, bytes);
memset(buf, 0, 512); memset(buf, 0, 512);
} }
if(NULL == (result = (unsigned char*)malloc(get_sha256_digest_length()))){ if(NULL == (result = (unsigned char*)malloc(get_sha256_digest_length()))){
EVP_MD_CTX_destroy(sha256ctx);
return NULL; return NULL;
} }
EVP_DigestFinal_ex(sha256ctx, result, NULL); EVP_DigestFinal_ex(sha256ctx, result, NULL);

1043
src/s3fs.cpp
View File

File diff suppressed because it is too large Load Diff

2
src/s3fs.h
View File

@ -84,8 +84,6 @@
#endif // HAVE_MALLOC_TRIM #endif // HAVE_MALLOC_TRIM
char* get_object_sseckey_md5(const char* path);
#endif // S3FS_S3_H_ #endif // S3FS_S3_H_
/* /*

5
src/s3fs_auth.h
View File

@ -20,14 +20,15 @@
#ifndef S3FS_AUTH_H_ #ifndef S3FS_AUTH_H_
#define S3FS_AUTH_H_ #define S3FS_AUTH_H_
#include <string>
#include <sys/types.h>
//------------------------------------------------------------------- //-------------------------------------------------------------------
// Utility functions for Authentication // Utility functions for Authentication
//------------------------------------------------------------------- //-------------------------------------------------------------------
// //
// in common_auth.cpp // in common_auth.cpp
// //
char* s3fs_base64(const unsigned char* input, size_t length);
unsigned char* s3fs_decode64(const char* input, size_t* plength);
std::string s3fs_get_content_md5(int fd); std::string s3fs_get_content_md5(int fd);
std::string s3fs_md5sum(int fd, off_t start, ssize_t size); std::string s3fs_md5sum(int fd, off_t start, ssize_t size);
std::string s3fs_sha256sum(int fd, off_t start, ssize_t size); std::string s3fs_sha256sum(int fd, off_t start, ssize_t size);

237
src/s3fs_util.cpp
View File

@ -233,7 +233,7 @@ bool S3ObjList::GetLastName(std::string& lastname) const
{ {
bool result = false; bool result = false;
lastname = ""; lastname = "";
for(s3obj_t::const_iterator iter = objects.begin(); iter != objects.end(); iter++){ for(s3obj_t::const_iterator iter = objects.begin(); iter != objects.end(); ++iter){
if((*iter).second.orgname.length()){ if((*iter).second.orgname.length()){
if(0 > strcmp(lastname.c_str(), (*iter).second.orgname.c_str())){ if(0 > strcmp(lastname.c_str(), (*iter).second.orgname.c_str())){
lastname = (*iter).second.orgname; lastname = (*iter).second.orgname;
@ -253,7 +253,7 @@ bool S3ObjList::GetNameList(s3obj_list_t& list, bool OnlyNormalized, bool CutSla
{ {
s3obj_t::const_iterator iter; s3obj_t::const_iterator iter;
for(iter = objects.begin(); objects.end() != iter; iter++){ for(iter = objects.begin(); objects.end() != iter; ++iter){
if(OnlyNormalized && 0 != (*iter).second.normalname.length()){ if(OnlyNormalized && 0 != (*iter).second.normalname.length()){
continue; continue;
} }
@ -275,7 +275,7 @@ bool S3ObjList::MakeHierarchizedList(s3obj_list_t& list, bool haveSlash)
s3obj_h_t::iterator hiter; s3obj_h_t::iterator hiter;
s3obj_list_t::const_iterator liter; s3obj_list_t::const_iterator liter;
for(liter = list.begin(); list.end() != liter; liter++){ for(liter = list.begin(); list.end() != liter; ++liter){
string strtmp = (*liter); string strtmp = (*liter);
if(1 < strtmp.length() && '/' == strtmp[strtmp.length() - 1]){ if(1 < strtmp.length() && '/' == strtmp[strtmp.length() - 1]){
strtmp = strtmp.substr(0, strtmp.length() - 1); strtmp = strtmp.substr(0, strtmp.length() - 1);
@ -425,51 +425,14 @@ void free_mvnodes(MVNODE *head)
//------------------------------------------------------------------- //-------------------------------------------------------------------
// Class AutoLock // Class AutoLock
//------------------------------------------------------------------- //-------------------------------------------------------------------
AutoLock::AutoLock(pthread_mutex_t* pmutex) : auto_mutex(pmutex), is_locked(false) AutoLock::AutoLock(pthread_mutex_t* pmutex) : auto_mutex(pmutex)
{ {
Lock(); pthread_mutex_lock(auto_mutex);
} }
AutoLock::~AutoLock() AutoLock::~AutoLock()
{ {
Unlock();
}
bool AutoLock::Lock(void)
{
if(!auto_mutex){
return false;
}
if(is_locked){
// already locked
return true;
}
try{
pthread_mutex_lock(auto_mutex);
is_locked = true;
}catch(exception& e){
is_locked = false;
return false;
}
return true;
}
bool AutoLock::Unlock(void)
{
if(!auto_mutex){
return false;
}
if(!is_locked){
// already unlocked
return true;
}
try{
pthread_mutex_unlock(auto_mutex); pthread_mutex_unlock(auto_mutex);
is_locked = false;
}catch(exception& e){
return false;
}
return true;
} }
//------------------------------------------------------------------- //-------------------------------------------------------------------
@ -479,7 +442,6 @@ bool AutoLock::Unlock(void)
string get_username(uid_t uid) string get_username(uid_t uid)
{ {
static size_t maxlen = 0; // set onece static size_t maxlen = 0; // set onece
int result;
char* pbuf; char* pbuf;
struct passwd pwinfo; struct passwd pwinfo;
struct passwd* ppwinfo = NULL; struct passwd* ppwinfo = NULL;
@ -488,19 +450,19 @@ string get_username(uid_t uid)
if(0 == maxlen){ if(0 == maxlen){
long res = sysconf(_SC_GETPW_R_SIZE_MAX); long res = sysconf(_SC_GETPW_R_SIZE_MAX);
if(0 > res){ if(0 > res){
DPRNNN("could not get max pw length."); S3FS_PRN_WARN("could not get max pw length.");
maxlen = 0; maxlen = 0;
return string(""); return string("");
} }
maxlen = res; maxlen = res;
} }
if(NULL == (pbuf = (char*)malloc(sizeof(char) * maxlen))){ if(NULL == (pbuf = (char*)malloc(sizeof(char) * maxlen))){
DPRNCRIT("failed to allocate memory."); S3FS_PRN_CRIT("failed to allocate memory.");
return string(""); return string("");
} }
// get group information // get group information
if(0 != (result = getpwuid_r(uid, &pwinfo, pbuf, maxlen, &ppwinfo))){ if(0 != getpwuid_r(uid, &pwinfo, pbuf, maxlen, &ppwinfo)){
DPRNNN("could not get pw information."); S3FS_PRN_WARN("could not get pw information.");
free(pbuf); free(pbuf);
return string(""); return string("");
} }
@ -526,19 +488,19 @@ int is_uid_inculde_group(uid_t uid, gid_t gid)
if(0 == maxlen){ if(0 == maxlen){
long res = sysconf(_SC_GETGR_R_SIZE_MAX); long res = sysconf(_SC_GETGR_R_SIZE_MAX);
if(0 > res){ if(0 > res){
DPRNNN("could not get max name length."); S3FS_PRN_ERR("could not get max name length.");
maxlen = 0; maxlen = 0;
return -ERANGE; return -ERANGE;
} }
maxlen = res; maxlen = res;
} }
if(NULL == (pbuf = (char*)malloc(sizeof(char) * maxlen))){ if(NULL == (pbuf = (char*)malloc(sizeof(char) * maxlen))){
DPRNCRIT("failed to allocate memory."); S3FS_PRN_CRIT("failed to allocate memory.");
return -ENOMEM; return -ENOMEM;
} }
// get group information // get group information
if(0 != (result = getgrgid_r(gid, &ginfo, pbuf, maxlen, &pginfo))){ if(0 != (result = getgrgid_r(gid, &ginfo, pbuf, maxlen, &pginfo))){
DPRNNN("could not get group information."); S3FS_PRN_ERR("could not get group information.");
free(pbuf); free(pbuf);
return -result; return -result;
} }
@ -589,18 +551,75 @@ int mkdirp(const string& path, mode_t mode)
stringstream ss(path); stringstream ss(path);
while (getline(ss, component, '/')) { while (getline(ss, component, '/')) {
base += "/" + component; base += "/" + component;
mkdir(base.c_str(), mode);
struct stat st;
if(0 == stat(base.c_str(), &st)){
if(!S_ISDIR(st.st_mode)){
return EPERM;
}
}else{
if(0 != mkdir(base.c_str(), mode)){
return errno;
}
}
} }
return 0; return 0;
} }
bool check_exist_dir_permission(const char* dirpath)
{
if(!dirpath || '\0' == dirpath[0]){
return false;
}
// exists
struct stat st;
if(0 != stat(dirpath, &st)){
if(ENOENT == errno){
// dir does not exitst
return true;
}
if(EACCES == errno){
// could not access directory
return false;
}
// somthing error occured
return false;
}
// check type
if(!S_ISDIR(st.st_mode)){
// path is not directory
return false;
}
// check permission
uid_t myuid = geteuid();
if(myuid == st.st_uid){
if(S_IRWXU != (st.st_mode & S_IRWXU)){
return false;
}
}else{
if(1 == is_uid_inculde_group(myuid, st.st_gid)){
if(S_IRWXG != (st.st_mode & S_IRWXG)){
return false;
}
}else{
if(S_IRWXO != (st.st_mode & S_IRWXO)){
return false;
}
}
}
return true;
}
bool delete_files_in_dir(const char* dir, bool is_remove_own) bool delete_files_in_dir(const char* dir, bool is_remove_own)
{ {
DIR* dp; DIR* dp;
struct dirent* dent; struct dirent* dent;
if(NULL == (dp = opendir(dir))){ if(NULL == (dp = opendir(dir))){
DPRNINFO("could not open dir(%s) - errno(%d)", dir, errno); S3FS_PRN_ERR("could not open dir(%s) - errno(%d)", dir, errno);
return false; return false;
} }
@ -613,20 +632,20 @@ bool delete_files_in_dir(const char* dir, bool is_remove_own)
fullpath += dent->d_name; fullpath += dent->d_name;
struct stat st; struct stat st;
if(0 != lstat(fullpath.c_str(), &st)){ if(0 != lstat(fullpath.c_str(), &st)){
DPRN("could not get stats of file(%s) - errno(%d)", fullpath.c_str(), errno); S3FS_PRN_ERR("could not get stats of file(%s) - errno(%d)", fullpath.c_str(), errno);
closedir(dp); closedir(dp);
return false; return false;
} }
if(S_ISDIR(st.st_mode)){ if(S_ISDIR(st.st_mode)){
// dir -> Reentrant // dir -> Reentrant
if(!delete_files_in_dir(fullpath.c_str(), true)){ if(!delete_files_in_dir(fullpath.c_str(), true)){
DPRNINFO("could not remove sub dir(%s) - errno(%d)", fullpath.c_str(), errno); S3FS_PRN_ERR("could not remove sub dir(%s) - errno(%d)", fullpath.c_str(), errno);
closedir(dp); closedir(dp);
return false; return false;
} }
}else{ }else{
if(0 != unlink(fullpath.c_str())){ if(0 != unlink(fullpath.c_str())){
DPRN("could not remove file(%s) - errno(%d)", fullpath.c_str(), errno); S3FS_PRN_ERR("could not remove file(%s) - errno(%d)", fullpath.c_str(), errno);
closedir(dp); closedir(dp);
return false; return false;
} }
@ -635,7 +654,7 @@ bool delete_files_in_dir(const char* dir, bool is_remove_own)
closedir(dp); closedir(dp);
if(is_remove_own && 0 != rmdir(dir)){ if(is_remove_own && 0 != rmdir(dir)){
DPRN("could not remove dir(%s) - errno(%d)", dir, errno); S3FS_PRN_ERR("could not remove dir(%s) - errno(%d)", dir, errno);
return false; return false;
} }
return true; return true;
@ -875,28 +894,54 @@ void show_help (void)
" del_cache (delete local file cache)\n" " del_cache (delete local file cache)\n"
" - delete local file cache when s3fs starts and exits.\n" " - delete local file cache when s3fs starts and exits.\n"
"\n" "\n"
" use_rrs (default is disable)\n" " storage_class (default=\"standard\")\n"
" - this option makes Amazon's Reduced Redundancy Storage enable.\n" " - store object with specified storage class. Possible values:\n"
" standard, standard_ia, and reduced_redundancy.\n"
"\n" "\n"
" use_sse (default is disable)\n" " use_sse (default is disable)\n"
" - use Amazon's Server-Site Encryption or Server-Side Encryption\n" " - Specify three type Amazon's Server-Site Encryption: SSE-S3,\n"
" with Customer-Provided Encryption Keys.\n" " SSE-C or SSE-KMS. SSE-S3 uses Amazon S3-managed encryption\n"
" this option can not be specified with use_rrs. specifying only \n" " keys, SSE-C uses customer-provided encryption keys, and\n"
" \"use_sse\" or \"use_sse=1\" enables Server-Side Encryption.\n" " SSE-KMS uses the master key which you manage in AWS KMS.\n"
" (use_sse=1 for old version)\n" " You can specify \"use_sse\" or \"use_sse=1\" enables SSE-S3\n"
" specifying this option with file path which has some SSE-C\n" " type(use_sse=1 is old type parameter).\n"
" secret key enables Server-Side Encryption with Customer-Provided\n" " Case of setting SSE-C, you can specify \"use_sse=custom\",\n"
" Encryption Keys.(use_sse=file)\n" " \"use_sse=custom:<custom key file path>\" or\n"
" the file must be 600 permission. the file can have some lines,\n" " \"use_sse=<custom key file path>\"(only <custom key file path>\n"
" each line is one SSE-C key. the first line in file is used as\n" " specified is old type parameter). You can use \"c\" for\n"
" Customer-Provided Encryption Keys for uploading and changing\n" " short \"custom\".\n"
" headers etc.\n" " The custom key file must be 600 permission. The file can\n"
" if there are some keys after first line, those are used\n" " have some lines, each line is one SSE-C key. The first line\n"
" downloading object which are encrypted by not first key.\n" " in file is used as Customer-Provided Encryption Keys for\n"
" so that, you can keep all SSE-C keys in file, that is SSE-C\n" " uploading and changing headers etc. If there are some keys\n"
" key history.\n" " after first line, those are used downloading object which\n"
" if AWSSSECKEYS environment is set, you can set SSE-C key instead\n" " are encrypted by not first key. So that, you can keep all\n"
" SSE-C keys in file, that is SSE-C key history.\n"
" If you specify \"custom\"(\"c\") without file path, you\n"
" need to set custom key by load_sse_c option or AWSSSECKEYS\n"
" environment.(AWSSSECKEYS environment has some SSE-C keys\n"
" with \":\" separator.) This option is used to decide the\n"
" SSE type. So that if you do not want to encrypt a object\n"
" object at uploading, but you need to decrypt encrypted\n"
" object at downloaing, you can use load_sse_c option instead\n"
" of this option.\n" " of this option.\n"
" For setting SSE-KMS, specify \"use_sse=kmsid\" or\n"
" \"use_sse=kmsid:<kms id>\". You can use \"k\" for short \"kmsid\".\n"
" If you san specify SSE-KMS type with your <kms id> in AWS\n"
" KMS, you can set it after \"kmsid:\"(or \"k:\"). If you\n"
" specify only \"kmsid\"(\"k\"), you need to set AWSSSEKMSID\n"
" environment which value is <kms id>. You must be careful\n"
" about that you can not use the KMS id which is not same EC2\n"
" region.\n"
"\n"
" load_sse_c - specify SSE-C keys\n"
" Specify the custom-provided encription keys file path for decrypting\n"
" at duwnloading.\n"
" If you use the custom-provided encription key at uploading, you\n"
" specify with \"use_sse=custom\". The file has many lines, one line\n"
" means one custom key. So that you can keep all SSE-C keys in file,\n"
" that is SSE-C key history. AWSSSECKEYS environment is as same as this\n"
" file contents.\n"
"\n" "\n"
" public_bucket (default=\"\" which means disabled)\n" " public_bucket (default=\"\" which means disabled)\n"
" - anonymously mount a public bucket when set to 1\n" " - anonymously mount a public bucket when set to 1\n"
@ -925,10 +970,10 @@ void show_help (void)
" If you specify this option for set \"Content-Encoding\" HTTP \n" " If you specify this option for set \"Content-Encoding\" HTTP \n"
" header, please take care for RFC 2616.\n" " header, please take care for RFC 2616.\n"
"\n" "\n"
" connect_timeout (default=\"10\" seconds)\n" " connect_timeout (default=\"300\" seconds)\n"
" - time to wait for connection before giving up\n" " - time to wait for connection before giving up\n"
"\n" "\n"
" readwrite_timeout (default=\"30\" seconds)\n" " readwrite_timeout (default=\"60\" seconds)\n"
" - time to wait between read/write activity before giving up\n" " - time to wait between read/write activity before giving up\n"
"\n" "\n"
" max_stat_cache_size (default=\"1000\" entries (about 4MB))\n" " max_stat_cache_size (default=\"1000\" entries (about 4MB))\n"
@ -948,7 +993,8 @@ void show_help (void)
" in stat cache that the object(file or directory) does not exist.\n" " in stat cache that the object(file or directory) does not exist.\n"
"\n" "\n"
" no_check_certificate\n" " no_check_certificate\n"
" - server certificate won't be checked against the available certificate authorities.\n" " - server certificate won't be checked against the available \n"
" certificate authorities.\n"
"\n" "\n"
" nodnscache (disable dns cache)\n" " nodnscache (disable dns cache)\n"
" - s3fs is always using dns cache, this option make dns cache disable.\n" " - s3fs is always using dns cache, this option make dns cache disable.\n"
@ -971,13 +1017,15 @@ void show_help (void)
" multipart_size (default=\"10\")\n" " multipart_size (default=\"10\")\n"
" - part size, in MB, for each multipart request.\n" " - part size, in MB, for each multipart request.\n"
"\n" "\n"
" fd_page_size (default=\"52428800\"(50MB))\n" " ensure_diskfree (default same multipart_size value)\n"
" - number of internal management page size for each file descriptor.\n" " - sets MB to ensure disk free space. s3fs makes file for\n"
" For delayed reading and writing by s3fs, s3fs manages pages which \n" " downloading, uploading and caching files. If the disk free\n"
" is separated from object. Each pages has a status that data is \n" " space is smaller than this value, s3fs do not use diskspace\n"
" already loaded(or not loaded yet).\n" " as possible in exchange for the performance.\n"
" This option should not be changed when you don't have a trouble \n" "\n"
" with performance.\n" " singlepart_copy_limit (default=\"5120\")\n"
" - maximum size, in MB, of a single-part copy before trying \n"
" multipart copy.\n"
"\n" "\n"
" url (default=\"http://s3.amazonaws.com\")\n" " url (default=\"http://s3.amazonaws.com\")\n"
" - sets the url to use to access amazon s3\n" " - sets the url to use to access amazon s3\n"
@ -1006,7 +1054,7 @@ void show_help (void)
" nomultipart (disable multipart uploads)\n" " nomultipart (disable multipart uploads)\n"
"\n" "\n"
" enable_content_md5 (default is disable)\n" " enable_content_md5 (default is disable)\n"
" - verifying uploaded object without multipart by content-md5 header.\n" " - ensure data integrity during writes with MD5 hash.\n"
"\n" "\n"
" iam_role (default is no role)\n" " iam_role (default is no role)\n"
" - set the IAM Role that will supply the credentials from the \n" " - set the IAM Role that will supply the credentials from the \n"
@ -1040,6 +1088,16 @@ void show_help (void)
" the virtual-host request style, by using the older path request\n" " the virtual-host request style, by using the older path request\n"
" style.\n" " style.\n"
"\n" "\n"
" dbglevel (default=\"crit\")\n"
" Set the debug message level. set value as crit(critical), err\n"
" (error), warn(warning), info(information) to debug level.\n"
" default debug level is critical. If s3fs run with \"-d\" option,\n"
" the debug level is set information. When s3fs catch the signal\n"
" SIGUSR2, the debug level is bumpup.\n"
"\n"
" curldbg - put curl debug message\n"
" Put the debug message from libcurl when this option is specified.\n"
"\n"
"FUSE/mount Options:\n" "FUSE/mount Options:\n"
"\n" "\n"
" Most of the generic mount options described in 'man mount' are\n" " Most of the generic mount options described in 'man mount' are\n"
@ -1062,8 +1120,7 @@ void show_help (void)
" disable multi-threaded operation\n" " disable multi-threaded operation\n"
"\n" "\n"
"\n" "\n"
"Report bugs to <s3fs-devel@googlegroups.com>\n" "s3fs home page: <https://github.com/s3fs-fuse/s3fs-fuse>\n"
"s3fs home page: <http://code.google.com/p/s3fs/>\n"
); );
return; return;
} }
@ -1071,12 +1128,12 @@ void show_help (void)
void show_version(void) void show_version(void)
{ {
printf( printf(
"Amazon Simple Storage Service File System V%s with %s\n" "Amazon Simple Storage Service File System V%s(commit:%s) with %s\n"
"Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>\n" "Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>\n"
"License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>\n" "License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>\n"
"This is free software: you are free to change and redistribute it.\n" "This is free software: you are free to change and redistribute it.\n"
"There is NO WARRANTY, to the extent permitted by law.\n", "There is NO WARRANTY, to the extent permitted by law.\n",
VERSION, s3fs_crypt_lib_name()); VERSION, COMMIT_HASH_VAL, s3fs_crypt_lib_name());
return; return;
} }

7
src/s3fs_util.h
View File

@ -88,14 +88,10 @@ class AutoLock
{ {
private: private:
pthread_mutex_t* auto_mutex; pthread_mutex_t* auto_mutex;
bool is_locked;
public: public:
AutoLock(pthread_mutex_t* pmutex = NULL); explicit AutoLock(pthread_mutex_t* pmutex);
~AutoLock(); ~AutoLock();
bool Lock(void);
bool Unlock(void);
}; };
//------------------------------------------------------------------- //-------------------------------------------------------------------
@ -113,6 +109,7 @@ int is_uid_inculde_group(uid_t uid, gid_t gid);
std::string mydirname(std::string path); std::string mydirname(std::string path);
std::string mybasename(std::string path); std::string mybasename(std::string path);
int mkdirp(const std::string& path, mode_t mode); int mkdirp(const std::string& path, mode_t mode);
bool check_exist_dir_permission(const char* dirpath);
bool delete_files_in_dir(const char* dir, bool is_remove_own); bool delete_files_in_dir(const char* dir, bool is_remove_own);
time_t get_mtime(const char *s); time_t get_mtime(const char *s);

106
src/string_util.cpp
View File

@ -17,6 +17,7 @@
* along with this program; if not, write to the Free Software * along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/ */
#include <limits.h>
#include <stdio.h> #include <stdio.h>
#include <stdlib.h> #include <stdlib.h>
#include <string.h> #include <string.h>
@ -85,13 +86,6 @@ string lower(string s)
return s; return s;
} }
string IntToStr(int n)
{
stringstream result;
result << n;
return result.str();
}
string trim_left(const string &s, const string &t /* = SPACES */) string trim_left(const string &s, const string &t /* = SPACES */)
{ {
string d(s); string d(s);
@ -275,6 +269,104 @@ string get_date_iso8601(time_t tm)
return buf; return buf;
} }
std::string s3fs_hex(const unsigned char* input, size_t length)
{
std::string hex;
for(size_t pos = 0; pos < length; ++pos){
char hexbuf[3];
snprintf(hexbuf, 3, "%02x", input[pos]);
hex += hexbuf;
}
return hex;
}
char* s3fs_base64(const unsigned char* input, size_t length)
{
static const char* base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
char* result;
if(!input || 0 >= length){
return NULL;
}
if(NULL == (result = (char*)malloc((((length / 3) + 1) * 4 + 1) * sizeof(char)))){
return NULL; // ENOMEM
}
unsigned char parts[4];
size_t rpos;
size_t wpos;
for(rpos = 0, wpos = 0; rpos < length; rpos += 3){
parts[0] = (input[rpos] & 0xfc) >> 2;
parts[1] = ((input[rpos] & 0x03) << 4) | ((((rpos + 1) < length ? input[rpos + 1] : 0x00) & 0xf0) >> 4);
parts[2] = (rpos + 1) < length ? (((input[rpos + 1] & 0x0f) << 2) | ((((rpos + 2) < length ? input[rpos + 2] : 0x00) & 0xc0) >> 6)) : 0x40;
parts[3] = (rpos + 2) < length ? (input[rpos + 2] & 0x3f) : 0x40;
result[wpos++] = base[parts[0]];
result[wpos++] = base[parts[1]];
result[wpos++] = base[parts[2]];
result[wpos++] = base[parts[3]];
}
result[wpos] = '\0';
return result;
}
inline unsigned char char_decode64(const char ch)
{
unsigned char by;
if('A' <= ch && ch <= 'Z'){ // A - Z
by = static_cast<unsigned char>(ch - 'A');
}else if('a' <= ch && ch <= 'z'){ // a - z
by = static_cast<unsigned char>(ch - 'a' + 26);
}else if('0' <= ch && ch <= '9'){ // 0 - 9
by = static_cast<unsigned char>(ch - '0' + 52);
}else if('+' == ch){ // +
by = 62;
}else if('/' == ch){ // /
by = 63;
}else if('=' == ch){ // =
by = 64;
}else{ // something wrong
by = UCHAR_MAX;
}
return by;
}
unsigned char* s3fs_decode64(const char* input, size_t* plength)
{
unsigned char* result;
if(!input || 0 == strlen(input) || !plength){
return NULL;
}
if(NULL == (result = (unsigned char*)malloc((strlen(input) + 1)))){
return NULL; // ENOMEM
}
unsigned char parts[4];
size_t input_len = strlen(input);
size_t rpos;
size_t wpos;
for(rpos = 0, wpos = 0; rpos < input_len; rpos += 4){
parts[0] = char_decode64(input[rpos]);
parts[1] = (rpos + 1) < input_len ? char_decode64(input[rpos + 1]) : 64;
parts[2] = (rpos + 2) < input_len ? char_decode64(input[rpos + 2]) : 64;
parts[3] = (rpos + 3) < input_len ? char_decode64(input[rpos + 3]) : 64;
result[wpos++] = ((parts[0] << 2) & 0xfc) | ((parts[1] >> 4) & 0x03);
if(64 == parts[2]){
break;
}
result[wpos++] = ((parts[1] << 4) & 0xf0) | ((parts[2] >> 2) & 0x0f);
if(64 == parts[3]){
break;
}
result[wpos++] = ((parts[2] << 6) & 0xc0) | (parts[3] & 0x3f);
}
result[wpos] = '\0';
*plength = wpos;
return result;
}
/* /*
* Local variables: * Local variables:
* tab-width: 4 * tab-width: 4

5
src/string_util.h
View File

@ -45,7 +45,6 @@ std::string trim_left(const std::string &s, const std::string &t = SPACES);
std::string trim_right(const std::string &s, const std::string &t = SPACES); std::string trim_right(const std::string &s, const std::string &t = SPACES);
std::string trim(const std::string &s, const std::string &t = SPACES); std::string trim(const std::string &s, const std::string &t = SPACES);
std::string lower(std::string s); std::string lower(std::string s);
std::string IntToStr(int);
std::string get_date_rfc850(void); std::string get_date_rfc850(void);
void get_date_sigv3(std::string& date, std::string& date8601); void get_date_sigv3(std::string& date, std::string& date8601);
std::string get_date_string(time_t tm); std::string get_date_string(time_t tm);
@ -56,6 +55,10 @@ std::string urlDecode(const std::string& s);
bool takeout_str_dquart(std::string& str); bool takeout_str_dquart(std::string& str);
bool get_keyword_value(std::string& target, const char* keyword, std::string& value); bool get_keyword_value(std::string& target, const char* keyword, std::string& value);
std::string s3fs_hex(const unsigned char* input, size_t length);
char* s3fs_base64(const unsigned char* input, size_t length);
unsigned char* s3fs_decode64(const char* input, size_t* plength);
#endif // S3FS_STRING_UTIL_H_ #endif // S3FS_STRING_UTIL_H_
/* /*

41
src/test_string_util.cpp
View File

@ -18,12 +18,14 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/ */
#include <limits>
#include <stdint.h>
#include <string> #include <string>
#include "string_util.h" #include "string_util.h"
#include "test_util.h" #include "test_util.h"
int main(int argc, char *argv[]) void test_trim()
{ {
ASSERT_EQUALS(std::string("1234"), trim(" 1234 ")); ASSERT_EQUALS(std::string("1234"), trim(" 1234 "));
ASSERT_EQUALS(std::string("1234"), trim("1234 ")); ASSERT_EQUALS(std::string("1234"), trim("1234 "));
@ -40,5 +42,42 @@ int main(int argc, char *argv[])
ASSERT_EQUALS(std::string(" 1234"), trim_right(" 1234")); ASSERT_EQUALS(std::string(" 1234"), trim_right(" 1234"));
ASSERT_EQUALS(std::string("1234"), trim_right("1234")); ASSERT_EQUALS(std::string("1234"), trim_right("1234"));
ASSERT_EQUALS(std::string("0"), str(0));
ASSERT_EQUALS(std::string("1"), str(1));
ASSERT_EQUALS(std::string("-1"), str(-1));
ASSERT_EQUALS(std::string("9223372036854775807"), str(std::numeric_limits<int64_t>::max()));
ASSERT_EQUALS(std::string("-9223372036854775808"), str(std::numeric_limits<int64_t>::min()));
ASSERT_EQUALS(std::string("0"), str(std::numeric_limits<uint64_t>::min()));
ASSERT_EQUALS(std::string("18446744073709551615"), str(std::numeric_limits<uint64_t>::max()));
}
void test_base64()
{
size_t len;
ASSERT_STREQUALS(s3fs_base64(NULL, 0), NULL);
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64(NULL, &len)), NULL);
ASSERT_STREQUALS(s3fs_base64(reinterpret_cast<const unsigned char *>(""), 0), NULL);
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64("", &len)), NULL);
ASSERT_STREQUALS(s3fs_base64(reinterpret_cast<const unsigned char *>("1"), 1), "MQ==");
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64("MQ==", &len)), "1");
ASSERT_EQUALS(len, static_cast<size_t>(1));
ASSERT_STREQUALS(s3fs_base64(reinterpret_cast<const unsigned char *>("12"), 2), "MTI=");
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64("MTI=", &len)), "12");
ASSERT_EQUALS(len, static_cast<size_t>(2));
ASSERT_STREQUALS(s3fs_base64(reinterpret_cast<const unsigned char *>("123"), 3), "MTIz");
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64("MTIz", &len)), "123");
ASSERT_EQUALS(len, static_cast<size_t>(3));
ASSERT_STREQUALS(s3fs_base64(reinterpret_cast<const unsigned char *>("1234"), 4), "MTIzNA==");
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64("MTIzNA==", &len)), "1234");
ASSERT_EQUALS(len, static_cast<size_t>(4));
// TODO: invalid input
}
int main(int argc, char *argv[])
{
test_trim();
test_base64();
return 0; return 0;
} }

12
src/test_util.h
View File

@ -29,6 +29,18 @@ template <typename T> void assert_equals(const T &x, const T &y, const char *fil
} }
} }
void assert_strequals(const char *x, const char *y, const char *file, int line)
{
if(x == NULL && y == NULL){
return;
} else if((x == NULL || y == NULL) || strcmp(x, y) != 0){
std::cerr << x << " != " << y << " at " << file << ":" << line << std::endl;
std::exit(1);
}
}
#define ASSERT_EQUALS(x, y) \ #define ASSERT_EQUALS(x, y) \
assert_equals((x), (y), __FILE__, __LINE__) assert_equals((x), (y), __FILE__, __LINE__)
#define ASSERT_STREQUALS(x, y) \
assert_strequals((x), (y), __FILE__, __LINE__)

5
test/Makefile.am
View File

@ -28,8 +28,3 @@ EXTRA_DIST = \
sample_ahbe.conf sample_ahbe.conf
testdir = test testdir = test
test_PROGRAMS=rename_before_close
rename_before_close_SOURCES = rename_before_close.c

12
test/integration-test-common.sh
View File

@ -1,10 +1,10 @@
#!/bin/bash -e #!/bin/bash -e
set -x
S3FS=../src/s3fs S3FS=../src/s3fs
S3FS_CREDENTIALS_FILE="passwd-s3fs" : ${S3FS_CREDENTIALS_FILE:="passwd-s3fs"}
TEST_BUCKET_1="s3fs-integration-test" : ${TEST_BUCKET_1:="s3fs-integration-test"}
TEST_BUCKET_MOUNT_POINT_1=${TEST_BUCKET_1} TEST_BUCKET_MOUNT_POINT_1=${TEST_BUCKET_1}
if [ ! -f "$S3FS_CREDENTIALS_FILE" ] if [ ! -f "$S3FS_CREDENTIALS_FILE" ]
@ -15,9 +15,9 @@ fi
chmod 600 "$S3FS_CREDENTIALS_FILE" chmod 600 "$S3FS_CREDENTIALS_FILE"
S3PROXY_VERSION="1.4.0" S3PROXY_VERSION="1.4.0"
S3PROXY_BINARY="s3proxy-${S3PROXY_VERSION}" S3PROXY_BINARY=${S3PROXY_BINARY-"s3proxy-${S3PROXY_VERSION}"}
if [ ! -e "${S3PROXY_BINARY}" ]; then if [ -n "${S3PROXY_BINARY}" ] && [ ! -e "${S3PROXY_BINARY}" ]; then
wget "https://github.com/andrewgaul/s3proxy/releases/download/s3proxy-${S3PROXY_VERSION}/s3proxy" \ wget "https://github.com/andrewgaul/s3proxy/releases/download/s3proxy-${S3PROXY_VERSION}/s3proxy" \
-O "${S3PROXY_BINARY}" --quiet -O "${S3PROXY_BINARY}"
chmod +x "${S3PROXY_BINARY}" chmod +x "${S3PROXY_BINARY}"
fi fi

194
test/integration-test-main.sh
View File

@ -61,25 +61,13 @@ function rm_test_dir {
fi fi
} }
CUR_DIR=`pwd` function test_append_file {
TEST_BUCKET_MOUNT_POINT_1=$1 echo "Testing append to file ..."
if [ "$TEST_BUCKET_MOUNT_POINT_1" == "" ]; then
echo "Mountpoint missing"
exit 1
fi
cd $TEST_BUCKET_MOUNT_POINT_1
if [ -e $TEST_TEXT_FILE ]
then
rm -f $TEST_TEXT_FILE
fi
# Write a small test file # Write a small test file
for x in `seq 1 $TEST_TEXT_FILE_LENGTH` for x in `seq 1 $TEST_TEXT_FILE_LENGTH`
do do
echo "echo ${TEST_TEXT} to ${TEST_TEXT_FILE}" echo "echo ${TEST_TEXT} to ${TEST_TEXT_FILE}"
echo $TEST_TEXT >> $TEST_TEXT_FILE done > ${TEST_TEXT_FILE}
done
# Verify contents of file # Verify contents of file
echo "Verifying length of test file" echo "Verifying length of test file"
@ -91,10 +79,27 @@ then
fi fi
rm_test_file rm_test_file
}
########################################################## function test_truncate_file {
# Rename test (individual file) echo "Testing truncate file ..."
########################################################## # Write a small test file
echo "${TEST_TEXT}" > ${TEST_TEXT_FILE}
# Truncate file to 0 length. This should trigger open(path, O_RDWR | O_TRUNC...)
: > ${TEST_TEXT_FILE}
# Verify file is zero length
if [ -s ${TEST_TEXT_FILE} ]
then
echo "error: expected ${TEST_TEXT_FILE} to be zero length"
exit 1
fi
rm_test_file
}
function test_mv_file {
echo "Testing mv file function ..." echo "Testing mv file function ..."
# if the rename file exists, delete it # if the rename file exists, delete it
@ -131,10 +136,9 @@ fi
# clean up # clean up
rm_test_file $ALT_TEST_TEXT_FILE rm_test_file $ALT_TEST_TEXT_FILE
}
########################################################## function test_mv_directory {
# Rename test (individual directory)
##########################################################
echo "Testing mv directory function ..." echo "Testing mv directory function ..."
if [ -e $TEST_DIR ]; then if [ -e $TEST_DIR ]; then
echo "Unexpected, this file/directory exists: ${TEST_DIR}" echo "Unexpected, this file/directory exists: ${TEST_DIR}"
@ -155,10 +159,9 @@ if [ -e "${TEST_DIR}_rename" ]; then
echo "Could not remove the test directory, it still exists: ${TEST_DIR}_rename" echo "Could not remove the test directory, it still exists: ${TEST_DIR}_rename"
exit 1 exit 1
fi fi
}
################################################################### function test_redirects {
# test redirects > and >>
###################################################################
echo "Testing redirects ..." echo "Testing redirects ..."
mk_test_file ABCDEF mk_test_file ABCDEF
@ -194,13 +197,11 @@ if [ ${LINE2} != "123456" ]; then
exit 1 exit 1
fi fi
# clean up # clean up
rm_test_file rm_test_file
}
##################################################################### function test_mkdir_rmdir {
# Simple directory test mkdir/rmdir
#####################################################################
echo "Testing creation/removal of a directory" echo "Testing creation/removal of a directory"
if [ -e $TEST_DIR ]; then if [ -e $TEST_DIR ]; then
@ -210,10 +211,9 @@ fi
mk_test_dir mk_test_dir
rm_test_dir rm_test_dir
}
########################################################## function test_chmod {
# File permissions test (individual file)
##########################################################
echo "Testing chmod file function ..." echo "Testing chmod file function ..."
# create the test file again # create the test file again
@ -232,10 +232,9 @@ fi
# clean up # clean up
rm_test_file rm_test_file
}
########################################################## function test_chown {
# File permissions test (individual file)
##########################################################
echo "Testing chown file function ..." echo "Testing chown file function ..."
# create the test file again # create the test file again
@ -254,10 +253,9 @@ fi
# clean up # clean up
rm_test_file rm_test_file
}
########################################################## function test_list {
# Testing list
##########################################################
echo "Testing list" echo "Testing list"
mk_test_file mk_test_file
mk_test_dir mk_test_dir
@ -270,25 +268,34 @@ fi
rm_test_file rm_test_file
rm_test_dir rm_test_dir
}
########################################################## function test_remove_nonempty_directory {
# Testing rename before close echo "Testing removing a non-empty directory"
########################################################## mk_test_dir
if false; then touch "${TEST_DIR}/file"
rmdir "${TEST_DIR}" 2>&1 | grep -q "Directory not empty"
rm "${TEST_DIR}/file"
rm_test_dir
}
function test_rename_before_close {
echo "Testing rename before close ..." echo "Testing rename before close ..."
$CUR_DIR/rename_before_close $TEST_TEXT_FILE (
if [ $? != 0 ]; then echo foo
mv $TEST_TEXT_FILE ${TEST_TEXT_FILE}.new
) > $TEST_TEXT_FILE
if ! cmp <(echo foo) ${TEST_TEXT_FILE}.new; then
echo "rename before close failed" echo "rename before close failed"
exit 1 exit 1
fi fi
# clean up rm_test_file ${TEST_TEXT_FILE}.new
rm_test_file rm -f ${TEST_TEXT_FILE}
fi }
########################################################## function test_multipart_upload {
# Testing multi-part upload
##########################################################
echo "Testing multi-part upload ..." echo "Testing multi-part upload ..."
dd if=/dev/urandom of="/tmp/${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1 dd if=/dev/urandom of="/tmp/${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1
dd if="/tmp/${BIG_FILE}" of="${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1 dd if="/tmp/${BIG_FILE}" of="${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1
@ -301,11 +308,27 @@ then
fi fi
rm -f "/tmp/${BIG_FILE}" rm -f "/tmp/${BIG_FILE}"
rm -f "${BIG_FILE}" rm_test_file "${BIG_FILE}"
}
########################################################## function test_multipart_copy {
# Testing special characters echo "Testing multi-part copy ..."
########################################################## dd if=/dev/urandom of="/tmp/${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1
dd if="/tmp/${BIG_FILE}" of="${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1
mv "${BIG_FILE}" "${BIG_FILE}-copy"
# Verify contents of file
echo "Comparing test file"
if ! cmp "/tmp/${BIG_FILE}" "${BIG_FILE}-copy"
then
exit 1
fi
rm -f "/tmp/${BIG_FILE}"
rm_test_file "${BIG_FILE}-copy"
}
function test_special_characters {
echo "Testing special characters ..." echo "Testing special characters ..."
ls 'special' 2>&1 | grep -q 'No such file or directory' ls 'special' 2>&1 | grep -q 'No such file or directory'
@ -313,10 +336,29 @@ ls 'special?' 2>&1 | grep -q 'No such file or directory'
ls 'special*' 2>&1 | grep -q 'No such file or directory' ls 'special*' 2>&1 | grep -q 'No such file or directory'
ls 'special~' 2>&1 | grep -q 'No such file or directory' ls 'special~' 2>&1 | grep -q 'No such file or directory'
ls 'specialµ' 2>&1 | grep -q 'No such file or directory' ls 'specialµ' 2>&1 | grep -q 'No such file or directory'
}
########################################################## function test_symlink {
# Testing extended attributes echo "Testing symlinks ..."
##########################################################
rm -f $TEST_TEXT_FILE
rm -f $ALT_TEST_TEXT_FILE
echo foo > $TEST_TEXT_FILE
ln -s $TEST_TEXT_FILE $ALT_TEST_TEXT_FILE
cmp $TEST_TEXT_FILE $ALT_TEST_TEXT_FILE
rm -f $TEST_TEXT_FILE
[ -L $ALT_TEST_TEXT_FILE ]
[ ! -f $ALT_TEST_TEXT_FILE ]
}
function test_extended_attributes {
command -v setfattr >/dev/null 2>&1 || \
{ echo "Skipping extended attribute tests" ; return; }
echo "Testing extended attributes ..."
rm -f $TEST_TEXT_FILE rm -f $TEST_TEXT_FILE
touch $TEST_TEXT_FILE touch $TEST_TEXT_FILE
@ -334,10 +376,44 @@ getfattr -n key2 --only-values $TEST_TEXT_FILE | grep -q '^value2$'
setfattr -x key1 $TEST_TEXT_FILE setfattr -x key1 $TEST_TEXT_FILE
! getfattr -n key1 --only-values $TEST_TEXT_FILE ! getfattr -n key1 --only-values $TEST_TEXT_FILE
getfattr -n key2 --only-values $TEST_TEXT_FILE | grep -q '^value2$' getfattr -n key2 --only-values $TEST_TEXT_FILE | grep -q '^value2$'
}
##################################################################### function run_all_tests {
# Tests are finished test_append_file
##################################################################### test_truncate_file
test_mv_file
test_mv_directory
test_redirects
test_mkdir_rmdir
test_chmod
test_chown
test_list
test_remove_nonempty_directory
# TODO: broken: https://github.com/s3fs-fuse/s3fs-fuse/issues/145
#test_rename_before_close
test_multipart_upload
# TODO: test disabled until S3Proxy 1.5.0 is released
#test_multipart_copy
test_special_characters
test_symlink
test_extended_attributes
}
# Mount the bucket
CUR_DIR=`pwd`
TEST_BUCKET_MOUNT_POINT_1=$1
if [ "$TEST_BUCKET_MOUNT_POINT_1" == "" ]; then
echo "Mountpoint missing"
exit 1
fi
cd $TEST_BUCKET_MOUNT_POINT_1
if [ -e $TEST_TEXT_FILE ]
then
rm -f $TEST_TEXT_FILE
fi
run_all_tests
# Unmount the bucket # Unmount the bucket
cd $CUR_DIR cd $CUR_DIR

88
test/rename_before_close.c
View File

@ -1,88 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
static const char FILE_CONTENT[] = "XXXX";
#define PROG "rename_before_close"
static char *
filename_to_mkstemp_template(const char *file)
{
size_t len = strlen(file);
static const char suffix[] = ".XXXXXX";
size_t new_len = len + sizeof(suffix);
char *ret_str = calloc(1, new_len);
int ret = snprintf(ret_str, new_len, "%s%s", file, suffix);
assert(ret == new_len - 1);
assert(ret_str[new_len] == '\0');
return ret_str;
}
static off_t
get_file_size(const char *file)
{
struct stat ss;
printf(PROG ": stat(%s)\n", file);
int ret = lstat(file, &ss);
assert(ret == 0);
return ss.st_size;
}
static void
test_rename_before_close(const char *file)
{
char *template = filename_to_mkstemp_template(file);
printf(PROG ": mkstemp(%s)\n", template);
int fd = mkstemp(template);
assert(fd >= 0);
sleep(1);
printf(PROG ": write(%s)\n", template);
int ret = write(fd, FILE_CONTENT, sizeof(FILE_CONTENT));
assert(ret == sizeof(FILE_CONTENT));
sleep(1);
printf(PROG ": fsync(%s)\n", template);
ret = fsync(fd);
assert(ret == 0);
sleep(1);
assert(get_file_size(template) == sizeof(FILE_CONTENT));
sleep(1);
printf(PROG ": rename(%s, %s)\n", template, file);
ret = rename(template, file);
assert(ret == 0);
sleep(1);
printf(PROG ": close(%s)\n", file);
ret = close(fd);
assert(ret == 0);
sleep(1);
assert(get_file_size(file) == sizeof(FILE_CONTENT));
}
int
main(int argc, char *argv[])
{
setvbuf(stdout, NULL, _IONBF, 0);
if (argc < 2) {
printf("Usage: %s <file>", argv[0]);
return 1;
}
test_rename_before_close(argv[1]);
return 0;
}

31
test/small-integration-test.sh
View File

@ -1,8 +1,24 @@
#!/bin/bash #!/bin/bash
#
# By default tests run against a local s3proxy instance. To run against
# Amazon S3, specify the following variables:
#
# S3FS_CREDENTIALS_FILE=keyfile s3fs format key file
# TEST_BUCKET_1=bucket Name of bucket to use
# S3PROXY_BINARY="" Leave empty
# S3_URL="http://s3.amazonaws.com" Specify Amazon server
#
# Example:
#
# S3FS_CREDENTIALS_FILE=keyfile TEST_BUCKET_1=bucket S3PROXY_BINARY="" S3_URL="http://s3.amazonaws.com" ./small-integration-test.sh
#
set -o xtrace set -o xtrace
set -o errexit set -o errexit
: ${S3_URL:="http://127.0.0.1:8080"}
# Require root # Require root
REQUIRE_ROOT=require-root.sh REQUIRE_ROOT=require-root.sh
#source $REQUIRE_ROOT #source $REQUIRE_ROOT
@ -29,17 +45,22 @@ function retry {
} }
function exit_handler { function exit_handler {
if [ -n "${S3PROXY_PID}" ]
then
kill $S3PROXY_PID kill $S3PROXY_PID
fi
retry 30 fusermount -u $TEST_BUCKET_MOUNT_POINT_1 retry 30 fusermount -u $TEST_BUCKET_MOUNT_POINT_1
} }
trap exit_handler EXIT trap exit_handler EXIT
if [ -n "${S3PROXY_BINARY}" ]
then
stdbuf -oL -eL java -jar "$S3PROXY_BINARY" --properties s3proxy.conf | stdbuf -oL -eL sed -u "s/^/s3proxy: /" & stdbuf -oL -eL java -jar "$S3PROXY_BINARY" --properties s3proxy.conf | stdbuf -oL -eL sed -u "s/^/s3proxy: /" &
# wait for S3Proxy to start # wait for S3Proxy to start
for i in $(seq 30); for i in $(seq 30);
do do
if exec 3<>"/dev/tcp/localhost/8080"; if exec 3<>"/dev/tcp/127.0.0.1/8080";
then then
exec 3<&- # Close for read exec 3<&- # Close for read
exec 3>&- # Close for write exec 3>&- # Close for write
@ -49,6 +70,7 @@ do
done done
S3PROXY_PID=$(netstat -lpnt | grep :8080 | awk '{ print $7 }' | sed -u 's|/java||') S3PROXY_PID=$(netstat -lpnt | grep :8080 | awk '{ print $7 }' | sed -u 's|/java||')
fi
# Mount the bucket # Mount the bucket
if [ ! -d $TEST_BUCKET_MOUNT_POINT_1 ] if [ ! -d $TEST_BUCKET_MOUNT_POINT_1 ]
@ -57,10 +79,13 @@ then
fi fi
stdbuf -oL -eL $S3FS $TEST_BUCKET_1 $TEST_BUCKET_MOUNT_POINT_1 \ stdbuf -oL -eL $S3FS $TEST_BUCKET_1 $TEST_BUCKET_MOUNT_POINT_1 \
-o createbucket \ -o createbucket \
-o enable_content_md5 \
-o passwd_file=$S3FS_CREDENTIALS_FILE \ -o passwd_file=$S3FS_CREDENTIALS_FILE \
-o sigv2 \ -o sigv2 \
-o url=http://127.0.0.1:8080 \ -o singlepart_copy_limit=$((10 * 1024)) \
-o use_path_request_style -f -o f2 -d -d |& stdbuf -oL -eL sed -u "s/^/s3fs: /" & -o url=${S3_URL} \
-o use_path_request_style \
-o dbglevel=info -f |& stdbuf -oL -eL sed -u "s/^/s3fs: /" &
retry 30 grep $TEST_BUCKET_MOUNT_POINT_1 /proc/mounts || exit 1 retry 30 grep $TEST_BUCKET_MOUNT_POINT_1 /proc/mounts || exit 1