octoleo/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2024-11-10 23:00:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into macosx

Browse Source
This commit is contained in:
Takeshi Nakatani 2015-11-08 06:06:05 +00:00
commit e1de134d94
33 changed files with 3379 additions and 2039 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
.travis.yml
View File

@ -1,17 +1,16 @@
language: cpp
sudo: required
dist: trusty
cache: apt
before_install:
- sudo apt-get update -qq
- sudo apt-get install -qq libfuse-dev
- sudo apt-get install -qq cppcheck libfuse-dev openjdk-7-jdk
script:
- ./autogen.sh
- ./configure
- make
- make cppcheck
- make check -C src
# Travis granted s3fs access to their upcoming alpha testing stack which may
# allow us to use FUSE.
# TODO: Travis changed their infrastructure some time in June 2015 such that
# this does not work currently
#- modprobe fuse
#- make check -C test
- modprobe fuse
- make check -C test
- cat test/test-suite.log

11
Makefile.am
View File

@ -19,7 +19,7 @@
######################################################################
SUBDIRS=src test doc
EXTRA_DIST=doc
EXTRA_DIST=doc default_commit_hash
dist-hook:
rm -rf `find $(distdir)/doc -type d -name .svn`
@ -28,3 +28,12 @@ dist-hook:
release : dist ../utils/release.sh
../utils/release.sh $(DIST_ARCHIVES)
cppcheck:
cppcheck --quiet --error-exitcode=1 \
-U CURLE_PEER_FAILED_VERIFICATION \
--enable=all \
--suppress=missingIncludeSystem \
--suppress=unsignedLessThanZero \
--suppress=unusedFunction \
--suppress=variableScope \
src/ test/

0
NEWS
View File

67
README
View File

@ -1,67 +0,0 @@
THIS README CONTAINS OUTDATED INFORMATION - please refer to the wiki or --help
S3FS-Fuse
S3FS is FUSE (File System in User Space) based solution to mount/unmount an Amazon S3 storage buckets and use system commands with S3 just like it was another Hard Disk.
In order to compile s3fs, You'll need the following requirements:
* Kernel-devel packages (or kernel source) installed that is the SAME version of your running kernel
* LibXML2-devel packages
* CURL-devel packages (or compile curl from sources at: curl.haxx.se/ use 7.15.X)
* GCC, GCC-C++
* pkgconfig
* FUSE (>= 2.8.4)
* FUSE Kernel module installed and running (RHEL 4.x/CentOS 4.x users - read below)
* OpenSSL-devel (0.9.8)
GnuTLS(gcrypt and nettle)
NSS
* Git
If you're using YUM or APT to install those packages, then it might require additional packaging, allow it to be installed.
Downloading & Compiling:
------------------------
In order to download s3fs, download from following url:
https://github.com/s3fs-fuse/s3fs-fuse/archive/master.zip
Or clone the following command:
git clone git://github.com/s3fs-fuse/s3fs-fuse.git
Go inside the directory that has been created (s3fs-fuse) and run: ./autogen.sh
This will generate a number of scripts in the project directory, including a configure script which you should run with: ./configure
If configure succeeded, you can now run: make. If it didn't, make sure you meet the dependencies above.
This should compile the code. If everything goes OK, you'll be greeted with "ok!" at the end and you'll have a binary file called "s3fs"
in the src/ directory.
As root (you can use su, su -, sudo) do: "make install" -this will copy the "s3fs" binary to /usr/local/bin.
Congratulations. S3fs is now compiled and installed.
Usage:
------
In order to use s3fs, make sure you have the Access Key and the Secret Key handy. (refer to the wiki)
First, create a directory where to mount the S3 bucket you want to use.
Example (as root): mkdir -p /mnt/s3
Then run: s3fs mybucket[:path] /mnt/s3
This will mount your bucket to /mnt/s3. You can do a simple "ls -l /mnt/s3" to see the content of your bucket.
If you want to allow other people access the same bucket in the same machine, you can add "-o allow_other" to read/write/delete content of the bucket.
You can add a fixed mount point in /etc/fstab, here's an example:
s3fs#mybucket /mnt/s3 fuse allow_other 0 0
This will mount upon reboot (or by launching: mount -a) your bucket on your machine.
If that does not work, probably you should specify with "_netdev" option in fstab.
All other options can be read at: https://github.com/s3fs-fuse/s3fs-fuse/wiki/Fuse-Over-Amazon
Known Issues:
-------------
s3fs should be working fine with S3 storage. However, There are couple of limitations:
* Currently s3fs could hang the CPU if you have lots of time-outs. This is *NOT* a fault of s3fs but rather libcurl. This happens when you try to copy thousands of files in 1 session, it doesn't happen when you upload hundreds of files or less.
* CentOS 4.x/RHEL 4.x users - if you use the kernel that shipped with your distribution and didn't upgrade to the latest kernel RedHat/CentOS gives, you might have a problem loading the "fuse" kernel. Please upgrade to the latest kernel (2.6.16 or above) and make sure "fuse" kernel module is compiled and loadable since FUSE requires this kernel module and s3fs requires it as well.
* Moving/renaming/erasing files takes time since the whole file needs to be accessed first. A workaround could be to use s3fs's cache support with the use_cache option.

108
README.md Normal file
View File

@ -0,0 +1,108 @@
s3fs
====
s3fs allows Linux and Mac OS X to mount an S3 bucket via FUSE.
s3fs preserves the native object format for files, allowing use of other tools like [s3cmd](http://s3tools.org/s3cmd).
Features
--------
* large subset of POSIX including reading/writing files, directories, symlinks, mode, uid/gid, and extended attributes
* compatible with Amazon S3, Google Cloud Storage, and other S3-based object stores
* large files via multi-part upload
* renames via server-side copy
* optional server-side encryption
* data integrity via MD5 hashes
* in-memory metadata caching
* local disk data caching
* user-specified regions, including Amazon GovCloud
* authenticate via v2 or v4 signatures
Installation
------------
Ensure you have all the dependencies:
On Ubuntu 14.04:
```
sudo apt-get install automake autotools-dev g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
```
On CentOS 7:
```
sudo yum install automake fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
```
Compile from master via the following commands:
```
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make
sudo make install
```
Examples
--------
Enter your S3 identity and credential in a file `/path/to/passwd`:
```
echo MYIDENTITY:MYCREDENTIAL > /path/to/passwd
```
Make sure the file has proper permissions (if you get 'permissions' error when mounting) `/path/to/passwd`:
```
chmod 600 /path/to/passwd
```
Run s3fs with an existing bucket `mybucket` and directory `/path/to/mountpoint`:
```
s3fs mybucket /path/to/mountpoint -o passwd_file=/path/to/passwd
```
If you encounter any errors, enable debug output:
```
s3fs mybucket /path/to/mountpoint -o passwd_file=/path/to/passwd -d -d -f -o f2 -o curldbg
```
You can also mount on boot by entering the following line to `/etc/fstab`:
```
s3fs#mybucket /path/to/mountpoint fuse _netdev,allow_other 0 0
```
Limitations
-----------
Generally S3 cannot offer the same performance or semantics as a local file system. More specifically:
* random writes or appends to files require rewriting the entire file
* metadata operations such as listing directories have poor performance due to network latency
* [eventual consistency](https://en.wikipedia.org/wiki/Eventual_consistency) can temporarily yield stale data
* no atomic renames of files or directories
* no coordination between multiple clients mounting the same bucket
* no hard links
References
----------
* [s3backer](https://github.com/archiecobbs/s3backer) - mount an S3 bucket as a single file
* [s3fs-python](https://fedorahosted.org/s3fs/) - an older and less complete implementation written in Python
* [S3Proxy](https://github.com/andrewgaul/s3proxy) - combine with s3fs to mount EMC Atmos, Microsoft Azure, and OpenStack Swift buckets
* [s3ql](https://bitbucket.org/nikratio/s3ql/) - similar to s3fs but uses its own object format
* [YAS3FS](https://github.com/danilop/yas3fs) - similar to s3fs but uses SNS to allow multiple clients to mount a bucket
License
-------
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
Licensed under the GNU GPL version 2

22
autogen.sh
View File

@ -19,6 +19,28 @@
#
# See the file ChangeLog for a revision history.
echo "--- Make commit hash file -------"
SHORTHASH="unknown"
type git > /dev/null 2>&1
if [ $? -eq 0 -a -d .git ]; then
RESULT=`git rev-parse --short HEAD`
if [ $? -eq 0 ]; then
SHORTHASH=${RESULT}
fi
fi
echo ${SHORTHASH} > default_commit_hash
echo "--- Finished commit hash file ---"
echo "--- Start autotools -------------"
aclocal \
&& autoheader \
&& automake --add-missing \
&& autoconf
echo "--- Finished autotools ----------"
exit 0

40
configure.ac
View File

@ -21,15 +21,19 @@ dnl Process this file with autoconf to produce a configure script.
AC_PREREQ(2.59)
AC_INIT(s3fs, 1.79)
AC_CONFIG_HEADER([config.h])
AC_CANONICAL_SYSTEM
AM_INIT_AUTOMAKE()
AM_INIT_AUTOMAKE([foreign])
AC_PROG_CXX
AC_PROG_CC
CXXFLAGS="$CXXFLAGS -Wall -D_FILE_OFFSET_BITS=64"
dnl ----------------------------------------------
dnl For OSX
dnl ----------------------------------------------
case "$target" in
*-darwin* )
# Do something specific for mac
@ -219,12 +223,40 @@ AM_CONDITIONAL([USE_GNUTLS_NETTLE], [test "$auth_lib" = nettle])
AM_CONDITIONAL([USE_SSL_NSS], [test "$auth_lib" = nss])
dnl ----------------------------------------------
dnl end of ssl library
dnl check functions
dnl ----------------------------------------------
dnl malloc_trim function
AC_CHECK_FUNCS(malloc_trim, , )
AC_CHECK_FUNCS([malloc_trim])
dnl ----------------------------------------------
dnl output files
dnl ----------------------------------------------
AC_CONFIG_FILES(Makefile src/Makefile test/Makefile doc/Makefile)
dnl ----------------------------------------------
dnl short commit hash
dnl ----------------------------------------------
AC_CHECK_PROG([GITCMD], [git —version], [yes], [no])
AC_CHECK_FILE([.git], [DOTGITDIR=yes], [DOTGITDIR=no])
AC_MSG_CHECKING([github short commit hash])
if test “x${GITCMD}” = “xyes” -a “x${DOTGITDIR}” = “xyes”; then
GITCOMMITHASH=`git rev-parse --short HEAD`
elif test -f default_commit_hash; then
GITCOMMITHASH=`cat default_commit_hash`
else
GITCOMMITHASH="unknown"
fi
AC_MSG_RESULT([${GITCOMMITHASH}])
AC_DEFINE_UNQUOTED([COMMIT_HASH_VAL], ["${GITCOMMITHASH}"], [short commit hash value on github])
dnl ----------------------------------------------
dnl put
dnl ----------------------------------------------
AC_OUTPUT
dnl ----------------------------------------------
dnl end configuration
dnl ----------------------------------------------

66
doc/man/s3fs.1
View File

@ -62,19 +62,41 @@ local folder to use for local file cache.
\fB\-o\fR del_cache - delete local file cache
delete local file cache when s3fs starts and exits.
.TP
\fB\-o\fR storage_class (default is standard)
store object with specified storage class.
this option replaces the old option use_rrs.
Possible values: standard, standard_ia, and reduced_redundancy.
.TP
\fB\-o\fR use_rrs (default is disable)
use Amazon's Reduced Redundancy Storage.
this option can not be specified with use_sse.
(can specify use_rrs=1 for old version)
this option has been replaced by new storage_class option.
.TP
\fB\-o\fR use_sse (default is disable)
use Amazon's Server-Site Encryption or Server-Side Encryption with Customer-Provided Encryption Keys.
this option can not be specified with use_rrs. specifying only "use_sse" or "use_sse=1" enables Server-Side Encryption.(use_sse=1 for old version)
specifying this option with file path which has some SSE-C secret key enables Server-Side Encryption with Customer-Provided Encryption Keys.(use_sse=file)
the file must be 600 permission. the file can have some lines, each line is one SSE-C key. the first line in file is used as Customer-Provided Encryption Keys for uploading and change headers etc.
if there are some keys after first line, those are used downloading object which are encripted by not first key.
so that, you can keep all SSE-C keys in file, that is SSE-C key history.
if AWSSSECKEYS environment is set, you can set SSE-C key instead of this option.
Specify three type Amazon's Server-Site Encryption: SSE-S3, SSE-C or SSE-KMS. SSE-S3 uses Amazon S3-managed encryption keys, SSE-C uses customer-provided encryption keys, and SSE-KMS uses the master key which you manage in AWS KMS.
You can specify "use_sse" or "use_sse=1" enables SSE-S3 type (use_sse=1 is old type parameter).
Case of setting SSE-C, you can specify "use_sse=custom", "use_sse=custom:<custom key file path>" or "use_sse=<custom key file path>"(only <custom key file path> specified is old type parameter).
You can use "c" for short "custom".
The custom key file must be 600 permission. The file can have some lines, each line is one SSE-C key.
The first line in file is used as Customer-Provided Encryption Keys for uploading and changing headers etc.
If there are some keys after first line, those are used downloading object which are encrypted by not first key.
So that, you can keep all SSE-C keys in file, that is SSE-C key history.
If you specify "custom"("c") without file path, you need to set custom key by load_sse_c option or AWSSSECKEYS environment.(AWSSSECKEYS environment has some SSE-C keys with ":" separator.)
This option is used to decide the SSE type.
So that if you do not want to encrypt a object at uploading, but you need to decrypt encrypted object at downloaing, you can use load_sse_c option instead of this option.
For setting SSE-KMS, specify "use_sse=kmsid" or "use_sse=kmsid:<kms id>".
You can use "k" for short "kmsid".
If you san specify SSE-KMS type with your <kms id> in AWS KMS, you can set it after "kmsid:"(or "k:").
If you specify only "kmsid"("k"), you need to set AWSSSEKMSID environment which value is <kms id>.
You must be careful about that you can not use the KMS id which is not same EC2 region.
.TP
\fB\-o\fR load_sse_c - specify SSE-C keys
Specify the custom-provided encription keys file path for decrypting at duwnloading.
If you use the custom-provided encription key at uploading, you specify with "use_sse=custom".
The file has many lines, one line means one custom key.
So that you can keep all SSE-C keys in file, that is SSE-C key history.
AWSSSECKEYS environment is as same as this file contents.
.TP
\fB\-o\fR passwd_file (default="")
specify the path to the password file, which which takes precedence over the password in $HOME/.passwd-s3fs and /etc/passwd-s3fs
@ -118,7 +140,8 @@ s3fs always has to check whether file(or sub directory) exists under object(path
It increases ListBucket request and makes performance bad.
You can specify this option for performance, s3fs memorizes in stat cache that the object(file or directory) does not exist.
.TP
\fB\-o\fR no_check_certificate (by default this option is disabled) - do not check ssl certificate.
\fB\-o\fR no_check_certificate (by default this option is disabled)
do not check ssl certificate.
server certificate won't be checked against the available certificate authorities.
.TP
\fB\-o\fR nodnscache - disable dns cache.
@ -135,26 +158,23 @@ number of parallel request for uploading big objects.
s3fs uploads large object(default:over 20MB) by multipart post request, and sends parallel requests.
This option limits parallel request count which s3fs requests at once.
It is necessary to set this value depending on a CPU and a network band.
This option is lated to fd_page_size option and affects it.
.TP
\fB\-o\fR fd_page_size(default="52428800"(50MB))
number of internal management page size for each file descriptor.
For delayed reading and writing by s3fs, s3fs manages pages which is separated from object. Each pages has a status that data is already loaded(or not loaded yet).
This option should not be changed when you don't have a trouble with performance.
This value is changed automatically by parallel_count and multipart_size values(fd_page_size value = parallel_count * multipart_size).
.TP
\fB\-o\fR multipart_size(default="10"(10MB))
number of one part size in multipart uploading request.
The default size is 10MB(10485760byte), this value is minimum size.
Specify number of MB and over 10(MB).
This option is lated to fd_page_size option and affects it.
The default size is 10MB(10485760byte), minimum value is 5MB(5242880byte).
Specify number of MB and over 5(MB).
.TP
\fB\-o\fR ensure_diskfree(default the same as multipart_size value)
sets MB to ensure disk free space. This option means the threshold of free space size on disk which is used for the cache file by s3fs.
s3fs makes file for downloading, and uploading and caching files.
If the disk free space is smaller than this value, s3fs do not use diskspace as possible in exchange for the performance.
.TP
\fB\-o\fR url (default="http://s3.amazonaws.com")
sets the url to use to access Amazon S3. If you want to use HTTPS, then you can set url=https://s3.amazonaws.com
.TP
\fB\-o\fR endpoint (default="us-east-1")
sets the endpoint to use.
If this option is not specified, s3fs uses \"us-east-1\" region as the default.
If this option is not specified, s3fs uses "us-east-1" region as the default.
If the s3fs could not connect to the region specified by this option, s3fs could not run.
But if you do not specify this option, and if you can not connect with the default region, s3fs will retry to automatically connect to the other region.
So s3fs can know the correct region name, because s3fs can find it in an error from the S3 server.
@ -194,6 +214,14 @@ If this option is specified with nocopapi, the s3fs ignores it.
.TP
\fB\-o\fR use_path_request_style (use legacy API calling style)
Enble compatibility with S3-like APIs which do not support the virtual-host request style, by using the older path request style.
.TP
\fB\-o\fR dbglevel (default="crit")
Set the debug message level. set value as crit(critical), err(error), warn(warning), info(information) to debug level. default debug level is critical.
If s3fs run with "-d" option, the debug level is set information.
When s3fs catch the signal SIGUSR2, the debug level is bumpup.
.TP
\fB\-o\fR curldbg - put curl debug message
Put the debug message from libcurl when this option is specified.
.SH FUSE/MOUNT OPTIONS
.TP
Most of the generic mount options described in 'man mount' are supported (ro, rw, suid, nosuid, dev, nodev, exec, noexec, atime, noatime, sync async, dirsync). Filesystems are mounted with '\-onodev,nosuid' by default, which can only be overridden by a privileged user.

2
src/Makefile.am
View File

@ -39,6 +39,6 @@ s3fs_LDADD = $(DEPS_LIBS)
noinst_PROGRAMS = test_string_util
test_string_util_SOURCES = string_util.cpp test_string_util.cpp
test_string_util_SOURCES = string_util.cpp test_string_util.cpp test_util.h
TESTS = test_string_util

43
src/cache.cpp
View File

@ -164,11 +164,11 @@ bool StatCache::GetStat(string& key, struct stat* pst, headers_t* meta, bool ove
}
if(is_delete_cache){
// not hit by different ETag
DPRNNN("stat cache not hit by ETag[path=%s][time=%jd][hit count=%lu][ETag(%s)!=(%s)]",
S3FS_PRN_DBG("stat cache not hit by ETag[path=%s][time=%jd][hit count=%lu][ETag(%s)!=(%s)]",
strpath.c_str(), (intmax_t)(ent->cache_date), ent->hit_count, petag ? petag : "null", ent->meta["ETag"].c_str());
}else{
// hit
DPRNNN("stat cache hit [path=%s][time=%jd][hit count=%lu]", strpath.c_str(), (intmax_t)(ent->cache_date), ent->hit_count);
S3FS_PRN_DBG("stat cache hit [path=%s][time=%jd][hit count=%lu]", strpath.c_str(), (intmax_t)(ent->cache_date), ent->hit_count);
if(pst!= NULL){
*pst= ent->stbuf;
@ -245,12 +245,19 @@ bool StatCache::AddStat(std::string& key, headers_t& meta, bool forcedir)
if(CacheSize< 1){
return true;
}
DPRNNN("add stat cache entry[path=%s]", key.c_str());
S3FS_PRN_INFO3("add stat cache entry[path=%s]", key.c_str());
if(stat_cache.end() != stat_cache.find(key)){
pthread_mutex_lock(&StatCache::stat_cache_lock);
bool found = stat_cache.end() != stat_cache.find(key);
bool do_truncate = stat_cache.size() > CacheSize;
pthread_mutex_unlock(&StatCache::stat_cache_lock);
if(found){
DelStat(key.c_str());
}else{
if(stat_cache.size() > CacheSize){
if(do_truncate){
if(!TruncateCache()){
return false;
}
@ -300,12 +307,19 @@ bool StatCache::AddNoObjectCache(string& key)
if(CacheSize < 1){
return true;
}
DPRNNN("add no object cache entry[path=%s]", key.c_str());
S3FS_PRN_INFO3("add no object cache entry[path=%s]", key.c_str());
if(stat_cache.end() != stat_cache.find(key)){
pthread_mutex_lock(&StatCache::stat_cache_lock);
bool found = stat_cache.end() != stat_cache.find(key);
bool do_truncate = stat_cache.size() > CacheSize;
pthread_mutex_unlock(&StatCache::stat_cache_lock);
if(found){
DelStat(key.c_str());
}else{
if(stat_cache.size() > CacheSize){
if(do_truncate){
if(!TruncateCache()){
return false;
}
@ -330,17 +344,18 @@ bool StatCache::AddNoObjectCache(string& key)
bool StatCache::TruncateCache(void)
{
if(0 == stat_cache.size()){
pthread_mutex_lock(&StatCache::stat_cache_lock);
if(stat_cache.empty()){
pthread_mutex_unlock(&StatCache::stat_cache_lock);
return true;
}
pthread_mutex_lock(&StatCache::stat_cache_lock);
time_t lowest_time = time(NULL) + 1;
stat_cache_t::iterator iter_to_delete = stat_cache.end();
stat_cache_t::iterator iter;
for(iter = stat_cache.begin(); iter != stat_cache.end(); iter++) {
for(iter = stat_cache.begin(); iter != stat_cache.end(); ++iter) {
if((*iter).second){
if(lowest_time > (*iter).second->cache_date){
lowest_time = (*iter).second->cache_date;
@ -349,7 +364,7 @@ bool StatCache::TruncateCache(void)
}
}
if(stat_cache.end() != iter_to_delete){
DPRNNN("truncate stat cache[path=%s]", (*iter_to_delete).first.c_str());
S3FS_PRN_DBG("truncate stat cache[path=%s]", (*iter_to_delete).first.c_str());
if((*iter_to_delete).second){
delete (*iter_to_delete).second;
}
@ -367,7 +382,7 @@ bool StatCache::DelStat(const char* key)
if(!key){
return false;
}
DPRNNN("delete stat cache entry[path=%s]", key);
S3FS_PRN_INFO3("delete stat cache entry[path=%s]", key);
pthread_mutex_lock(&StatCache::stat_cache_lock);

135
src/common.h
View File

@ -17,64 +17,91 @@
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef S3FS_COMMON_H_
#define S3FS_COMMON_H_
#include "../config.h"
//
// Macro
//
#define SAFESTRPTR(strptr) (strptr ? strptr : "")
// for debug
#define FPRINT_NEST_SPACE_0 ""
#define FPRINT_NEST_SPACE_1 " "
#define FPRINT_NEST_SPACE_2 " "
#define FPRINT_NEST_CHECK(NEST) \
(0 == NEST ? FPRINT_NEST_SPACE_0 : 1 == NEST ? FPRINT_NEST_SPACE_1 : FPRINT_NEST_SPACE_2)
//
// Debug level
//
enum s3fs_log_level{
S3FS_LOG_CRIT = 0, // LOG_CRIT
S3FS_LOG_ERR = 1, // LOG_ERR
S3FS_LOG_WARN = 3, // LOG_WARNING
S3FS_LOG_INFO = 7, // LOG_INFO
S3FS_LOG_DBG = 15 // LOG_DEBUG
};
#define LOWFPRINT(NEST, ...) \
printf("%s%s(%d): ", FPRINT_NEST_CHECK(NEST), __func__, __LINE__); \
printf(__VA_ARGS__); \
printf("\n"); \
//
// Debug macros
//
#define IS_S3FS_LOG_CRIT() (S3FS_LOG_CRIT == debug_level)
#define IS_S3FS_LOG_ERR() (S3FS_LOG_ERR == (debug_level & S3FS_LOG_DBG))
#define IS_S3FS_LOG_WARN() (S3FS_LOG_WARN == (debug_level & S3FS_LOG_DBG))
#define IS_S3FS_LOG_INFO() (S3FS_LOG_INFO == (debug_level & S3FS_LOG_DBG))
#define IS_S3FS_LOG_DBG() (S3FS_LOG_DBG == (debug_level & S3FS_LOG_DBG))
#define FPRINT(NEST, ...) \
if(foreground){ \
LOWFPRINT(NEST, __VA_ARGS__); \
}
#define S3FS_LOG_LEVEL_TO_SYSLOG(level) \
( S3FS_LOG_DBG == (level & S3FS_LOG_DBG) ? LOG_DEBUG : \
S3FS_LOG_INFO == (level & S3FS_LOG_DBG) ? LOG_INFO : \
S3FS_LOG_WARN == (level & S3FS_LOG_DBG) ? LOG_WARNING : \
S3FS_LOG_ERR == (level & S3FS_LOG_DBG) ? LOG_ERR : LOG_CRIT )
#define FPRINT2(NEST, ...) \
if(foreground2){ \
LOWFPRINT(NEST, __VA_ARGS__); \
}
#define S3FS_LOG_LEVEL_STRING(level) \
( S3FS_LOG_DBG == (level & S3FS_LOG_DBG) ? "[DBG] " : \
S3FS_LOG_INFO == (level & S3FS_LOG_DBG) ? "[INF] " : \
S3FS_LOG_WARN == (level & S3FS_LOG_DBG) ? "[WAN] " : \
S3FS_LOG_ERR == (level & S3FS_LOG_DBG) ? "[ERR] " : "[CRT] " )
#define LOWSYSLOGPRINT(LEVEL, ...) \
syslog(LEVEL, __VA_ARGS__);
#define S3FS_LOG_NEST_MAX 4
#define S3FS_LOG_NEST(nest) (nest < S3FS_LOG_NEST_MAX ? s3fs_log_nest[nest] : s3fs_log_nest[S3FS_LOG_NEST_MAX - 1])
#define SYSLOGPRINT(LEVEL, ...) \
if(LEVEL <= LOG_CRIT || debug){ \
LOWSYSLOGPRINT(LEVEL, __VA_ARGS__); \
}
#define S3FS_LOW_LOGPRN(level, fmt, ...) \
if(S3FS_LOG_CRIT == level || (S3FS_LOG_CRIT != debug_level && level == (debug_level & level))){ \
if(foreground){ \
fprintf(stdout, "%s%s:%s(%d): " fmt "%s\n", S3FS_LOG_LEVEL_STRING(level), __FILE__, __func__, __LINE__, __VA_ARGS__); \
}else{ \
syslog(S3FS_LOG_LEVEL_TO_SYSLOG(level), "%s:%s(%d): " fmt "%s", __FILE__, __func__, __LINE__, __VA_ARGS__); \
} \
}
#define DPRINT(LEVEL, NEST, ...) \
FPRINT(NEST, __VA_ARGS__); \
SYSLOGPRINT(LEVEL, __VA_ARGS__);
#define S3FS_LOW_LOGPRN2(level, nest, fmt, ...) \
if(S3FS_LOG_CRIT == level || (S3FS_LOG_CRIT != debug_level && level == (debug_level & level))){ \
if(foreground){ \
fprintf(stdout, "%s%s%s:%s(%d): " fmt "%s\n", S3FS_LOG_LEVEL_STRING(level), S3FS_LOG_NEST(nest), __FILE__, __func__, __LINE__, __VA_ARGS__); \
}else{ \
syslog(S3FS_LOG_LEVEL_TO_SYSLOG(level), "%s" fmt "%s", S3FS_LOG_NEST(nest), __VA_ARGS__); \
} \
}
#define DPRINT2(LEVEL, ...) \
FPRINT2(2, __VA_ARGS__); \
SYSLOGPRINT(LEVEL, __VA_ARGS__);
#define S3FS_LOW_LOGPRN_EXIT(fmt, ...) \
if(foreground){ \
fprintf(stderr, "s3fs: " fmt "%s\n", __VA_ARGS__); \
}else{ \
syslog(S3FS_LOG_CRIT, "s3fs: " fmt "%s", __VA_ARGS__); \
}
// print debug message
#define FPRN(...) FPRINT(0, __VA_ARGS__)
#define FPRNN(...) FPRINT(1, __VA_ARGS__)
#define FPRNNN(...) FPRINT(2, __VA_ARGS__)
#define FPRNINFO(...) FPRINT2(2, __VA_ARGS__)
// print debug message with putting syslog
#define DPRNCRIT(...) DPRINT(LOG_CRIT, 0, __VA_ARGS__)
#define DPRN(...) DPRINT(LOG_ERR, 0, __VA_ARGS__)
#define DPRNN(...) DPRINT(LOG_DEBUG, 1, __VA_ARGS__)
#define DPRNNN(...) DPRINT(LOG_DEBUG, 2, __VA_ARGS__)
#define DPRNINFO(...) DPRINT2(LOG_INFO, __VA_ARGS__)
// [NOTE]
// small trick for VA_ARGS
//
#define S3FS_PRN_EXIT(fmt, ...) S3FS_LOW_LOGPRN_EXIT(fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_CRIT(fmt, ...) S3FS_LOW_LOGPRN(S3FS_LOG_CRIT, fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_ERR(fmt, ...) S3FS_LOW_LOGPRN(S3FS_LOG_ERR, fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_WARN(fmt, ...) S3FS_LOW_LOGPRN(S3FS_LOG_WARN, fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_DBG(fmt, ...) S3FS_LOW_LOGPRN(S3FS_LOG_DBG, fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_INFO(fmt, ...) S3FS_LOW_LOGPRN2(S3FS_LOG_INFO, 0, fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_INFO0(fmt, ...) S3FS_LOG_INFO(fmt, __VA_ARGS__)
#define S3FS_PRN_INFO1(fmt, ...) S3FS_LOW_LOGPRN2(S3FS_LOG_INFO, 1, fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_INFO2(fmt, ...) S3FS_LOW_LOGPRN2(S3FS_LOG_INFO, 2, fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_INFO3(fmt, ...) S3FS_LOW_LOGPRN2(S3FS_LOG_INFO, 3, fmt, ##__VA_ARGS__, "")
#define S3FS_PRN_CURL(fmt, ...) S3FS_LOW_LOGPRN2(S3FS_LOG_CRIT, 0, fmt, ##__VA_ARGS__, "")
//
// Typedef
@ -90,7 +117,7 @@ typedef struct xattr_value{
unsigned char* pvalue;
size_t length;
xattr_value(unsigned char* pval = NULL, size_t len = 0) : pvalue(pval), length(len) {}
explicit xattr_value(unsigned char* pval = NULL, size_t len = 0) : pvalue(pval), length(len) {}
~xattr_value()
{
if(pvalue){
@ -104,17 +131,17 @@ typedef std::map<std::string, PXATTRVAL> xattrs_t;
//
// Global valiables
//
extern bool debug;
extern bool foreground;
extern bool foreground2;
extern bool nomultipart;
extern bool pathrequeststyle;
extern std::string program_name;
extern std::string service_path;
extern std::string host;
extern std::string bucket;
extern std::string mount_prefix;
extern std::string endpoint;
extern bool foreground;
extern bool nomultipart;
extern bool pathrequeststyle;
extern std::string program_name;
extern std::string service_path;
extern std::string host;
extern std::string bucket;
extern std::string mount_prefix;
extern std::string endpoint;
extern s3fs_log_level debug_level;
extern const char* s3fs_log_nest[S3FS_LOG_NEST_MAX];
#endif // S3FS_COMMON_H_

98
src/common_auth.cpp
View File

@ -18,104 +18,20 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <string>
#include "s3fs_auth.h"
#include "string_util.h"
using namespace std;
//-------------------------------------------------------------------
// Utility Function
//-------------------------------------------------------------------
char* s3fs_base64(const unsigned char* input, size_t length)
{
static const char* base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
char* result;
if(!input || 0 >= length){
return NULL;
}
if(NULL == (result = (char*)malloc((((length / 3) + 1) * 4 + 1) * sizeof(char)))){
return NULL; // ENOMEM
}
unsigned char parts[4];
size_t rpos;
size_t wpos;
for(rpos = 0, wpos = 0; rpos < length; rpos += 3){
parts[0] = (input[rpos] & 0xfc) >> 2;
parts[1] = ((input[rpos] & 0x03) << 4) | ((((rpos + 1) < length ? input[rpos + 1] : 0x00) & 0xf0) >> 4);
parts[2] = (rpos + 1) < length ? (((input[rpos + 1] & 0x0f) << 2) | ((((rpos + 2) < length ? input[rpos + 2] : 0x00) & 0xc0) >> 6)) : 0x40;
parts[3] = (rpos + 2) < length ? (input[rpos + 2] & 0x3f) : 0x40;
result[wpos++] = base[parts[0]];
result[wpos++] = base[parts[1]];
result[wpos++] = base[parts[2]];
result[wpos++] = base[parts[3]];
}
result[wpos] = '\0';
return result;
}
inline unsigned char char_decode64(const char ch)
{
unsigned char by;
if('A' <= ch && ch <= 'Z'){ // A - Z
by = static_cast<unsigned char>(ch - 'A');
}else if('a' <= ch && ch <= 'z'){ // a - z
by = static_cast<unsigned char>(ch - 'a' + 26);
}else if('0' <= ch && ch <= '9'){ // 0 - 9
by = static_cast<unsigned char>(ch - '0' + 52);
}else if('+' == ch){ // +
by = 62;
}else if('/' == ch){ // /
by = 63;
}else if('=' == ch){ // =
by = 64;
}else{ // something wrong
by = 64;
}
return by;
}
unsigned char* s3fs_decode64(const char* input, size_t* plength)
{
unsigned char* result;
if(!input || 0 == strlen(input) || !plength){
return NULL;
}
if(NULL == (result = (unsigned char*)malloc((strlen(input) + 1)))){
return NULL; // ENOMEM
}
unsigned char parts[4];
size_t input_len = strlen(input);
size_t rpos;
size_t wpos;
for(rpos = 0, wpos = 0; rpos < input_len; rpos += 4){
parts[0] = char_decode64(input[rpos]);
parts[1] = (rpos + 1) < input_len ? char_decode64(input[rpos + 1]) : 64;
parts[2] = (rpos + 2) < input_len ? char_decode64(input[rpos + 2]) : 64;
parts[3] = (rpos + 3) < input_len ? char_decode64(input[rpos + 3]) : 64;
result[wpos++] = ((parts[0] << 2) & 0xfc) | ((parts[1] >> 4) & 0x03);
if(64 == parts[2]){
break;
}
result[wpos++] = ((parts[1] << 4) & 0xf0) | ((parts[2] >> 2) & 0x0f);
if(64 == parts[3]){
break;
}
result[wpos++] = ((parts[2] << 6) & 0xc0) | (parts[3] & 0x3f);
}
*plength = wpos;
return result;
}
string s3fs_get_content_md5(int fd)
{
unsigned char* md5hex;
@ -139,22 +55,16 @@ string s3fs_get_content_md5(int fd)
string s3fs_md5sum(int fd, off_t start, ssize_t size)
{
size_t digestlen = get_md5_digest_length();
char md5[2 * digestlen + 1];
char hexbuf[3];
unsigned char* md5hex;
if(NULL == (md5hex = s3fs_md5hexsum(fd, start, size))){
return string("");
}
memset(md5, 0, 2 * digestlen + 1);
for(size_t pos = 0; pos < digestlen; pos++){
snprintf(hexbuf, 3, "%02x", md5hex[pos]);
strncat(md5, hexbuf, 2);
}
std::string md5 = s3fs_hex(md5hex, digestlen);
free(md5hex);
return string(md5);
return md5;
}
string s3fs_sha256sum(int fd, off_t start, ssize_t size)

824
src/curl.cpp
View File

File diff suppressed because it is too large Load Diff

69
src/curl.h
View File

@ -20,6 +20,11 @@
#ifndef S3FS_CURL_H_
#define S3FS_CURL_H_
//----------------------------------------------
// Symbols
//----------------------------------------------
#define MIN_MULTIPART_SIZE 5242880 // 5MB
//----------------------------------------------
// class BodyData
//----------------------------------------------
@ -122,6 +127,21 @@ typedef std::map<std::string, std::string> iamcredmap_t;
typedef std::map<std::string, std::string> sseckeymap_t;
typedef std::list<sseckeymap_t> sseckeylist_t;
// strage class(rrs)
enum storage_class_t {
STANDARD,
STANDARD_IA,
REDUCED_REDUNDANCY,
};
// sse type
enum sse_type_t {
SSE_DISABLE = 0, // not use server side encrypting
SSE_S3, // server side encrypting by S3 key
SSE_C, // server side encrypting by custom key
SSE_KMS // server side encrypting by kms id
};
// share
#define SHARE_MUTEX_DNS 0
#define SHARE_MUTEX_SSL_SESSION 1
@ -165,9 +185,10 @@ class S3fsCurl
static int retries;
static bool is_public_bucket;
static std::string default_acl; // TODO: to enum
static bool is_use_rrs;
static storage_class_t storage_class;
static sseckeylist_t sseckeys;
static bool is_use_sse;
static std::string ssekmsid;
static sse_type_t ssetype;
static bool is_content_md5;
static bool is_verbose;
static std::string AWSAccessKeyId;
@ -206,12 +227,13 @@ class S3fsCurl
int b_postdata_remaining; // backup for retrying
off_t b_partdata_startpos; // backup for retrying
ssize_t b_partdata_size; // backup for retrying
bool b_ssekey_pos; // backup for retrying
std::string b_ssekey_md5; // backup for retrying
int b_ssekey_pos; // backup for retrying
std::string b_ssevalue; // backup for retrying
sse_type_t b_ssetype; // backup for retrying
public:
// constructor/destructor
S3fsCurl(bool ahbe = false);
explicit S3fsCurl(bool ahbe = false);
~S3fsCurl();
private:
@ -240,8 +262,12 @@ class S3fsCurl
static bool ParseIAMCredentialResponse(const char* response, iamcredmap_t& keyval);
static bool SetIAMCredentials(const char* response);
static bool LoadEnvSseCKeys(void);
static bool LoadEnvSseKmsid(void);
static bool PushbackSseKeys(std::string& onekey);
static int CurlDebugFunc(CURL* hcurl, curl_infotype type, char* data, size_t size, void* userptr);
// methods
bool ResetHandle(void);
bool RemakeHandle(void);
@ -252,10 +278,7 @@ class S3fsCurl
bool GetUploadId(std::string& upload_id);
int GetIAMCredentials(void);
int PreMultipartPostRequest(const char* tpath, headers_t& meta, std::string& upload_id, bool is_copy);
int CompleteMultipartPostRequest(const char* tpath, std::string& upload_id, etaglist_t& parts);
int UploadMultipartPostSetup(const char* tpath, int part_num, std::string& upload_id);
int UploadMultipartPostRequest(const char* tpath, int part_num, std::string& upload_id);
int CopyMultipartPostRequest(const char* from, const char* to, int part_num, std::string& upload_id, headers_t& meta);
public:
@ -278,16 +301,23 @@ class S3fsCurl
static bool SetPublicBucket(bool flag);
static bool IsPublicBucket(void) { return S3fsCurl::is_public_bucket; }
static std::string SetDefaultAcl(const char* acl);
static bool SetUseRrs(bool flag);
static bool GetUseRrs(void) { return S3fsCurl::is_use_rrs; }
static bool SetSseKeys(const char* filepath);
static bool LoadEnvSseKeys(void);
static storage_class_t SetStorageClass(storage_class_t storage_class);
static storage_class_t GetStorageClass() { return S3fsCurl::storage_class; }
static bool LoadEnvSse(void) { return (S3fsCurl::LoadEnvSseCKeys() && S3fsCurl::LoadEnvSseKmsid()); }
static sse_type_t SetSseType(sse_type_t type);
static sse_type_t GetSseType(void) { return S3fsCurl::ssetype; }
static bool IsSseDisable(void) { return (SSE_DISABLE == S3fsCurl::ssetype); }
static bool IsSseS3Type(void) { return (SSE_S3 == S3fsCurl::ssetype); }
static bool IsSseCType(void) { return (SSE_C == S3fsCurl::ssetype); }
static bool IsSseKmsType(void) { return (SSE_KMS == S3fsCurl::ssetype); }
static bool FinalCheckSse(void);
static bool SetSseCKeys(const char* filepath);
static bool SetSseKmsid(const char* kmsid);
static bool IsSetSseKmsId(void) { return !S3fsCurl::ssekmsid.empty(); }
static const char* GetSseKmsId(void) { return S3fsCurl::ssekmsid.c_str(); }
static bool GetSseKey(std::string& md5, std::string& ssekey);
static bool GetSseKeyMd5(int pos, std::string& md5);
static int GetSseKeyCount(void);
static bool IsSseCustomMode(void);
static bool SetUseSse(bool flag);
static bool GetUseSse(void) { return S3fsCurl::is_use_sse; }
static bool SetContentMd5(bool flag);
static bool SetVerbose(bool flag);
static bool GetVerbose(void) { return S3fsCurl::is_verbose; }
@ -310,7 +340,7 @@ class S3fsCurl
bool CreateCurlHandle(bool force = false);
bool DestroyCurlHandle(void);
bool AddSseKeyRequestHead(std::string& md5, bool is_copy);
bool AddSseRequestHead(sse_type_t ssetype, std::string& ssevalue, bool is_only_c, bool is_copy);
bool GetResponseCode(long& responseCode);
int RequestPerform(void);
int DeleteRequest(const char* tpath);
@ -321,14 +351,18 @@ class S3fsCurl
int HeadRequest(const char* tpath, headers_t& meta);
int PutHeadRequest(const char* tpath, headers_t& meta, bool is_copy);
int PutRequest(const char* tpath, headers_t& meta, int fd);
int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size, std::string& ssekeymd5);
int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size, sse_type_t ssetype, std::string& ssevalue);
int GetObjectRequest(const char* tpath, int fd, off_t start = -1, ssize_t size = -1);
int CheckBucket(void);
int ListBucketRequest(const char* tpath, const char* query);
int PreMultipartPostRequest(const char* tpath, headers_t& meta, std::string& upload_id, bool is_copy);
int CompleteMultipartPostRequest(const char* tpath, std::string& upload_id, etaglist_t& parts);
int UploadMultipartPostRequest(const char* tpath, int part_num, std::string& upload_id);
int MultipartListRequest(std::string& body);
int AbortMultipartUpload(const char* tpath, std::string& upload_id);
int MultipartHeadRequest(const char* tpath, off_t size, headers_t& meta, bool is_copy);
int MultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool is_copy);
int MultipartUploadRequest(std::string upload_id, const char* tpath, int fd, off_t offset, size_t size, etaglist_t& list);
int MultipartRenameRequest(const char* from, const char* to, headers_t& meta, off_t size);
// methods(valiables)
@ -433,6 +467,7 @@ std::string get_sorted_header_keys(const struct curl_slist* list);
std::string get_canonical_headers(const struct curl_slist* list, bool only_amz = false);
bool MakeUrlResource(const char* realpath, std::string& resourcepath, std::string& url);
std::string prepare_url(const char* url);
bool get_object_sse_type(const char* path, sse_type_t& ssetype, std::string& ssevalue); // implement in s3fs.cpp
#endif // S3FS_CURL_H_

1469
src/fdcache.cpp
View File

File diff suppressed because it is too large Load Diff

117
src/fdcache.h
View File

@ -20,6 +20,9 @@
#ifndef FD_CACHE_H_
#define FD_CACHE_H_
#include <sys/statvfs.h>
#include "curl.h"
//------------------------------------------------
// CacheFileStat
//------------------------------------------------
@ -34,8 +37,9 @@ class CacheFileStat
public:
static bool DeleteCacheFileStat(const char* path);
static bool CheckCacheFileStatTopDir(void);
CacheFileStat(const char* tpath = NULL);
explicit CacheFileStat(const char* tpath = NULL);
~CacheFileStat();
bool Open(void);
@ -52,40 +56,49 @@ struct fdpage
{
off_t offset;
size_t bytes;
bool init;
bool loaded;
fdpage(off_t start = 0, size_t size = 0, bool is_init = false)
: offset(start), bytes(size), init(is_init) {}
fdpage(off_t start = 0, size_t size = 0, bool is_loaded = false)
: offset(start), bytes(size), loaded(is_loaded) {}
off_t next(void) const { return (offset + bytes); }
off_t end(void) const { return (0 < bytes ? offset + bytes - 1 : 0); }
};
typedef std::list<struct fdpage*> fdpage_list_t;
class FdEntity;
//
// Management of loading area/modifying
//
class PageList
{
friend class FdEntity; // only one method access directly pages.
private:
fdpage_list_t pages;
private:
void Clear(void);
bool Compress(void);
bool Parse(off_t new_pos);
public:
static void FreeList(fdpage_list_t& list);
PageList(off_t size = 0, bool is_init = false);
explicit PageList(size_t size = 0, bool is_loaded = false);
~PageList();
off_t Size(void) const;
int Resize(off_t size, bool is_init);
int Init(off_t size, bool is_init);
bool IsInit(off_t start, off_t size);
bool SetInit(off_t start, off_t size, bool is_init = true);
bool FindUninitPage(off_t start, off_t& resstart, size_t& ressize);
int GetUninitPages(fdpage_list_t& uninit_list, off_t start = 0, off_t size = -1);
bool Init(size_t size, bool is_loaded);
size_t Size(void) const;
bool Resize(size_t size, bool is_loaded);
bool IsPageLoaded(off_t start = 0, size_t size = 0) const; // size=0 is checking to end of list
bool SetPageLoadedStatus(off_t start, size_t size, bool is_loaded = true, bool is_compress = true);
bool FindUnloadedPage(off_t start, off_t& resstart, size_t& ressize) const;
size_t GetTotalUnloadedPageSize(off_t start = 0, size_t size = 0) const; // size=0 is checking to end of list
int GetUnloadedPages(fdpage_list_t& unloaded_list, off_t start = 0, size_t size = 0) const; // size=0 is checking to end of list
bool Serialize(CacheFileStat& file, bool is_output);
void Dump(void);
};
@ -99,39 +112,61 @@ class FdEntity
pthread_mutex_t fdent_lock;
bool is_lock_init;
PageList pagelist;
int refcnt; // reference count
std::string path; // object path
std::string cachepath; // local cache file path
int fd; // file descriptor(tmp file or cache file)
FILE* file; // file pointer(tmp file or cache file)
bool is_modify; // if file is changed, this flag is true
int refcnt; // reference count
std::string path; // object path
std::string cachepath; // local cache file path
// (if this is empty, does not load/save pagelist.)
int fd; // file descriptor(tmp file or cache file)
FILE* pfile; // file pointer(tmp file or cache file)
bool is_modify; // if file is changed, this flag is true
headers_t orgmeta; // original headers at opening
size_t size_orgmeta; // original file size in original headers
std::string upload_id; // for no cached multipart uploading when no disk space
etaglist_t etaglist; // for no cached multipart uploading when no disk space
off_t mp_start; // start position for no cached multipart(write method only)
size_t mp_size; // size for no cached multipart(write method only)
private:
static int FillFile(int fd, unsigned char byte, size_t size, off_t start);
void Clear(void);
int Dup(void);
bool SetAllStatus(bool is_enable);
bool SetAllStatus(bool is_loaded); // [NOTE] not locking
//bool SetAllStatusLoaded(void) { return SetAllStatus(true); }
bool SetAllStatusUnloaded(void) { return SetAllStatus(false); }
public:
FdEntity(const char* tpath = NULL, const char* cpath = NULL);
explicit FdEntity(const char* tpath = NULL, const char* cpath = NULL);
~FdEntity();
void Close(void);
bool IsOpen(void) const { return (-1 != fd); }
int Open(off_t size = -1, time_t time = -1);
int Open(headers_t* pmeta = NULL, ssize_t size = -1, time_t time = -1);
bool OpenAndLoadAll(headers_t* pmeta = NULL, size_t* size = NULL, bool force_load = false);
int Dup(void);
const char* GetPath(void) const { return path.c_str(); }
void SetPath(const std::string &newpath) { path = newpath; }
int GetFd(void) const { return fd; }
int SetMtime(time_t time);
bool GetSize(off_t& size);
bool GetMtime(time_t& time);
bool GetStats(struct stat& st);
bool SetAllEnable(void) { return SetAllStatus(true); }
bool SetAllDisable(void) { return SetAllStatus(false); }
bool LoadFull(off_t* size = NULL, bool force_load = false);
int Load(off_t start, off_t size);
int RowFlush(const char* tpath, headers_t& meta, bool force_sync = false);
int Flush(headers_t& meta, bool force_sync = false) { return RowFlush(NULL, meta, force_sync); }
bool GetStats(struct stat& st);
int SetMtime(time_t time);
bool UpdateMtime(void);
bool GetSize(size_t& size);
bool SetMode(mode_t mode);
bool SetUId(uid_t uid);
bool SetGId(gid_t gid);
bool SetContentType(const char* path);
int Load(off_t start = 0, size_t size = 0); // size=0 means loading to end
int NoCacheLoadAndPost(off_t start = 0, size_t size = 0); // size=0 means loading to end
int NoCachePreMultipartPost(void);
int NoCacheMultipartPost(int tgfd, off_t start, size_t size);
int NoCacheCompleteMultipartPost(void);
int RowFlush(const char* tpath, bool force_sync = false);
int Flush(bool force_sync = false) { return RowFlush(NULL, force_sync); }
ssize_t Read(char* bytes, off_t start, size_t size, bool force_load = false);
ssize_t Write(const char* bytes, off_t start, size_t size);
};
@ -147,9 +182,12 @@ class FdManager
static pthread_mutex_t fd_manager_lock;
static bool is_lock_init;
static std::string cache_dir;
static size_t page_size;
static size_t free_disk_space; // limit free disk space
fdent_map_t fent;
fdent_map_t fent;
private:
static fsblkcnt_t GetFreeDiskSpace(const char* path);
public:
FdManager();
@ -163,16 +201,21 @@ class FdManager
static bool SetCacheDir(const char* dir);
static bool IsCacheDir(void) { return (0 < FdManager::cache_dir.size()); }
static const char* GetCacheDir(void) { return FdManager::cache_dir.c_str(); }
static size_t SetPageSize(size_t size);
static size_t GetPageSize(void) { return FdManager::page_size; }
static bool MakeCachePath(const char* path, std::string& cache_path, bool is_create_dir = true);
static bool CheckCacheTopDir(void);
static bool MakeRandomTempPath(const char* path, std::string& tmppath);
static size_t GetEnsureFreeDiskSpace(void) { return FdManager::free_disk_space; }
static size_t SetEnsureFreeDiskSpace(size_t size);
static size_t InitEnsureFreeDiskSpace(void) { return SetEnsureFreeDiskSpace(0); }
static bool IsSafeDiskSpace(const char* path, size_t size);
FdEntity* GetFdEntity(const char* path, int existfd = -1);
FdEntity* Open(const char* path, off_t size = -1, time_t time = -1, bool force_tmpfile = false, bool is_create = true);
FdEntity* Open(const char* path, headers_t* pmeta = NULL, ssize_t size = -1, time_t time = -1, bool force_tmpfile = false, bool is_create = true);
FdEntity* ExistOpen(const char* path, int existfd = -1);
void Rename(const std::string &from, const std::string &to);
bool Close(FdEntity* ent);
bool ChangeEntityToTempPath(FdEntity* ent, const char* path);
};
#endif // FD_CACHE_H_

14
src/gnutls_auth.cpp
View File

@ -217,7 +217,7 @@ unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
S3FS_PRN_ERR("file read error(%d)", errno);
return NULL;
}
md5_update(&ctx_md5, bytes, buf);
@ -261,7 +261,7 @@ unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
memset(buf, 0, 512);
if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_md5, GCRY_MD_MD5, 0))){
DPRNN("MD5 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
S3FS_PRN_ERR("MD5 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
return NULL;
}
@ -273,7 +273,7 @@ unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
S3FS_PRN_ERR("file read error(%d)", errno);
return NULL;
}
gcry_md_write(ctx_md5, buf, bytes);
@ -344,7 +344,7 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
S3FS_PRN_ERR("file read error(%d)", errno);
return NULL;
}
sha256_update(&ctx_sha256, bytes, buf);
@ -375,7 +375,7 @@ bool s3fs_sha256(const unsigned char* data, unsigned int datalen, unsigned char*
gcry_md_hd_t ctx_sha256;
gcry_error_t err;
if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_sha256, GCRY_MD_SHA256, 0))){
DPRNN("SHA256 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
S3FS_PRN_ERR("SHA256 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
free(*digest);
return false;
}
@ -409,7 +409,7 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
memset(buf, 0, 512);
if(GPG_ERR_NO_ERROR != (err = gcry_md_open(&ctx_sha256, GCRY_MD_SHA256, 0))){
DPRNN("SHA256 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
S3FS_PRN_ERR("SHA256 context creation failure: %s/%s", gcry_strsource(err), gcry_strerror(err));
return NULL;
}
@ -421,7 +421,7 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
S3FS_PRN_ERR("file read error(%d)", errno);
return NULL;
}
gcry_md_write(ctx_sha256, buf, bytes);

5
src/nss_auth.cpp
View File

@ -182,7 +182,7 @@ unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
S3FS_PRN_ERR("file read error(%d)", errno);
return NULL;
}
PK11_DigestOp(md5ctx, buf, bytes);
@ -262,7 +262,8 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
S3FS_PRN_ERR("file read error(%d)", errno);
PK11_DestroyContext(sha256ctx, PR_TRUE);
return NULL;
}
PK11_DigestOp(sha256ctx, buf, bytes);

23
src/openssl_auth.cpp
View File

@ -105,7 +105,7 @@ static struct CRYPTO_dynlock_value* s3fs_dyn_crypt_mutex(const char* file, int l
struct CRYPTO_dynlock_value* dyndata;
if(NULL == (dyndata = static_cast<struct CRYPTO_dynlock_value*>(malloc(sizeof(struct CRYPTO_dynlock_value))))){
DPRNCRIT("Could not allocate memory for CRYPTO_dynlock_value");
S3FS_PRN_CRIT("Could not allocate memory for CRYPTO_dynlock_value");
return NULL;
}
pthread_mutex_init(&(dyndata->dyn_mutex), NULL);
@ -134,14 +134,14 @@ static void s3fs_destroy_dyn_crypt_mutex(struct CRYPTO_dynlock_value* dyndata, c
bool s3fs_init_crypt_mutex(void)
{
if(s3fs_crypt_mutex){
FPRNNN("s3fs_crypt_mutex is not NULL, destroy it.");
S3FS_PRN_DBG("s3fs_crypt_mutex is not NULL, destroy it.");
if(!s3fs_destroy_crypt_mutex()){
DPRN("Failed to s3fs_crypt_mutex");
S3FS_PRN_ERR("Failed to s3fs_crypt_mutex");
return false;
}
}
if(NULL == (s3fs_crypt_mutex = static_cast<pthread_mutex_t*>(malloc(CRYPTO_num_locks() * sizeof(pthread_mutex_t))))){
DPRNCRIT("Could not allocate memory for s3fs_crypt_mutex");
S3FS_PRN_CRIT("Could not allocate memory for s3fs_crypt_mutex");
return false;
}
for(int cnt = 0; cnt < CRYPTO_num_locks(); cnt++){
@ -250,7 +250,7 @@ unsigned char* s3fs_md5hexsum(int fd, off_t start, ssize_t size)
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
S3FS_PRN_ERR("file read error(%d)", errno);
return NULL;
}
MD5_Update(&md5ctx, buf, bytes);
@ -297,10 +297,8 @@ bool s3fs_sha256(const unsigned char* data, unsigned int datalen, unsigned char*
unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
{
const EVP_MD* md = EVP_get_digestbyname("sha256");
EVP_MD_CTX* sha256ctx = EVP_MD_CTX_create();
EVP_DigestInit_ex(sha256ctx, md, NULL);
const EVP_MD* md = EVP_get_digestbyname("sha256");
EVP_MD_CTX* sha256ctx;
char buf[512];
ssize_t bytes;
unsigned char* result;
@ -318,6 +316,9 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
return NULL;
}
sha256ctx = EVP_MD_CTX_create();
EVP_DigestInit_ex(sha256ctx, md, NULL);
memset(buf, 0, 512);
for(ssize_t total = 0; total < size; total += bytes){
bytes = 512 < (size - total) ? 512 : (size - total);
@ -327,13 +328,15 @@ unsigned char* s3fs_sha256hexsum(int fd, off_t start, ssize_t size)
break;
}else if(-1 == bytes){
// error
DPRNNN("file read error(%d)", errno);
S3FS_PRN_ERR("file read error(%d)", errno);
EVP_MD_CTX_destroy(sha256ctx);
return NULL;
}
EVP_DigestUpdate(sha256ctx, buf, bytes);
memset(buf, 0, 512);
}
if(NULL == (result = (unsigned char*)malloc(get_sha256_digest_length()))){
EVP_MD_CTX_destroy(sha256ctx);
return NULL;
}
EVP_DigestFinal_ex(sha256ctx, result, NULL);

1105
src/s3fs.cpp
View File

File diff suppressed because it is too large Load Diff

2
src/s3fs.h
View File

@ -84,8 +84,6 @@
#endif // HAVE_MALLOC_TRIM
char* get_object_sseckey_md5(const char* path);
#endif // S3FS_S3_H_
/*

5
src/s3fs_auth.h
View File

@ -20,14 +20,15 @@
#ifndef S3FS_AUTH_H_
#define S3FS_AUTH_H_
#include <string>
#include <sys/types.h>
//-------------------------------------------------------------------
// Utility functions for Authentication
//-------------------------------------------------------------------
//
// in common_auth.cpp
//
char* s3fs_base64(const unsigned char* input, size_t length);
unsigned char* s3fs_decode64(const char* input, size_t* plength);
std::string s3fs_get_content_md5(int fd);
std::string s3fs_md5sum(int fd, off_t start, ssize_t size);
std::string s3fs_sha256sum(int fd, off_t start, ssize_t size);

243
src/s3fs_util.cpp
View File

@ -233,7 +233,7 @@ bool S3ObjList::GetLastName(std::string& lastname) const
{
bool result = false;
lastname = "";
for(s3obj_t::const_iterator iter = objects.begin(); iter != objects.end(); iter++){
for(s3obj_t::const_iterator iter = objects.begin(); iter != objects.end(); ++iter){
if((*iter).second.orgname.length()){
if(0 > strcmp(lastname.c_str(), (*iter).second.orgname.c_str())){
lastname = (*iter).second.orgname;
@ -253,7 +253,7 @@ bool S3ObjList::GetNameList(s3obj_list_t& list, bool OnlyNormalized, bool CutSla
{
s3obj_t::const_iterator iter;
for(iter = objects.begin(); objects.end() != iter; iter++){
for(iter = objects.begin(); objects.end() != iter; ++iter){
if(OnlyNormalized && 0 != (*iter).second.normalname.length()){
continue;
}
@ -275,7 +275,7 @@ bool S3ObjList::MakeHierarchizedList(s3obj_list_t& list, bool haveSlash)
s3obj_h_t::iterator hiter;
s3obj_list_t::const_iterator liter;
for(liter = list.begin(); list.end() != liter; liter++){
for(liter = list.begin(); list.end() != liter; ++liter){
string strtmp = (*liter);
if(1 < strtmp.length() && '/' == strtmp[strtmp.length() - 1]){
strtmp = strtmp.substr(0, strtmp.length() - 1);
@ -425,51 +425,14 @@ void free_mvnodes(MVNODE *head)
//-------------------------------------------------------------------
// Class AutoLock
//-------------------------------------------------------------------
AutoLock::AutoLock(pthread_mutex_t* pmutex) : auto_mutex(pmutex), is_locked(false)
AutoLock::AutoLock(pthread_mutex_t* pmutex) : auto_mutex(pmutex)
{
Lock();
pthread_mutex_lock(auto_mutex);
}
AutoLock::~AutoLock()
{
Unlock();
}
bool AutoLock::Lock(void)
{
if(!auto_mutex){
return false;
}
if(is_locked){
// already locked
return true;
}
try{
pthread_mutex_lock(auto_mutex);
is_locked = true;
}catch(exception& e){
is_locked = false;
return false;
}
return true;
}
bool AutoLock::Unlock(void)
{
if(!auto_mutex){
return false;
}
if(!is_locked){
// already unlocked
return true;
}
try{
pthread_mutex_unlock(auto_mutex);
is_locked = false;
}catch(exception& e){
return false;
}
return true;
pthread_mutex_unlock(auto_mutex);
}
//-------------------------------------------------------------------
@ -479,7 +442,6 @@ bool AutoLock::Unlock(void)
string get_username(uid_t uid)
{
static size_t maxlen = 0; // set onece
int result;
char* pbuf;
struct passwd pwinfo;
struct passwd* ppwinfo = NULL;
@ -488,19 +450,19 @@ string get_username(uid_t uid)
if(0 == maxlen){
long res = sysconf(_SC_GETPW_R_SIZE_MAX);
if(0 > res){
DPRNNN("could not get max pw length.");
S3FS_PRN_WARN("could not get max pw length.");
maxlen = 0;
return string("");
}
maxlen = res;
}
if(NULL == (pbuf = (char*)malloc(sizeof(char) * maxlen))){
DPRNCRIT("failed to allocate memory.");
S3FS_PRN_CRIT("failed to allocate memory.");
return string("");
}
// get group information
if(0 != (result = getpwuid_r(uid, &pwinfo, pbuf, maxlen, &ppwinfo))){
DPRNNN("could not get pw information.");
if(0 != getpwuid_r(uid, &pwinfo, pbuf, maxlen, &ppwinfo)){
S3FS_PRN_WARN("could not get pw information.");
free(pbuf);
return string("");
}
@ -526,19 +488,19 @@ int is_uid_inculde_group(uid_t uid, gid_t gid)
if(0 == maxlen){
long res = sysconf(_SC_GETGR_R_SIZE_MAX);
if(0 > res){
DPRNNN("could not get max name length.");
S3FS_PRN_ERR("could not get max name length.");
maxlen = 0;
return -ERANGE;
}
maxlen = res;
}
if(NULL == (pbuf = (char*)malloc(sizeof(char) * maxlen))){
DPRNCRIT("failed to allocate memory.");
S3FS_PRN_CRIT("failed to allocate memory.");
return -ENOMEM;
}
// get group information
if(0 != (result = getgrgid_r(gid, &ginfo, pbuf, maxlen, &pginfo))){
DPRNNN("could not get group information.");
S3FS_PRN_ERR("could not get group information.");
free(pbuf);
return -result;
}
@ -584,23 +546,80 @@ string mybasename(string path)
// mkdir --parents
int mkdirp(const string& path, mode_t mode)
{
string base;
string component;
string base;
string component;
stringstream ss(path);
while (getline(ss, component, '/')) {
base += "/" + component;
mkdir(base.c_str(), mode);
struct stat st;
if(0 == stat(base.c_str(), &st)){
if(!S_ISDIR(st.st_mode)){
return EPERM;
}
}else{
if(0 != mkdir(base.c_str(), mode)){
return errno;
}
}
}
return 0;
}
bool check_exist_dir_permission(const char* dirpath)
{
if(!dirpath || '\0' == dirpath[0]){
return false;
}
// exists
struct stat st;
if(0 != stat(dirpath, &st)){
if(ENOENT == errno){
// dir does not exitst
return true;
}
if(EACCES == errno){
// could not access directory
return false;
}
// somthing error occured
return false;
}
// check type
if(!S_ISDIR(st.st_mode)){
// path is not directory
return false;
}
// check permission
uid_t myuid = geteuid();
if(myuid == st.st_uid){
if(S_IRWXU != (st.st_mode & S_IRWXU)){
return false;
}
}else{
if(1 == is_uid_inculde_group(myuid, st.st_gid)){
if(S_IRWXG != (st.st_mode & S_IRWXG)){
return false;
}
}else{
if(S_IRWXO != (st.st_mode & S_IRWXO)){
return false;
}
}
}
return true;
}
bool delete_files_in_dir(const char* dir, bool is_remove_own)
{
DIR* dp;
struct dirent* dent;
if(NULL == (dp = opendir(dir))){
DPRNINFO("could not open dir(%s) - errno(%d)", dir, errno);
S3FS_PRN_ERR("could not open dir(%s) - errno(%d)", dir, errno);
return false;
}
@ -613,20 +632,20 @@ bool delete_files_in_dir(const char* dir, bool is_remove_own)
fullpath += dent->d_name;
struct stat st;
if(0 != lstat(fullpath.c_str(), &st)){
DPRN("could not get stats of file(%s) - errno(%d)", fullpath.c_str(), errno);
S3FS_PRN_ERR("could not get stats of file(%s) - errno(%d)", fullpath.c_str(), errno);
closedir(dp);
return false;
}
if(S_ISDIR(st.st_mode)){
// dir -> Reentrant
if(!delete_files_in_dir(fullpath.c_str(), true)){
DPRNINFO("could not remove sub dir(%s) - errno(%d)", fullpath.c_str(), errno);
S3FS_PRN_ERR("could not remove sub dir(%s) - errno(%d)", fullpath.c_str(), errno);
closedir(dp);
return false;
}
}else{
if(0 != unlink(fullpath.c_str())){
DPRN("could not remove file(%s) - errno(%d)", fullpath.c_str(), errno);
S3FS_PRN_ERR("could not remove file(%s) - errno(%d)", fullpath.c_str(), errno);
closedir(dp);
return false;
}
@ -635,7 +654,7 @@ bool delete_files_in_dir(const char* dir, bool is_remove_own)
closedir(dp);
if(is_remove_own && 0 != rmdir(dir)){
DPRN("could not remove dir(%s) - errno(%d)", dir, errno);
S3FS_PRN_ERR("could not remove dir(%s) - errno(%d)", dir, errno);
return false;
}
return true;
@ -875,28 +894,54 @@ void show_help (void)
" del_cache (delete local file cache)\n"
" - delete local file cache when s3fs starts and exits.\n"
"\n"
" use_rrs (default is disable)\n"
" - this option makes Amazon's Reduced Redundancy Storage enable.\n"
" storage_class (default=\"standard\")\n"
" - store object with specified storage class. Possible values:\n"
" standard, standard_ia, and reduced_redundancy.\n"
"\n"
" use_sse (default is disable)\n"
" - use Amazon's Server-Site Encryption or Server-Side Encryption\n"
" with Customer-Provided Encryption Keys.\n"
" this option can not be specified with use_rrs. specifying only \n"
" \"use_sse\" or \"use_sse=1\" enables Server-Side Encryption.\n"
" (use_sse=1 for old version)\n"
" specifying this option with file path which has some SSE-C\n"
" secret key enables Server-Side Encryption with Customer-Provided\n"
" Encryption Keys.(use_sse=file)\n"
" the file must be 600 permission. the file can have some lines,\n"
" each line is one SSE-C key. the first line in file is used as\n"
" Customer-Provided Encryption Keys for uploading and changing\n"
" headers etc.\n"
" if there are some keys after first line, those are used\n"
" downloading object which are encrypted by not first key.\n"
" so that, you can keep all SSE-C keys in file, that is SSE-C\n"
" key history.\n"
" if AWSSSECKEYS environment is set, you can set SSE-C key instead\n"
" - Specify three type Amazon's Server-Site Encryption: SSE-S3,\n"
" SSE-C or SSE-KMS. SSE-S3 uses Amazon S3-managed encryption\n"
" keys, SSE-C uses customer-provided encryption keys, and\n"
" SSE-KMS uses the master key which you manage in AWS KMS.\n"
" You can specify \"use_sse\" or \"use_sse=1\" enables SSE-S3\n"
" type(use_sse=1 is old type parameter).\n"
" Case of setting SSE-C, you can specify \"use_sse=custom\",\n"
" \"use_sse=custom:<custom key file path>\" or\n"
" \"use_sse=<custom key file path>\"(only <custom key file path>\n"
" specified is old type parameter). You can use \"c\" for\n"
" short \"custom\".\n"
" The custom key file must be 600 permission. The file can\n"
" have some lines, each line is one SSE-C key. The first line\n"
" in file is used as Customer-Provided Encryption Keys for\n"
" uploading and changing headers etc. If there are some keys\n"
" after first line, those are used downloading object which\n"
" are encrypted by not first key. So that, you can keep all\n"
" SSE-C keys in file, that is SSE-C key history.\n"
" If you specify \"custom\"(\"c\") without file path, you\n"
" need to set custom key by load_sse_c option or AWSSSECKEYS\n"
" environment.(AWSSSECKEYS environment has some SSE-C keys\n"
" with \":\" separator.) This option is used to decide the\n"
" SSE type. So that if you do not want to encrypt a object\n"
" object at uploading, but you need to decrypt encrypted\n"
" object at downloaing, you can use load_sse_c option instead\n"
" of this option.\n"
" For setting SSE-KMS, specify \"use_sse=kmsid\" or\n"
" \"use_sse=kmsid:<kms id>\". You can use \"k\" for short \"kmsid\".\n"
" If you san specify SSE-KMS type with your <kms id> in AWS\n"
" KMS, you can set it after \"kmsid:\"(or \"k:\"). If you\n"
" specify only \"kmsid\"(\"k\"), you need to set AWSSSEKMSID\n"
" environment which value is <kms id>. You must be careful\n"
" about that you can not use the KMS id which is not same EC2\n"
" region.\n"
"\n"
" load_sse_c - specify SSE-C keys\n"
" Specify the custom-provided encription keys file path for decrypting\n"
" at duwnloading.\n"
" If you use the custom-provided encription key at uploading, you\n"
" specify with \"use_sse=custom\". The file has many lines, one line\n"
" means one custom key. So that you can keep all SSE-C keys in file,\n"
" that is SSE-C key history. AWSSSECKEYS environment is as same as this\n"
" file contents.\n"
"\n"
" public_bucket (default=\"\" which means disabled)\n"
" - anonymously mount a public bucket when set to 1\n"
@ -925,10 +970,10 @@ void show_help (void)
" If you specify this option for set \"Content-Encoding\" HTTP \n"
" header, please take care for RFC 2616.\n"
"\n"
" connect_timeout (default=\"10\" seconds)\n"
" connect_timeout (default=\"300\" seconds)\n"
" - time to wait for connection before giving up\n"
"\n"
" readwrite_timeout (default=\"30\" seconds)\n"
" readwrite_timeout (default=\"60\" seconds)\n"
" - time to wait between read/write activity before giving up\n"
"\n"
" max_stat_cache_size (default=\"1000\" entries (about 4MB))\n"
@ -948,7 +993,8 @@ void show_help (void)
" in stat cache that the object(file or directory) does not exist.\n"
"\n"
" no_check_certificate\n"
" - server certificate won't be checked against the available certificate authorities.\n"
" - server certificate won't be checked against the available \n"
" certificate authorities.\n"
"\n"
" nodnscache (disable dns cache)\n"
" - s3fs is always using dns cache, this option make dns cache disable.\n"
@ -971,13 +1017,15 @@ void show_help (void)
" multipart_size (default=\"10\")\n"
" - part size, in MB, for each multipart request.\n"
"\n"
" fd_page_size (default=\"52428800\"(50MB))\n"
" - number of internal management page size for each file descriptor.\n"
" For delayed reading and writing by s3fs, s3fs manages pages which \n"
" is separated from object. Each pages has a status that data is \n"
" already loaded(or not loaded yet).\n"
" This option should not be changed when you don't have a trouble \n"
" with performance.\n"
" ensure_diskfree (default same multipart_size value)\n"
" - sets MB to ensure disk free space. s3fs makes file for\n"
" downloading, uploading and caching files. If the disk free\n"
" space is smaller than this value, s3fs do not use diskspace\n"
" as possible in exchange for the performance.\n"
"\n"
" singlepart_copy_limit (default=\"5120\")\n"
" - maximum size, in MB, of a single-part copy before trying \n"
" multipart copy.\n"
"\n"
" url (default=\"http://s3.amazonaws.com\")\n"
" - sets the url to use to access amazon s3\n"
@ -1006,7 +1054,7 @@ void show_help (void)
" nomultipart (disable multipart uploads)\n"
"\n"
" enable_content_md5 (default is disable)\n"
" - verifying uploaded object without multipart by content-md5 header.\n"
" - ensure data integrity during writes with MD5 hash.\n"
"\n"
" iam_role (default is no role)\n"
" - set the IAM Role that will supply the credentials from the \n"
@ -1040,6 +1088,16 @@ void show_help (void)
" the virtual-host request style, by using the older path request\n"
" style.\n"
"\n"
" dbglevel (default=\"crit\")\n"
" Set the debug message level. set value as crit(critical), err\n"
" (error), warn(warning), info(information) to debug level.\n"
" default debug level is critical. If s3fs run with \"-d\" option,\n"
" the debug level is set information. When s3fs catch the signal\n"
" SIGUSR2, the debug level is bumpup.\n"
"\n"
" curldbg - put curl debug message\n"
" Put the debug message from libcurl when this option is specified.\n"
"\n"
"FUSE/mount Options:\n"
"\n"
" Most of the generic mount options described in 'man mount' are\n"
@ -1062,8 +1120,7 @@ void show_help (void)
" disable multi-threaded operation\n"
"\n"
"\n"
"Report bugs to <s3fs-devel@googlegroups.com>\n"
"s3fs home page: <http://code.google.com/p/s3fs/>\n"
"s3fs home page: <https://github.com/s3fs-fuse/s3fs-fuse>\n"
);
return;
}
@ -1071,12 +1128,12 @@ void show_help (void)
void show_version(void)
{
printf(
"Amazon Simple Storage Service File System V%s with %s\n"
"Amazon Simple Storage Service File System V%s(commit:%s) with %s\n"
"Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>\n"
"License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>\n"
"This is free software: you are free to change and redistribute it.\n"
"There is NO WARRANTY, to the extent permitted by law.\n",
VERSION, s3fs_crypt_lib_name());
VERSION, COMMIT_HASH_VAL, s3fs_crypt_lib_name());
return;
}

7
src/s3fs_util.h
View File

@ -88,14 +88,10 @@ class AutoLock
{
private:
pthread_mutex_t* auto_mutex;
bool is_locked;
public:
AutoLock(pthread_mutex_t* pmutex = NULL);
explicit AutoLock(pthread_mutex_t* pmutex);
~AutoLock();
bool Lock(void);
bool Unlock(void);
};
//-------------------------------------------------------------------
@ -113,6 +109,7 @@ int is_uid_inculde_group(uid_t uid, gid_t gid);
std::string mydirname(std::string path);
std::string mybasename(std::string path);
int mkdirp(const std::string& path, mode_t mode);
bool check_exist_dir_permission(const char* dirpath);
bool delete_files_in_dir(const char* dir, bool is_remove_own);
time_t get_mtime(const char *s);

106
src/string_util.cpp
View File

@ -17,6 +17,7 @@
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
@ -85,13 +86,6 @@ string lower(string s)
return s;
}
string IntToStr(int n)
{
stringstream result;
result << n;
return result.str();
}
string trim_left(const string &s, const string &t /* = SPACES */)
{
string d(s);
@ -275,6 +269,104 @@ string get_date_iso8601(time_t tm)
return buf;
}
std::string s3fs_hex(const unsigned char* input, size_t length)
{
std::string hex;
for(size_t pos = 0; pos < length; ++pos){
char hexbuf[3];
snprintf(hexbuf, 3, "%02x", input[pos]);
hex += hexbuf;
}
return hex;
}
char* s3fs_base64(const unsigned char* input, size_t length)
{
static const char* base = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";
char* result;
if(!input || 0 >= length){
return NULL;
}
if(NULL == (result = (char*)malloc((((length / 3) + 1) * 4 + 1) * sizeof(char)))){
return NULL; // ENOMEM
}
unsigned char parts[4];
size_t rpos;
size_t wpos;
for(rpos = 0, wpos = 0; rpos < length; rpos += 3){
parts[0] = (input[rpos] & 0xfc) >> 2;
parts[1] = ((input[rpos] & 0x03) << 4) | ((((rpos + 1) < length ? input[rpos + 1] : 0x00) & 0xf0) >> 4);
parts[2] = (rpos + 1) < length ? (((input[rpos + 1] & 0x0f) << 2) | ((((rpos + 2) < length ? input[rpos + 2] : 0x00) & 0xc0) >> 6)) : 0x40;
parts[3] = (rpos + 2) < length ? (input[rpos + 2] & 0x3f) : 0x40;
result[wpos++] = base[parts[0]];
result[wpos++] = base[parts[1]];
result[wpos++] = base[parts[2]];
result[wpos++] = base[parts[3]];
}
result[wpos] = '\0';
return result;
}
inline unsigned char char_decode64(const char ch)
{
unsigned char by;
if('A' <= ch && ch <= 'Z'){ // A - Z
by = static_cast<unsigned char>(ch - 'A');
}else if('a' <= ch && ch <= 'z'){ // a - z
by = static_cast<unsigned char>(ch - 'a' + 26);
}else if('0' <= ch && ch <= '9'){ // 0 - 9
by = static_cast<unsigned char>(ch - '0' + 52);
}else if('+' == ch){ // +
by = 62;
}else if('/' == ch){ // /
by = 63;
}else if('=' == ch){ // =
by = 64;
}else{ // something wrong
by = UCHAR_MAX;
}
return by;
}
unsigned char* s3fs_decode64(const char* input, size_t* plength)
{
unsigned char* result;
if(!input || 0 == strlen(input) || !plength){
return NULL;
}
if(NULL == (result = (unsigned char*)malloc((strlen(input) + 1)))){
return NULL; // ENOMEM
}
unsigned char parts[4];
size_t input_len = strlen(input);
size_t rpos;
size_t wpos;
for(rpos = 0, wpos = 0; rpos < input_len; rpos += 4){
parts[0] = char_decode64(input[rpos]);
parts[1] = (rpos + 1) < input_len ? char_decode64(input[rpos + 1]) : 64;
parts[2] = (rpos + 2) < input_len ? char_decode64(input[rpos + 2]) : 64;
parts[3] = (rpos + 3) < input_len ? char_decode64(input[rpos + 3]) : 64;
result[wpos++] = ((parts[0] << 2) & 0xfc) | ((parts[1] >> 4) & 0x03);
if(64 == parts[2]){
break;
}
result[wpos++] = ((parts[1] << 4) & 0xf0) | ((parts[2] >> 2) & 0x0f);
if(64 == parts[3]){
break;
}
result[wpos++] = ((parts[2] << 6) & 0xc0) | (parts[3] & 0x3f);
}
result[wpos] = '\0';
*plength = wpos;
return result;
}
/*
* Local variables:
* tab-width: 4

5
src/string_util.h
View File

@ -45,7 +45,6 @@ std::string trim_left(const std::string &s, const std::string &t = SPACES);
std::string trim_right(const std::string &s, const std::string &t = SPACES);
std::string trim(const std::string &s, const std::string &t = SPACES);
std::string lower(std::string s);
std::string IntToStr(int);
std::string get_date_rfc850(void);
void get_date_sigv3(std::string& date, std::string& date8601);
std::string get_date_string(time_t tm);
@ -56,6 +55,10 @@ std::string urlDecode(const std::string& s);
bool takeout_str_dquart(std::string& str);
bool get_keyword_value(std::string& target, const char* keyword, std::string& value);
std::string s3fs_hex(const unsigned char* input, size_t length);
char* s3fs_base64(const unsigned char* input, size_t length);
unsigned char* s3fs_decode64(const char* input, size_t* plength);
#endif // S3FS_STRING_UTIL_H_
/*

41
src/test_string_util.cpp
View File

@ -18,12 +18,14 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include <limits>
#include <stdint.h>
#include <string>
#include "string_util.h"
#include "test_util.h"
int main(int argc, char *argv[])
void test_trim()
{
ASSERT_EQUALS(std::string("1234"), trim(" 1234 "));
ASSERT_EQUALS(std::string("1234"), trim("1234 "));
@ -40,5 +42,42 @@ int main(int argc, char *argv[])
ASSERT_EQUALS(std::string(" 1234"), trim_right(" 1234"));
ASSERT_EQUALS(std::string("1234"), trim_right("1234"));
ASSERT_EQUALS(std::string("0"), str(0));
ASSERT_EQUALS(std::string("1"), str(1));
ASSERT_EQUALS(std::string("-1"), str(-1));
ASSERT_EQUALS(std::string("9223372036854775807"), str(std::numeric_limits<int64_t>::max()));
ASSERT_EQUALS(std::string("-9223372036854775808"), str(std::numeric_limits<int64_t>::min()));
ASSERT_EQUALS(std::string("0"), str(std::numeric_limits<uint64_t>::min()));
ASSERT_EQUALS(std::string("18446744073709551615"), str(std::numeric_limits<uint64_t>::max()));
}
void test_base64()
{
size_t len;
ASSERT_STREQUALS(s3fs_base64(NULL, 0), NULL);
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64(NULL, &len)), NULL);
ASSERT_STREQUALS(s3fs_base64(reinterpret_cast<const unsigned char *>(""), 0), NULL);
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64("", &len)), NULL);
ASSERT_STREQUALS(s3fs_base64(reinterpret_cast<const unsigned char *>("1"), 1), "MQ==");
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64("MQ==", &len)), "1");
ASSERT_EQUALS(len, static_cast<size_t>(1));
ASSERT_STREQUALS(s3fs_base64(reinterpret_cast<const unsigned char *>("12"), 2), "MTI=");
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64("MTI=", &len)), "12");
ASSERT_EQUALS(len, static_cast<size_t>(2));
ASSERT_STREQUALS(s3fs_base64(reinterpret_cast<const unsigned char *>("123"), 3), "MTIz");
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64("MTIz", &len)), "123");
ASSERT_EQUALS(len, static_cast<size_t>(3));
ASSERT_STREQUALS(s3fs_base64(reinterpret_cast<const unsigned char *>("1234"), 4), "MTIzNA==");
ASSERT_STREQUALS(reinterpret_cast<const char *>(s3fs_decode64("MTIzNA==", &len)), "1234");
ASSERT_EQUALS(len, static_cast<size_t>(4));
// TODO: invalid input
}
int main(int argc, char *argv[])
{
test_trim();
test_base64();
return 0;
}

12
src/test_util.h
View File

@ -29,6 +29,18 @@ template <typename T> void assert_equals(const T &x, const T &y, const char *fil
}
}
void assert_strequals(const char *x, const char *y, const char *file, int line)
{
if(x == NULL && y == NULL){
return;
} else if((x == NULL || y == NULL) || strcmp(x, y) != 0){
std::cerr << x << " != " << y << " at " << file << ":" << line << std::endl;
std::exit(1);
}
}
#define ASSERT_EQUALS(x, y) \
assert_equals((x), (y), __FILE__, __LINE__)
#define ASSERT_STREQUALS(x, y) \
assert_strequals((x), (y), __FILE__, __LINE__)

5
test/Makefile.am
View File

@ -28,8 +28,3 @@ EXTRA_DIST = \
sample_ahbe.conf
testdir = test
test_PROGRAMS=rename_before_close
rename_before_close_SOURCES = rename_before_close.c

12
test/integration-test-common.sh
View File

@ -1,10 +1,10 @@
#!/bin/bash -e
set -x
S3FS=../src/s3fs
S3FS_CREDENTIALS_FILE="passwd-s3fs"
: ${S3FS_CREDENTIALS_FILE:="passwd-s3fs"}
TEST_BUCKET_1="s3fs-integration-test"
: ${TEST_BUCKET_1:="s3fs-integration-test"}
TEST_BUCKET_MOUNT_POINT_1=${TEST_BUCKET_1}
if [ ! -f "$S3FS_CREDENTIALS_FILE" ]
@ -15,9 +15,9 @@ fi
chmod 600 "$S3FS_CREDENTIALS_FILE"
S3PROXY_VERSION="1.4.0"
S3PROXY_BINARY="s3proxy-${S3PROXY_VERSION}"
if [ ! -e "${S3PROXY_BINARY}" ]; then
S3PROXY_BINARY=${S3PROXY_BINARY-"s3proxy-${S3PROXY_VERSION}"}
if [ -n "${S3PROXY_BINARY}" ] && [ ! -e "${S3PROXY_BINARY}" ]; then
wget "https://github.com/andrewgaul/s3proxy/releases/download/s3proxy-${S3PROXY_VERSION}/s3proxy" \
-O "${S3PROXY_BINARY}"
--quiet -O "${S3PROXY_BINARY}"
chmod +x "${S3PROXY_BINARY}"
fi

604
test/integration-test-main.sh
View File

@ -61,6 +61,345 @@ function rm_test_dir {
fi
}
function test_append_file {
echo "Testing append to file ..."
# Write a small test file
for x in `seq 1 $TEST_TEXT_FILE_LENGTH`
do
echo "echo ${TEST_TEXT} to ${TEST_TEXT_FILE}"
done > ${TEST_TEXT_FILE}
# Verify contents of file
echo "Verifying length of test file"
FILE_LENGTH=`wc -l $TEST_TEXT_FILE | awk '{print $1}'`
if [ "$FILE_LENGTH" -ne "$TEST_TEXT_FILE_LENGTH" ]
then
echo "error: expected $TEST_TEXT_FILE_LENGTH , got $FILE_LENGTH"
exit 1
fi
rm_test_file
}
function test_truncate_file {
echo "Testing truncate file ..."
# Write a small test file
echo "${TEST_TEXT}" > ${TEST_TEXT_FILE}
# Truncate file to 0 length. This should trigger open(path, O_RDWR | O_TRUNC...)
: > ${TEST_TEXT_FILE}
# Verify file is zero length
if [ -s ${TEST_TEXT_FILE} ]
then
echo "error: expected ${TEST_TEXT_FILE} to be zero length"
exit 1
fi
rm_test_file
}
function test_mv_file {
echo "Testing mv file function ..."
# if the rename file exists, delete it
if [ -e $ALT_TEST_TEXT_FILE ]
then
rm $ALT_TEST_TEXT_FILE
fi
if [ -e $ALT_TEST_TEXT_FILE ]
then
echo "Could not delete file ${ALT_TEST_TEXT_FILE}, it still exists"
exit 1
fi
# create the test file again
mk_test_file
#rename the test file
mv $TEST_TEXT_FILE $ALT_TEST_TEXT_FILE
if [ ! -e $ALT_TEST_TEXT_FILE ]
then
echo "Could not move file"
exit 1
fi
# Check the contents of the alt file
ALT_TEXT_LENGTH=`echo $TEST_TEXT | wc -c | awk '{print $1}'`
ALT_FILE_LENGTH=`wc -c $ALT_TEST_TEXT_FILE | awk '{print $1}'`
if [ "$ALT_FILE_LENGTH" -ne "$ALT_TEXT_LENGTH" ]
then
echo "moved file length is not as expected expected: $ALT_TEXT_LENGTH got: $ALT_FILE_LENGTH"
exit 1
fi
# clean up
rm_test_file $ALT_TEST_TEXT_FILE
}
function test_mv_directory {
echo "Testing mv directory function ..."
if [ -e $TEST_DIR ]; then
echo "Unexpected, this file/directory exists: ${TEST_DIR}"
exit 1
fi
mk_test_dir
mv ${TEST_DIR} ${TEST_DIR}_rename
if [ ! -d "${TEST_DIR}_rename" ]; then
echo "Directory ${TEST_DIR} was not renamed"
exit 1
fi
rmdir ${TEST_DIR}_rename
if [ -e "${TEST_DIR}_rename" ]; then
echo "Could not remove the test directory, it still exists: ${TEST_DIR}_rename"
exit 1
fi
}
function test_redirects {
echo "Testing redirects ..."
mk_test_file ABCDEF
CONTENT=`cat $TEST_TEXT_FILE`
if [ ${CONTENT} != "ABCDEF" ]; then
echo "CONTENT read is unexpected, got ${CONTENT}, expected ABCDEF"
exit 1
fi
echo XYZ > $TEST_TEXT_FILE
CONTENT=`cat $TEST_TEXT_FILE`
if [ ${CONTENT} != "XYZ" ]; then
echo "CONTENT read is unexpected, got ${CONTENT}, expected XYZ"
exit 1
fi
echo 123456 >> $TEST_TEXT_FILE
LINE1=`sed -n '1,1p' $TEST_TEXT_FILE`
LINE2=`sed -n '2,2p' $TEST_TEXT_FILE`
if [ ${LINE1} != "XYZ" ]; then
echo "LINE1 was not as expected, got ${LINE1}, expected XYZ"
exit 1
fi
if [ ${LINE2} != "123456" ]; then
echo "LINE2 was not as expected, got ${LINE2}, expected 123456"
exit 1
fi
# clean up
rm_test_file
}
function test_mkdir_rmdir {
echo "Testing creation/removal of a directory"
if [ -e $TEST_DIR ]; then
echo "Unexpected, this file/directory exists: ${TEST_DIR}"
exit 1
fi
mk_test_dir
rm_test_dir
}
function test_chmod {
echo "Testing chmod file function ..."
# create the test file again
mk_test_file
ORIGINAL_PERMISSIONS=$(stat --format=%a $TEST_TEXT_FILE)
chmod 777 $TEST_TEXT_FILE;
# if they're the same, we have a problem.
if [ $(stat --format=%a $TEST_TEXT_FILE) == $ORIGINAL_PERMISSIONS ]
then
echo "Could not modify $TEST_TEXT_FILE permissions"
exit 1
fi
# clean up
rm_test_file
}
function test_chown {
echo "Testing chown file function ..."
# create the test file again
mk_test_file
ORIGINAL_PERMISSIONS=$(stat --format=%u:%g $TEST_TEXT_FILE)
chown 1000:1000 $TEST_TEXT_FILE;
# if they're the same, we have a problem.
if [ $(stat --format=%a $TEST_TEXT_FILE) == $ORIGINAL_PERMISSIONS ]
then
echo "Could not modify $TEST_TEXT_FILE ownership"
exit 1
fi
# clean up
rm_test_file
}
function test_list {
echo "Testing list"
mk_test_file
mk_test_dir
file_cnt=$(ls -1 | wc -l)
if [ $file_cnt != 2 ]; then
echo "Expected 2 file but got $file_cnt"
exit 1
fi
rm_test_file
rm_test_dir
}
function test_remove_nonempty_directory {
echo "Testing removing a non-empty directory"
mk_test_dir
touch "${TEST_DIR}/file"
rmdir "${TEST_DIR}" 2>&1 | grep -q "Directory not empty"
rm "${TEST_DIR}/file"
rm_test_dir
}
function test_rename_before_close {
echo "Testing rename before close ..."
(
echo foo
mv $TEST_TEXT_FILE ${TEST_TEXT_FILE}.new
) > $TEST_TEXT_FILE
if ! cmp <(echo foo) ${TEST_TEXT_FILE}.new; then
echo "rename before close failed"
exit 1
fi
rm_test_file ${TEST_TEXT_FILE}.new
rm -f ${TEST_TEXT_FILE}
}
function test_multipart_upload {
echo "Testing multi-part upload ..."
dd if=/dev/urandom of="/tmp/${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1
dd if="/tmp/${BIG_FILE}" of="${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1
# Verify contents of file
echo "Comparing test file"
if ! cmp "/tmp/${BIG_FILE}" "${BIG_FILE}"
then
exit 1
fi
rm -f "/tmp/${BIG_FILE}"
rm_test_file "${BIG_FILE}"
}
function test_multipart_copy {
echo "Testing multi-part copy ..."
dd if=/dev/urandom of="/tmp/${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1
dd if="/tmp/${BIG_FILE}" of="${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1
mv "${BIG_FILE}" "${BIG_FILE}-copy"
# Verify contents of file
echo "Comparing test file"
if ! cmp "/tmp/${BIG_FILE}" "${BIG_FILE}-copy"
then
exit 1
fi
rm -f "/tmp/${BIG_FILE}"
rm_test_file "${BIG_FILE}-copy"
}
function test_special_characters {
echo "Testing special characters ..."
ls 'special' 2>&1 | grep -q 'No such file or directory'
ls 'special?' 2>&1 | grep -q 'No such file or directory'
ls 'special*' 2>&1 | grep -q 'No such file or directory'
ls 'special~' 2>&1 | grep -q 'No such file or directory'
ls 'specialµ' 2>&1 | grep -q 'No such file or directory'
}
function test_symlink {
echo "Testing symlinks ..."
rm -f $TEST_TEXT_FILE
rm -f $ALT_TEST_TEXT_FILE
echo foo > $TEST_TEXT_FILE
ln -s $TEST_TEXT_FILE $ALT_TEST_TEXT_FILE
cmp $TEST_TEXT_FILE $ALT_TEST_TEXT_FILE
rm -f $TEST_TEXT_FILE
[ -L $ALT_TEST_TEXT_FILE ]
[ ! -f $ALT_TEST_TEXT_FILE ]
}
function test_extended_attributes {
command -v setfattr >/dev/null 2>&1 || \
{ echo "Skipping extended attribute tests" ; return; }
echo "Testing extended attributes ..."
rm -f $TEST_TEXT_FILE
touch $TEST_TEXT_FILE
# set value
setfattr -n key1 -v value1 $TEST_TEXT_FILE
getfattr -n key1 --only-values $TEST_TEXT_FILE | grep -q '^value1$'
# append value
setfattr -n key2 -v value2 $TEST_TEXT_FILE
getfattr -n key1 --only-values $TEST_TEXT_FILE | grep -q '^value1$'
getfattr -n key2 --only-values $TEST_TEXT_FILE | grep -q '^value2$'
# remove value
setfattr -x key1 $TEST_TEXT_FILE
! getfattr -n key1 --only-values $TEST_TEXT_FILE
getfattr -n key2 --only-values $TEST_TEXT_FILE | grep -q '^value2$'
}
function run_all_tests {
test_append_file
test_truncate_file
test_mv_file
test_mv_directory
test_redirects
test_mkdir_rmdir
test_chmod
test_chown
test_list
test_remove_nonempty_directory
# TODO: broken: https://github.com/s3fs-fuse/s3fs-fuse/issues/145
#test_rename_before_close
test_multipart_upload
# TODO: test disabled until S3Proxy 1.5.0 is released
#test_multipart_copy
test_special_characters
test_symlink
test_extended_attributes
}
# Mount the bucket
CUR_DIR=`pwd`
TEST_BUCKET_MOUNT_POINT_1=$1
if [ "$TEST_BUCKET_MOUNT_POINT_1" == "" ]; then
@ -74,270 +413,7 @@ then
rm -f $TEST_TEXT_FILE
fi
# Write a small test file
for x in `seq 1 $TEST_TEXT_FILE_LENGTH`
do
echo "echo ${TEST_TEXT} to ${TEST_TEXT_FILE}"
echo $TEST_TEXT >> $TEST_TEXT_FILE
done
# Verify contents of file
echo "Verifying length of test file"
FILE_LENGTH=`wc -l $TEST_TEXT_FILE | awk '{print $1}'`
if [ "$FILE_LENGTH" -ne "$TEST_TEXT_FILE_LENGTH" ]
then
echo "error: expected $TEST_TEXT_FILE_LENGTH , got $FILE_LENGTH"
exit 1
fi
rm_test_file
##########################################################
# Rename test (individual file)
##########################################################
echo "Testing mv file function ..."
# if the rename file exists, delete it
if [ -e $ALT_TEST_TEXT_FILE ]
then
rm $ALT_TEST_TEXT_FILE
fi
if [ -e $ALT_TEST_TEXT_FILE ]
then
echo "Could not delete file ${ALT_TEST_TEXT_FILE}, it still exists"
exit 1
fi
# create the test file again
mk_test_file
#rename the test file
mv $TEST_TEXT_FILE $ALT_TEST_TEXT_FILE
if [ ! -e $ALT_TEST_TEXT_FILE ]
then
echo "Could not move file"
exit 1
fi
# Check the contents of the alt file
ALT_TEXT_LENGTH=`echo $TEST_TEXT | wc -c | awk '{print $1}'`
ALT_FILE_LENGTH=`wc -c $ALT_TEST_TEXT_FILE | awk '{print $1}'`
if [ "$ALT_FILE_LENGTH" -ne "$ALT_TEXT_LENGTH" ]
then
echo "moved file length is not as expected expected: $ALT_TEXT_LENGTH got: $ALT_FILE_LENGTH"
exit 1
fi
# clean up
rm_test_file $ALT_TEST_TEXT_FILE
##########################################################
# Rename test (individual directory)
##########################################################
echo "Testing mv directory function ..."
if [ -e $TEST_DIR ]; then
echo "Unexpected, this file/directory exists: ${TEST_DIR}"
exit 1
fi
mk_test_dir
mv ${TEST_DIR} ${TEST_DIR}_rename
if [ ! -d "${TEST_DIR}_rename" ]; then
echo "Directory ${TEST_DIR} was not renamed"
exit 1
fi
rmdir ${TEST_DIR}_rename
if [ -e "${TEST_DIR}_rename" ]; then
echo "Could not remove the test directory, it still exists: ${TEST_DIR}_rename"
exit 1
fi
###################################################################
# test redirects > and >>
###################################################################
echo "Testing redirects ..."
mk_test_file ABCDEF
CONTENT=`cat $TEST_TEXT_FILE`
if [ ${CONTENT} != "ABCDEF" ]; then
echo "CONTENT read is unexpected, got ${CONTENT}, expected ABCDEF"
exit 1
fi
echo XYZ > $TEST_TEXT_FILE
CONTENT=`cat $TEST_TEXT_FILE`
if [ ${CONTENT} != "XYZ" ]; then
echo "CONTENT read is unexpected, got ${CONTENT}, expected XYZ"
exit 1
fi
echo 123456 >> $TEST_TEXT_FILE
LINE1=`sed -n '1,1p' $TEST_TEXT_FILE`
LINE2=`sed -n '2,2p' $TEST_TEXT_FILE`
if [ ${LINE1} != "XYZ" ]; then
echo "LINE1 was not as expected, got ${LINE1}, expected XYZ"
exit 1
fi
if [ ${LINE2} != "123456" ]; then
echo "LINE2 was not as expected, got ${LINE2}, expected 123456"
exit 1
fi
# clean up
rm_test_file
#####################################################################
# Simple directory test mkdir/rmdir
#####################################################################
echo "Testing creation/removal of a directory"
if [ -e $TEST_DIR ]; then
echo "Unexpected, this file/directory exists: ${TEST_DIR}"
exit 1
fi
mk_test_dir
rm_test_dir
##########################################################
# File permissions test (individual file)
##########################################################
echo "Testing chmod file function ..."
# create the test file again
mk_test_file
ORIGINAL_PERMISSIONS=$(stat --format=%a $TEST_TEXT_FILE)
chmod 777 $TEST_TEXT_FILE;
# if they're the same, we have a problem.
if [ $(stat --format=%a $TEST_TEXT_FILE) == $ORIGINAL_PERMISSIONS ]
then
echo "Could not modify $TEST_TEXT_FILE permissions"
exit 1
fi
# clean up
rm_test_file
##########################################################
# File permissions test (individual file)
##########################################################
echo "Testing chown file function ..."
# create the test file again
mk_test_file
ORIGINAL_PERMISSIONS=$(stat --format=%u:%g $TEST_TEXT_FILE)
chown 1000:1000 $TEST_TEXT_FILE;
# if they're the same, we have a problem.
if [ $(stat --format=%a $TEST_TEXT_FILE) == $ORIGINAL_PERMISSIONS ]
then
echo "Could not modify $TEST_TEXT_FILE ownership"
exit 1
fi
# clean up
rm_test_file
##########################################################
# Testing list
##########################################################
echo "Testing list"
mk_test_file
mk_test_dir
file_cnt=$(ls -1 | wc -l)
if [ $file_cnt != 2 ]; then
echo "Expected 2 file but got $file_cnt"
exit 1
fi
rm_test_file
rm_test_dir
##########################################################
# Testing rename before close
##########################################################
if false; then
echo "Testing rename before close ..."
$CUR_DIR/rename_before_close $TEST_TEXT_FILE
if [ $? != 0 ]; then
echo "rename before close failed"
exit 1
fi
# clean up
rm_test_file
fi
##########################################################
# Testing multi-part upload
##########################################################
echo "Testing multi-part upload ..."
dd if=/dev/urandom of="/tmp/${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1
dd if="/tmp/${BIG_FILE}" of="${BIG_FILE}" bs=$BIG_FILE_LENGTH count=1
# Verify contents of file
echo "Comparing test file"
if ! cmp "/tmp/${BIG_FILE}" "${BIG_FILE}"
then
exit 1
fi
rm -f "/tmp/${BIG_FILE}"
rm -f "${BIG_FILE}"
##########################################################
# Testing special characters
##########################################################
echo "Testing special characters ..."
ls 'special' 2>&1 | grep -q 'No such file or directory'
ls 'special?' 2>&1 | grep -q 'No such file or directory'
ls 'special*' 2>&1 | grep -q 'No such file or directory'
ls 'special~' 2>&1 | grep -q 'No such file or directory'
ls 'specialµ' 2>&1 | grep -q 'No such file or directory'
##########################################################
# Testing extended attributes
##########################################################
rm -f $TEST_TEXT_FILE
touch $TEST_TEXT_FILE
# set value
setfattr -n key1 -v value1 $TEST_TEXT_FILE
getfattr -n key1 --only-values $TEST_TEXT_FILE | grep -q '^value1$'
# append value
setfattr -n key2 -v value2 $TEST_TEXT_FILE
getfattr -n key1 --only-values $TEST_TEXT_FILE | grep -q '^value1$'
getfattr -n key2 --only-values $TEST_TEXT_FILE | grep -q '^value2$'
# remove value
setfattr -x key1 $TEST_TEXT_FILE
! getfattr -n key1 --only-values $TEST_TEXT_FILE
getfattr -n key2 --only-values $TEST_TEXT_FILE | grep -q '^value2$'
#####################################################################
# Tests are finished
#####################################################################
run_all_tests
# Unmount the bucket
cd $CUR_DIR

88
test/rename_before_close.c
View File

@ -1,88 +0,0 @@
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
static const char FILE_CONTENT[] = "XXXX";
#define PROG "rename_before_close"
static char *
filename_to_mkstemp_template(const char *file)
{
size_t len = strlen(file);
static const char suffix[] = ".XXXXXX";
size_t new_len = len + sizeof(suffix);
char *ret_str = calloc(1, new_len);
int ret = snprintf(ret_str, new_len, "%s%s", file, suffix);
assert(ret == new_len - 1);
assert(ret_str[new_len] == '\0');
return ret_str;
}
static off_t
get_file_size(const char *file)
{
struct stat ss;
printf(PROG ": stat(%s)\n", file);
int ret = lstat(file, &ss);
assert(ret == 0);
return ss.st_size;
}
static void
test_rename_before_close(const char *file)
{
char *template = filename_to_mkstemp_template(file);
printf(PROG ": mkstemp(%s)\n", template);
int fd = mkstemp(template);
assert(fd >= 0);
sleep(1);
printf(PROG ": write(%s)\n", template);
int ret = write(fd, FILE_CONTENT, sizeof(FILE_CONTENT));
assert(ret == sizeof(FILE_CONTENT));
sleep(1);
printf(PROG ": fsync(%s)\n", template);
ret = fsync(fd);
assert(ret == 0);
sleep(1);
assert(get_file_size(template) == sizeof(FILE_CONTENT));
sleep(1);
printf(PROG ": rename(%s, %s)\n", template, file);
ret = rename(template, file);
assert(ret == 0);
sleep(1);
printf(PROG ": close(%s)\n", file);
ret = close(fd);
assert(ret == 0);
sleep(1);
assert(get_file_size(file) == sizeof(FILE_CONTENT));
}
int
main(int argc, char *argv[])
{
setvbuf(stdout, NULL, _IONBF, 0);
if (argc < 2) {
printf("Usage: %s <file>", argv[0]);
return 1;
}
test_rename_before_close(argv[1]);
return 0;
}

57
test/small-integration-test.sh
View File

@ -1,8 +1,24 @@
#!/bin/bash
#
# By default tests run against a local s3proxy instance. To run against
# Amazon S3, specify the following variables:
#
# S3FS_CREDENTIALS_FILE=keyfile s3fs format key file
# TEST_BUCKET_1=bucket Name of bucket to use
# S3PROXY_BINARY="" Leave empty
# S3_URL="http://s3.amazonaws.com" Specify Amazon server
#
# Example:
#
# S3FS_CREDENTIALS_FILE=keyfile TEST_BUCKET_1=bucket S3PROXY_BINARY="" S3_URL="http://s3.amazonaws.com" ./small-integration-test.sh
#
set -o xtrace
set -o errexit
: ${S3_URL:="http://127.0.0.1:8080"}
# Require root
REQUIRE_ROOT=require-root.sh
#source $REQUIRE_ROOT
@ -29,26 +45,32 @@ function retry {
}
function exit_handler {
kill $S3PROXY_PID
if [ -n "${S3PROXY_PID}" ]
then
kill $S3PROXY_PID
fi
retry 30 fusermount -u $TEST_BUCKET_MOUNT_POINT_1
}
trap exit_handler EXIT
stdbuf -oL -eL java -jar "$S3PROXY_BINARY" --properties s3proxy.conf | stdbuf -oL -eL sed -u "s/^/s3proxy: /" &
if [ -n "${S3PROXY_BINARY}" ]
then
stdbuf -oL -eL java -jar "$S3PROXY_BINARY" --properties s3proxy.conf | stdbuf -oL -eL sed -u "s/^/s3proxy: /" &
# wait for S3Proxy to start
for i in $(seq 30);
do
if exec 3<>"/dev/tcp/localhost/8080";
then
exec 3<&- # Close for read
exec 3>&- # Close for write
break
fi
sleep 1
done
# wait for S3Proxy to start
for i in $(seq 30);
do
if exec 3<>"/dev/tcp/127.0.0.1/8080";
then
exec 3<&- # Close for read
exec 3>&- # Close for write
break
fi
sleep 1
done
S3PROXY_PID=$(netstat -lpnt | grep :8080 | awk '{ print $7 }' | sed -u 's|/java||')
S3PROXY_PID=$(netstat -lpnt | grep :8080 | awk '{ print $7 }' | sed -u 's|/java||')
fi
# Mount the bucket
if [ ! -d $TEST_BUCKET_MOUNT_POINT_1 ]
@ -57,10 +79,13 @@ then
fi
stdbuf -oL -eL $S3FS $TEST_BUCKET_1 $TEST_BUCKET_MOUNT_POINT_1 \
-o createbucket \
-o enable_content_md5 \
-o passwd_file=$S3FS_CREDENTIALS_FILE \
-o sigv2 \
-o url=http://127.0.0.1:8080 \
-o use_path_request_style -f -o f2 -d -d |& stdbuf -oL -eL sed -u "s/^/s3fs: /" &
-o singlepart_copy_limit=$((10 * 1024)) \
-o url=${S3_URL} \
-o use_path_request_style \
-o dbglevel=info -f |& stdbuf -oL -eL sed -u "s/^/s3fs: /" &
retry 30 grep $TEST_BUCKET_MOUNT_POINT_1 /proc/mounts || exit 1