FUSE-based file system backed by Amazon S3
Go to file
Andrew Gaul f531e6aff2 Configure S3Proxy for SSL
This also demonstrates that SSL certificate checking occurs and the
tests must disable it for S3Proxy's self-signed certificate.
References #640.
2017-09-17 16:16:18 -07:00
.github Added issue and PR templates. 2016-12-04 12:25:11 +00:00
doc Not fallback to HTTP - #596 2017-05-13 15:47:39 +00:00
src Fix intermittent upload failures on macOS 2017-09-08 15:23:10 -04:00
test Configure S3Proxy for SSL 2017-09-17 16:16:18 -07:00
.gitignore Added use_xattr option for #467 and #460 2016-09-19 04:28:01 +00:00
.mailmap Add .mailmap 2015-04-27 11:17:39 -07:00
.travis.yml Added travis test on osx for #601 2017-08-11 14:09:43 +00:00
AUTHORS Summary of Changes(1.72 -> 1.73) 2013-08-23 17:24:47 +00:00
autogen.sh s3fs can print version with short commit hash - #228 2015-08-21 16:19:31 +00:00
ChangeLog Updated ChangeLog and configure.ac for release 1.82 2017-05-13 16:12:46 +00:00
configure.ac Updated ChangeLog and configure.ac for release 1.82 2017-05-13 16:12:46 +00:00
COPYING In preparation to remove the unnecessary "s3fs" 2010-11-13 23:59:23 +00:00
INSTALL In preparation to remove the unnecessary "s3fs" 2010-11-13 23:59:23 +00:00
Makefile.am Added travis test on osx for #601 2017-08-11 14:09:43 +00:00
README.md Merge pull request #639 from andrewgaul/homebrew 2017-09-17 15:43:03 +09:00

s3fs

s3fs allows Linux and Mac OS X to mount an S3 bucket via FUSE. s3fs preserves the native object format for files, allowing use of other tools like s3cmd.
Build Status

Features

  • large subset of POSIX including reading/writing files, directories, symlinks, mode, uid/gid, and extended attributes
  • compatible with Amazon S3, Google Cloud Storage, and other S3-based object stores
  • large files via multi-part upload
  • renames via server-side copy
  • optional server-side encryption
  • data integrity via MD5 hashes
  • in-memory metadata caching
  • local disk data caching
  • user-specified regions, including Amazon GovCloud
  • authenticate via v2 or v4 signatures

Installation

  • On Linux, ensure you have all the dependencies:

On Ubuntu 14.04:

sudo apt-get install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

On CentOS 7:

sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

Then compile from master via the following commands:

git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make
sudo make install
$ brew cask install osxfuse
$ brew install s3fs

Examples

Enter your S3 identity and credential in a file /path/to/passwd and set owner-only permissions:

echo MYIDENTITY:MYCREDENTIAL > /path/to/passwd
chmod 600 /path/to/passwd

Run s3fs with an existing bucket mybucket and directory /path/to/mountpoint:

s3fs mybucket /path/to/mountpoint -o passwd_file=/path/to/passwd

If you use s3fs with a non-Amazon S3 implementation, specify the URL and path-style requests:

s3fs mybucket /path/to/mountpoint -o passwd_file=/path/to/passwd -o url=http://url.to.s3/ -o use_path_request_style

If you encounter any errors, enable debug output:

s3fs mybucket /path/to/mountpoint -o passwd_file=/path/to/passwd -d -d -f -o f2 -o curldbg

You can also mount on boot by entering the following line to /etc/fstab:

s3fs#mybucket /path/to/mountpoint fuse _netdev,allow_other 0 0

or

mybucket /path/to/mountpoint fuse.s3fs _netdev,allow_other 0 0

Note: You may also want to create the global credential file first

echo MYIDENTITY:MYCREDENTIAL > /etc/passwd-s3fs
chmod 600 /etc/passwd-s3fs

Note2: You may also need to make sure netfs service is start on boot

Limitations

Generally S3 cannot offer the same performance or semantics as a local file system. More specifically:

  • random writes or appends to files require rewriting the entire file
  • metadata operations such as listing directories have poor performance due to network latency
  • eventual consistency can temporarily yield stale data(Amazon S3 Data Consistency Model)
  • no atomic renames of files or directories
  • no coordination between multiple clients mounting the same bucket
  • no hard links

References

  • goofys - similar to s3fs but has better performance and less POSIX compatibility
  • s3backer - mount an S3 bucket as a single file
  • s3fs-python - an older and less complete implementation written in Python
  • S3Proxy - combine with s3fs to mount EMC Atmos, Microsoft Azure, and OpenStack Swift buckets
  • s3ql - similar to s3fs but uses its own object format
  • YAS3FS - similar to s3fs but uses SNS to allow multiple clients to mount a bucket

Frequently Asked Questions

License

Copyright (C) 2010 Randy Rizun rrizun@gmail.com

Licensed under the GNU GPL version 2