FUSE-based file system backed by Amazon S3
Go to file
Petr Vaněk 4dfe2bfdd7 Include climits to support musl libc
PATH_MAX constant is not visible from any of currently included header
files in system with musl libc, where compilation fails with an error
below. The constant is defined in limits.h which is directly include via
climits header file.

fdcache.cpp: In static member function 'static FILE* FdManager::MakeTempFile()':
fdcache.cpp:381:14: error: 'PATH_MAX' was not declared in this scope
  381 |     char cfn[PATH_MAX];
      |              ^~~~~~~~

Fixes: d67b83e671 ("Allow configuration for temporary files directory")
2022-02-14 09:19:30 +09:00
.github Fixed some Github Actions errors. (#1886) 2022-02-13 14:23:35 +09:00
doc Fix typo in -o enable_unsigned_payload 2022-01-12 22:50:49 +09:00
src Include climits to support musl libc 2022-02-14 09:19:30 +09:00
test Fixed test_external_directory_creation test when cache enabled (#1885) 2022-02-13 13:32:19 +09:00
.clang-tidy Add new suppressions for clang-tidy 13 (#1847) 2022-01-09 21:42:36 +09:00
.gitattributes Added a missing extension to .gitignore, and formatted dot files 2019-01-20 09:18:40 +00:00
.gitignore Remove duplicates in .gitignore 2022-01-23 21:49:51 +09:00
.mailmap Add .mailmap 2015-04-27 11:17:39 -07:00
AUTHORS Updated ChangeLog and configure.ac etc for release 1.85 2019-03-11 11:53:57 +00:00
autogen.sh Added test by a shell script static analysis tool(ShellCheck) 2022-01-22 22:23:08 +09:00
ChangeLog Update ChangeLog and configure.ac for 1.90 (#1739) 2021-08-08 14:01:19 +09:00
COMPILATION.md Add instructions for Windows compilation 2022-01-23 21:49:51 +09:00
configure.ac Remove unnecessary conditional for automake 2022-01-23 21:49:51 +09:00
COPYING In preparation to remove the unnecessary "s3fs" 2010-11-13 23:59:23 +00:00
INSTALL Replace all mentions to MacOS X to macOS 2018-10-14 23:23:19 +02:00
Makefile.am Added test by a shell script static analysis tool(ShellCheck) 2022-01-22 22:23:08 +09:00
README.md Always call clock_gettime(2) (#1871) 2022-01-25 08:36:27 +09:00

s3fs

s3fs allows Linux, macOS, and FreeBSD to mount an S3 bucket via FUSE. s3fs preserves the native object format for files, allowing use of other tools like AWS CLI.
s3fs-fuse CI Twitter Follow

Features

  • large subset of POSIX including reading/writing files, directories, symlinks, mode, uid/gid, and extended attributes
  • compatible with Amazon S3, and other S3-based object stores
  • allows random writes and appends
  • large files via multi-part upload
  • renames via server-side copy
  • optional server-side encryption
  • data integrity via MD5 hashes
  • in-memory metadata caching
  • local disk data caching
  • user-specified regions, including Amazon GovCloud
  • authenticate via v2 or v4 signatures

Installation

Many systems provide pre-built packages:

  • Amazon Linux via EPEL:

    sudo amazon-linux-extras install epel
    sudo yum install s3fs-fuse
    
  • Arch Linux:

    sudo pacman -S s3fs-fuse
    
  • Debian 9 and Ubuntu 16.04 or newer:

    sudo apt install s3fs
    
  • Fedora 27 or newer:

    sudo dnf install s3fs-fuse
    
  • Gentoo:

    sudo emerge net-fs/s3fs
    
  • RHEL and CentOS 7 or newer via EPEL:

    sudo yum install epel-release
    sudo yum install s3fs-fuse
    
  • SUSE 12 and openSUSE 42.1 or newer:

    sudo zypper install s3fs
    
  • macOS 10.12 and newer via Homebrew:

    brew install --cask osxfuse
    brew install gromgit/fuse/s3fs-mac
    
  • FreeBSD:

    pkg install fusefs-s3fs
    

Note: Homebrew has deprecated osxfuse and s3fs may not install any more, see #1618.

Otherwise consult the compilation instructions.

Examples

s3fs supports the standard AWS credentials file stored in ${HOME}/.aws/credentials. Alternatively, s3fs supports a custom passwd file.

The default location for the s3fs password file can be created:

  • using a .passwd-s3fs file in the users home directory (i.e. ${HOME}/.passwd-s3fs)
  • using the system-wide /etc/passwd-s3fs file

Enter your credentials in a file ${HOME}/.passwd-s3fs and set owner-only permissions:

echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ${HOME}/.passwd-s3fs
chmod 600 ${HOME}/.passwd-s3fs

Run s3fs with an existing bucket mybucket and directory /path/to/mountpoint:

s3fs mybucket /path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs

If you encounter any errors, enable debug output:

s3fs mybucket /path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs -o dbglevel=info -f -o curldbg

You can also mount on boot by entering the following line to /etc/fstab:

mybucket /path/to/mountpoint fuse.s3fs _netdev,allow_other 0 0

If you use s3fs with a non-Amazon S3 implementation, specify the URL and path-style requests:

s3fs mybucket /path/to/mountpoint -o passwd_file=${HOME}/.passwd-s3fs -o url=https://url.to.s3/ -o use_path_request_style

or(fstab)

mybucket /path/to/mountpoint fuse.s3fs _netdev,allow_other,use_path_request_style,url=https://url.to.s3/ 0 0

Note: You may also want to create the global credential file first

echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > /etc/passwd-s3fs
chmod 600 /etc/passwd-s3fs

Note2: You may also need to make sure netfs service is start on boot

Limitations

Generally S3 cannot offer the same performance or semantics as a local file system. More specifically:

  • random writes or appends to files require rewriting the entire object, optimized with multi-part upload copy
  • metadata operations such as listing directories have poor performance due to network latency
  • non-AWS providers may have eventual consistency so reads can temporarily yield stale data (AWS offers read-after-write consistency since Dec 2020)
  • no atomic renames of files or directories
  • no coordination between multiple clients mounting the same bucket
  • no hard links
  • inotify detects only local modifications, not external ones by other clients or tools

References

  • goofys - similar to s3fs but has better performance and less POSIX compatibility
  • s3backer - mount an S3 bucket as a single file
  • S3Proxy - combine with s3fs to mount Backblaze B2, EMC Atmos, Microsoft Azure, and OpenStack Swift buckets
  • s3ql - similar to s3fs but uses its own object format
  • YAS3FS - similar to s3fs but uses SNS to allow multiple clients to mount a bucket

Frequently Asked Questions

License

Copyright (C) 2010 Randy Rizun rrizun@gmail.com

Licensed under the GNU GPL version 2