Previously s3fs experienced data loss when writing to the middle of a
file. Corrupt files would have the expected data from 0..offset+size
but unexpected NUL bytes from offset+size..EOF. References #808.
There were multiple problems with the FdManager::GetFreeDiskSpace() function
on macOS:
1) When calling statvfs(), f_frsize should be used instead of f_bsize when
converting available blocks to bytes. This was causing the free space
calculation to be incorrect.
2) On macOS, fsblkcnt_t is a 32-bit integer. Thus, when calculating available
disk space, there were frequently overflows. This caused s3fs to incorrectly
determine that the cache location was out of space in the middle of a transfer
which caused uploads to fail. Changing this to a uint64_t resolves the
problem.
When the prefetch size is limited to the multipart size, the entire parallel logic of the read flow does not have an opportunity to use parallel get.
This fix increases the read performance significantly over our own s3 on-premise solution.
nautilus does this when you drag and drop to overwrite a file:
1) create .goutputstream-XXXXXX to write to
2) fsync the fd for .goutputstream-XXXXXX
3) rename .goutputstream-XXXXXX to target file
4) close the fd for .goutputstream-XXXXXX
previously, doing this on s3fs would result in an empty target file
because after the rename, s3fs would not flush the content of
.goutputstream-XXXXXX to target file.
this change moves the FdEntity from the old path to the new path
whenever rename happens. On flush s3fs would now flush the correct
content to the rename target.
1) Changed buffer size for file size
Changes a internal buffer size from size_t to offt_t.
It is a bug for 32 bit OS enviroment.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@496 df820570-a93a-0410-bd06-b72b767a4274
1) Overflow
About over 4GB file, when st_size which is member stat structure,
the value is overflow.
Fixed this bug and fixed like this bug in all sources.
2) Changed retrying request
If s3fs gets 500 HTTP status for multipart request, s3fs retry
to send same request.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@495 df820570-a93a-0410-bd06-b72b767a4274
1) Changed codes about memory leak
For memory leak, below codes are changed.
* calls malloc_trim function
* calls initializing NSS function, and adds configure
option "--enable-nss-init".
If libcurl is with NSS, s3fs initializes NSS manually.
This initializing NSS is enabled by "--enable-nss-init"
option at configure. if this option is specified, you
need "nss-devel" package.
* calls initializing libxml2(xmlInitParser).
* BIO functions have memory leak, calls CRYPTO_free_ex_data.
* changes cache structure.
* changes cache out logic to LRU.
* sets alignment for allcated memory in body data structure.
* adds ssl session into share handle. and adds nosscache option.
* deletes unused allocated memory.(bug)
* changes defaule parallel count of head request in readdir
(500->20)
* fixes some bugs.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@482 df820570-a93a-0410-bd06-b72b767a4274
1) Fixed Issue 368
Fixed a bug that s3fs could not update local cache.
(Issue 368)1.73: Updating existing file on server 'a' does not change length of file on server 'b'
git-svn-id: http://s3fs.googlecode.com/svn/trunk@481 df820570-a93a-0410-bd06-b72b767a4274
1) Fixed Issue 363
Fixed a bug which has below reason.
Fuse does not wait finishing "release file descriptor" function
called by fuse, and fuse runs(calls) next processing(commands).
Then s3fs could not clear stats cache information for that file
before calling next processing, and s3fs uses old stats cache
information.
So that, s3fs clears stats cache in release function at first.
And found two bad codes(but these codes do not influence normal
movement) in fdcache.cpp and fixed these.
Issue 363: make check failing inconsistently
git-svn-id: http://s3fs.googlecode.com/svn/trunk@471 df820570-a93a-0410-bd06-b72b767a4274
1) Fixed bugs
Fixes below bugs( format error and undefined fund ).
* 1.72 Will not compile on Ubuntu 12.04.2 (precise) i686(Issue 360)
* complie time error after running #make(Issue 361)
I'll close these Issue if I can confirm that these problem was solved.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@466 df820570-a93a-0410-bd06-b72b767a4274
1) Set metadata "Content-Encoding" automatically(Issue 292)
For this issue, s3fs is added new option "ahbe_conf".
New option means the configuration file path, and this file specifies
additional HTTP header by file(object) extension.
Thus you can specify any HTTP header for each object by extension.
* ahbe_conf file format:
-----------
line = [file suffix] HTTP-header [HTTP-header-values]
file suffix = file(object) suffix, if this field is empty,
it means "*"(all object).
HTTP-header = additional HTTP header name
HTTP-header-values = additional HTTP header value
-----------
* Example:
-----------
.gz Content-Encoding gzip
.Z Content-Encoding compress
X-S3FS-MYHTTPHEAD myvalue
-----------
A sample configuration file is uploaded in "test" directory.
If ahbe_conf parameter is specified, s3fs loads it's configuration
and compares extension(suffix) of object(file) when uploading
(PUT/POST) it. If the extension is same, s3fs adds/sends specified
HTTP header and value.
A case of sample configuration file, if a object(it's extension is
".gz") which already has Content-Encoding HTTP header is renamed
to ".txt" extension, s3fs does not set Content-Encoding. Because
".txt" is not match any line in configuration file.
So, s3fs matches the extension by each PUT/POST action.
* Please take care about "Content-Encoding".
This new option allows setting ANY HTTP header by object extension.
For example, you can specify "Content-Encoding" for ".gz"/etc
extension in configuration. But this means that S3 always returns
"Content-Encoding: gzip" when a client requests with other
"Accept-Encoding:" header. It SHOULD NOT be good.
Please see RFC 2616.
2) Changes about allow_other/uid/gid option for mount point
I reviewed about mount point permission and allow_other/uid/gid
options, and found bugs about these.
s3fs is fixed bugs and changed to the following specifications.
* s3fs only allows uid(gid) options as 0(root), when the effective
user is zero(root).
* A mount point(directory) must have a permission to allow
accessing by effective user/group.
* If allow_other option is specified, the mount point permission
is set 0777(all users allow all access).
In another case, the mount point is set 0700(only allows
effective user).
* When uid/gid option is specified, the mount point owner/group
is set uid/gid option value.
If uid/gid is not set, it is set effective user/group id.
This changes maybe fixes some issue(321, 338).
3) Changes a logic about (Issue 229)
The chmod command returns -EIO when changing the mount point.
It is correct, s3fs can not changed owner/group/mtime for the
mount point, but s3fs sends a request for changing the bucket.
This revision does not send the request, and returns EIO as
soon as possible.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@465 df820570-a93a-0410-bd06-b72b767a4274
1 ) Fixed a bug
Since r458, s3fs uses stat files of cache files. But s3fs forgot removing these
stat files when s3fs removed cache files.
Fixed this bug.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@459 df820570-a93a-0410-bd06-b72b767a4274
* Summay
This revision includes big change about temporary file and local cache file.
By this big change, s3fs works with good performance when s3fs opens/
closes/syncs/reads object.
I made a big change about the handling about temporary file and local cache
file to do this implementation.
* Detail
1) About temporary file(local file)
s3fs uses a temporary file on local file system when s3fs does download/
upload/open/seek object on S3.
After this revision, s3fs calls ftruncate() function when s3fs makes the
temporary file.
In this way s3fs can set a file size of precisely length without downloading.
(Notice - ftruncate function is for XSI-compliant systems, so that possibly
you have a problem on non-XSI-compliant systems.)
By this change, s3fs can download a part of a object by requesting with
"Range" http header. It seems like downloading by each block unit.
The default block(part) size is 50MB, it is caused the result which is default
parallel requests count(5) by default multipart upload size(10MB).
If you need to change this block size, you can change by new option
"fd_page_size". This option can take from 1MB(1024 * 1024) to any bytes.
So that, you have to take care about that fdcache.cpp(and fdcache.h) were
changed a lot.
2) About local cache
Local cache files which are in directory specified by "use_cache" option do
not have always all of object data.
This cause is that s3fs uses ftruncate function and reads(writes) each block
unit of a temporary file.
s3fs manages each block unit's status which are "downloaded area" or "not".
For this status, s3fs makes new temporary file in cache directory which is
specified by "use_cache" option. This status files is in a directory which is
named "<use_cache sirectory>/.<bucket_name>/".
When s3fs opens this status file, s3fs locks this file for exclusive control by
calling flock function. You need to take care about this, the status files can
not be laid on network drive(like NFS).
This revision changes about file open mode, s3fs always opens a local cache
file and each status file with writable mode.
Last, this revision adds new option "del_cache", this option means that s3fs
deletes all local cache file when s3fs starts and exits.
3) Uploading
When s3fs writes data to file descriptor through FUSE request, old s3fs
revision downloads all of the object. But new revision does not download all,
it downloads only small percial area(some block units) including writing data
area.
And when s3fs closes or flushes the file descriptor, s3fs downloads other area
which is not downloaded from server. After that, s3fs uploads all of data.
Already r456 revision has parallel upload function, then this revision with
r456 and r457 are very big change for performance.
4) Downloading
By changing a temporary file and a local cache file, when s3fs downloads a
object, it downloads only the required range(some block units).
And s3fs downloads units by parallel GET request, it is same as a case of
uploading. (Maximum parallel request count and each download size are
specified same parameters for uploading.)
In the new revision, when s3fs opens file, s3fs returns file descriptor soon.
Because s3fs only opens(makes) the file descriptor with no downloading
data. And when s3fs reads a data, s3fs downloads only some block unit
including specified area.
This result is good for performance.
5) Changes option name
The option "parallel_upload" which added at r456 is changed to new option
name as "parallel_count". This reason is this option value is not only used by
uploading object, but a uploading object also uses this option. (For a while,
you can use old option name "parallel_upload" for compatibility.)
git-svn-id: http://s3fs.googlecode.com/svn/trunk@458 df820570-a93a-0410-bd06-b72b767a4274
1) Changes FdCache class(cleanup codes)
The FdCache class is for caching file discriptor.
This class is modified as adding reference count for file discriptor and
removing pid for each path.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@450 df820570-a93a-0410-bd06-b72b767a4274
1) problems with fseek and s3fs(Issue 241)
This problem is that s3fs returns file stat(size) when client already has opened file discriptor and client modifies file before saving fd.
So that, client adds bytes into file, but s3fs_getattr() function returns original size before changing.
Changes codes for that s3fs returns size from opened file discriptor, if client already opens the file.
* Changes s3fs.cpp
* Adds fdcache.cpp fdcache.h
git-svn-id: http://s3fs.googlecode.com/svn/trunk@432 df820570-a93a-0410-bd06-b72b767a4274
