mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2025-01-22 21:38:24 +00:00
Changes codes for performance(part 3)
* Summay
This revision includes big change about temporary file and local cache file.
By this big change, s3fs works with good performance when s3fs opens/
closes/syncs/reads object.
I made a big change about the handling about temporary file and local cache
file to do this implementation.
* Detail
1) About temporary file(local file)
s3fs uses a temporary file on local file system when s3fs does download/
upload/open/seek object on S3.
After this revision, s3fs calls ftruncate() function when s3fs makes the
temporary file.
In this way s3fs can set a file size of precisely length without downloading.
(Notice - ftruncate function is for XSI-compliant systems, so that possibly
you have a problem on non-XSI-compliant systems.)
By this change, s3fs can download a part of a object by requesting with
"Range" http header. It seems like downloading by each block unit.
The default block(part) size is 50MB, it is caused the result which is default
parallel requests count(5) by default multipart upload size(10MB).
If you need to change this block size, you can change by new option
"fd_page_size". This option can take from 1MB(1024 * 1024) to any bytes.
So that, you have to take care about that fdcache.cpp(and fdcache.h) were
changed a lot.
2) About local cache
Local cache files which are in directory specified by "use_cache" option do
not have always all of object data.
This cause is that s3fs uses ftruncate function and reads(writes) each block
unit of a temporary file.
s3fs manages each block unit's status which are "downloaded area" or "not".
For this status, s3fs makes new temporary file in cache directory which is
specified by "use_cache" option. This status files is in a directory which is
named "<use_cache sirectory>/.<bucket_name>/".
When s3fs opens this status file, s3fs locks this file for exclusive control by
calling flock function. You need to take care about this, the status files can
not be laid on network drive(like NFS).
This revision changes about file open mode, s3fs always opens a local cache
file and each status file with writable mode.
Last, this revision adds new option "del_cache", this option means that s3fs
deletes all local cache file when s3fs starts and exits.
3) Uploading
When s3fs writes data to file descriptor through FUSE request, old s3fs
revision downloads all of the object. But new revision does not download all,
it downloads only small percial area(some block units) including writing data
area.
And when s3fs closes or flushes the file descriptor, s3fs downloads other area
which is not downloaded from server. After that, s3fs uploads all of data.
Already r456 revision has parallel upload function, then this revision with
r456 and r457 are very big change for performance.
4) Downloading
By changing a temporary file and a local cache file, when s3fs downloads a
object, it downloads only the required range(some block units).
And s3fs downloads units by parallel GET request, it is same as a case of
uploading. (Maximum parallel request count and each download size are
specified same parameters for uploading.)
In the new revision, when s3fs opens file, s3fs returns file descriptor soon.
Because s3fs only opens(makes) the file descriptor with no downloading
data. And when s3fs reads a data, s3fs downloads only some block unit
including specified area.
This result is good for performance.
5) Changes option name
The option "parallel_upload" which added at r456 is changed to new option
name as "parallel_count". This reason is this option value is not only used by
uploading object, but a uploading object also uses this option. (For a while,
you can use old option name "parallel_upload" for compatibility.)
git-svn-id: http://s3fs.googlecode.com/svn/trunk@458 df820570-a93a-0410-bd06-b72b767a4274
This commit is contained in:
ggtakec@gmail.com
parent
1c93dd30c1
commit
3274f58948
@ -59,6 +59,9 @@ number of times to retry a failed S3 transaction.
|
||||
\fB\-o\fR use_cache (default="" which means disabled)
|
||||
local folder to use for local file cache.
|
||||
.TP
|
||||
\fB\-o\fR del_cache - delete local file cache
|
||||
delete local file cache when s3fs starts and exits.
|
||||
.TP
|
||||
\fB\-o\fR use_rrs (default is disable)
|
||||
use Amazon's Reduced Redundancy Storage.
|
||||
this option can not be specified with use_sse.
|
||||
@ -99,12 +102,17 @@ s3fs is always using dns cache, this option make dns cache disable.
|
||||
\fB\-o\fR multireq_max (default="500")
|
||||
maximum number of parallel request for listing objects.
|
||||
.TP
|
||||
\fB\-o\fR parallel_upload (default="5")
|
||||
\fB\-o\fR parallel_count (default="5")
|
||||
number of parallel request for uploading big objects.
|
||||
s3fs uploads large object(over 20MB) by multipart post request, and sends parallel requests.
|
||||
This option limits parallel request count which s3fs requests at once.
|
||||
It is necessary to set this value depending on a CPU and a network band.
|
||||
.TP
|
||||
\fB\-o\fR fd_page_size(default="52428800"(50MB))
|
||||
number of internal management page size for each file discriptor.
|
||||
For delayed reading and writing by s3fs, s3fs manages pages which is separated from object. Each pages has a status that data is already loaded(or not loaded yet).
|
||||
This option should not be changed when you don't have a trouble with performance.
|
||||
.TP
|
||||
\fB\-o\fR url (default="http://s3.amazonaws.com")
|
||||
sets the url to use to access Amazon S3. If you want to use HTTPS, then you can set url=https://s3.amazonaws.com
|
||||
.TP
|
||||
|
||||
@ -23,6 +23,11 @@
|
||||
printf(__VA_ARGS__); \
|
||||
}
|
||||
|
||||
#define FGPRINT2(...) \
|
||||
if(foreground2){ \
|
||||
printf(__VA_ARGS__); \
|
||||
}
|
||||
|
||||
#define SAFESTRPTR(strptr) (strptr ? strptr : "")
|
||||
|
||||
//
|
||||
@ -35,6 +40,8 @@ typedef std::map<std::string, std::string> headers_t;
|
||||
//
|
||||
extern bool debug;
|
||||
extern bool foreground;
|
||||
extern bool foreground2;
|
||||
extern bool nomultipart;
|
||||
extern std::string program_name;
|
||||
extern std::string service_path;
|
||||
extern std::string host;
|
||||
|
||||
184
src/curl.cpp
184
src/curl.cpp
@ -150,7 +150,7 @@ curltime_t S3fsCurl::curl_times;
|
||||
curlprogress_t S3fsCurl::curl_progress;
|
||||
string S3fsCurl::curl_ca_bundle;
|
||||
mimes_t S3fsCurl::mimeTypes;
|
||||
int S3fsCurl::max_parallel_upload = 5; // default
|
||||
int S3fsCurl::max_parallel_cnt = 5; // default
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Class methods for S3fsCurl
|
||||
@ -554,6 +554,41 @@ size_t S3fsCurl::UploadReadCallback(void *ptr, size_t size, size_t nmemb, void *
|
||||
return totalread;
|
||||
}
|
||||
|
||||
size_t S3fsCurl::DownloadWriteCallback(void* ptr, size_t size, size_t nmemb, void* userp)
|
||||
{
|
||||
S3fsCurl* pCurl = reinterpret_cast<S3fsCurl*>(userp);
|
||||
|
||||
if(1 > (size * nmemb)){
|
||||
return 0;
|
||||
}
|
||||
if(-1 == pCurl->partdata.fd || 0 >= pCurl->partdata.size){
|
||||
return 0;
|
||||
}
|
||||
|
||||
// write size
|
||||
ssize_t copysize = (size * nmemb) < (size_t)pCurl->partdata.size ? (size * nmemb) : (size_t)pCurl->partdata.size;
|
||||
ssize_t writebytes;
|
||||
ssize_t totalwrite;
|
||||
|
||||
// write
|
||||
for(totalwrite = 0, writebytes = 0; totalwrite < copysize; totalwrite += writebytes){
|
||||
writebytes = pwrite(pCurl->partdata.fd, &((char*)ptr)[totalwrite], (copysize - totalwrite), pCurl->partdata.startpos + totalwrite);
|
||||
if(0 == writebytes){
|
||||
// eof?
|
||||
break;
|
||||
}else if(-1 == writebytes){
|
||||
// error
|
||||
FGPRINT("S3fsCurl::DownloadWriteCallback: write file error(%d).\n", errno);
|
||||
SYSLOGERR("write file error(%d).", errno);
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
pCurl->partdata.startpos += totalwrite;
|
||||
pCurl->partdata.size -= totalwrite;
|
||||
|
||||
return totalwrite;
|
||||
}
|
||||
|
||||
bool S3fsCurl::SetDnsCache(bool isCache)
|
||||
{
|
||||
bool old = S3fsCurl::is_dns_cache;
|
||||
@ -637,10 +672,10 @@ long S3fsCurl::SetSslVerifyHostname(long value)
|
||||
return old;
|
||||
}
|
||||
|
||||
int S3fsCurl::SetMaxParallelUpload(int value)
|
||||
int S3fsCurl::SetMaxParallelCount(int value)
|
||||
{
|
||||
int old = S3fsCurl::max_parallel_upload;
|
||||
S3fsCurl::max_parallel_upload = value;
|
||||
int old = S3fsCurl::max_parallel_cnt;
|
||||
S3fsCurl::max_parallel_cnt = value;
|
||||
return old;
|
||||
}
|
||||
|
||||
@ -705,7 +740,6 @@ int S3fsCurl::ParallelMultipartUploadRequest(const char* tpath, headers_t& meta,
|
||||
etaglist_t list;
|
||||
off_t remaining_bytes;
|
||||
unsigned char* buf;
|
||||
char tmpfile[256];
|
||||
S3fsCurl s3fscurl;
|
||||
|
||||
FGPRINT(" S3fsCurl::ParallelMultipartUploadRequest[tpath=%s][fd=%d]\n", SAFESTRPTR(tpath), fd);
|
||||
@ -752,7 +786,7 @@ int S3fsCurl::ParallelMultipartUploadRequest(const char* tpath, headers_t& meta,
|
||||
curlmulti.SetRetryCallback(S3fsCurl::UploadMultipartPostRetryCallback);
|
||||
|
||||
// Loop for setup parallel upload(multipart) request.
|
||||
for(para_cnt = 0; para_cnt < S3fsCurl::max_parallel_upload && 0 < remaining_bytes; para_cnt++, remaining_bytes -= chunk){
|
||||
for(para_cnt = 0; para_cnt < S3fsCurl::max_parallel_cnt && 0 < remaining_bytes; para_cnt++, remaining_bytes -= chunk){
|
||||
// chunk size
|
||||
chunk = remaining_bytes > MULTIPART_SIZE ? MULTIPART_SIZE : remaining_bytes;
|
||||
|
||||
@ -775,12 +809,12 @@ int S3fsCurl::ParallelMultipartUploadRequest(const char* tpath, headers_t& meta,
|
||||
|
||||
// set into parallel object
|
||||
if(!curlmulti.SetS3fsCurlObject(s3fscurl_para)){
|
||||
FGPRINT("S3fsCurl::ParallelMultipartUploadRequest: Could not set curl object into multi curl(%s).\n", tmpfile);
|
||||
SYSLOGERR("Could not make curl object into multi curl(%s).", tmpfile);
|
||||
FGPRINT("S3fsCurl::ParallelMultipartUploadRequest: Could not set curl object into multi curl(%s).\n", tpath);
|
||||
SYSLOGERR("Could not make curl object into multi curl(%s).", tpath);
|
||||
free(buf);
|
||||
fclose(file);
|
||||
delete s3fscurl_para;
|
||||
return result;
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
|
||||
@ -803,6 +837,78 @@ int S3fsCurl::ParallelMultipartUploadRequest(const char* tpath, headers_t& meta,
|
||||
return 0;
|
||||
}
|
||||
|
||||
S3fsCurl* S3fsCurl::ParallelGetObjectRetryCallback(S3fsCurl* s3fscurl)
|
||||
{
|
||||
int result;
|
||||
|
||||
if(!s3fscurl){
|
||||
return NULL;
|
||||
}
|
||||
// duplicate request(setup new curl object)
|
||||
S3fsCurl* newcurl = new S3fsCurl();
|
||||
if(0 != (result = newcurl->PreGetObjectRequest(
|
||||
s3fscurl->path.c_str(), s3fscurl->partdata.fd, s3fscurl->partdata.startpos, s3fscurl->partdata.size))){
|
||||
FGPRINT("S3fsCurl::ParallelGetObjectRetryCallback: failed downloading part setup(%d)\n", result);
|
||||
SYSLOGERR("failed downloading part setup(%d)", result);
|
||||
delete newcurl;
|
||||
return NULL;;
|
||||
}
|
||||
return newcurl;
|
||||
}
|
||||
|
||||
int S3fsCurl::ParallelGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size)
|
||||
{
|
||||
FGPRINT(" S3fsCurl::ParallelGetObjectRequest[tpath=%s][fd=%d]\n", SAFESTRPTR(tpath), fd);
|
||||
|
||||
int result = 0;
|
||||
ssize_t remaining_bytes;
|
||||
|
||||
// cycle through open fd, pulling off 10MB chunks at a time
|
||||
for(remaining_bytes = size; 0 < remaining_bytes; ){
|
||||
S3fsMultiCurl curlmulti;
|
||||
int para_cnt;
|
||||
off_t chunk;
|
||||
|
||||
// Initialize S3fsMultiCurl
|
||||
//curlmulti.SetSuccessCallback(NULL); // not need to set success callback
|
||||
curlmulti.SetRetryCallback(S3fsCurl::ParallelGetObjectRetryCallback);
|
||||
|
||||
// Loop for setup parallel upload(multipart) request.
|
||||
for(para_cnt = 0; para_cnt < S3fsCurl::max_parallel_cnt && 0 < remaining_bytes; para_cnt++, remaining_bytes -= chunk){
|
||||
// chunk size
|
||||
chunk = remaining_bytes > MULTIPART_SIZE ? MULTIPART_SIZE : remaining_bytes;
|
||||
|
||||
// s3fscurl sub object
|
||||
S3fsCurl* s3fscurl_para = new S3fsCurl();
|
||||
if(0 != (result = s3fscurl_para->PreGetObjectRequest(tpath, fd, (start + size - remaining_bytes), chunk))){
|
||||
FGPRINT("S3fsCurl::ParallelGetObjectRequest: failed downloading part setup(%d)\n", result);
|
||||
SYSLOGERR("failed downloading part setup(%d)", result);
|
||||
delete s3fscurl_para;
|
||||
return result;
|
||||
}
|
||||
|
||||
// set into parallel object
|
||||
if(!curlmulti.SetS3fsCurlObject(s3fscurl_para)){
|
||||
FGPRINT("S3fsCurl::ParallelGetObjectRequest: Could not set curl object into multi curl(%s).\n", tpath);
|
||||
SYSLOGERR("Could not make curl object into multi curl(%s).", tpath);
|
||||
delete s3fscurl_para;
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
|
||||
// Multi request
|
||||
if(0 != (result = curlmulti.Request())){
|
||||
FGPRINT("S3fsCurl::ParallelGetObjectRequest: error occuered in multi request(errno=%d).\n", result);
|
||||
SYSLOGERR("error occuered in multi request(errno=%d).", result);
|
||||
break;
|
||||
}
|
||||
|
||||
// reinit for loop.
|
||||
curlmulti.Clear();
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Methods for S3fsCurl
|
||||
//-------------------------------------------------------------------
|
||||
@ -1558,21 +1664,13 @@ int S3fsCurl::PutRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse
|
||||
return result;
|
||||
}
|
||||
|
||||
int S3fsCurl::GetObjectRequest(const char* tpath, int fd)
|
||||
int S3fsCurl::PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size)
|
||||
{
|
||||
FILE* file;
|
||||
int fd2;
|
||||
FGPRINT(" S3fsCurl::GetRequest [tpath=%s]\n", SAFESTRPTR(tpath));
|
||||
FGPRINT(" S3fsCurl::PreGetRequest [tpath=%s][start=%zd][size=%zd]\n", SAFESTRPTR(tpath), start, size);
|
||||
|
||||
if(!tpath){
|
||||
if(!tpath || -1 == fd || 0 > start || 0 >= size){
|
||||
return -1;
|
||||
}
|
||||
// duplicate fd
|
||||
if(-1 == (fd2 = dup(fd)) || 0 != lseek(fd2, 0, SEEK_SET) || NULL == (file = fdopen(fd2, "w+"))){
|
||||
FGPRINT("S3fsCurl::GetRequest : Cloud not duplicate file discriptor(errno=%d)\n", errno);
|
||||
SYSLOGERR("Cloud not duplicate file discriptor(errno=%d)", errno);
|
||||
return -errno;
|
||||
}
|
||||
|
||||
if(!CreateCurlHandle(true)){
|
||||
return -1;
|
||||
@ -1589,6 +1687,13 @@ int S3fsCurl::GetObjectRequest(const char* tpath, int fd)
|
||||
string date = get_date();
|
||||
requestHeaders = curl_slist_sort_insert(requestHeaders, string("Date: " + date).c_str());
|
||||
requestHeaders = curl_slist_sort_insert(requestHeaders, "Content-Type: ");
|
||||
if(-1 != start && -1 != size){
|
||||
string range = "Range: bytes=";
|
||||
range += str(start);
|
||||
range += "-";
|
||||
range += str(start + size - 1);
|
||||
requestHeaders = curl_slist_sort_insert(requestHeaders, range.c_str());
|
||||
}
|
||||
|
||||
if(!S3fsCurl::IsPublicBucket()){
|
||||
requestHeaders = curl_slist_sort_insert(
|
||||
@ -1600,20 +1705,37 @@ int S3fsCurl::GetObjectRequest(const char* tpath, int fd)
|
||||
// setopt
|
||||
curl_easy_setopt(hCurl, CURLOPT_URL, url.c_str());
|
||||
curl_easy_setopt(hCurl, CURLOPT_HTTPHEADER, requestHeaders);
|
||||
curl_easy_setopt(hCurl, CURLOPT_FILE, file);
|
||||
curl_easy_setopt(hCurl, CURLOPT_WRITEFUNCTION, S3fsCurl::DownloadWriteCallback);
|
||||
curl_easy_setopt(hCurl, CURLOPT_WRITEDATA, (void*)this);
|
||||
|
||||
// set info for callback func.
|
||||
// (use only fd, startpos and size, other member is not used.)
|
||||
partdata.clear();
|
||||
partdata.fd = fd;
|
||||
partdata.startpos = start;
|
||||
partdata.size = size;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int S3fsCurl::GetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size)
|
||||
{
|
||||
int result;
|
||||
|
||||
FGPRINT(" S3fsCurl::GetRequest [tpath=%s][start=%zd][size=%zd]\n", SAFESTRPTR(tpath), start, size);
|
||||
|
||||
if(!tpath){
|
||||
return -1;
|
||||
}
|
||||
if(0 != (result = PreGetObjectRequest(tpath, fd, start, size))){
|
||||
return result;
|
||||
}
|
||||
|
||||
FGPRINT(" downloading... [path=%s][fd=%d]\n", tpath, fd);
|
||||
SYSLOGDBG("LOCAL FD");
|
||||
|
||||
int result = RequestPerform();
|
||||
|
||||
fflush(file);
|
||||
fclose(file);
|
||||
if(0 != lseek(fd, 0, SEEK_SET)){
|
||||
FGPRINT("S3fsCurl::GetRequest : Cloud not seek file discriptor(errno=%d)\n", errno);
|
||||
SYSLOGERR("Cloud not seek file discriptor(errno=%d)", errno);
|
||||
return -errno;
|
||||
}
|
||||
result = RequestPerform();
|
||||
partdata.clear();
|
||||
|
||||
return result;
|
||||
}
|
||||
@ -1811,7 +1933,7 @@ int S3fsCurl::CompleteMultipartPostRequest(const char* tpath, string& upload_id,
|
||||
for(int cnt = 0; cnt < (int)parts.size(); cnt++){
|
||||
if(0 == parts[cnt].length()){
|
||||
FGPRINT("S3fsCurl::CompleteMultipartPostRequest : %d file part is not finished uploading.\n", cnt + 1);
|
||||
return false;
|
||||
return -1;
|
||||
}
|
||||
postContent += "<Part>\n";
|
||||
postContent += " <PartNumber>" + IntToStr(cnt + 1) + "</PartNumber>\n";
|
||||
|
||||
14
src/curl.h
14
src/curl.h
@ -58,7 +58,7 @@ struct filepart
|
||||
clear();
|
||||
}
|
||||
|
||||
void clear(bool isfree = true)
|
||||
void clear(void)
|
||||
{
|
||||
uploaded = false;
|
||||
etag = "";
|
||||
@ -128,7 +128,7 @@ class S3fsCurl
|
||||
static curlprogress_t curl_progress;
|
||||
static std::string curl_ca_bundle;
|
||||
static mimes_t mimeTypes;
|
||||
static int max_parallel_upload;
|
||||
static int max_parallel_cnt;
|
||||
|
||||
// variables
|
||||
CURL* hCurl;
|
||||
@ -143,7 +143,7 @@ class S3fsCurl
|
||||
long LastResponseCode;
|
||||
const unsigned char* postdata; // use by post method and read callback function.
|
||||
int postdata_remaining; // use by post method and read callback function.
|
||||
filepart partdata; // use by multipart upload
|
||||
filepart partdata; // use by multipart upload/get object callback
|
||||
|
||||
public:
|
||||
// constructor/destructor
|
||||
@ -162,9 +162,11 @@ class S3fsCurl
|
||||
static size_t WriteMemoryCallback(void *ptr, size_t blockSize, size_t numBlocks, void *data);
|
||||
static size_t ReadCallback(void *ptr, size_t size, size_t nmemb, void *userp);
|
||||
static size_t UploadReadCallback(void *ptr, size_t size, size_t nmemb, void *userp);
|
||||
static size_t DownloadWriteCallback(void* ptr, size_t size, size_t nmemb, void* userp);
|
||||
|
||||
static bool UploadMultipartPostCallback(S3fsCurl* s3fscurl);
|
||||
static S3fsCurl* UploadMultipartPostRetryCallback(S3fsCurl* s3fscurl);
|
||||
static S3fsCurl* ParallelGetObjectRetryCallback(S3fsCurl* s3fscurl);
|
||||
|
||||
// methods
|
||||
bool ClearInternalData(void);
|
||||
@ -186,6 +188,7 @@ class S3fsCurl
|
||||
static bool InitShareCurl(void);
|
||||
static bool DestroyShareCurl(void);
|
||||
static int ParallelMultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse_flg);
|
||||
static int ParallelGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size);
|
||||
|
||||
// class methods(valiables)
|
||||
static std::string LookupMimeType(std::string name);
|
||||
@ -206,7 +209,7 @@ class S3fsCurl
|
||||
static bool IsSetAccessKeyId(void) { return (0 < S3fsCurl::AWSAccessKeyId.size() && 0 < S3fsCurl::AWSSecretAccessKey.size()); }
|
||||
static long SetSslVerifyHostname(long value);
|
||||
static long GetSslVerifyHostname(void) { return S3fsCurl::ssl_verify_hostname; }
|
||||
static int SetMaxParallelUpload(int value);
|
||||
static int SetMaxParallelCount(int value);
|
||||
|
||||
// methods
|
||||
bool CreateCurlHandle(bool force = false);
|
||||
@ -222,7 +225,8 @@ class S3fsCurl
|
||||
int HeadRequest(const char* tpath, headers_t& meta);
|
||||
int PutHeadRequest(const char* tpath, headers_t& meta, bool ow_sse_flg);
|
||||
int PutRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse_flg);
|
||||
int GetObjectRequest(const char* tpath, int fd);
|
||||
int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size);
|
||||
int GetObjectRequest(const char* tpath, int fd, off_t start = -1, ssize_t size = -1);
|
||||
int CheckBucket(void);
|
||||
int ListBucketRequest(const char* tpath, const char* query);
|
||||
int MultipartListRequest(std::string& body);
|
||||
|
||||
1324
src/fdcache.cpp
1324
src/fdcache.cpp
File diff suppressed because it is too large
Load Diff
178
src/fdcache.h
178
src/fdcache.h
@ -1,50 +1,154 @@
|
||||
#ifndef FD_CACHE_H_
|
||||
#define FD_CACHE_H_
|
||||
|
||||
#include "common.h"
|
||||
|
||||
//
|
||||
// Struct for fuse file handle cache
|
||||
//
|
||||
struct fd_cache_entry {
|
||||
int refcnt;
|
||||
int fd;
|
||||
int flags;
|
||||
|
||||
fd_cache_entry() : refcnt(0), fd(0), flags(0) {}
|
||||
};
|
||||
|
||||
typedef std::list<int> fd_list_t;
|
||||
typedef std::list<struct fd_cache_entry> fd_cache_entlist_t;
|
||||
typedef std::map<std::string, fd_cache_entlist_t*> fd_cache_t; // key=path, value=<list>*
|
||||
typedef std::map<int, int> fd_flags_t; // key=file discriptor, value=flags
|
||||
|
||||
//
|
||||
// Class for fuse file handle cache
|
||||
//
|
||||
class FdCache
|
||||
//------------------------------------------------
|
||||
// CacheFileStat
|
||||
//------------------------------------------------
|
||||
class CacheFileStat
|
||||
{
|
||||
private:
|
||||
static FdCache singleton;
|
||||
static pthread_mutex_t fd_cache_lock;
|
||||
fd_cache_t fd_cache;
|
||||
fd_flags_t fd_flags;
|
||||
std::string path;
|
||||
int fd;
|
||||
|
||||
private:
|
||||
static bool MakeCacheFileStatPath(const char* path, std::string& sfile_path, bool is_create_dir = true);
|
||||
|
||||
public:
|
||||
FdCache();
|
||||
~FdCache();
|
||||
CacheFileStat(const char* tpath = NULL);
|
||||
~CacheFileStat();
|
||||
|
||||
bool Open(void);
|
||||
bool Release(void);
|
||||
bool SetPath(const char* tpath, bool is_open = true);
|
||||
int GetFd(void) const { return fd; }
|
||||
};
|
||||
|
||||
//------------------------------------------------
|
||||
// fdpage & PageList
|
||||
//------------------------------------------------
|
||||
// page block information
|
||||
struct fdpage
|
||||
{
|
||||
off_t offset;
|
||||
size_t bytes;
|
||||
bool init;
|
||||
|
||||
fdpage(off_t start = 0, size_t size = 0, bool is_init = false)
|
||||
: offset(start), bytes(size), init(is_init) {}
|
||||
|
||||
off_t next(void) const { return (offset + bytes); }
|
||||
off_t end(void) const { return (0 < bytes ? offset + bytes - 1 : 0); }
|
||||
};
|
||||
typedef std::list<struct fdpage*> fdpage_list_t;
|
||||
|
||||
//
|
||||
// Management of loading area/modifying
|
||||
//
|
||||
class PageList
|
||||
{
|
||||
private:
|
||||
fdpage_list_t pages;
|
||||
|
||||
private:
|
||||
void Clear(void);
|
||||
|
||||
public:
|
||||
static void FreeList(fdpage_list_t& list);
|
||||
|
||||
PageList(size_t size = 0, bool is_init = false);
|
||||
~PageList();
|
||||
|
||||
size_t Size(void) const;
|
||||
int Resize(size_t size, bool is_init);
|
||||
int Init(size_t size, bool is_init);
|
||||
bool IsInit(off_t start, size_t size);
|
||||
bool SetInit(off_t start, size_t size, bool is_init = true);
|
||||
bool FindUninitPage(off_t start, off_t& resstart, size_t& ressize);
|
||||
int GetUninitPages(fdpage_list_t& uninit_list, off_t start = 0);
|
||||
bool Serialize(CacheFileStat& file, bool is_output);
|
||||
void Dump(void);
|
||||
};
|
||||
|
||||
//------------------------------------------------
|
||||
// class FdEntity
|
||||
//------------------------------------------------
|
||||
class FdEntity
|
||||
{
|
||||
private:
|
||||
pthread_mutex_t fdent_lock;
|
||||
bool is_lock_init;
|
||||
PageList pagelist;
|
||||
int refcnt; // reference count
|
||||
std::string path; // object path
|
||||
std::string cachepath; // local cache file path
|
||||
int fd; // file discriptor(tmp file or cache file)
|
||||
FILE* file; // file pointer(tmp file or cache file)
|
||||
bool is_modify; // if file is changed, this flag is true
|
||||
|
||||
private:
|
||||
void Clear(void);
|
||||
int Dup(void);
|
||||
bool SetAllStatus(bool is_enable);
|
||||
|
||||
public:
|
||||
FdEntity(const char* tpath = NULL, const char* cpath = NULL);
|
||||
~FdEntity();
|
||||
|
||||
void Close(void);
|
||||
bool IsOpen(void) const { return (-1 != fd); }
|
||||
int Open(ssize_t size = -1, time_t time = -1);
|
||||
const char* GetPath(void) const { return path.c_str(); }
|
||||
int GetFd(void) const { return fd; }
|
||||
int SetMtime(time_t time);
|
||||
bool GetSize(size_t& size);
|
||||
bool GetMtime(time_t& time);
|
||||
bool GetStats(struct stat& st);
|
||||
|
||||
bool SetAllEnable(void) { return SetAllStatus(true); }
|
||||
bool SetAllDisable(void) { return SetAllStatus(false); }
|
||||
bool LoadFull(size_t* size = NULL, bool force_load = false);
|
||||
int Load(off_t start, ssize_t size);
|
||||
int RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool force_sync = false);
|
||||
int Flush(headers_t& meta, bool ow_sse_flg, bool force_sync = false) { return RowFlush(NULL, meta, ow_sse_flg, force_sync); }
|
||||
ssize_t Read(char* bytes, off_t start, size_t size, bool force_load = false);
|
||||
ssize_t Write(const char* bytes, off_t start, size_t size);
|
||||
};
|
||||
typedef std::map<std::string, class FdEntity*> fdent_map_t; // key=path, value=FdEntity*
|
||||
|
||||
//------------------------------------------------
|
||||
// class FdManager
|
||||
//------------------------------------------------
|
||||
class FdManager
|
||||
{
|
||||
private:
|
||||
static FdManager singleton;
|
||||
static pthread_mutex_t fd_manager_lock;
|
||||
static bool is_lock_init;
|
||||
static std::string cache_dir;
|
||||
static size_t page_size;
|
||||
|
||||
fdent_map_t fent;
|
||||
|
||||
public:
|
||||
FdManager();
|
||||
~FdManager();
|
||||
|
||||
// Reference singleton
|
||||
static FdCache* getFdCacheData(void) {
|
||||
return &singleton;
|
||||
}
|
||||
static FdManager* get(void) { return &singleton; }
|
||||
|
||||
bool Add(const char* path, int fd, int flags);
|
||||
bool Del(const char* path, int fd);
|
||||
bool Del(const char* path);
|
||||
bool Del(int fd);
|
||||
bool Get(const char* path, int* pfd = NULL, int* pflags = NULL) const;
|
||||
bool Get(int fd, int* pflags = NULL) const;
|
||||
static bool DeleteCacheDirectory(void);
|
||||
static int DeleteCacheFile(const char* path);
|
||||
static bool SetCacheDir(const char* dir);
|
||||
static bool IsCacheDir(void) { return (0 < FdManager::cache_dir.size()); }
|
||||
static const char* GetCacheDir(void) { return FdManager::cache_dir.c_str(); }
|
||||
static size_t SetPageSize(size_t size);
|
||||
static size_t GetPageSize(void) { return FdManager::page_size; }
|
||||
static bool MakeCachePath(const char* path, std::string& cache_path, bool is_create_dir = true);
|
||||
|
||||
FdEntity* GetFdEntity(const char* path);
|
||||
FdEntity* Open(const char* path, ssize_t size = -1, time_t time = -1, bool force_tmpfile = false, bool is_create = true);
|
||||
FdEntity* ExistOpen(const char* path) { return Open(path, -1, -1, false, false); }
|
||||
bool Close(FdEntity* ent);
|
||||
};
|
||||
|
||||
#endif // FD_CACHE_H_
|
||||
|
||||
812
src/s3fs.cpp
812
src/s3fs.cpp
File diff suppressed because it is too large
Load Diff
@ -28,6 +28,9 @@
|
||||
#include <pwd.h>
|
||||
#include <grp.h>
|
||||
#include <syslog.h>
|
||||
#include <pthread.h>
|
||||
#include <sys/types.h>
|
||||
#include <dirent.h>
|
||||
|
||||
#include <string>
|
||||
#include <sstream>
|
||||
@ -397,6 +400,56 @@ void free_mvnodes(MVNODE *head)
|
||||
return;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Class AutoLock
|
||||
//-------------------------------------------------------------------
|
||||
AutoLock::AutoLock(pthread_mutex_t* pmutex) : auto_mutex(pmutex), is_locked(false)
|
||||
{
|
||||
Lock();
|
||||
}
|
||||
|
||||
AutoLock::~AutoLock()
|
||||
{
|
||||
Unlock();
|
||||
}
|
||||
|
||||
bool AutoLock::Lock(void)
|
||||
{
|
||||
if(!auto_mutex){
|
||||
return false;
|
||||
}
|
||||
if(is_locked){
|
||||
// already locked
|
||||
return true;
|
||||
}
|
||||
try{
|
||||
pthread_mutex_lock(auto_mutex);
|
||||
is_locked = true;
|
||||
}catch(exception& e){
|
||||
is_locked = false;
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool AutoLock::Unlock(void)
|
||||
{
|
||||
if(!auto_mutex){
|
||||
return false;
|
||||
}
|
||||
if(!is_locked){
|
||||
// already unlocked
|
||||
return true;
|
||||
}
|
||||
try{
|
||||
pthread_mutex_unlock(auto_mutex);
|
||||
is_locked = false;
|
||||
}catch(exception& e){
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility for UID/GID
|
||||
//-------------------------------------------------------------------
|
||||
@ -494,6 +547,53 @@ int mkdirp(const string& path, mode_t mode)
|
||||
return 0;
|
||||
}
|
||||
|
||||
bool delete_files_in_dir(const char* dir, bool is_remove_own)
|
||||
{
|
||||
DIR* dp;
|
||||
struct dirent* dent;
|
||||
|
||||
if(NULL == (dp = opendir(dir))){
|
||||
//FGPRINT("delete_files_in_dir: could not open dir(%s) - errno(%d)\n", dir, errno);
|
||||
return false;
|
||||
}
|
||||
|
||||
for(dent = readdir(dp); dent; dent = readdir(dp)){
|
||||
if(0 == strcmp(dent->d_name, "..") || 0 == strcmp(dent->d_name, ".")){
|
||||
continue;
|
||||
}
|
||||
string fullpath = dir;
|
||||
fullpath += "/";
|
||||
fullpath += dent->d_name;
|
||||
struct stat st;
|
||||
if(0 != lstat(fullpath.c_str(), &st)){
|
||||
FGPRINT("delete_files_in_dir: could not get stats of file(%s) - errno(%d)\n", fullpath.c_str(), errno);
|
||||
closedir(dp);
|
||||
return false;
|
||||
}
|
||||
if(S_ISDIR(st.st_mode)){
|
||||
// dir -> Reentrant
|
||||
if(!delete_files_in_dir(fullpath.c_str(), true)){
|
||||
//FGPRINT("delete_files_in_dir: could not remove sub dir(%s) - errno(%d)\n", fullpath.c_str(), errno);
|
||||
closedir(dp);
|
||||
return false;
|
||||
}
|
||||
}else{
|
||||
if(0 != unlink(fullpath.c_str())){
|
||||
FGPRINT("delete_files_in_dir: could not remove file(%s) - errno(%d)\n", fullpath.c_str(), errno);
|
||||
closedir(dp);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
closedir(dp);
|
||||
|
||||
if(is_remove_own && 0 != rmdir(dir)){
|
||||
FGPRINT("delete_files_in_dir: could not remove dir(%s) - errno(%d)\n", dir, errno);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Utility functions for convert
|
||||
//-------------------------------------------------------------------
|
||||
@ -709,6 +809,9 @@ void show_help (void)
|
||||
" use_cache (default=\"\" which means disabled)\n"
|
||||
" - local folder to use for local file cache\n"
|
||||
"\n"
|
||||
" del_cache (delete local file cache)\n"
|
||||
" - delete local file cache when s3fs starts and exits.\n"
|
||||
"\n"
|
||||
" use_rrs (default is disable)\n"
|
||||
" - this option makes Amazon's Reduced Redundancy Storage enable.\n"
|
||||
"\n"
|
||||
@ -749,7 +852,7 @@ void show_help (void)
|
||||
" multireq_max (default=\"500\")\n"
|
||||
" - maximum number of parallel request for listing objects.\n"
|
||||
"\n"
|
||||
" parallel_upload (default=\"5\")\n"
|
||||
" parallel_count (default=\"5\")\n"
|
||||
" - number of parallel request for uploading big objects.\n"
|
||||
" s3fs uploads large object(over 20MB) by multipart post request, \n"
|
||||
" and sends parallel requests.\n"
|
||||
@ -757,6 +860,14 @@ void show_help (void)
|
||||
" at once. It is necessary to set this value depending on a CPU \n"
|
||||
" and a network band.\n"
|
||||
"\n"
|
||||
" fd_page_size (default=\"52428800\"(50MB))\n"
|
||||
" - number of internal management page size for each file discriptor.\n"
|
||||
" For delayed reading and writing by s3fs, s3fs manages pages which \n"
|
||||
" is separated from object. Each pages has a status that data is \n"
|
||||
" already loaded(or not loaded yet).\n"
|
||||
" This option should not be changed when you don't have a trouble \n"
|
||||
" with performance.\n"
|
||||
"\n"
|
||||
" url (default=\"http://s3.amazonaws.com\")\n"
|
||||
" - sets the url to use to access amazon s3\n"
|
||||
"\n"
|
||||
|
||||
@ -64,6 +64,20 @@ typedef struct mvnode {
|
||||
struct mvnode *next;
|
||||
} MVNODE;
|
||||
|
||||
class AutoLock
|
||||
{
|
||||
private:
|
||||
pthread_mutex_t* auto_mutex;
|
||||
bool is_locked;
|
||||
|
||||
public:
|
||||
AutoLock(pthread_mutex_t* pmutex = NULL);
|
||||
~AutoLock();
|
||||
|
||||
bool Lock(void);
|
||||
bool Unlock(void);
|
||||
};
|
||||
|
||||
//-------------------------------------------------------------------
|
||||
// Functions
|
||||
//-------------------------------------------------------------------
|
||||
@ -79,6 +93,7 @@ int is_uid_inculde_group(uid_t uid, gid_t gid);
|
||||
std::string mydirname(std::string path);
|
||||
std::string mybasename(std::string path);
|
||||
int mkdirp(const std::string& path, mode_t mode);
|
||||
bool delete_files_in_dir(const char* dir, bool is_remove_own);
|
||||
|
||||
time_t get_mtime(const char *s);
|
||||
time_t get_mtime(headers_t& meta, bool overcheck = true);
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user