Changes codes for performance(part 3)

* Summay
   This revision includes big change about temporary file and local cache file. 
   By this big change, s3fs works with good performance when s3fs opens/
   closes/syncs/reads object.
   I made a big change about the handling about temporary file and local cache
   file to do this implementation.

* Detail
1) About temporary file(local file)
   s3fs uses a temporary file on local file system when s3fs does download/
   upload/open/seek object on S3.
   After this revision, s3fs calls ftruncate() function when s3fs makes the 
   temporary file.
   In this way s3fs can set a file size of precisely length without downloading.
   (Notice - ftruncate function is for XSI-compliant systems, so that possibly
    you have a problem on non-XSI-compliant systems.)

   By this change, s3fs can download a part of a object by requesting with 
   "Range" http header. It seems like downloading by each block unit.
   The default block(part) size is 50MB, it is caused the result which is default 
   parallel requests count(5) by default multipart upload size(10MB).
   If you need to change this block size, you can change by new option 
   "fd_page_size". This option can take from 1MB(1024 * 1024) to any bytes.

   So that, you have to take care about that fdcache.cpp(and fdcache.h) were 
   changed a lot.

2) About local cache
   Local cache files which are in directory specified by "use_cache" option do 
   not have always all of object data.
   This cause is that s3fs uses ftruncate function and reads(writes) each block 
   unit of a temporary file.
   s3fs manages each block unit's status which are "downloaded area" or "not".
   For this status, s3fs makes new temporary file in cache directory which is 
   specified by "use_cache" option. This status files is in a directory which is 
   named "<use_cache sirectory>/.<bucket_name>/".

   When s3fs opens this status file, s3fs locks this file for exclusive control by 
   calling flock function. You need to take care about this, the status files can 
   not be laid on network drive(like NFS).

   This revision changes about file open mode, s3fs always opens a local cache 
   file and each status file with writable mode.
   Last, this revision adds new option "del_cache", this option means that s3fs 
   deletes all local cache file when s3fs starts and exits.

3) Uploading
   When s3fs writes data to file descriptor through FUSE request, old s3fs 
   revision downloads all of the object. But new revision does not download all, 
   it downloads only small percial area(some block units) including writing data 
   area.
   And when s3fs closes or flushes the file descriptor, s3fs downloads other area 
   which is not downloaded from server. After that,  s3fs uploads all of data.
   Already r456 revision has parallel upload function, then this revision with 
   r456 and r457 are very big change for performance.

4) Downloading
   By changing a temporary file and a local cache file, when s3fs downloads a 
   object, it downloads only the required range(some block units). 
   And s3fs downloads units by parallel GET request, it is same as a case of 
   uploading. (Maximum parallel request count and each download size are 
   specified same parameters for uploading.)

   In the new revision, when s3fs opens file, s3fs returns file descriptor soon.
   Because s3fs only opens(makes) the file descriptor with no downloading 
   data. And when s3fs reads a data, s3fs downloads only some block unit 
   including specified area.
   This result is good for performance.

5) Changes option name
   The option "parallel_upload" which added at r456 is changed to new option 
   name as "parallel_count". This reason is this option value is not only used by 
   uploading object, but a uploading object also uses this option. (For a while, 
   you can use old option name "parallel_upload" for compatibility.)



git-svn-id: http://s3fs.googlecode.com/svn/trunk@458 df820570-a93a-0410-bd06-b72b767a4274
This commit is contained in:
ggtakec@gmail.com 2013-07-23 16:01:48 +00:00
parent 1c93dd30c1
commit 3274f58948
9 changed files with 1881 additions and 798 deletions

View File

@ -59,6 +59,9 @@ number of times to retry a failed S3 transaction.
\fB\-o\fR use_cache (default="" which means disabled)
local folder to use for local file cache.
.TP
\fB\-o\fR del_cache - delete local file cache
delete local file cache when s3fs starts and exits.
.TP
\fB\-o\fR use_rrs (default is disable)
use Amazon's Reduced Redundancy Storage.
this option can not be specified with use_sse.
@ -99,12 +102,17 @@ s3fs is always using dns cache, this option make dns cache disable.
\fB\-o\fR multireq_max (default="500")
maximum number of parallel request for listing objects.
.TP
\fB\-o\fR parallel_upload (default="5")
\fB\-o\fR parallel_count (default="5")
number of parallel request for uploading big objects.
s3fs uploads large object(over 20MB) by multipart post request, and sends parallel requests.
This option limits parallel request count which s3fs requests at once.
It is necessary to set this value depending on a CPU and a network band.
.TP
\fB\-o\fR fd_page_size(default="52428800"(50MB))
number of internal management page size for each file discriptor.
For delayed reading and writing by s3fs, s3fs manages pages which is separated from object. Each pages has a status that data is already loaded(or not loaded yet).
This option should not be changed when you don't have a trouble with performance.
.TP
\fB\-o\fR url (default="http://s3.amazonaws.com")
sets the url to use to access Amazon S3. If you want to use HTTPS, then you can set url=https://s3.amazonaws.com
.TP

View File

@ -23,6 +23,11 @@
printf(__VA_ARGS__); \
}
#define FGPRINT2(...) \
if(foreground2){ \
printf(__VA_ARGS__); \
}
#define SAFESTRPTR(strptr) (strptr ? strptr : "")
//
@ -35,6 +40,8 @@ typedef std::map<std::string, std::string> headers_t;
//
extern bool debug;
extern bool foreground;
extern bool foreground2;
extern bool nomultipart;
extern std::string program_name;
extern std::string service_path;
extern std::string host;

View File

@ -150,7 +150,7 @@ curltime_t S3fsCurl::curl_times;
curlprogress_t S3fsCurl::curl_progress;
string S3fsCurl::curl_ca_bundle;
mimes_t S3fsCurl::mimeTypes;
int S3fsCurl::max_parallel_upload = 5; // default
int S3fsCurl::max_parallel_cnt = 5; // default
//-------------------------------------------------------------------
// Class methods for S3fsCurl
@ -469,7 +469,7 @@ bool S3fsCurl::LocateBundle(void)
return true;
}
size_t S3fsCurl::WriteMemoryCallback(void *ptr, size_t blockSize, size_t numBlocks, void *data)
size_t S3fsCurl::WriteMemoryCallback(void* ptr, size_t blockSize, size_t numBlocks, void* data)
{
BodyData* body = (BodyData*)data;
@ -481,7 +481,7 @@ size_t S3fsCurl::WriteMemoryCallback(void *ptr, size_t blockSize, size_t numBloc
return (blockSize * numBlocks);
}
size_t S3fsCurl::ReadCallback(void *ptr, size_t size, size_t nmemb, void *userp)
size_t S3fsCurl::ReadCallback(void* ptr, size_t size, size_t nmemb, void* userp)
{
S3fsCurl* pCurl = reinterpret_cast<S3fsCurl*>(userp);
@ -500,7 +500,7 @@ size_t S3fsCurl::ReadCallback(void *ptr, size_t size, size_t nmemb, void *userp)
return copysize;
}
size_t S3fsCurl::HeaderCallback(void *data, size_t blockSize, size_t numBlocks, void *userPtr)
size_t S3fsCurl::HeaderCallback(void* data, size_t blockSize, size_t numBlocks, void* userPtr)
{
headers_t* headers = reinterpret_cast<headers_t*>(userPtr);
string header(reinterpret_cast<char*>(data), blockSize * numBlocks);
@ -521,7 +521,7 @@ size_t S3fsCurl::HeaderCallback(void *data, size_t blockSize, size_t numBlocks,
return blockSize * numBlocks;
}
size_t S3fsCurl::UploadReadCallback(void *ptr, size_t size, size_t nmemb, void *userp)
size_t S3fsCurl::UploadReadCallback(void* ptr, size_t size, size_t nmemb, void* userp)
{
S3fsCurl* pCurl = reinterpret_cast<S3fsCurl*>(userp);
@ -554,6 +554,41 @@ size_t S3fsCurl::UploadReadCallback(void *ptr, size_t size, size_t nmemb, void *
return totalread;
}
size_t S3fsCurl::DownloadWriteCallback(void* ptr, size_t size, size_t nmemb, void* userp)
{
S3fsCurl* pCurl = reinterpret_cast<S3fsCurl*>(userp);
if(1 > (size * nmemb)){
return 0;
}
if(-1 == pCurl->partdata.fd || 0 >= pCurl->partdata.size){
return 0;
}
// write size
ssize_t copysize = (size * nmemb) < (size_t)pCurl->partdata.size ? (size * nmemb) : (size_t)pCurl->partdata.size;
ssize_t writebytes;
ssize_t totalwrite;
// write
for(totalwrite = 0, writebytes = 0; totalwrite < copysize; totalwrite += writebytes){
writebytes = pwrite(pCurl->partdata.fd, &((char*)ptr)[totalwrite], (copysize - totalwrite), pCurl->partdata.startpos + totalwrite);
if(0 == writebytes){
// eof?
break;
}else if(-1 == writebytes){
// error
FGPRINT("S3fsCurl::DownloadWriteCallback: write file error(%d).\n", errno);
SYSLOGERR("write file error(%d).", errno);
return 0;
}
}
pCurl->partdata.startpos += totalwrite;
pCurl->partdata.size -= totalwrite;
return totalwrite;
}
bool S3fsCurl::SetDnsCache(bool isCache)
{
bool old = S3fsCurl::is_dns_cache;
@ -637,10 +672,10 @@ long S3fsCurl::SetSslVerifyHostname(long value)
return old;
}
int S3fsCurl::SetMaxParallelUpload(int value)
int S3fsCurl::SetMaxParallelCount(int value)
{
int old = S3fsCurl::max_parallel_upload;
S3fsCurl::max_parallel_upload = value;
int old = S3fsCurl::max_parallel_cnt;
S3fsCurl::max_parallel_cnt = value;
return old;
}
@ -705,7 +740,6 @@ int S3fsCurl::ParallelMultipartUploadRequest(const char* tpath, headers_t& meta,
etaglist_t list;
off_t remaining_bytes;
unsigned char* buf;
char tmpfile[256];
S3fsCurl s3fscurl;
FGPRINT(" S3fsCurl::ParallelMultipartUploadRequest[tpath=%s][fd=%d]\n", SAFESTRPTR(tpath), fd);
@ -752,7 +786,7 @@ int S3fsCurl::ParallelMultipartUploadRequest(const char* tpath, headers_t& meta,
curlmulti.SetRetryCallback(S3fsCurl::UploadMultipartPostRetryCallback);
// Loop for setup parallel upload(multipart) request.
for(para_cnt = 0; para_cnt < S3fsCurl::max_parallel_upload && 0 < remaining_bytes; para_cnt++, remaining_bytes -= chunk){
for(para_cnt = 0; para_cnt < S3fsCurl::max_parallel_cnt && 0 < remaining_bytes; para_cnt++, remaining_bytes -= chunk){
// chunk size
chunk = remaining_bytes > MULTIPART_SIZE ? MULTIPART_SIZE : remaining_bytes;
@ -775,12 +809,12 @@ int S3fsCurl::ParallelMultipartUploadRequest(const char* tpath, headers_t& meta,
// set into parallel object
if(!curlmulti.SetS3fsCurlObject(s3fscurl_para)){
FGPRINT("S3fsCurl::ParallelMultipartUploadRequest: Could not set curl object into multi curl(%s).\n", tmpfile);
SYSLOGERR("Could not make curl object into multi curl(%s).", tmpfile);
FGPRINT("S3fsCurl::ParallelMultipartUploadRequest: Could not set curl object into multi curl(%s).\n", tpath);
SYSLOGERR("Could not make curl object into multi curl(%s).", tpath);
free(buf);
fclose(file);
delete s3fscurl_para;
return result;
return -1;
}
}
@ -803,6 +837,78 @@ int S3fsCurl::ParallelMultipartUploadRequest(const char* tpath, headers_t& meta,
return 0;
}
S3fsCurl* S3fsCurl::ParallelGetObjectRetryCallback(S3fsCurl* s3fscurl)
{
int result;
if(!s3fscurl){
return NULL;
}
// duplicate request(setup new curl object)
S3fsCurl* newcurl = new S3fsCurl();
if(0 != (result = newcurl->PreGetObjectRequest(
s3fscurl->path.c_str(), s3fscurl->partdata.fd, s3fscurl->partdata.startpos, s3fscurl->partdata.size))){
FGPRINT("S3fsCurl::ParallelGetObjectRetryCallback: failed downloading part setup(%d)\n", result);
SYSLOGERR("failed downloading part setup(%d)", result);
delete newcurl;
return NULL;;
}
return newcurl;
}
int S3fsCurl::ParallelGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size)
{
FGPRINT(" S3fsCurl::ParallelGetObjectRequest[tpath=%s][fd=%d]\n", SAFESTRPTR(tpath), fd);
int result = 0;
ssize_t remaining_bytes;
// cycle through open fd, pulling off 10MB chunks at a time
for(remaining_bytes = size; 0 < remaining_bytes; ){
S3fsMultiCurl curlmulti;
int para_cnt;
off_t chunk;
// Initialize S3fsMultiCurl
//curlmulti.SetSuccessCallback(NULL); // not need to set success callback
curlmulti.SetRetryCallback(S3fsCurl::ParallelGetObjectRetryCallback);
// Loop for setup parallel upload(multipart) request.
for(para_cnt = 0; para_cnt < S3fsCurl::max_parallel_cnt && 0 < remaining_bytes; para_cnt++, remaining_bytes -= chunk){
// chunk size
chunk = remaining_bytes > MULTIPART_SIZE ? MULTIPART_SIZE : remaining_bytes;
// s3fscurl sub object
S3fsCurl* s3fscurl_para = new S3fsCurl();
if(0 != (result = s3fscurl_para->PreGetObjectRequest(tpath, fd, (start + size - remaining_bytes), chunk))){
FGPRINT("S3fsCurl::ParallelGetObjectRequest: failed downloading part setup(%d)\n", result);
SYSLOGERR("failed downloading part setup(%d)", result);
delete s3fscurl_para;
return result;
}
// set into parallel object
if(!curlmulti.SetS3fsCurlObject(s3fscurl_para)){
FGPRINT("S3fsCurl::ParallelGetObjectRequest: Could not set curl object into multi curl(%s).\n", tpath);
SYSLOGERR("Could not make curl object into multi curl(%s).", tpath);
delete s3fscurl_para;
return -1;
}
}
// Multi request
if(0 != (result = curlmulti.Request())){
FGPRINT("S3fsCurl::ParallelGetObjectRequest: error occuered in multi request(errno=%d).\n", result);
SYSLOGERR("error occuered in multi request(errno=%d).", result);
break;
}
// reinit for loop.
curlmulti.Clear();
}
return result;
}
//-------------------------------------------------------------------
// Methods for S3fsCurl
//-------------------------------------------------------------------
@ -1558,21 +1664,13 @@ int S3fsCurl::PutRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse
return result;
}
int S3fsCurl::GetObjectRequest(const char* tpath, int fd)
int S3fsCurl::PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size)
{
FILE* file;
int fd2;
FGPRINT(" S3fsCurl::GetRequest [tpath=%s]\n", SAFESTRPTR(tpath));
FGPRINT(" S3fsCurl::PreGetRequest [tpath=%s][start=%zd][size=%zd]\n", SAFESTRPTR(tpath), start, size);
if(!tpath){
if(!tpath || -1 == fd || 0 > start || 0 >= size){
return -1;
}
// duplicate fd
if(-1 == (fd2 = dup(fd)) || 0 != lseek(fd2, 0, SEEK_SET) || NULL == (file = fdopen(fd2, "w+"))){
FGPRINT("S3fsCurl::GetRequest : Cloud not duplicate file discriptor(errno=%d)\n", errno);
SYSLOGERR("Cloud not duplicate file discriptor(errno=%d)", errno);
return -errno;
}
if(!CreateCurlHandle(true)){
return -1;
@ -1589,6 +1687,13 @@ int S3fsCurl::GetObjectRequest(const char* tpath, int fd)
string date = get_date();
requestHeaders = curl_slist_sort_insert(requestHeaders, string("Date: " + date).c_str());
requestHeaders = curl_slist_sort_insert(requestHeaders, "Content-Type: ");
if(-1 != start && -1 != size){
string range = "Range: bytes=";
range += str(start);
range += "-";
range += str(start + size - 1);
requestHeaders = curl_slist_sort_insert(requestHeaders, range.c_str());
}
if(!S3fsCurl::IsPublicBucket()){
requestHeaders = curl_slist_sort_insert(
@ -1600,20 +1705,37 @@ int S3fsCurl::GetObjectRequest(const char* tpath, int fd)
// setopt
curl_easy_setopt(hCurl, CURLOPT_URL, url.c_str());
curl_easy_setopt(hCurl, CURLOPT_HTTPHEADER, requestHeaders);
curl_easy_setopt(hCurl, CURLOPT_FILE, file);
curl_easy_setopt(hCurl, CURLOPT_WRITEFUNCTION, S3fsCurl::DownloadWriteCallback);
curl_easy_setopt(hCurl, CURLOPT_WRITEDATA, (void*)this);
// set info for callback func.
// (use only fd, startpos and size, other member is not used.)
partdata.clear();
partdata.fd = fd;
partdata.startpos = start;
partdata.size = size;
return 0;
}
int S3fsCurl::GetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size)
{
int result;
FGPRINT(" S3fsCurl::GetRequest [tpath=%s][start=%zd][size=%zd]\n", SAFESTRPTR(tpath), start, size);
if(!tpath){
return -1;
}
if(0 != (result = PreGetObjectRequest(tpath, fd, start, size))){
return result;
}
FGPRINT(" downloading... [path=%s][fd=%d]\n", tpath, fd);
SYSLOGDBG("LOCAL FD");
int result = RequestPerform();
fflush(file);
fclose(file);
if(0 != lseek(fd, 0, SEEK_SET)){
FGPRINT("S3fsCurl::GetRequest : Cloud not seek file discriptor(errno=%d)\n", errno);
SYSLOGERR("Cloud not seek file discriptor(errno=%d)", errno);
return -errno;
}
result = RequestPerform();
partdata.clear();
return result;
}
@ -1811,7 +1933,7 @@ int S3fsCurl::CompleteMultipartPostRequest(const char* tpath, string& upload_id,
for(int cnt = 0; cnt < (int)parts.size(); cnt++){
if(0 == parts[cnt].length()){
FGPRINT("S3fsCurl::CompleteMultipartPostRequest : %d file part is not finished uploading.\n", cnt + 1);
return false;
return -1;
}
postContent += "<Part>\n";
postContent += " <PartNumber>" + IntToStr(cnt + 1) + "</PartNumber>\n";

View File

@ -58,7 +58,7 @@ struct filepart
clear();
}
void clear(bool isfree = true)
void clear(void)
{
uploaded = false;
etag = "";
@ -128,7 +128,7 @@ class S3fsCurl
static curlprogress_t curl_progress;
static std::string curl_ca_bundle;
static mimes_t mimeTypes;
static int max_parallel_upload;
static int max_parallel_cnt;
// variables
CURL* hCurl;
@ -143,7 +143,7 @@ class S3fsCurl
long LastResponseCode;
const unsigned char* postdata; // use by post method and read callback function.
int postdata_remaining; // use by post method and read callback function.
filepart partdata; // use by multipart upload
filepart partdata; // use by multipart upload/get object callback
public:
// constructor/destructor
@ -162,9 +162,11 @@ class S3fsCurl
static size_t WriteMemoryCallback(void *ptr, size_t blockSize, size_t numBlocks, void *data);
static size_t ReadCallback(void *ptr, size_t size, size_t nmemb, void *userp);
static size_t UploadReadCallback(void *ptr, size_t size, size_t nmemb, void *userp);
static size_t DownloadWriteCallback(void* ptr, size_t size, size_t nmemb, void* userp);
static bool UploadMultipartPostCallback(S3fsCurl* s3fscurl);
static S3fsCurl* UploadMultipartPostRetryCallback(S3fsCurl* s3fscurl);
static S3fsCurl* ParallelGetObjectRetryCallback(S3fsCurl* s3fscurl);
// methods
bool ClearInternalData(void);
@ -186,6 +188,7 @@ class S3fsCurl
static bool InitShareCurl(void);
static bool DestroyShareCurl(void);
static int ParallelMultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse_flg);
static int ParallelGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size);
// class methods(valiables)
static std::string LookupMimeType(std::string name);
@ -206,7 +209,7 @@ class S3fsCurl
static bool IsSetAccessKeyId(void) { return (0 < S3fsCurl::AWSAccessKeyId.size() && 0 < S3fsCurl::AWSSecretAccessKey.size()); }
static long SetSslVerifyHostname(long value);
static long GetSslVerifyHostname(void) { return S3fsCurl::ssl_verify_hostname; }
static int SetMaxParallelUpload(int value);
static int SetMaxParallelCount(int value);
// methods
bool CreateCurlHandle(bool force = false);
@ -222,7 +225,8 @@ class S3fsCurl
int HeadRequest(const char* tpath, headers_t& meta);
int PutHeadRequest(const char* tpath, headers_t& meta, bool ow_sse_flg);
int PutRequest(const char* tpath, headers_t& meta, int fd, bool ow_sse_flg);
int GetObjectRequest(const char* tpath, int fd);
int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size);
int GetObjectRequest(const char* tpath, int fd, off_t start = -1, ssize_t size = -1);
int CheckBucket(void);
int ListBucketRequest(const char* tpath, const char* query);
int MultipartListRequest(std::string& body);

File diff suppressed because it is too large Load Diff

View File

@ -1,50 +1,154 @@
#ifndef FD_CACHE_H_
#define FD_CACHE_H_
#include "common.h"
//
// Struct for fuse file handle cache
//
struct fd_cache_entry {
int refcnt;
int fd;
int flags;
fd_cache_entry() : refcnt(0), fd(0), flags(0) {}
};
typedef std::list<int> fd_list_t;
typedef std::list<struct fd_cache_entry> fd_cache_entlist_t;
typedef std::map<std::string, fd_cache_entlist_t*> fd_cache_t; // key=path, value=<list>*
typedef std::map<int, int> fd_flags_t; // key=file discriptor, value=flags
//
// Class for fuse file handle cache
//
class FdCache
//------------------------------------------------
// CacheFileStat
//------------------------------------------------
class CacheFileStat
{
private:
static FdCache singleton;
static pthread_mutex_t fd_cache_lock;
fd_cache_t fd_cache;
fd_flags_t fd_flags;
std::string path;
int fd;
private:
static bool MakeCacheFileStatPath(const char* path, std::string& sfile_path, bool is_create_dir = true);
public:
FdCache();
~FdCache();
CacheFileStat(const char* tpath = NULL);
~CacheFileStat();
bool Open(void);
bool Release(void);
bool SetPath(const char* tpath, bool is_open = true);
int GetFd(void) const { return fd; }
};
//------------------------------------------------
// fdpage & PageList
//------------------------------------------------
// page block information
struct fdpage
{
off_t offset;
size_t bytes;
bool init;
fdpage(off_t start = 0, size_t size = 0, bool is_init = false)
: offset(start), bytes(size), init(is_init) {}
off_t next(void) const { return (offset + bytes); }
off_t end(void) const { return (0 < bytes ? offset + bytes - 1 : 0); }
};
typedef std::list<struct fdpage*> fdpage_list_t;
//
// Management of loading area/modifying
//
class PageList
{
private:
fdpage_list_t pages;
private:
void Clear(void);
public:
static void FreeList(fdpage_list_t& list);
PageList(size_t size = 0, bool is_init = false);
~PageList();
size_t Size(void) const;
int Resize(size_t size, bool is_init);
int Init(size_t size, bool is_init);
bool IsInit(off_t start, size_t size);
bool SetInit(off_t start, size_t size, bool is_init = true);
bool FindUninitPage(off_t start, off_t& resstart, size_t& ressize);
int GetUninitPages(fdpage_list_t& uninit_list, off_t start = 0);
bool Serialize(CacheFileStat& file, bool is_output);
void Dump(void);
};
//------------------------------------------------
// class FdEntity
//------------------------------------------------
class FdEntity
{
private:
pthread_mutex_t fdent_lock;
bool is_lock_init;
PageList pagelist;
int refcnt; // reference count
std::string path; // object path
std::string cachepath; // local cache file path
int fd; // file discriptor(tmp file or cache file)
FILE* file; // file pointer(tmp file or cache file)
bool is_modify; // if file is changed, this flag is true
private:
void Clear(void);
int Dup(void);
bool SetAllStatus(bool is_enable);
public:
FdEntity(const char* tpath = NULL, const char* cpath = NULL);
~FdEntity();
void Close(void);
bool IsOpen(void) const { return (-1 != fd); }
int Open(ssize_t size = -1, time_t time = -1);
const char* GetPath(void) const { return path.c_str(); }
int GetFd(void) const { return fd; }
int SetMtime(time_t time);
bool GetSize(size_t& size);
bool GetMtime(time_t& time);
bool GetStats(struct stat& st);
bool SetAllEnable(void) { return SetAllStatus(true); }
bool SetAllDisable(void) { return SetAllStatus(false); }
bool LoadFull(size_t* size = NULL, bool force_load = false);
int Load(off_t start, ssize_t size);
int RowFlush(const char* tpath, headers_t& meta, bool ow_sse_flg, bool force_sync = false);
int Flush(headers_t& meta, bool ow_sse_flg, bool force_sync = false) { return RowFlush(NULL, meta, ow_sse_flg, force_sync); }
ssize_t Read(char* bytes, off_t start, size_t size, bool force_load = false);
ssize_t Write(const char* bytes, off_t start, size_t size);
};
typedef std::map<std::string, class FdEntity*> fdent_map_t; // key=path, value=FdEntity*
//------------------------------------------------
// class FdManager
//------------------------------------------------
class FdManager
{
private:
static FdManager singleton;
static pthread_mutex_t fd_manager_lock;
static bool is_lock_init;
static std::string cache_dir;
static size_t page_size;
fdent_map_t fent;
public:
FdManager();
~FdManager();
// Reference singleton
static FdCache* getFdCacheData(void) {
return &singleton;
}
static FdManager* get(void) { return &singleton; }
bool Add(const char* path, int fd, int flags);
bool Del(const char* path, int fd);
bool Del(const char* path);
bool Del(int fd);
bool Get(const char* path, int* pfd = NULL, int* pflags = NULL) const;
bool Get(int fd, int* pflags = NULL) const;
static bool DeleteCacheDirectory(void);
static int DeleteCacheFile(const char* path);
static bool SetCacheDir(const char* dir);
static bool IsCacheDir(void) { return (0 < FdManager::cache_dir.size()); }
static const char* GetCacheDir(void) { return FdManager::cache_dir.c_str(); }
static size_t SetPageSize(size_t size);
static size_t GetPageSize(void) { return FdManager::page_size; }
static bool MakeCachePath(const char* path, std::string& cache_path, bool is_create_dir = true);
FdEntity* GetFdEntity(const char* path);
FdEntity* Open(const char* path, ssize_t size = -1, time_t time = -1, bool force_tmpfile = false, bool is_create = true);
FdEntity* ExistOpen(const char* path) { return Open(path, -1, -1, false, false); }
bool Close(FdEntity* ent);
};
#endif // FD_CACHE_H_

File diff suppressed because it is too large Load Diff

View File

@ -28,6 +28,9 @@
#include <pwd.h>
#include <grp.h>
#include <syslog.h>
#include <pthread.h>
#include <sys/types.h>
#include <dirent.h>
#include <string>
#include <sstream>
@ -397,6 +400,56 @@ void free_mvnodes(MVNODE *head)
return;
}
//-------------------------------------------------------------------
// Class AutoLock
//-------------------------------------------------------------------
AutoLock::AutoLock(pthread_mutex_t* pmutex) : auto_mutex(pmutex), is_locked(false)
{
Lock();
}
AutoLock::~AutoLock()
{
Unlock();
}
bool AutoLock::Lock(void)
{
if(!auto_mutex){
return false;
}
if(is_locked){
// already locked
return true;
}
try{
pthread_mutex_lock(auto_mutex);
is_locked = true;
}catch(exception& e){
is_locked = false;
return false;
}
return true;
}
bool AutoLock::Unlock(void)
{
if(!auto_mutex){
return false;
}
if(!is_locked){
// already unlocked
return true;
}
try{
pthread_mutex_unlock(auto_mutex);
is_locked = false;
}catch(exception& e){
return false;
}
return true;
}
//-------------------------------------------------------------------
// Utility for UID/GID
//-------------------------------------------------------------------
@ -494,6 +547,53 @@ int mkdirp(const string& path, mode_t mode)
return 0;
}
bool delete_files_in_dir(const char* dir, bool is_remove_own)
{
DIR* dp;
struct dirent* dent;
if(NULL == (dp = opendir(dir))){
//FGPRINT("delete_files_in_dir: could not open dir(%s) - errno(%d)\n", dir, errno);
return false;
}
for(dent = readdir(dp); dent; dent = readdir(dp)){
if(0 == strcmp(dent->d_name, "..") || 0 == strcmp(dent->d_name, ".")){
continue;
}
string fullpath = dir;
fullpath += "/";
fullpath += dent->d_name;
struct stat st;
if(0 != lstat(fullpath.c_str(), &st)){
FGPRINT("delete_files_in_dir: could not get stats of file(%s) - errno(%d)\n", fullpath.c_str(), errno);
closedir(dp);
return false;
}
if(S_ISDIR(st.st_mode)){
// dir -> Reentrant
if(!delete_files_in_dir(fullpath.c_str(), true)){
//FGPRINT("delete_files_in_dir: could not remove sub dir(%s) - errno(%d)\n", fullpath.c_str(), errno);
closedir(dp);
return false;
}
}else{
if(0 != unlink(fullpath.c_str())){
FGPRINT("delete_files_in_dir: could not remove file(%s) - errno(%d)\n", fullpath.c_str(), errno);
closedir(dp);
return false;
}
}
}
closedir(dp);
if(is_remove_own && 0 != rmdir(dir)){
FGPRINT("delete_files_in_dir: could not remove dir(%s) - errno(%d)\n", dir, errno);
return false;
}
return true;
}
//-------------------------------------------------------------------
// Utility functions for convert
//-------------------------------------------------------------------
@ -709,6 +809,9 @@ void show_help (void)
" use_cache (default=\"\" which means disabled)\n"
" - local folder to use for local file cache\n"
"\n"
" del_cache (delete local file cache)\n"
" - delete local file cache when s3fs starts and exits.\n"
"\n"
" use_rrs (default is disable)\n"
" - this option makes Amazon's Reduced Redundancy Storage enable.\n"
"\n"
@ -749,7 +852,7 @@ void show_help (void)
" multireq_max (default=\"500\")\n"
" - maximum number of parallel request for listing objects.\n"
"\n"
" parallel_upload (default=\"5\")\n"
" parallel_count (default=\"5\")\n"
" - number of parallel request for uploading big objects.\n"
" s3fs uploads large object(over 20MB) by multipart post request, \n"
" and sends parallel requests.\n"
@ -757,6 +860,14 @@ void show_help (void)
" at once. It is necessary to set this value depending on a CPU \n"
" and a network band.\n"
"\n"
" fd_page_size (default=\"52428800\"(50MB))\n"
" - number of internal management page size for each file discriptor.\n"
" For delayed reading and writing by s3fs, s3fs manages pages which \n"
" is separated from object. Each pages has a status that data is \n"
" already loaded(or not loaded yet).\n"
" This option should not be changed when you don't have a trouble \n"
" with performance.\n"
"\n"
" url (default=\"http://s3.amazonaws.com\")\n"
" - sets the url to use to access amazon s3\n"
"\n"

View File

@ -64,6 +64,20 @@ typedef struct mvnode {
struct mvnode *next;
} MVNODE;
class AutoLock
{
private:
pthread_mutex_t* auto_mutex;
bool is_locked;
public:
AutoLock(pthread_mutex_t* pmutex = NULL);
~AutoLock();
bool Lock(void);
bool Unlock(void);
};
//-------------------------------------------------------------------
// Functions
//-------------------------------------------------------------------
@ -79,6 +93,7 @@ int is_uid_inculde_group(uid_t uid, gid_t gid);
std::string mydirname(std::string path);
std::string mybasename(std::string path);
int mkdirp(const std::string& path, mode_t mode);
bool delete_files_in_dir(const char* dir, bool is_remove_own);
time_t get_mtime(const char *s);
time_t get_mtime(headers_t& meta, bool overcheck = true);