s3fs-fuse/src/curl.h

489 lines
20 KiB
C
Raw Normal View History

2014-09-07 15:08:27 +00:00
/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#ifndef S3FS_CURL_H_
#define S3FS_CURL_H_
//----------------------------------------------
// Symbols
//----------------------------------------------
#define MIN_MULTIPART_SIZE 5242880 // 5MB
//----------------------------------------------
// class BodyData
//----------------------------------------------
// memory class for curl write memory callback
//
class BodyData
{
private:
char* text;
size_t lastpos;
size_t bufsize;
private:
bool IsSafeSize(size_t addbytes) const {
return ((lastpos + addbytes + 1) > bufsize ? false : true);
}
bool Resize(size_t addbytes);
public:
BodyData() : text(NULL), lastpos(0), bufsize(0) {}
~BodyData() {
Clear();
}
void Clear(void);
bool Append(void* ptr, size_t bytes);
bool Append(void* ptr, size_t blockSize, size_t numBlocks) {
return Append(ptr, (blockSize * numBlocks));
}
const char* str() const;
size_t size() const {
return lastpos;
}
};
//----------------------------------------------
// Utility structs & typedefs
//----------------------------------------------
typedef std::vector<std::string> etaglist_t;
// Each part information for Multipart upload
struct filepart
{
bool uploaded; // does finish uploading
std::string etag; // expected etag value
2015-07-10 18:50:40 +00:00
int fd; // base file(temporary full file) descriptor
off_t startpos; // seek fd point for uploading
ssize_t size; // uploading size
etaglist_t* etaglist; // use only parallel upload
int etagpos; // use only parallel upload
filepart() : uploaded(false), fd(-1), startpos(0), size(-1), etaglist(NULL), etagpos(-1) {}
~filepart()
{
clear();
}
Changes codes for performance(part 3) * Summay This revision includes big change about temporary file and local cache file. By this big change, s3fs works with good performance when s3fs opens/ closes/syncs/reads object. I made a big change about the handling about temporary file and local cache file to do this implementation. * Detail 1) About temporary file(local file) s3fs uses a temporary file on local file system when s3fs does download/ upload/open/seek object on S3. After this revision, s3fs calls ftruncate() function when s3fs makes the temporary file. In this way s3fs can set a file size of precisely length without downloading. (Notice - ftruncate function is for XSI-compliant systems, so that possibly you have a problem on non-XSI-compliant systems.) By this change, s3fs can download a part of a object by requesting with "Range" http header. It seems like downloading by each block unit. The default block(part) size is 50MB, it is caused the result which is default parallel requests count(5) by default multipart upload size(10MB). If you need to change this block size, you can change by new option "fd_page_size". This option can take from 1MB(1024 * 1024) to any bytes. So that, you have to take care about that fdcache.cpp(and fdcache.h) were changed a lot. 2) About local cache Local cache files which are in directory specified by "use_cache" option do not have always all of object data. This cause is that s3fs uses ftruncate function and reads(writes) each block unit of a temporary file. s3fs manages each block unit's status which are "downloaded area" or "not". For this status, s3fs makes new temporary file in cache directory which is specified by "use_cache" option. This status files is in a directory which is named "<use_cache sirectory>/.<bucket_name>/". When s3fs opens this status file, s3fs locks this file for exclusive control by calling flock function. You need to take care about this, the status files can not be laid on network drive(like NFS). This revision changes about file open mode, s3fs always opens a local cache file and each status file with writable mode. Last, this revision adds new option "del_cache", this option means that s3fs deletes all local cache file when s3fs starts and exits. 3) Uploading When s3fs writes data to file descriptor through FUSE request, old s3fs revision downloads all of the object. But new revision does not download all, it downloads only small percial area(some block units) including writing data area. And when s3fs closes or flushes the file descriptor, s3fs downloads other area which is not downloaded from server. After that, s3fs uploads all of data. Already r456 revision has parallel upload function, then this revision with r456 and r457 are very big change for performance. 4) Downloading By changing a temporary file and a local cache file, when s3fs downloads a object, it downloads only the required range(some block units). And s3fs downloads units by parallel GET request, it is same as a case of uploading. (Maximum parallel request count and each download size are specified same parameters for uploading.) In the new revision, when s3fs opens file, s3fs returns file descriptor soon. Because s3fs only opens(makes) the file descriptor with no downloading data. And when s3fs reads a data, s3fs downloads only some block unit including specified area. This result is good for performance. 5) Changes option name The option "parallel_upload" which added at r456 is changed to new option name as "parallel_count". This reason is this option value is not only used by uploading object, but a uploading object also uses this option. (For a while, you can use old option name "parallel_upload" for compatibility.) git-svn-id: http://s3fs.googlecode.com/svn/trunk@458 df820570-a93a-0410-bd06-b72b767a4274
2013-07-23 16:01:48 +00:00
void clear(void)
{
uploaded = false;
etag = "";
fd = -1;
startpos = 0;
size = -1;
etaglist = NULL;
etagpos = - 1;
}
void add_etag_list(etaglist_t* list)
{
if(list){
list->push_back(std::string(""));
etaglist = list;
etagpos = list->size() - 1;
}else{
etaglist = NULL;
etagpos = - 1;
}
}
};
// for progress
struct case_insensitive_compare_func
{
bool operator()(const std::string& a, const std::string& b) const {
return strcasecmp(a.c_str(), b.c_str()) < 0;
}
};
typedef std::map<std::string, std::string, case_insensitive_compare_func> mimes_t;
typedef std::pair<double, double> progress_t;
typedef std::map<CURL*, time_t> curltime_t;
typedef std::map<CURL*, progress_t> curlprogress_t;
class S3fsMultiCurl;
//----------------------------------------------
// class S3fsCurl
//----------------------------------------------
typedef std::map<std::string, std::string> iamcredmap_t;
2014-07-19 19:02:55 +00:00
typedef std::map<std::string, std::string> sseckeymap_t;
typedef std::list<sseckeymap_t> sseckeylist_t;
2015-10-06 14:46:14 +00:00
// strage class(rrs)
enum storage_class_t {
STANDARD,
STANDARD_IA,
REDUCED_REDUNDANCY
};
2015-10-06 14:46:14 +00:00
// sse type
enum sse_type_t {
SSE_DISABLE = 0, // not use server side encrypting
SSE_S3, // server side encrypting by S3 key
SSE_C, // server side encrypting by custom key
SSE_KMS // server side encrypting by kms id
};
// share
#define SHARE_MUTEX_DNS 0
#define SHARE_MUTEX_SSL_SESSION 1
#define SHARE_MUTEX_MAX 2
// Class for lapping curl
//
class S3fsCurl
{
friend class S3fsMultiCurl;
private:
enum REQTYPE {
REQTYPE_UNSET = -1,
REQTYPE_DELETE = 0,
REQTYPE_HEAD,
REQTYPE_PUTHEAD,
REQTYPE_PUT,
REQTYPE_GET,
REQTYPE_CHKBUCKET,
REQTYPE_LISTBUCKET,
REQTYPE_PREMULTIPOST,
REQTYPE_COMPLETEMULTIPOST,
REQTYPE_UPLOADMULTIPOST,
REQTYPE_COPYMULTIPOST,
REQTYPE_MULTILIST,
REQTYPE_IAMCRED,
REQTYPE_ABORTMULTIUPLOAD
};
// class variables
static pthread_mutex_t curl_handles_lock;
static pthread_mutex_t curl_share_lock[SHARE_MUTEX_MAX];
static bool is_initglobal_done;
static CURLSH* hCurlShare;
static bool is_cert_check;
static bool is_dns_cache;
static bool is_ssl_session_cache;
static long connect_timeout;
static time_t readwrite_timeout;
static int retries;
static bool is_public_bucket;
static std::string default_acl; // TODO: to enum
static storage_class_t storage_class;
2014-07-19 19:02:55 +00:00
static sseckeylist_t sseckeys;
2015-10-06 14:46:14 +00:00
static std::string ssekmsid;
static sse_type_t ssetype;
static bool is_content_md5;
static bool is_verbose;
static std::string AWSAccessKeyId;
static std::string AWSSecretAccessKey;
static std::string AWSAccessToken;
static time_t AWSAccessTokenExpire;
static std::string IAM_role;
static long ssl_verify_hostname;
static curltime_t curl_times;
static curlprogress_t curl_progress;
static std::string curl_ca_bundle;
static mimes_t mimeTypes;
static int max_parallel_cnt;
2014-03-30 07:53:41 +00:00
static off_t multipart_size;
static bool is_sigv4;
// variables
CURL* hCurl;
REQTYPE type; // type of request
std::string path; // target object path
std::string base_path; // base path (for multi curl head request)
std::string saved_path; // saved path = cache key (for multi curl head request)
std::string url; // target object path(url)
struct curl_slist* requestHeaders;
headers_t responseHeaders; // header data by HeaderCallback
BodyData* bodydata; // body data by WriteMemoryCallback
BodyData* headdata; // header data by WriteMemoryCallback
long LastResponseCode;
const unsigned char* postdata; // use by post method and read callback function.
int postdata_remaining; // use by post method and read callback function.
filepart partdata; // use by multipart upload/get object callback
bool is_use_ahbe; // additional header by extension
int retry_count; // retry count for multipart
FILE* b_infile; // backup for retrying
const unsigned char* b_postdata; // backup for retrying
int b_postdata_remaining; // backup for retrying
off_t b_partdata_startpos; // backup for retrying
ssize_t b_partdata_size; // backup for retrying
2015-10-06 14:46:14 +00:00
int b_ssekey_pos; // backup for retrying
std::string b_ssevalue; // backup for retrying
sse_type_t b_ssetype; // backup for retrying
public:
// constructor/destructor
explicit S3fsCurl(bool ahbe = false);
~S3fsCurl();
private:
// class methods
static bool InitGlobalCurl(void);
static bool DestroyGlobalCurl(void);
static bool InitShareCurl(void);
static bool DestroyShareCurl(void);
static void LockCurlShare(CURL* handle, curl_lock_data nLockData, curl_lock_access laccess, void* useptr);
static void UnlockCurlShare(CURL* handle, curl_lock_data nLockData, void* useptr);
static bool InitCryptMutex(void);
static bool DestroyCryptMutex(void);
static int CurlProgress(void *clientp, double dltotal, double dlnow, double ultotal, double ulnow);
static bool InitMimeType(const char* MimeFile = NULL);
static bool LocateBundle(void);
static size_t HeaderCallback(void *data, size_t blockSize, size_t numBlocks, void *userPtr);
static size_t WriteMemoryCallback(void *ptr, size_t blockSize, size_t numBlocks, void *data);
static size_t ReadCallback(void *ptr, size_t size, size_t nmemb, void *userp);
static size_t UploadReadCallback(void *ptr, size_t size, size_t nmemb, void *userp);
Changes codes for performance(part 3) * Summay This revision includes big change about temporary file and local cache file. By this big change, s3fs works with good performance when s3fs opens/ closes/syncs/reads object. I made a big change about the handling about temporary file and local cache file to do this implementation. * Detail 1) About temporary file(local file) s3fs uses a temporary file on local file system when s3fs does download/ upload/open/seek object on S3. After this revision, s3fs calls ftruncate() function when s3fs makes the temporary file. In this way s3fs can set a file size of precisely length without downloading. (Notice - ftruncate function is for XSI-compliant systems, so that possibly you have a problem on non-XSI-compliant systems.) By this change, s3fs can download a part of a object by requesting with "Range" http header. It seems like downloading by each block unit. The default block(part) size is 50MB, it is caused the result which is default parallel requests count(5) by default multipart upload size(10MB). If you need to change this block size, you can change by new option "fd_page_size". This option can take from 1MB(1024 * 1024) to any bytes. So that, you have to take care about that fdcache.cpp(and fdcache.h) were changed a lot. 2) About local cache Local cache files which are in directory specified by "use_cache" option do not have always all of object data. This cause is that s3fs uses ftruncate function and reads(writes) each block unit of a temporary file. s3fs manages each block unit's status which are "downloaded area" or "not". For this status, s3fs makes new temporary file in cache directory which is specified by "use_cache" option. This status files is in a directory which is named "<use_cache sirectory>/.<bucket_name>/". When s3fs opens this status file, s3fs locks this file for exclusive control by calling flock function. You need to take care about this, the status files can not be laid on network drive(like NFS). This revision changes about file open mode, s3fs always opens a local cache file and each status file with writable mode. Last, this revision adds new option "del_cache", this option means that s3fs deletes all local cache file when s3fs starts and exits. 3) Uploading When s3fs writes data to file descriptor through FUSE request, old s3fs revision downloads all of the object. But new revision does not download all, it downloads only small percial area(some block units) including writing data area. And when s3fs closes or flushes the file descriptor, s3fs downloads other area which is not downloaded from server. After that, s3fs uploads all of data. Already r456 revision has parallel upload function, then this revision with r456 and r457 are very big change for performance. 4) Downloading By changing a temporary file and a local cache file, when s3fs downloads a object, it downloads only the required range(some block units). And s3fs downloads units by parallel GET request, it is same as a case of uploading. (Maximum parallel request count and each download size are specified same parameters for uploading.) In the new revision, when s3fs opens file, s3fs returns file descriptor soon. Because s3fs only opens(makes) the file descriptor with no downloading data. And when s3fs reads a data, s3fs downloads only some block unit including specified area. This result is good for performance. 5) Changes option name The option "parallel_upload" which added at r456 is changed to new option name as "parallel_count". This reason is this option value is not only used by uploading object, but a uploading object also uses this option. (For a while, you can use old option name "parallel_upload" for compatibility.) git-svn-id: http://s3fs.googlecode.com/svn/trunk@458 df820570-a93a-0410-bd06-b72b767a4274
2013-07-23 16:01:48 +00:00
static size_t DownloadWriteCallback(void* ptr, size_t size, size_t nmemb, void* userp);
static bool UploadMultipartPostCallback(S3fsCurl* s3fscurl);
static S3fsCurl* UploadMultipartPostRetryCallback(S3fsCurl* s3fscurl);
Changes codes for performance(part 3) * Summay This revision includes big change about temporary file and local cache file. By this big change, s3fs works with good performance when s3fs opens/ closes/syncs/reads object. I made a big change about the handling about temporary file and local cache file to do this implementation. * Detail 1) About temporary file(local file) s3fs uses a temporary file on local file system when s3fs does download/ upload/open/seek object on S3. After this revision, s3fs calls ftruncate() function when s3fs makes the temporary file. In this way s3fs can set a file size of precisely length without downloading. (Notice - ftruncate function is for XSI-compliant systems, so that possibly you have a problem on non-XSI-compliant systems.) By this change, s3fs can download a part of a object by requesting with "Range" http header. It seems like downloading by each block unit. The default block(part) size is 50MB, it is caused the result which is default parallel requests count(5) by default multipart upload size(10MB). If you need to change this block size, you can change by new option "fd_page_size". This option can take from 1MB(1024 * 1024) to any bytes. So that, you have to take care about that fdcache.cpp(and fdcache.h) were changed a lot. 2) About local cache Local cache files which are in directory specified by "use_cache" option do not have always all of object data. This cause is that s3fs uses ftruncate function and reads(writes) each block unit of a temporary file. s3fs manages each block unit's status which are "downloaded area" or "not". For this status, s3fs makes new temporary file in cache directory which is specified by "use_cache" option. This status files is in a directory which is named "<use_cache sirectory>/.<bucket_name>/". When s3fs opens this status file, s3fs locks this file for exclusive control by calling flock function. You need to take care about this, the status files can not be laid on network drive(like NFS). This revision changes about file open mode, s3fs always opens a local cache file and each status file with writable mode. Last, this revision adds new option "del_cache", this option means that s3fs deletes all local cache file when s3fs starts and exits. 3) Uploading When s3fs writes data to file descriptor through FUSE request, old s3fs revision downloads all of the object. But new revision does not download all, it downloads only small percial area(some block units) including writing data area. And when s3fs closes or flushes the file descriptor, s3fs downloads other area which is not downloaded from server. After that, s3fs uploads all of data. Already r456 revision has parallel upload function, then this revision with r456 and r457 are very big change for performance. 4) Downloading By changing a temporary file and a local cache file, when s3fs downloads a object, it downloads only the required range(some block units). And s3fs downloads units by parallel GET request, it is same as a case of uploading. (Maximum parallel request count and each download size are specified same parameters for uploading.) In the new revision, when s3fs opens file, s3fs returns file descriptor soon. Because s3fs only opens(makes) the file descriptor with no downloading data. And when s3fs reads a data, s3fs downloads only some block unit including specified area. This result is good for performance. 5) Changes option name The option "parallel_upload" which added at r456 is changed to new option name as "parallel_count". This reason is this option value is not only used by uploading object, but a uploading object also uses this option. (For a while, you can use old option name "parallel_upload" for compatibility.) git-svn-id: http://s3fs.googlecode.com/svn/trunk@458 df820570-a93a-0410-bd06-b72b767a4274
2013-07-23 16:01:48 +00:00
static S3fsCurl* ParallelGetObjectRetryCallback(S3fsCurl* s3fscurl);
static bool ParseIAMCredentialResponse(const char* response, iamcredmap_t& keyval);
static bool SetIAMCredentials(const char* response);
2015-10-06 14:46:14 +00:00
static bool LoadEnvSseCKeys(void);
static bool LoadEnvSseKmsid(void);
2014-08-26 17:11:10 +00:00
static bool PushbackSseKeys(std::string& onekey);
static int CurlDebugFunc(CURL* hcurl, curl_infotype type, char* data, size_t size, void* userptr);
// methods
bool ResetHandle(void);
bool RemakeHandle(void);
bool ClearInternalData(void);
void insertV4Headers(const std::string &op, const std::string &path, const std::string &query_string, const std::string &payload_hash);
2015-08-17 04:42:45 +00:00
std::string CalcSignatureV2(const std::string& method, const std::string& strMD5, const std::string& content_type, const std::string& date, const std::string& resource);
std::string CalcSignature(const std::string& method, const std::string& canonical_uri, const std::string& query_string, const std::string& strdate, const std::string& payload_hash, const std::string& date8601);
bool GetUploadId(std::string& upload_id);
int GetIAMCredentials(void);
2015-08-17 04:42:45 +00:00
int UploadMultipartPostSetup(const char* tpath, int part_num, const std::string& upload_id);
int CopyMultipartPostRequest(const char* from, const char* to, int part_num, std::string& upload_id, headers_t& meta);
public:
// class methods
static bool InitS3fsCurl(const char* MimeFile = NULL);
static bool DestroyS3fsCurl(void);
2014-08-26 17:11:10 +00:00
static int ParallelMultipartUploadRequest(const char* tpath, headers_t& meta, int fd);
Changes codes for performance(part 3) * Summay This revision includes big change about temporary file and local cache file. By this big change, s3fs works with good performance when s3fs opens/ closes/syncs/reads object. I made a big change about the handling about temporary file and local cache file to do this implementation. * Detail 1) About temporary file(local file) s3fs uses a temporary file on local file system when s3fs does download/ upload/open/seek object on S3. After this revision, s3fs calls ftruncate() function when s3fs makes the temporary file. In this way s3fs can set a file size of precisely length without downloading. (Notice - ftruncate function is for XSI-compliant systems, so that possibly you have a problem on non-XSI-compliant systems.) By this change, s3fs can download a part of a object by requesting with "Range" http header. It seems like downloading by each block unit. The default block(part) size is 50MB, it is caused the result which is default parallel requests count(5) by default multipart upload size(10MB). If you need to change this block size, you can change by new option "fd_page_size". This option can take from 1MB(1024 * 1024) to any bytes. So that, you have to take care about that fdcache.cpp(and fdcache.h) were changed a lot. 2) About local cache Local cache files which are in directory specified by "use_cache" option do not have always all of object data. This cause is that s3fs uses ftruncate function and reads(writes) each block unit of a temporary file. s3fs manages each block unit's status which are "downloaded area" or "not". For this status, s3fs makes new temporary file in cache directory which is specified by "use_cache" option. This status files is in a directory which is named "<use_cache sirectory>/.<bucket_name>/". When s3fs opens this status file, s3fs locks this file for exclusive control by calling flock function. You need to take care about this, the status files can not be laid on network drive(like NFS). This revision changes about file open mode, s3fs always opens a local cache file and each status file with writable mode. Last, this revision adds new option "del_cache", this option means that s3fs deletes all local cache file when s3fs starts and exits. 3) Uploading When s3fs writes data to file descriptor through FUSE request, old s3fs revision downloads all of the object. But new revision does not download all, it downloads only small percial area(some block units) including writing data area. And when s3fs closes or flushes the file descriptor, s3fs downloads other area which is not downloaded from server. After that, s3fs uploads all of data. Already r456 revision has parallel upload function, then this revision with r456 and r457 are very big change for performance. 4) Downloading By changing a temporary file and a local cache file, when s3fs downloads a object, it downloads only the required range(some block units). And s3fs downloads units by parallel GET request, it is same as a case of uploading. (Maximum parallel request count and each download size are specified same parameters for uploading.) In the new revision, when s3fs opens file, s3fs returns file descriptor soon. Because s3fs only opens(makes) the file descriptor with no downloading data. And when s3fs reads a data, s3fs downloads only some block unit including specified area. This result is good for performance. 5) Changes option name The option "parallel_upload" which added at r456 is changed to new option name as "parallel_count". This reason is this option value is not only used by uploading object, but a uploading object also uses this option. (For a while, you can use old option name "parallel_upload" for compatibility.) git-svn-id: http://s3fs.googlecode.com/svn/trunk@458 df820570-a93a-0410-bd06-b72b767a4274
2013-07-23 16:01:48 +00:00
static int ParallelGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size);
static bool CheckIAMCredentialUpdate(void);
// class methods(valiables)
2015-08-17 04:42:45 +00:00
static std::string LookupMimeType(const std::string& name);
static bool SetCheckCertificate(bool isCertCheck);
static bool SetDnsCache(bool isCache);
static bool SetSslSessionCache(bool isCache);
static long SetConnectTimeout(long timeout);
static time_t SetReadwriteTimeout(time_t timeout);
static time_t GetReadwriteTimeout(void) { return S3fsCurl::readwrite_timeout; }
static int SetRetries(int count);
static bool SetPublicBucket(bool flag);
static bool IsPublicBucket(void) { return S3fsCurl::is_public_bucket; }
static std::string SetDefaultAcl(const char* acl);
static storage_class_t SetStorageClass(storage_class_t storage_class);
static storage_class_t GetStorageClass() { return S3fsCurl::storage_class; }
2015-10-06 14:46:14 +00:00
static bool LoadEnvSse(void) { return (S3fsCurl::LoadEnvSseCKeys() && S3fsCurl::LoadEnvSseKmsid()); }
static sse_type_t SetSseType(sse_type_t type);
static sse_type_t GetSseType(void) { return S3fsCurl::ssetype; }
static bool IsSseDisable(void) { return (SSE_DISABLE == S3fsCurl::ssetype); }
static bool IsSseS3Type(void) { return (SSE_S3 == S3fsCurl::ssetype); }
static bool IsSseCType(void) { return (SSE_C == S3fsCurl::ssetype); }
static bool IsSseKmsType(void) { return (SSE_KMS == S3fsCurl::ssetype); }
static bool FinalCheckSse(void);
static bool SetSseCKeys(const char* filepath);
static bool SetSseKmsid(const char* kmsid);
static bool IsSetSseKmsId(void) { return !S3fsCurl::ssekmsid.empty(); }
static const char* GetSseKmsId(void) { return S3fsCurl::ssekmsid.c_str(); }
2014-07-19 19:02:55 +00:00
static bool GetSseKey(std::string& md5, std::string& ssekey);
static bool GetSseKeyMd5(int pos, std::string& md5);
static int GetSseKeyCount(void);
static bool SetContentMd5(bool flag);
static bool SetVerbose(bool flag);
static bool GetVerbose(void) { return S3fsCurl::is_verbose; }
static bool SetAccessKey(const char* AccessKeyId, const char* SecretAccessKey);
static bool IsSetAccessKeyId(void){
return (0 < S3fsCurl::IAM_role.size() || (0 < S3fsCurl::AWSAccessKeyId.size() && 0 < S3fsCurl::AWSSecretAccessKey.size()));
}
static long SetSslVerifyHostname(long value);
static long GetSslVerifyHostname(void) { return S3fsCurl::ssl_verify_hostname; }
Changes codes for performance(part 3) * Summay This revision includes big change about temporary file and local cache file. By this big change, s3fs works with good performance when s3fs opens/ closes/syncs/reads object. I made a big change about the handling about temporary file and local cache file to do this implementation. * Detail 1) About temporary file(local file) s3fs uses a temporary file on local file system when s3fs does download/ upload/open/seek object on S3. After this revision, s3fs calls ftruncate() function when s3fs makes the temporary file. In this way s3fs can set a file size of precisely length without downloading. (Notice - ftruncate function is for XSI-compliant systems, so that possibly you have a problem on non-XSI-compliant systems.) By this change, s3fs can download a part of a object by requesting with "Range" http header. It seems like downloading by each block unit. The default block(part) size is 50MB, it is caused the result which is default parallel requests count(5) by default multipart upload size(10MB). If you need to change this block size, you can change by new option "fd_page_size". This option can take from 1MB(1024 * 1024) to any bytes. So that, you have to take care about that fdcache.cpp(and fdcache.h) were changed a lot. 2) About local cache Local cache files which are in directory specified by "use_cache" option do not have always all of object data. This cause is that s3fs uses ftruncate function and reads(writes) each block unit of a temporary file. s3fs manages each block unit's status which are "downloaded area" or "not". For this status, s3fs makes new temporary file in cache directory which is specified by "use_cache" option. This status files is in a directory which is named "<use_cache sirectory>/.<bucket_name>/". When s3fs opens this status file, s3fs locks this file for exclusive control by calling flock function. You need to take care about this, the status files can not be laid on network drive(like NFS). This revision changes about file open mode, s3fs always opens a local cache file and each status file with writable mode. Last, this revision adds new option "del_cache", this option means that s3fs deletes all local cache file when s3fs starts and exits. 3) Uploading When s3fs writes data to file descriptor through FUSE request, old s3fs revision downloads all of the object. But new revision does not download all, it downloads only small percial area(some block units) including writing data area. And when s3fs closes or flushes the file descriptor, s3fs downloads other area which is not downloaded from server. After that, s3fs uploads all of data. Already r456 revision has parallel upload function, then this revision with r456 and r457 are very big change for performance. 4) Downloading By changing a temporary file and a local cache file, when s3fs downloads a object, it downloads only the required range(some block units). And s3fs downloads units by parallel GET request, it is same as a case of uploading. (Maximum parallel request count and each download size are specified same parameters for uploading.) In the new revision, when s3fs opens file, s3fs returns file descriptor soon. Because s3fs only opens(makes) the file descriptor with no downloading data. And when s3fs reads a data, s3fs downloads only some block unit including specified area. This result is good for performance. 5) Changes option name The option "parallel_upload" which added at r456 is changed to new option name as "parallel_count". This reason is this option value is not only used by uploading object, but a uploading object also uses this option. (For a while, you can use old option name "parallel_upload" for compatibility.) git-svn-id: http://s3fs.googlecode.com/svn/trunk@458 df820570-a93a-0410-bd06-b72b767a4274
2013-07-23 16:01:48 +00:00
static int SetMaxParallelCount(int value);
2014-03-30 07:53:41 +00:00
static int GetMaxParallelCount(void) { return S3fsCurl::max_parallel_cnt; }
static std::string SetIAMRole(const char* role);
static const char* GetIAMRole(void) { return S3fsCurl::IAM_role.c_str(); }
2014-03-30 07:53:41 +00:00
static bool SetMultipartSize(off_t size);
static off_t GetMultipartSize(void) { return S3fsCurl::multipart_size; }
static bool SetSignatureV4(bool isset) { bool bresult = S3fsCurl::is_sigv4; S3fsCurl::is_sigv4 = isset; return bresult; }
static bool IsSignatureV4(void) { return S3fsCurl::is_sigv4; }
// methods
bool CreateCurlHandle(bool force = false);
bool DestroyCurlHandle(void);
2015-10-06 14:46:14 +00:00
bool AddSseRequestHead(sse_type_t ssetype, std::string& ssevalue, bool is_only_c, bool is_copy);
bool GetResponseCode(long& responseCode);
int RequestPerform(void);
int DeleteRequest(const char* tpath);
2014-07-19 19:02:55 +00:00
bool PreHeadRequest(const char* tpath, const char* bpath = NULL, const char* savedpath = NULL, int ssekey_pos = -1);
bool PreHeadRequest(std::string& tpath, std::string& bpath, std::string& savedpath, int ssekey_pos = -1) {
return PreHeadRequest(tpath.c_str(), bpath.c_str(), savedpath.c_str(), ssekey_pos);
}
int HeadRequest(const char* tpath, headers_t& meta);
2014-08-26 17:11:10 +00:00
int PutHeadRequest(const char* tpath, headers_t& meta, bool is_copy);
int PutRequest(const char* tpath, headers_t& meta, int fd);
2015-10-06 14:46:14 +00:00
int PreGetObjectRequest(const char* tpath, int fd, off_t start, ssize_t size, sse_type_t ssetype, std::string& ssevalue);
Changes codes for performance(part 3) * Summay This revision includes big change about temporary file and local cache file. By this big change, s3fs works with good performance when s3fs opens/ closes/syncs/reads object. I made a big change about the handling about temporary file and local cache file to do this implementation. * Detail 1) About temporary file(local file) s3fs uses a temporary file on local file system when s3fs does download/ upload/open/seek object on S3. After this revision, s3fs calls ftruncate() function when s3fs makes the temporary file. In this way s3fs can set a file size of precisely length without downloading. (Notice - ftruncate function is for XSI-compliant systems, so that possibly you have a problem on non-XSI-compliant systems.) By this change, s3fs can download a part of a object by requesting with "Range" http header. It seems like downloading by each block unit. The default block(part) size is 50MB, it is caused the result which is default parallel requests count(5) by default multipart upload size(10MB). If you need to change this block size, you can change by new option "fd_page_size". This option can take from 1MB(1024 * 1024) to any bytes. So that, you have to take care about that fdcache.cpp(and fdcache.h) were changed a lot. 2) About local cache Local cache files which are in directory specified by "use_cache" option do not have always all of object data. This cause is that s3fs uses ftruncate function and reads(writes) each block unit of a temporary file. s3fs manages each block unit's status which are "downloaded area" or "not". For this status, s3fs makes new temporary file in cache directory which is specified by "use_cache" option. This status files is in a directory which is named "<use_cache sirectory>/.<bucket_name>/". When s3fs opens this status file, s3fs locks this file for exclusive control by calling flock function. You need to take care about this, the status files can not be laid on network drive(like NFS). This revision changes about file open mode, s3fs always opens a local cache file and each status file with writable mode. Last, this revision adds new option "del_cache", this option means that s3fs deletes all local cache file when s3fs starts and exits. 3) Uploading When s3fs writes data to file descriptor through FUSE request, old s3fs revision downloads all of the object. But new revision does not download all, it downloads only small percial area(some block units) including writing data area. And when s3fs closes or flushes the file descriptor, s3fs downloads other area which is not downloaded from server. After that, s3fs uploads all of data. Already r456 revision has parallel upload function, then this revision with r456 and r457 are very big change for performance. 4) Downloading By changing a temporary file and a local cache file, when s3fs downloads a object, it downloads only the required range(some block units). And s3fs downloads units by parallel GET request, it is same as a case of uploading. (Maximum parallel request count and each download size are specified same parameters for uploading.) In the new revision, when s3fs opens file, s3fs returns file descriptor soon. Because s3fs only opens(makes) the file descriptor with no downloading data. And when s3fs reads a data, s3fs downloads only some block unit including specified area. This result is good for performance. 5) Changes option name The option "parallel_upload" which added at r456 is changed to new option name as "parallel_count". This reason is this option value is not only used by uploading object, but a uploading object also uses this option. (For a while, you can use old option name "parallel_upload" for compatibility.) git-svn-id: http://s3fs.googlecode.com/svn/trunk@458 df820570-a93a-0410-bd06-b72b767a4274
2013-07-23 16:01:48 +00:00
int GetObjectRequest(const char* tpath, int fd, off_t start = -1, ssize_t size = -1);
int CheckBucket(void);
int ListBucketRequest(const char* tpath, const char* query);
int PreMultipartPostRequest(const char* tpath, headers_t& meta, std::string& upload_id, bool is_copy);
int CompleteMultipartPostRequest(const char* tpath, std::string& upload_id, etaglist_t& parts);
2015-08-17 04:42:45 +00:00
int UploadMultipartPostRequest(const char* tpath, int part_num, const std::string& upload_id);
int MultipartListRequest(std::string& body);
int AbortMultipartUpload(const char* tpath, std::string& upload_id);
2014-08-26 17:11:10 +00:00
int MultipartHeadRequest(const char* tpath, off_t size, headers_t& meta, bool is_copy);
int MultipartUploadRequest(const char* tpath, headers_t& meta, int fd, bool is_copy);
2015-08-17 04:42:45 +00:00
int MultipartUploadRequest(const std::string& upload_id, const char* tpath, int fd, off_t offset, size_t size, etaglist_t& list);
int MultipartRenameRequest(const char* from, const char* to, headers_t& meta, off_t size);
// methods(valiables)
CURL* GetCurlHandle(void) const { return hCurl; }
std::string GetPath(void) const { return path; }
std::string GetBasePath(void) const { return base_path; }
std::string GetSpacialSavedPath(void) const { return saved_path; }
std::string GetUrl(void) const { return url; }
headers_t* GetResponseHeaders(void) { return &responseHeaders; }
BodyData* GetBodyData(void) const { return bodydata; }
BodyData* GetHeadData(void) const { return headdata; }
long GetLastResponseCode(void) const { return LastResponseCode; }
Fixed Issue 229 and Changes codes 1) Set metadata "Content-Encoding" automatically(Issue 292) For this issue, s3fs is added new option "ahbe_conf". New option means the configuration file path, and this file specifies additional HTTP header by file(object) extension. Thus you can specify any HTTP header for each object by extension. * ahbe_conf file format: ----------- line = [file suffix] HTTP-header [HTTP-header-values] file suffix = file(object) suffix, if this field is empty, it means "*"(all object). HTTP-header = additional HTTP header name HTTP-header-values = additional HTTP header value ----------- * Example: ----------- .gz Content-Encoding gzip .Z Content-Encoding compress X-S3FS-MYHTTPHEAD myvalue ----------- A sample configuration file is uploaded in "test" directory. If ahbe_conf parameter is specified, s3fs loads it's configuration and compares extension(suffix) of object(file) when uploading (PUT/POST) it. If the extension is same, s3fs adds/sends specified HTTP header and value. A case of sample configuration file, if a object(it's extension is ".gz") which already has Content-Encoding HTTP header is renamed to ".txt" extension, s3fs does not set Content-Encoding. Because ".txt" is not match any line in configuration file. So, s3fs matches the extension by each PUT/POST action. * Please take care about "Content-Encoding". This new option allows setting ANY HTTP header by object extension. For example, you can specify "Content-Encoding" for ".gz"/etc extension in configuration. But this means that S3 always returns "Content-Encoding: gzip" when a client requests with other "Accept-Encoding:" header. It SHOULD NOT be good. Please see RFC 2616. 2) Changes about allow_other/uid/gid option for mount point I reviewed about mount point permission and allow_other/uid/gid options, and found bugs about these. s3fs is fixed bugs and changed to the following specifications. * s3fs only allows uid(gid) options as 0(root), when the effective user is zero(root). * A mount point(directory) must have a permission to allow accessing by effective user/group. * If allow_other option is specified, the mount point permission is set 0777(all users allow all access). In another case, the mount point is set 0700(only allows effective user). * When uid/gid option is specified, the mount point owner/group is set uid/gid option value. If uid/gid is not set, it is set effective user/group id. This changes maybe fixes some issue(321, 338). 3) Changes a logic about (Issue 229) The chmod command returns -EIO when changing the mount point. It is correct, s3fs can not changed owner/group/mtime for the mount point, but s3fs sends a request for changing the bucket. This revision does not send the request, and returns EIO as soon as possible. git-svn-id: http://s3fs.googlecode.com/svn/trunk@465 df820570-a93a-0410-bd06-b72b767a4274
2013-08-16 19:24:01 +00:00
bool SetUseAhbe(bool ahbe);
bool EnableUseAhbe(void) { return SetUseAhbe(true); }
bool DisableUseAhbe(void) { return SetUseAhbe(false); }
bool IsUseAhbe(void) const { return is_use_ahbe; }
int GetMultipartRetryCount(void) const { return retry_count; }
void SetMultipartRetryCount(int retrycnt) { retry_count = retrycnt; }
bool IsOverMultipartRetryCount(void) const { return (retry_count >= S3fsCurl::retries); }
2014-07-19 19:02:55 +00:00
int GetLastPreHeadSeecKeyPos(void) const { return b_ssekey_pos; }
};
//----------------------------------------------
// class S3fsMultiCurl
//----------------------------------------------
// Class for lapping multi curl
//
typedef std::map<CURL*, S3fsCurl*> s3fscurlmap_t;
typedef bool (*S3fsMultiSuccessCallback)(S3fsCurl* s3fscurl); // callback for succeed multi request
2014-10-01 10:42:39 +00:00
typedef S3fsCurl* (*S3fsMultiRetryCallback)(S3fsCurl* s3fscurl); // callback for failure and retrying
class S3fsMultiCurl
{
private:
static int max_multireq;
CURLM* hMulti;
s3fscurlmap_t cMap_all; // all of curl requests
s3fscurlmap_t cMap_req; // curl requests are sent
S3fsMultiSuccessCallback SuccessCallback;
S3fsMultiRetryCallback RetryCallback;
private:
bool ClearEx(bool is_all);
int MultiPerform(void);
int MultiRead(void);
public:
S3fsMultiCurl();
~S3fsMultiCurl();
static int SetMaxMultiRequest(int max);
static int GetMaxMultiRequest(void) { return S3fsMultiCurl::max_multireq; }
S3fsMultiSuccessCallback SetSuccessCallback(S3fsMultiSuccessCallback function);
S3fsMultiRetryCallback SetRetryCallback(S3fsMultiRetryCallback function);
bool Clear(void) { return ClearEx(true); }
bool SetS3fsCurlObject(S3fsCurl* s3fscurl);
int Request(void);
};
Fixed Issue 229 and Changes codes 1) Set metadata "Content-Encoding" automatically(Issue 292) For this issue, s3fs is added new option "ahbe_conf". New option means the configuration file path, and this file specifies additional HTTP header by file(object) extension. Thus you can specify any HTTP header for each object by extension. * ahbe_conf file format: ----------- line = [file suffix] HTTP-header [HTTP-header-values] file suffix = file(object) suffix, if this field is empty, it means "*"(all object). HTTP-header = additional HTTP header name HTTP-header-values = additional HTTP header value ----------- * Example: ----------- .gz Content-Encoding gzip .Z Content-Encoding compress X-S3FS-MYHTTPHEAD myvalue ----------- A sample configuration file is uploaded in "test" directory. If ahbe_conf parameter is specified, s3fs loads it's configuration and compares extension(suffix) of object(file) when uploading (PUT/POST) it. If the extension is same, s3fs adds/sends specified HTTP header and value. A case of sample configuration file, if a object(it's extension is ".gz") which already has Content-Encoding HTTP header is renamed to ".txt" extension, s3fs does not set Content-Encoding. Because ".txt" is not match any line in configuration file. So, s3fs matches the extension by each PUT/POST action. * Please take care about "Content-Encoding". This new option allows setting ANY HTTP header by object extension. For example, you can specify "Content-Encoding" for ".gz"/etc extension in configuration. But this means that S3 always returns "Content-Encoding: gzip" when a client requests with other "Accept-Encoding:" header. It SHOULD NOT be good. Please see RFC 2616. 2) Changes about allow_other/uid/gid option for mount point I reviewed about mount point permission and allow_other/uid/gid options, and found bugs about these. s3fs is fixed bugs and changed to the following specifications. * s3fs only allows uid(gid) options as 0(root), when the effective user is zero(root). * A mount point(directory) must have a permission to allow accessing by effective user/group. * If allow_other option is specified, the mount point permission is set 0777(all users allow all access). In another case, the mount point is set 0700(only allows effective user). * When uid/gid option is specified, the mount point owner/group is set uid/gid option value. If uid/gid is not set, it is set effective user/group id. This changes maybe fixes some issue(321, 338). 3) Changes a logic about (Issue 229) The chmod command returns -EIO when changing the mount point. It is correct, s3fs can not changed owner/group/mtime for the mount point, but s3fs sends a request for changing the bucket. This revision does not send the request, and returns EIO as soon as possible. git-svn-id: http://s3fs.googlecode.com/svn/trunk@465 df820570-a93a-0410-bd06-b72b767a4274
2013-08-16 19:24:01 +00:00
//----------------------------------------------
// class AdditionalHeader
//----------------------------------------------
#include <regex.h>
typedef struct add_header{
regex_t* pregex; // not NULL means using regex, NULL means comparing suffix directly.
std::string basestring;
std::string headkey;
std::string headvalue;
}ADDHEAD, *PADDHEAD;
typedef std::vector<PADDHEAD> addheadlist_t;
Fixed Issue 229 and Changes codes 1) Set metadata "Content-Encoding" automatically(Issue 292) For this issue, s3fs is added new option "ahbe_conf". New option means the configuration file path, and this file specifies additional HTTP header by file(object) extension. Thus you can specify any HTTP header for each object by extension. * ahbe_conf file format: ----------- line = [file suffix] HTTP-header [HTTP-header-values] file suffix = file(object) suffix, if this field is empty, it means "*"(all object). HTTP-header = additional HTTP header name HTTP-header-values = additional HTTP header value ----------- * Example: ----------- .gz Content-Encoding gzip .Z Content-Encoding compress X-S3FS-MYHTTPHEAD myvalue ----------- A sample configuration file is uploaded in "test" directory. If ahbe_conf parameter is specified, s3fs loads it's configuration and compares extension(suffix) of object(file) when uploading (PUT/POST) it. If the extension is same, s3fs adds/sends specified HTTP header and value. A case of sample configuration file, if a object(it's extension is ".gz") which already has Content-Encoding HTTP header is renamed to ".txt" extension, s3fs does not set Content-Encoding. Because ".txt" is not match any line in configuration file. So, s3fs matches the extension by each PUT/POST action. * Please take care about "Content-Encoding". This new option allows setting ANY HTTP header by object extension. For example, you can specify "Content-Encoding" for ".gz"/etc extension in configuration. But this means that S3 always returns "Content-Encoding: gzip" when a client requests with other "Accept-Encoding:" header. It SHOULD NOT be good. Please see RFC 2616. 2) Changes about allow_other/uid/gid option for mount point I reviewed about mount point permission and allow_other/uid/gid options, and found bugs about these. s3fs is fixed bugs and changed to the following specifications. * s3fs only allows uid(gid) options as 0(root), when the effective user is zero(root). * A mount point(directory) must have a permission to allow accessing by effective user/group. * If allow_other option is specified, the mount point permission is set 0777(all users allow all access). In another case, the mount point is set 0700(only allows effective user). * When uid/gid option is specified, the mount point owner/group is set uid/gid option value. If uid/gid is not set, it is set effective user/group id. This changes maybe fixes some issue(321, 338). 3) Changes a logic about (Issue 229) The chmod command returns -EIO when changing the mount point. It is correct, s3fs can not changed owner/group/mtime for the mount point, but s3fs sends a request for changing the bucket. This revision does not send the request, and returns EIO as soon as possible. git-svn-id: http://s3fs.googlecode.com/svn/trunk@465 df820570-a93a-0410-bd06-b72b767a4274
2013-08-16 19:24:01 +00:00
class AdditionalHeader
{
private:
static AdditionalHeader singleton;
bool is_enable;
addheadlist_t addheadlist;
protected:
AdditionalHeader();
~AdditionalHeader();
Fixed Issue 229 and Changes codes 1) Set metadata "Content-Encoding" automatically(Issue 292) For this issue, s3fs is added new option "ahbe_conf". New option means the configuration file path, and this file specifies additional HTTP header by file(object) extension. Thus you can specify any HTTP header for each object by extension. * ahbe_conf file format: ----------- line = [file suffix] HTTP-header [HTTP-header-values] file suffix = file(object) suffix, if this field is empty, it means "*"(all object). HTTP-header = additional HTTP header name HTTP-header-values = additional HTTP header value ----------- * Example: ----------- .gz Content-Encoding gzip .Z Content-Encoding compress X-S3FS-MYHTTPHEAD myvalue ----------- A sample configuration file is uploaded in "test" directory. If ahbe_conf parameter is specified, s3fs loads it's configuration and compares extension(suffix) of object(file) when uploading (PUT/POST) it. If the extension is same, s3fs adds/sends specified HTTP header and value. A case of sample configuration file, if a object(it's extension is ".gz") which already has Content-Encoding HTTP header is renamed to ".txt" extension, s3fs does not set Content-Encoding. Because ".txt" is not match any line in configuration file. So, s3fs matches the extension by each PUT/POST action. * Please take care about "Content-Encoding". This new option allows setting ANY HTTP header by object extension. For example, you can specify "Content-Encoding" for ".gz"/etc extension in configuration. But this means that S3 always returns "Content-Encoding: gzip" when a client requests with other "Accept-Encoding:" header. It SHOULD NOT be good. Please see RFC 2616. 2) Changes about allow_other/uid/gid option for mount point I reviewed about mount point permission and allow_other/uid/gid options, and found bugs about these. s3fs is fixed bugs and changed to the following specifications. * s3fs only allows uid(gid) options as 0(root), when the effective user is zero(root). * A mount point(directory) must have a permission to allow accessing by effective user/group. * If allow_other option is specified, the mount point permission is set 0777(all users allow all access). In another case, the mount point is set 0700(only allows effective user). * When uid/gid option is specified, the mount point owner/group is set uid/gid option value. If uid/gid is not set, it is set effective user/group id. This changes maybe fixes some issue(321, 338). 3) Changes a logic about (Issue 229) The chmod command returns -EIO when changing the mount point. It is correct, s3fs can not changed owner/group/mtime for the mount point, but s3fs sends a request for changing the bucket. This revision does not send the request, and returns EIO as soon as possible. git-svn-id: http://s3fs.googlecode.com/svn/trunk@465 df820570-a93a-0410-bd06-b72b767a4274
2013-08-16 19:24:01 +00:00
public:
// Reference singleton
static AdditionalHeader* get(void) { return &singleton; }
bool Load(const char* file);
void Unload(void);
bool AddHeader(headers_t& meta, const char* path) const;
struct curl_slist* AddHeader(struct curl_slist* list, const char* path) const;
bool Dump(void) const;
};
//----------------------------------------------
// Utility Functions
//----------------------------------------------
std::string GetContentMD5(int fd);
unsigned char* md5hexsum(int fd, off_t start, ssize_t size);
std::string md5sum(int fd, off_t start, ssize_t size);
struct curl_slist* curl_slist_sort_insert(struct curl_slist* list, const char* data);
struct curl_slist* curl_slist_sort_insert(struct curl_slist* list, const char* key, const char* value);
std::string get_sorted_header_keys(const struct curl_slist* list);
std::string get_canonical_headers(const struct curl_slist* list, bool only_amz = false);
bool MakeUrlResource(const char* realpath, std::string& resourcepath, std::string& url);
std::string prepare_url(const char* url);
2015-10-06 14:46:14 +00:00
bool get_object_sse_type(const char* path, sse_type_t& ssetype, std::string& ssevalue); // implement in s3fs.cpp
#endif // S3FS_CURL_H_
2014-09-07 15:08:27 +00:00
/*
* Local variables:
* tab-width: 4
* c-basic-offset: 4
* End:
* vim600: noet sw=4 ts=4 fdm=marker
* vim<600: noet sw=4 ts=4
*/