s3fs-fuse/src/cache.cpp

122 lines
3.4 KiB
C++
Raw Normal View History

/*
* s3fs - FUSE-based file system backed by Amazon S3
*
* Copyright 2007-2008 Randy Rizun <rrizun@gmail.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; either version 2
* of the License, or (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include <pthread.h>
#include <iostream>
#include <string>
#include <map>
#include "cache.h"
using namespace std;
typedef std::map<std::string, struct stat_cache_entry> stat_cache_t; // key=path
static stat_cache_t stat_cache;
pthread_mutex_t stat_cache_lock;
int get_stat_cache_entry(const char *path, struct stat *buf) {
Summary of Changes(1.62 -> 1.63) 1) Lifetime for the stats cache Added the new option "stat_cache_expire". This option which is specified by seconds means the lifetime for each stats cache entry. If this option is not specified, the stats cache is kept in s3fs process until the stats cache grown to maximum size. (default) If this option is specified, the stats cache entry is out from the memory when the entry expires time. 2) Enable file permission s3fs before 1.62 did not consider the file access permission. s3fs after this version can consider it. For access permission, the s3fs_getattr() function was divided into sub function which can check the file access permission. It is like access() function. And the function calling the s3fs_getattr() calls this new sub function instead of s3fs_getattr(). Last the s3fs_opendir() function which is called by FUSE was added for checking directory access permission when listing the files in directory. 3) UID/GUID When a file or a directory was created, the s3fs could not set the UID/GID as the user who executed a command. (Almost the UID/GID are root, because the s3fs run by root.) After this version, the s3fs set correct UID/GID as the user who executes the commond. 4) About the mtime If the object does not have "x-amz-meta-mtime" meta, the s3fs uses the "Last-Modified" header instead of it. But the s3fs had a bug in this code, and this version fixed this bug. When user modified the file, the s3fs did not update the mtime of the file. This version fixed this bug. In the get_local_fd() function, the local file's mtime was changed only when s3fs run with "use_cache" option. This version always updates the mtime whether the local cache file is used or not. And s3fs_flush ( ) function set the mtime of local cache file from S3 object mtime, but it was wrong . This version is that the s3fs_flush ( ) changes the mtime of S3 object from the local cache file or the tmpfile . The s3fs cuts some requests, because the s3fs can always check mtime whether the s3fs uses or does not use the local cache file. 5) A case of no "x-amz-meta-mode" If the object did not have "x-amz-meta-mtime" mete, the s3fs recognized the file as not regular file. After this version, the s3fs recognizes the file as regular file. 6) "." and ".." directory The s3fs_readdir() did not return "X" and "XX" directory name. After this version, the s3fs is changed that it returns "X" and "XX". Example, the result of "ls" lists "X" and "XX" directory. 7) Fixed a bug The insert_object() had a bug, and it is fixed. git-svn-id: http://s3fs.googlecode.com/svn/trunk@390 df820570-a93a-0410-bd06-b72b767a4274
2013-02-24 08:58:54 +00:00
int is_delete_cache = 0;
pthread_mutex_lock(&stat_cache_lock);
stat_cache_t::iterator iter = stat_cache.find(path);
if(iter != stat_cache.end()) {
Summary of Changes(1.62 -> 1.63) 1) Lifetime for the stats cache Added the new option "stat_cache_expire". This option which is specified by seconds means the lifetime for each stats cache entry. If this option is not specified, the stats cache is kept in s3fs process until the stats cache grown to maximum size. (default) If this option is specified, the stats cache entry is out from the memory when the entry expires time. 2) Enable file permission s3fs before 1.62 did not consider the file access permission. s3fs after this version can consider it. For access permission, the s3fs_getattr() function was divided into sub function which can check the file access permission. It is like access() function. And the function calling the s3fs_getattr() calls this new sub function instead of s3fs_getattr(). Last the s3fs_opendir() function which is called by FUSE was added for checking directory access permission when listing the files in directory. 3) UID/GUID When a file or a directory was created, the s3fs could not set the UID/GID as the user who executed a command. (Almost the UID/GID are root, because the s3fs run by root.) After this version, the s3fs set correct UID/GID as the user who executes the commond. 4) About the mtime If the object does not have "x-amz-meta-mtime" meta, the s3fs uses the "Last-Modified" header instead of it. But the s3fs had a bug in this code, and this version fixed this bug. When user modified the file, the s3fs did not update the mtime of the file. This version fixed this bug. In the get_local_fd() function, the local file's mtime was changed only when s3fs run with "use_cache" option. This version always updates the mtime whether the local cache file is used or not. And s3fs_flush ( ) function set the mtime of local cache file from S3 object mtime, but it was wrong . This version is that the s3fs_flush ( ) changes the mtime of S3 object from the local cache file or the tmpfile . The s3fs cuts some requests, because the s3fs can always check mtime whether the s3fs uses or does not use the local cache file. 5) A case of no "x-amz-meta-mode" If the object did not have "x-amz-meta-mtime" mete, the s3fs recognized the file as not regular file. After this version, the s3fs recognizes the file as regular file. 6) "." and ".." directory The s3fs_readdir() did not return "X" and "XX" directory name. After this version, the s3fs is changed that it returns "X" and "XX". Example, the result of "ls" lists "X" and "XX" directory. 7) Fixed a bug The insert_object() had a bug, and it is fixed. git-svn-id: http://s3fs.googlecode.com/svn/trunk@390 df820570-a93a-0410-bd06-b72b767a4274
2013-02-24 08:58:54 +00:00
if(!is_stat_cache_expire_time || ((*iter).second.cache_date + stat_cache_expire_time) >= time(NULL)){
// hit
if(foreground)
cout << " stat cache hit [path=" << path << "]"
<< " [time=" << (*iter).second.cache_date << "]"
<< " [hit count=" << (*iter).second.hit_count << "]" << endl;
Summary of Changes(1.62 -> 1.63) 1) Lifetime for the stats cache Added the new option "stat_cache_expire". This option which is specified by seconds means the lifetime for each stats cache entry. If this option is not specified, the stats cache is kept in s3fs process until the stats cache grown to maximum size. (default) If this option is specified, the stats cache entry is out from the memory when the entry expires time. 2) Enable file permission s3fs before 1.62 did not consider the file access permission. s3fs after this version can consider it. For access permission, the s3fs_getattr() function was divided into sub function which can check the file access permission. It is like access() function. And the function calling the s3fs_getattr() calls this new sub function instead of s3fs_getattr(). Last the s3fs_opendir() function which is called by FUSE was added for checking directory access permission when listing the files in directory. 3) UID/GUID When a file or a directory was created, the s3fs could not set the UID/GID as the user who executed a command. (Almost the UID/GID are root, because the s3fs run by root.) After this version, the s3fs set correct UID/GID as the user who executes the commond. 4) About the mtime If the object does not have "x-amz-meta-mtime" meta, the s3fs uses the "Last-Modified" header instead of it. But the s3fs had a bug in this code, and this version fixed this bug. When user modified the file, the s3fs did not update the mtime of the file. This version fixed this bug. In the get_local_fd() function, the local file's mtime was changed only when s3fs run with "use_cache" option. This version always updates the mtime whether the local cache file is used or not. And s3fs_flush ( ) function set the mtime of local cache file from S3 object mtime, but it was wrong . This version is that the s3fs_flush ( ) changes the mtime of S3 object from the local cache file or the tmpfile . The s3fs cuts some requests, because the s3fs can always check mtime whether the s3fs uses or does not use the local cache file. 5) A case of no "x-amz-meta-mode" If the object did not have "x-amz-meta-mtime" mete, the s3fs recognized the file as not regular file. After this version, the s3fs recognizes the file as regular file. 6) "." and ".." directory The s3fs_readdir() did not return "X" and "XX" directory name. After this version, the s3fs is changed that it returns "X" and "XX". Example, the result of "ls" lists "X" and "XX" directory. 7) Fixed a bug The insert_object() had a bug, and it is fixed. git-svn-id: http://s3fs.googlecode.com/svn/trunk@390 df820570-a93a-0410-bd06-b72b767a4274
2013-02-24 08:58:54 +00:00
if(buf != NULL)
*buf = (*iter).second.stbuf;
Summary of Changes(1.62 -> 1.63) 1) Lifetime for the stats cache Added the new option "stat_cache_expire". This option which is specified by seconds means the lifetime for each stats cache entry. If this option is not specified, the stats cache is kept in s3fs process until the stats cache grown to maximum size. (default) If this option is specified, the stats cache entry is out from the memory when the entry expires time. 2) Enable file permission s3fs before 1.62 did not consider the file access permission. s3fs after this version can consider it. For access permission, the s3fs_getattr() function was divided into sub function which can check the file access permission. It is like access() function. And the function calling the s3fs_getattr() calls this new sub function instead of s3fs_getattr(). Last the s3fs_opendir() function which is called by FUSE was added for checking directory access permission when listing the files in directory. 3) UID/GUID When a file or a directory was created, the s3fs could not set the UID/GID as the user who executed a command. (Almost the UID/GID are root, because the s3fs run by root.) After this version, the s3fs set correct UID/GID as the user who executes the commond. 4) About the mtime If the object does not have "x-amz-meta-mtime" meta, the s3fs uses the "Last-Modified" header instead of it. But the s3fs had a bug in this code, and this version fixed this bug. When user modified the file, the s3fs did not update the mtime of the file. This version fixed this bug. In the get_local_fd() function, the local file's mtime was changed only when s3fs run with "use_cache" option. This version always updates the mtime whether the local cache file is used or not. And s3fs_flush ( ) function set the mtime of local cache file from S3 object mtime, but it was wrong . This version is that the s3fs_flush ( ) changes the mtime of S3 object from the local cache file or the tmpfile . The s3fs cuts some requests, because the s3fs can always check mtime whether the s3fs uses or does not use the local cache file. 5) A case of no "x-amz-meta-mode" If the object did not have "x-amz-meta-mtime" mete, the s3fs recognized the file as not regular file. After this version, the s3fs recognizes the file as regular file. 6) "." and ".." directory The s3fs_readdir() did not return "X" and "XX" directory name. After this version, the s3fs is changed that it returns "X" and "XX". Example, the result of "ls" lists "X" and "XX" directory. 7) Fixed a bug The insert_object() had a bug, and it is fixed. git-svn-id: http://s3fs.googlecode.com/svn/trunk@390 df820570-a93a-0410-bd06-b72b767a4274
2013-02-24 08:58:54 +00:00
(*iter).second.hit_count++;
pthread_mutex_unlock(&stat_cache_lock);
return 0;
}else{
// timeout
is_delete_cache = 1;
}
}
pthread_mutex_unlock(&stat_cache_lock);
Summary of Changes(1.62 -> 1.63) 1) Lifetime for the stats cache Added the new option "stat_cache_expire". This option which is specified by seconds means the lifetime for each stats cache entry. If this option is not specified, the stats cache is kept in s3fs process until the stats cache grown to maximum size. (default) If this option is specified, the stats cache entry is out from the memory when the entry expires time. 2) Enable file permission s3fs before 1.62 did not consider the file access permission. s3fs after this version can consider it. For access permission, the s3fs_getattr() function was divided into sub function which can check the file access permission. It is like access() function. And the function calling the s3fs_getattr() calls this new sub function instead of s3fs_getattr(). Last the s3fs_opendir() function which is called by FUSE was added for checking directory access permission when listing the files in directory. 3) UID/GUID When a file or a directory was created, the s3fs could not set the UID/GID as the user who executed a command. (Almost the UID/GID are root, because the s3fs run by root.) After this version, the s3fs set correct UID/GID as the user who executes the commond. 4) About the mtime If the object does not have "x-amz-meta-mtime" meta, the s3fs uses the "Last-Modified" header instead of it. But the s3fs had a bug in this code, and this version fixed this bug. When user modified the file, the s3fs did not update the mtime of the file. This version fixed this bug. In the get_local_fd() function, the local file's mtime was changed only when s3fs run with "use_cache" option. This version always updates the mtime whether the local cache file is used or not. And s3fs_flush ( ) function set the mtime of local cache file from S3 object mtime, but it was wrong . This version is that the s3fs_flush ( ) changes the mtime of S3 object from the local cache file or the tmpfile . The s3fs cuts some requests, because the s3fs can always check mtime whether the s3fs uses or does not use the local cache file. 5) A case of no "x-amz-meta-mode" If the object did not have "x-amz-meta-mtime" mete, the s3fs recognized the file as not regular file. After this version, the s3fs recognizes the file as regular file. 6) "." and ".." directory The s3fs_readdir() did not return "X" and "XX" directory name. After this version, the s3fs is changed that it returns "X" and "XX". Example, the result of "ls" lists "X" and "XX" directory. 7) Fixed a bug The insert_object() had a bug, and it is fixed. git-svn-id: http://s3fs.googlecode.com/svn/trunk@390 df820570-a93a-0410-bd06-b72b767a4274
2013-02-24 08:58:54 +00:00
if(is_delete_cache){
delete_stat_cache_entry(path);
}
return -1;
}
void add_stat_cache_entry(const char *path, struct stat *st) {
if(foreground)
cout << " add_stat_cache_entry[path=" << path << "]" << endl;
if(max_stat_cache_size < 1)
return;
if(stat_cache.size() > max_stat_cache_size)
truncate_stat_cache();
pthread_mutex_lock(&stat_cache_lock);
stat_cache[path].stbuf = *st;
Summary of Changes(1.62 -> 1.63) 1) Lifetime for the stats cache Added the new option "stat_cache_expire". This option which is specified by seconds means the lifetime for each stats cache entry. If this option is not specified, the stats cache is kept in s3fs process until the stats cache grown to maximum size. (default) If this option is specified, the stats cache entry is out from the memory when the entry expires time. 2) Enable file permission s3fs before 1.62 did not consider the file access permission. s3fs after this version can consider it. For access permission, the s3fs_getattr() function was divided into sub function which can check the file access permission. It is like access() function. And the function calling the s3fs_getattr() calls this new sub function instead of s3fs_getattr(). Last the s3fs_opendir() function which is called by FUSE was added for checking directory access permission when listing the files in directory. 3) UID/GUID When a file or a directory was created, the s3fs could not set the UID/GID as the user who executed a command. (Almost the UID/GID are root, because the s3fs run by root.) After this version, the s3fs set correct UID/GID as the user who executes the commond. 4) About the mtime If the object does not have "x-amz-meta-mtime" meta, the s3fs uses the "Last-Modified" header instead of it. But the s3fs had a bug in this code, and this version fixed this bug. When user modified the file, the s3fs did not update the mtime of the file. This version fixed this bug. In the get_local_fd() function, the local file's mtime was changed only when s3fs run with "use_cache" option. This version always updates the mtime whether the local cache file is used or not. And s3fs_flush ( ) function set the mtime of local cache file from S3 object mtime, but it was wrong . This version is that the s3fs_flush ( ) changes the mtime of S3 object from the local cache file or the tmpfile . The s3fs cuts some requests, because the s3fs can always check mtime whether the s3fs uses or does not use the local cache file. 5) A case of no "x-amz-meta-mode" If the object did not have "x-amz-meta-mtime" mete, the s3fs recognized the file as not regular file. After this version, the s3fs recognizes the file as regular file. 6) "." and ".." directory The s3fs_readdir() did not return "X" and "XX" directory name. After this version, the s3fs is changed that it returns "X" and "XX". Example, the result of "ls" lists "X" and "XX" directory. 7) Fixed a bug The insert_object() had a bug, and it is fixed. git-svn-id: http://s3fs.googlecode.com/svn/trunk@390 df820570-a93a-0410-bd06-b72b767a4274
2013-02-24 08:58:54 +00:00
stat_cache[path].cache_date = time(NULL); // Set time.
pthread_mutex_unlock(&stat_cache_lock);
}
void delete_stat_cache_entry(const char *path) {
if(foreground)
cout << " delete_stat_cache_entry[path=" << path << "]" << endl;
pthread_mutex_lock(&stat_cache_lock);
stat_cache_t::iterator iter = stat_cache.find(path);
if(iter != stat_cache.end())
stat_cache.erase(iter);
pthread_mutex_unlock(&stat_cache_lock);
}
void truncate_stat_cache() {
string path_to_delete;
unsigned int hit_count = 0;
unsigned int lowest_hit_count = 0;
pthread_mutex_lock(&stat_cache_lock);
stat_cache_t::iterator iter;
for(iter = stat_cache.begin(); iter != stat_cache.end(); iter++) {
hit_count = (* iter).second.hit_count;
if(!lowest_hit_count) {
lowest_hit_count = hit_count;
path_to_delete = (* iter).first;
}
if(lowest_hit_count > hit_count)
path_to_delete = (* iter).first;
}
stat_cache.erase(path_to_delete);
pthread_mutex_unlock(&stat_cache_lock);
printf(" purged %s from the stat cache\n", path_to_delete.c_str());
}