Commit Graph

68 Commits

Author SHA1 Message Date
ben.lemasurier@gmail.com
cf43a2a260 Fixes a compile-time bug in 32bit systems and a memory leak in md5sum()
git-svn-id: http://s3fs.googlecode.com/svn/trunk@367 df820570-a93a-0410-bd06-b72b767a4274
2011-08-30 15:20:49 +00:00
ben.lemasurier@gmail.com
c933b6a9b1 Support for modifying files > 5GB (fixes issue #215)
Modified rename_object and put_headers to handle objects larger than
5GB. Files larger than 5GB are required to use the multi interface.


git-svn-id: http://s3fs.googlecode.com/svn/trunk@363 df820570-a93a-0410-bd06-b72b767a4274
2011-08-29 22:01:32 +00:00
ben.lemasurier@gmail.com
f623a395bc fixes memory leaks associated with the fuse option parser and curl handles
Passing s3fs through valgrind revealed a few more memory leaks assocated with
the fuse option parser. We'll no longer directly return main from fuse_main().

In the event that s3fs_check_service failed, many of the curl handles were not
being cleaned up properly.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@362 df820570-a93a-0410-bd06-b72b767a4274
2011-08-25 20:32:56 +00:00
ben.lemasurier@gmail.com
979ed21d5a close directory handle after checking mountpoint
running s3fs through several valgrind checks found a bug in s3fs_check_service()


git-svn-id: http://s3fs.googlecode.com/svn/trunk@361 df820570-a93a-0410-bd06-b72b767a4274
2011-08-25 17:11:25 +00:00
ben.lemasurier@gmail.com
f29986f4c6 Simplified s3fs_check_service
simply using strstr to see if the bucket is available.
removed the check to see if the service has any buckets at all.


git-svn-id: http://s3fs.googlecode.com/svn/trunk@360 df820570-a93a-0410-bd06-b72b767a4274
2011-08-25 16:34:10 +00:00
ben.lemasurier@gmail.com
5a6ef54b36 trivial change to fix some valgrind warnings regarding unititialized values.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@359 df820570-a93a-0410-bd06-b72b767a4274
2011-08-02 15:08:28 +00:00
ben.lemasurier@gmail.com
b41f982d59 Fixes issue #205
when s3fs is linked against libcurl < 7.20.0 we'd receive CURLM_CALL_MULTI_PERFORM



git-svn-id: http://s3fs.googlecode.com/svn/trunk@358 df820570-a93a-0410-bd06-b72b767a4274
2011-07-29 20:03:38 +00:00
ben.lemasurier@gmail.com
40fde1e0ea Possible fix for issue #206
- handle bad xml data from Amazon



git-svn-id: http://s3fs.googlecode.com/svn/trunk@355 df820570-a93a-0410-bd06-b72b767a4274
2011-07-28 15:43:52 +00:00
mooredan@suncup.net
b54b2ddfda Fixed issue #210
Applied user supplied patch


git-svn-id: http://s3fs.googlecode.com/svn/trunk@353 df820570-a93a-0410-bd06-b72b767a4274
2011-07-19 19:52:38 +00:00
ben.lemasurier@gmail.com
f484ea00ae Fixes issue #172
git-svn-id: http://s3fs.googlecode.com/svn/trunk@352 df820570-a93a-0410-bd06-b72b767a4274
2011-07-07 22:09:40 +00:00
ben.lemasurier@gmail.com
913fee101e fix for issue #205
git-svn-id: http://s3fs.googlecode.com/svn/trunk@350 df820570-a93a-0410-bd06-b72b767a4274
2011-07-07 16:25:04 +00:00
ben.lemasurier@gmail.com
2eafa487d7 Massive speed improvements for readdir operations
complete s3fs_readdir() refactor
    - multi interface now batches HTTP requests
      - proper HTTP KeepAlive sessions are back! (CURLOPT_FORBID_REUSE is no longer required)
    - use xpath to quickly grab xml nodes
    - lots of cleanup
    - fixes some strange stat cache behavior
    - huge readdir performance benefits (8-14x in my case) on large directories



git-svn-id: http://s3fs.googlecode.com/svn/trunk@348 df820570-a93a-0410-bd06-b72b767a4274
2011-07-02 02:11:54 +00:00
ben.lemasurier@gmail.com
1a6885359c Ability to disable multipart uploads
- Added 'nomultipart' option to disable multipart uploads
  - see issue #181


git-svn-id: http://s3fs.googlecode.com/svn/trunk@347 df820570-a93a-0410-bd06-b72b767a4274
2011-06-27 02:21:38 +00:00
ben.lemasurier@gmail.com
6db8dafca4 bump version number
- removed debugging line


git-svn-id: http://s3fs.googlecode.com/svn/trunk@346 df820570-a93a-0410-bd06-b72b767a4274
2011-06-26 00:42:45 +00:00
ben.lemasurier@gmail.com
1597dfe659 Support for mounting a remote directory (issue #7).
- fixed a bug in the file cache, it was attempting to set the mtime
    on symlinks
  - general code cleanup; moved some string functions to string_util.cpp


git-svn-id: http://s3fs.googlecode.com/svn/trunk@345 df820570-a93a-0410-bd06-b72b767a4274
2011-06-26 00:37:52 +00:00
mooredan@suncup.net
30abdff53f resolves issue #155
git-svn-id: http://s3fs.googlecode.com/svn/trunk@340 df820570-a93a-0410-bd06-b72b767a4274
2011-04-06 15:06:28 +00:00
ben.lemasurier@gmail.com
2e09e5201b fixes issue #23 and issue #160. validate the file cache by comparing the local/remote size/mtime values instead of an md5 sum
git-svn-id: http://s3fs.googlecode.com/svn/trunk@338 df820570-a93a-0410-bd06-b72b767a4274
2011-03-10 00:11:55 +00:00
mooredan@suncup.net
ebdde932a6 Strip the trailing '/' characters off of the url= option's
argument

Resolves issue #171


git-svn-id: http://s3fs.googlecode.com/svn/trunk@336 df820570-a93a-0410-bd06-b72b767a4274
2011-03-09 20:55:31 +00:00
ben.lemasurier@gmail.com
6cd9e9e65d moved generic curl routines to their own file
git-svn-id: http://s3fs.googlecode.com/svn/trunk@332 df820570-a93a-0410-bd06-b72b767a4274
2011-03-01 19:35:55 +00:00
mooredan@suncup.net
c16925bb10 Added check to check for illegal characters in bucket name
Resolves issue #163



git-svn-id: http://s3fs.googlecode.com/svn/trunk@330 df820570-a93a-0410-bd06-b72b767a4274
2011-02-26 14:48:02 +00:00
mooredan@suncup.net
2fe1abc66b First attempt to resolve issue 161 -- added handler for
curl error code 23 - CURLE_WRITE_ERROR

When encountered, it does a retry.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@329 df820570-a93a-0410-bd06-b72b767a4274
2011-02-26 14:11:46 +00:00
ben.lemasurier@gmail.com
48d1a73e06 cleanup only; s3fs.cpp is getting huge, split caching to its own file
git-svn-id: http://s3fs.googlecode.com/svn/trunk@328 df820570-a93a-0410-bd06-b72b767a4274
2011-02-25 17:35:12 +00:00
ben.lemasurier@gmail.com
ec4c5b3627 reduced the number of keys returned to see if a directory is empty, we only need to know if 1 exists. Minor cleanups.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@327 df820570-a93a-0410-bd06-b72b767a4274
2011-02-23 17:16:12 +00:00
ben.lemasurier@gmail.com
6c89527aed minor cleanup, no functional changes.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@326 df820570-a93a-0410-bd06-b72b767a4274
2011-02-23 16:03:08 +00:00
ben.lemasurier@gmail.com
c07e27eff1 only delete stat cache entries when file could have been modified.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@324 df820570-a93a-0410-bd06-b72b767a4274
2011-02-22 23:01:42 +00:00
ben.lemasurier@gmail.com
9b5f694497 update to r322, this only works when local file cache is turned on. Initial 'make check' erroneously passed
git-svn-id: http://s3fs.googlecode.com/svn/trunk@323 df820570-a93a-0410-bd06-b72b767a4274
2011-02-22 22:28:57 +00:00
ben.lemasurier@gmail.com
0fb4427444 Fixes issues #134, "double upload". This _should_ result in a large performance improvment.
- s3fs_flush() now checks to see whether the file on the remote end is the same as the local copy.
  - md5sum() now requires a file descriptor instead of a path.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@322 df820570-a93a-0410-bd06-b72b767a4274
2011-02-22 21:28:01 +00:00
ben.lemasurier@gmail.com
1496f6a81e Further multipart cleanup/error checking, in preparation for multi-threaded uploads.
"Last-Modified" is now returned with get_headers() data


git-svn-id: http://s3fs.googlecode.com/svn/trunk@320 df820570-a93a-0410-bd06-b72b767a4274
2011-02-17 17:31:43 +00:00
ben.lemasurier@gmail.com
cfa0fd2992 clean up get_local_fd() to use md5sum()
git-svn-id: http://s3fs.googlecode.com/svn/trunk@319 df820570-a93a-0410-bd06-b72b767a4274
2011-02-16 16:52:45 +00:00
ben.lemasurier@gmail.com
00bde54d0a A large amount of cleanup for multipart uploads; preparation work for upcoming multi-threaded upload support.
Functional changes are limited to the multipart upload process. Each uploaded part is now verified against a local md5sum.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@318 df820570-a93a-0410-bd06-b72b767a4274
2011-02-15 23:32:27 +00:00
ben.lemasurier@gmail.com
1a79d451c5 Fixes an issue reintroduced in r315: s3fs_readdir was not populating the file stat cache
git-svn-id: http://s3fs.googlecode.com/svn/trunk@317 df820570-a93a-0410-bd06-b72b767a4274
2011-02-14 18:54:30 +00:00
mooredan@suncup.net
ecaaf4d324 Implemented max_stat_cache_size as an option
Resolves issue #157



git-svn-id: http://s3fs.googlecode.com/svn/trunk@316 df820570-a93a-0410-bd06-b72b767a4274
2011-02-12 16:48:23 +00:00
mooredan@suncup.net
6a3a68b01c Bound the size of stat_cache as described in issue #157
git-svn-id: http://s3fs.googlecode.com/svn/trunk@315 df820570-a93a-0410-bd06-b72b767a4274
2011-02-12 15:02:44 +00:00
mooredan@suncup.net
2c0456680e Resolves issue #156
s3fs_readdir() now looks at stat_cache


git-svn-id: http://s3fs.googlecode.com/svn/trunk@312 df820570-a93a-0410-bd06-b72b767a4274
2011-02-11 03:52:31 +00:00
mooredan@suncup.net
c8d5b35f8f Resolves issue #154
Installed and tested fix for file permissions/cache issue


git-svn-id: http://s3fs.googlecode.com/svn/trunk@311 df820570-a93a-0410-bd06-b72b767a4274
2011-02-11 03:30:02 +00:00
mooredan@suncup.net
6f7e180133 Resolves issue #152
- added move directory test
- fix bug introduced with fixing issue #150



git-svn-id: http://s3fs.googlecode.com/svn/trunk@310 df820570-a93a-0410-bd06-b72b767a4274
2011-02-10 01:07:46 +00:00
apetresc
fd2a6c120a Committing patch contributed by Ben LeMasurier to cache directory attributes. Details and tests on Issue 150.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@309 df820570-a93a-0410-bd06-b72b767a4274
2011-02-08 18:23:38 +00:00
mooredan@suncup.net
0a233011a5 potential fix for issue 148
- increase max_keys in readdir from 50 to 500
- handle the curle_couldnt_resolve_host error better
- add the curl forbid reuse option



git-svn-id: http://s3fs.googlecode.com/svn/trunk@308 df820570-a93a-0410-bd06-b72b767a4274
2011-02-05 01:35:18 +00:00
mooredan@suncup.net
b3726191b2 Clean up a couple of compile warnings seen in CentOS
git-svn-id: http://s3fs.googlecode.com/svn/trunk@304 df820570-a93a-0410-bd06-b72b767a4274
2011-01-21 01:12:09 +00:00
mooredan@suncup.net
a5e7745589 Another checkpoint
git-svn-id: http://s3fs.googlecode.com/svn/trunk@303 df820570-a93a-0410-bd06-b72b767a4274
2011-01-21 00:50:55 +00:00
mooredan@suncup.net
8e4c89fdec Checkpoint for large file (> 2GB) upload capability
needs more testing.


git-svn-id: http://s3fs.googlecode.com/svn/trunk@302 df820570-a93a-0410-bd06-b72b767a4274
2011-01-20 22:40:59 +00:00
mooredan@suncup.net
0f18298886 Fix for issue #145 and additional tests in make check
git-svn-id: http://s3fs.googlecode.com/svn/trunk@301 df820570-a93a-0410-bd06-b72b767a4274
2011-01-19 05:26:01 +00:00
mooredan@suncup.net
f94bbd70f9 Cleaned up compile time warnings as reported by -Wall
Beginning of s3fs "utility" mode - initially -u option
just reports in progress multipart uploads for the
bucket. Eventually this mode could be used for
other S3 tasks not accessible through typical
file system operations

For multipart upload, use safer mkstemp() instead
of tmpnam() for temporary file

Increased the curl connect and readwrite timeouts
to 10 and 30 seconds respectively.

Autodetect when a big file is being uploaded,
increase the readwrite timeout to 120 seconds. This
was found through experimentation.  When uploading
a big file, it is suspected that time is needed
for S3 to assemble the file before it is available
for access. It was found that when a large file
was uploaded via rsync, the final mtime and
chmod modifications were timing out, even though
the upload itself was successful.


Multipart upload is ready for use. A couple of
error checks are still needed in the function and
some cleanup.  Need some feedback on how it
is working though.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@298 df820570-a93a-0410-bd06-b72b767a4274
2010-12-30 03:13:21 +00:00
mooredan@suncup.net
acc7363433 Checkpoint for implementation of multipart upload
Check issue #142 for details

Code is operational, but not quite ready for
prime time -- needs some clean up


git-svn-id: http://s3fs.googlecode.com/svn/trunk@297 df820570-a93a-0410-bd06-b72b767a4274
2010-12-28 04:15:23 +00:00
mooredan@suncup.net
784d51d805 separated out a common function to mknod and create
git-svn-id: http://s3fs.googlecode.com/svn/trunk@290 df820570-a93a-0410-bd06-b72b767a4274
2010-12-22 17:19:52 +00:00
mooredan@suncup.net
d3d850596b Implemented directory rename - Resolves issue #17
Implemented create() function - Resolves issue #18

Issues will be updated with more detail.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@289 df820570-a93a-0410-bd06-b72b767a4274
2010-12-21 15:24:46 +00:00
mooredan@suncup.net
5c64ff83cf Restructing to take care of the directory rename.
This is a checkpoint. No functional changes in this commit.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@288 df820570-a93a-0410-bd06-b72b767a4274
2010-12-20 05:26:27 +00:00
mooredan@suncup.net
68774b5960 Added code to address the potential short write
associated with BIO_write

Resolves issue #6


git-svn-id: http://s3fs.googlecode.com/svn/trunk@287 df820570-a93a-0410-bd06-b72b767a4274
2010-12-20 00:06:56 +00:00
mooredan@suncup.net
90ee6b8f9b Some more unwinding of the C++ classes, should make
refactoring easier and the code easier to understand (for me anyway)

Opened up the VERIFY macro so that memory cleanup can be done
before returning from a function.

Make the file descriptor function calls a bit more robust,
check the return codes.

Current code tested on Debian sid, CentOS (with FUSE 2.84) and Ubuntu 10.10




git-svn-id: http://s3fs.googlecode.com/svn/trunk@286 df820570-a93a-0410-bd06-b72b767a4274
2010-12-19 22:27:56 +00:00
mooredan@suncup.net
f56b95f11e Fixed memory leak issues as outlined in issue #104
No tarball until further testing on other platforms.
Extensively tested on Debian sid 64bit

Resolves issue #104


git-svn-id: http://s3fs.googlecode.com/svn/trunk@285 df820570-a93a-0410-bd06-b72b767a4274
2010-12-19 01:34:27 +00:00