Commit Graph

776 Commits

Author SHA1 Message Date
mooredan@suncup.net
68774b5960 Added code to address the potential short write
associated with BIO_write

Resolves issue #6


git-svn-id: http://s3fs.googlecode.com/svn/trunk@287 df820570-a93a-0410-bd06-b72b767a4274
2010-12-20 00:06:56 +00:00
mooredan@suncup.net
90ee6b8f9b Some more unwinding of the C++ classes, should make
refactoring easier and the code easier to understand (for me anyway)

Opened up the VERIFY macro so that memory cleanup can be done
before returning from a function.

Make the file descriptor function calls a bit more robust,
check the return codes.

Current code tested on Debian sid, CentOS (with FUSE 2.84) and Ubuntu 10.10




git-svn-id: http://s3fs.googlecode.com/svn/trunk@286 df820570-a93a-0410-bd06-b72b767a4274
2010-12-19 22:27:56 +00:00
mooredan@suncup.net
f56b95f11e Fixed memory leak issues as outlined in issue #104
No tarball until further testing on other platforms.
Extensively tested on Debian sid 64bit

Resolves issue #104


git-svn-id: http://s3fs.googlecode.com/svn/trunk@285 df820570-a93a-0410-bd06-b72b767a4274
2010-12-19 01:34:27 +00:00
mooredan@suncup.net
147dd86215 Minimum FUSE version is now 2.8.4 !
re-wrote the curl write-to-memory callback function
(this helped eliminate a memory leak)

eliminated another memory leak

more debug messages



git-svn-id: http://s3fs.googlecode.com/svn/trunk@281 df820570-a93a-0410-bd06-b72b767a4274
2010-12-17 04:40:15 +00:00
mooredan@suncup.net
dfd6d6c1b6 Turn off CURL_FAILONERROR and parse the HTTP return
code with CURLE_OK is returned.

clean up a few debug/stdout messages

This commit doesn't really fix anything, but eliminates
the suspicous HTTP 404 errors from the syslog -- these are
usually normal



git-svn-id: http://s3fs.googlecode.com/svn/trunk@280 df820570-a93a-0410-bd06-b72b767a4274
2010-12-11 04:42:52 +00:00
mooredan@suncup.net
9c6d671fec Service another curl error appropriately
do not output via cout unless we are in "foreground" mode


git-svn-id: http://s3fs.googlecode.com/svn/trunk@279 df820570-a93a-0410-bd06-b72b767a4274
2010-12-09 20:56:29 +00:00
mooredan@suncup.net
d3c42255b9 Handle a couple of more specific curl errors.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@278 df820570-a93a-0410-bd06-b72b767a4274
2010-12-09 02:59:49 +00:00
mooredan@suncup.net
7358b3512e Now servicing the CURLE_COULDNT_CONNECT error better.
Rather than erroring out, it is treated like the
CURLE_OPERATION_TIMEOUT in that it doesn't exit the
timeout loop while the retry count is > 0.  I
also added a short duration sleep to not retry
immediately.

Resolves issue #132


git-svn-id: http://s3fs.googlecode.com/svn/trunk@277 df820570-a93a-0410-bd06-b72b767a4274
2010-12-08 04:52:19 +00:00
mooredan@suncup.net
99aa781701 Clean up from last time just in case last time had
an error and left the test file in the bucket


git-svn-id: http://s3fs.googlecode.com/svn/trunk@276 df820570-a93a-0410-bd06-b72b767a4274
2010-12-08 03:08:10 +00:00
mooredan@suncup.net
412afb6953 During the check_service function, parse a unsuccessful HTTP
return for more specific information as to why the communication
failed. Most common reasons are the "time too skewed" or
credentials failure.

This could be extended to the curl routine that is used
during normal operation. However, the check_service routine
is a good first pass.


Resolves issue #133


git-svn-id: http://s3fs.googlecode.com/svn/trunk@275 df820570-a93a-0410-bd06-b72b767a4274
2010-12-08 02:39:13 +00:00
mooredan@suncup.net
99781e70bc Added -s option to pass onto FUSE -- I'm not entirely
sure of the usefulness or purpose of this option, but
am including it for completeness

Re-categorized SYS_INFO messages ot SYS_DEBUG messages
to reduce the amount of spew into syslog -- these can
be turned back on with the -d option
Resolves issue #120



git-svn-id: http://s3fs.googlecode.com/svn/trunk@274 df820570-a93a-0410-bd06-b72b767a4274
2010-12-04 20:07:08 +00:00
mooredan@suncup.net
5d2c29976a CentOS 5.5 used curl version 7.15 and CURLE_PEER_FAILED_VERIFICATION is not
defined. Added #defines around appropriate code. 

Latest revision now compiles and works on CentOS



git-svn-id: http://s3fs.googlecode.com/svn/trunk@271 df820570-a93a-0410-bd06-b72b767a4274
2010-11-26 22:44:56 +00:00
mooredan@suncup.net
8b10de5559 Added an additional check in check_service to
expose the curl compiled with openssl vs. nss issue.

If the issue is seen, emit an informational message
and give the user an option to over-ride checking of
the hostname -- it's recommended not to use bucket
names with periods and https

As implied, added an option ssl_verify_hostname=[0|1]

Tested on fedora 14. Will check on Ubuntu/Debian/CentOS
after check in.

Resolves issue #128



git-svn-id: http://s3fs.googlecode.com/svn/trunk@270 df820570-a93a-0410-bd06-b72b767a4274
2010-11-26 22:11:48 +00:00
mooredan@suncup.net
49c3687a52 This is a fix for Issue #125. Add a more robust way of
trapping the CURLE_SSL_CACERT error and trying to automatically
correct it. 

Tested on CentOS 5.5 and ensured that Debian/Ubuntu doesn't break
because of this. This only applies to the https usage.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@269 df820570-a93a-0410-bd06-b72b767a4274
2010-11-26 03:13:53 +00:00
mooredan@suncup.net
90f90f6b42 Trap the certificate issue outlined in issue 125
During the service_check() function, if curl returns
a CURLE_SSL_CACERT error then report it and do not
start the s3fs service.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@265 df820570-a93a-0410-bd06-b72b767a4274
2010-11-24 23:00:31 +00:00
mooredan@suncup.net
ded2dd527d Added check for left square bracket character at the
beginning of the line in the password file.

Resolves issue #126


git-svn-id: http://s3fs.googlecode.com/svn/trunk@263 df820570-a93a-0410-bd06-b72b767a4274
2010-11-24 02:44:15 +00:00
mooredan@suncup.net
b7fc931962 Renaming directories currently have an issue. If you
rename (mv) a directory, its contents "disappear". To
get the contents back, just re-create the directory.

This issue has led to more than one support issues.
This patch disallows renaming of directories and
returns a operation not supported message, e.g.

% mv dir1 dir2
mv: cannot move `dir1' to `dir2': Operation not supported

Another observation is that when a directory was renamed
it loses its directory status and becomes a normal file.

This is a first step to allow for supporting directory
renames. 



git-svn-id: http://s3fs.googlecode.com/svn/trunk@260 df820570-a93a-0410-bd06-b72b767a4274
2010-11-23 22:41:58 +00:00
mooredan@suncup.net
58be75f036 Added process to release
git-svn-id: http://s3fs.googlecode.com/svn/trunk@257 df820570-a93a-0410-bd06-b72b767a4274
2010-11-23 04:46:21 +00:00
mooredan@suncup.net
f7f7003985 Added release target to automatically add tarball and
update Installation Notes wiki - not finished yet



git-svn-id: http://s3fs.googlecode.com/svn/trunk@248 df820570-a93a-0410-bd06-b72b767a4274
2010-11-23 01:32:41 +00:00
mooredan@suncup.net
45b1256966 test/*.sh files were not being included in the
tarball created by "make dist"


git-svn-id: http://s3fs.googlecode.com/svn/trunk@247 df820570-a93a-0410-bd06-b72b767a4274
2010-11-22 22:30:17 +00:00
mooredan@suncup.net
4187f03089 Don't try to read from /etc/mime.types if it
is not present/readable.

Don't process blank lines in mime.types

There wasn't an issue seen with this, just
cleaning up the code a bit.


git-svn-id: http://s3fs.googlecode.com/svn/trunk@245 df820570-a93a-0410-bd06-b72b767a4274
2010-11-22 18:28:07 +00:00
mooredan@suncup.net
7429227922 Added a couple of casts to take care of compile warnings
Resolves issue #88

Specifying credentials on the command line is no longer
supported. There are several other ways to do this now.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@243 df820570-a93a-0410-bd06-b72b767a4274
2010-11-21 01:30:25 +00:00
mooredan@suncup.net
d239780a29 Added check for preliminary checking of credentials
and if the bucket exists on the amazon server.

Resolves issue #121


git-svn-id: http://s3fs.googlecode.com/svn/trunk@241 df820570-a93a-0410-bd06-b72b767a4274
2010-11-20 17:55:15 +00:00
mooredan@suncup.net
65e0a2ff84 Re-enable the -f option for FUSE. (This got disabled
with the conversion to get_opt

-f is passed along to FUSE. This makes FUSE run
in foreground (non-daemon) mode and some debugging
messages now appear on STDOUT



git-svn-id: http://s3fs.googlecode.com/svn/trunk@240 df820570-a93a-0410-bd06-b72b767a4274
2010-11-19 22:23:38 +00:00
mooredan@suncup.net
a587656914 Deleted empty directories. Now the whole repository
has been moved up one directory.


git-svn-id: http://s3fs.googlecode.com/svn/trunk@239 df820570-a93a-0410-bd06-b72b767a4274
2010-11-14 02:29:04 +00:00
mooredan@suncup.net
8864850ca8 Updated svn:ignore props on top level
git-svn-id: http://s3fs.googlecode.com/svn/trunk@238 df820570-a93a-0410-bd06-b72b767a4274
2010-11-14 02:26:57 +00:00
mooredan@suncup.net
1ada6609f0 autoreconf, configure, make, make check, install, etc
all work from the copy of source files up on directory

...and the history is still there.

Not deleting the directories yet because need to
preserve svn properties -- not sure how to copy
vs. recreate -- need to look it up

On the todo list: remove s3fs, s3fs/src and s3fs/test


git-svn-id: http://s3fs.googlecode.com/svn/trunk@237 df820570-a93a-0410-bd06-b72b767a4274
2010-11-14 00:41:12 +00:00
mooredan@suncup.net
e9b8216d21 In preparation to remove the unnecessary "s3fs"
directory from the trunk directory.

First do a svn cp of all of the source up to
trunk.  This is supposed to preserve change
history -- we'll see.

The source remains untouched until this gets
worked out.

Also in preparation of bringing in the source
collateral for the debian package into the
repository. I expect that the top level will
look like this:

svn/
   s3fs/
      trunk/
      tags/
      branches/
   dpkg/
      trunk/
      tags/
      branches/


So far that's how it is looking.  I'll be
very careful to ensure integrity of the data.
As a result this may be a multistep process.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@236 df820570-a93a-0410-bd06-b72b767a4274
2010-11-13 23:59:23 +00:00
mooredan@suncup.net
41cf008523 Added command line option for debug (-d or --debug)
Sending DEBUG messages to syslog is off by default.
-d (or --debug) enables DEBUG messages to go to
syslog

Additionally, if an additional -d command line option
is given, then the -d option is passed to FUSE, upon
which FUSE outputs debug messages to STDOUT

resolves issue #120



git-svn-id: http://s3fs.googlecode.com/svn/trunk@235 df820570-a93a-0410-bd06-b72b767a4274
2010-11-12 23:18:39 +00:00
mooredan@suncup.net
b6768464af A couple of small changes to the check collateral:
- The script was looking for /root/.passwd-s3fs on my debian
machine (it worked correctly on Ubuntu). Fixed the common.sh
file to always look in the SUDO_USER's home directory

- Added the passwd_file option to the s3fs command. Since the
test runs as root, the $SUDO_USER's password file wasn't
being looked for by the program itself. Instead the normal
precedence was being followed and the /etc/passwd-s3fs file
was being used.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@234 df820570-a93a-0410-bd06-b72b767a4274
2010-11-11 21:07:45 +00:00
mooredan@suncup.net
c0ca9dd0c3 Fixed issue #123 - Check permissions on password file used
If any password file is used, regardless if it is specified
on the command line, ~/.passwd-s3fs or /etc/passwd-s3fs it
is checked for appropriate permissions.

No password file is allowed to have any others permissions

Only the /etc/passwd-s3fs file is allowed to have any
group permissions, all others are not allowed to have
any group permissions.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@233 df820570-a93a-0410-bd06-b72b767a4274
2010-11-11 05:14:16 +00:00
apetresc
ec822185da Usability improvement: use the ${SUDO_USER} environment provided by sudo to create the bucket based on the user who invoked the test using sudo. Much better this way.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@232 df820570-a93a-0410-bd06-b72b767a4274
2010-11-09 07:22:30 +00:00
apetresc
ac294bc885 Making the test bucket be based on ${USER}. This requires the bucket to already exist. S3FS doesn't report an error when mounting if it doesn't exist, so the test fails strangely in such cases.
The correct way to solve this is to consistently faily when mounting. Also offer a flag to forcefully create the bucket if it doesn't yet exist.


git-svn-id: http://s3fs.googlecode.com/svn/trunk@231 df820570-a93a-0410-bd06-b72b767a4274
2010-11-09 07:13:30 +00:00
apetresc
3022b87738 Added a simple integration test, and properly integrated it with autotools.
To run integration tests, just use: sudo make check
This will give you a report of all passes and failures.

To add an integration test, just add it to the TESTS variable in s3fs/test/Makefile.am
Make sure your test sources integration-test-common.sh and require-root.sh

Having many of these for every feature will help us avoid regressions and develop with confidence.


git-svn-id: http://s3fs.googlecode.com/svn/trunk@230 df820570-a93a-0410-bd06-b72b767a4274
2010-11-09 07:04:47 +00:00
mooredan@suncup.net
cc2e440276 Revamped the read_passwd_file function to resolve issue #114
Add support for mulitple access id/secret access keys in passwd-s3fs file

This change applies to any passwd-s3fs files.

The format of the file is more robust, error checked and extended:

  - as before, any line beginning with # is ignored
  - any empty line is ignored
  - any non-ignored line which contains a space or tab is an error
  - any non-ignored line which does not contain a : separator is an error
  
The format of the file is:

[bucket:]AccessKeyId:SecretAccessKey

The bucket can now be specified to allow for multiple credentials.
A default entry is as before:

AccessKeyId:SecretAccessKey

Only one default entry is allowed, if more than one default entry
is found, that is an error.  A default entry is not required, if the
bucket that is being mounted has its own entry.

If the user's .passwd-s3fs file is present but credentials cannot
be determined from it, then the system-wide /etc/passwd-s3fs will
be consulted (if readable by the current user).

This change is completely backward compatable with the existing
scheme and has been well tested.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@229 df820570-a93a-0410-bd06-b72b767a4274
2010-11-08 03:42:16 +00:00
mooredan@suncup.net
a80c900894 Fixes for several issues:
Issue #102 --help command line 
Issue #107 command line checking and other error checking
Issue #112 uses getopt_long now, but FUSE's own parser is still there
           some command line checking is split between main() and the
           helper function
Issue #117 command line option for credentials file
Issue #118 use of environment variables for credentials
Issue #119 support for a local passwd-s3fs file

Big checkin, many new lines of code.  

Comand line option checking is now more robust and does quite
a bit of error checking before handing the mount off to FUSE.

Cleaned up the code and made several new functions from
existing code

Many ways to supply credentials now and there is a precedance
order followed.

--help is helpful




git-svn-id: http://s3fs.googlecode.com/svn/trunk@227 df820570-a93a-0410-bd06-b72b767a4274
2010-11-07 01:39:30 +00:00
mooredan@suncup.net
08a611217a Reduced minimum versions for fuse and libxml-2
Resolves issue #110 - see issue for more details



git-svn-id: http://s3fs.googlecode.com/svn/trunk@223 df820570-a93a-0410-bd06-b72b767a4274
2010-10-30 02:32:49 +00:00
mooredan@suncup.net
ef630253d6 Added support for anonymously mounting a public bucket.
If the "public_bucket=1" option is used, then
the command line options for Access Key ID and
Secret Access Key are ignored as well as the
/etc/passwd-s3fs file

Internally, the "Authorization: AWS ..." header
line is not included in the header.

Tested on a public bucket and it appears to work.



git-svn-id: http://s3fs.googlecode.com/svn/trunk@221 df820570-a93a-0410-bd06-b72b767a4274
2010-10-29 03:25:06 +00:00
apetresc
28e56b759d Updated the README to reflect the new compilation method.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@220 df820570-a93a-0410-bd06-b72b767a4274
2010-10-25 19:58:59 +00:00
mooredan@suncup.net
bf06d02536 Added autom4te.cache to svn ignore list
Incremented the release version number


git-svn-id: http://s3fs.googlecode.com/svn/trunk@219 df820570-a93a-0410-bd06-b72b767a4274
2010-10-25 05:06:58 +00:00
mooredan@suncup.net
6eb6b1e238 M s3fs/src/s3fs.cpp
=========================
Removed extra space in the message emitted by --version


M    s3fs/src/Makefile.am
==========================
Added the .h files to the s3fs_SOURCES. These files
were not being included in the .tar.gz file when
"make dist" was being executed.  This was sourced
as an error with the "make distcheck" target.




git-svn-id: http://s3fs.googlecode.com/svn/trunk@217 df820570-a93a-0410-bd06-b72b767a4274
2010-10-24 22:23:09 +00:00
apetresc
184a31d676 Factored out purely string-based operations into string_util
git-svn-id: http://s3fs.googlecode.com/svn/trunk@216 df820570-a93a-0410-bd06-b72b767a4274
2010-10-24 09:36:50 +00:00
apetresc
22f260e582 Beginning the refactoring my making the first separation of s3fs.cpp into s3fs.h. Many further divisions along logical boundaries will be coming soon.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@215 df820570-a93a-0410-bd06-b72b767a4274
2010-10-24 08:32:15 +00:00
apetresc
639e087d4f Added .project and .cproject to svn:ignore.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@214 df820570-a93a-0410-bd06-b72b767a4274
2010-10-24 07:42:55 +00:00
mooredan@suncup.net
4dd73defab Fixed issue #19 added a --version switch
Used the compile time VERSION define
Version message conforms to GNU Coding Standards


git-svn-id: http://s3fs.googlecode.com/svn/trunk@213 df820570-a93a-0410-bd06-b72b767a4274
2010-10-21 14:23:43 +00:00
mooredan@suncup.net
0d130efe60 A bit more cleanup of the directory.
readme.txt has been moved to README

removal of Eclipse CDT project files


git-svn-id: http://s3fs.googlecode.com/svn/trunk@212 df820570-a93a-0410-bd06-b72b767a4274
2010-10-21 00:05:53 +00:00
apetresc
8e2652b233 Code reformatting so it conforms to the Google C++ style guide. I chose that one because it's generally-accepted, but any style guide is okay with me. The important thing is to no longer have the random assortment of formatting that was there before.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@211 df820570-a93a-0410-bd06-b72b767a4274
2010-10-20 07:08:50 +00:00
apetresc
aebff8b8d2 Adding Autotools cruft to svn:ignore.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@210 df820570-a93a-0410-bd06-b72b767a4274
2010-10-20 06:16:06 +00:00
apetresc
a15b72145e Converting to Autotools.
git-svn-id: http://s3fs.googlecode.com/svn/trunk@209 df820570-a93a-0410-bd06-b72b767a4274
2010-10-20 06:06:09 +00:00
mooredan@suncup.net
cb85e65591 Fixed issue #101 - mv (rename) does not update the
Content-Type metadata if the file extension changes 



git-svn-id: http://s3fs.googlecode.com/svn/trunk@208 df820570-a93a-0410-bd06-b72b767a4274
2010-10-20 02:54:01 +00:00