Previously s3fs calculated the strlen of a binary input instead of
using the stored length. This yielded IO errors when interacting with
SSE-encrypted objects. Fixes#696.
Workers now notify the master thread when they complete, unifying the
Linux and macOS code paths. This also avoids excessive
pthread_tryjoin_np calls. Follows on to
88cd8feb05.
S3 can copy multipart much faster than single part due to IO
parallelization. Renaming a 4 GB file reduces from 72 to 20 seconds
with bigger gains with larger files.
This simplifies the code and fixes an issue with unlocked access.
Also use a recursive lock for StatCache to avoid races between
lock..unlock..lock sequences.
Previously s3fs would issue a batch of HEAD requests and wait for all
to succeed before issuing the next batch. Now it issues the first
batch and only waits for a single call to succeed before issuing the
next call. This can improve performance when one call lags due to
network errors. I measured 25% improvement with the same level of
parallelism. This commit also reparents parallelism knobs for
consistency. Follows on to 88cd8feb05.
Fixes#223.
Previously this appended to an empty list, zeroed the list, then
appended to the list. Instead zero the list first and then append.
This also enables sending Content-MD5 which can eagerly detect
transmission errors. Found via Valgrind.
Previously s3fs considered 3xx codes as successful. When writing an
object to a misconfigured region, s3fs did not propagate the 301 as a
write error to the client. References #693.
Previously s3fs would issue a batch of uploads and wait for all to
succeed before issuing the next batch. Now it issues the first batch
and only waits for a single part to succeed before uploading the next
part. This can improve performance when one part lags due to network
errors. Fixes#183.
Amazon SSE-C https://tinyurl.com/ychug4cg writes:
> Use this header to provide the 256-bit, base64-encoded encryption key
> for Amazon S3 to use to encrypt or decrypt your data.
It seems likely that future keys would be as long, or longer, so
this change assumes that text longer than 32 characters are keys
that are base64 encoded.
Signed-off-by: Earl Chew <earl_chew@yahoo.com>
