How to Mount an Amazon S3 Bucket to a Ubuntu Linux File System

After many hours of struggling to utilize the vast power of Amazon Web Services for my internship, I finally have paid the time price and done enough research into how one actually puts together some of the services. One such connection is utilizing Amazon S3 storage as a filesystem in a Linux distibution, more particularly an Amazon EC2 Ubuntu distro. For information on how to set these up visit this site (it is really easy and so I won't cover it here). The following tutorial is more of an aggregation of several other tutorials with a few other useful techniques to make your server as current as possible.

Installing the Basics

The following is copied from an article posted on the AWS Developers forum (link here):

1. Download the latest version of s3fs (which is a FUSE backend for S3) from http://code.google.com/p/s3fs/downloads/list wget http://s3fs.googlecode.com/files/s3fs-r177-source.tar.gz
2. Unpack it: tar zxf s3fs-r177-source.tar.gz
3. Install dependencies for building. In my Ubuntu AMI was:apt-get install build-essential libxml2-dev libfuse-dev libcurl4-openssl-devBuild and install the code (this copies the binary s3fs to /usr/bin):
4. cd s3fs

A Few Items Left Out

We will depart from the tutorial at this point and cover a few things that it does not explicitly state that will be confusing if you are not familiar with Linux. Once in the s3fs directory you do not simply enter make install as this will throw a very unhelpful error. If you go to the FuseOverAmazon website, you will note that there is step missing (which turns out is necessary in most make calls):

1. Enter ./configure --prefix=/usr
2. Enter sudo make install

Creating a Config File for S3FS Access

Part of the listed tutorial covers a method on how to create a config file that s3fs will access when mounting a bucket to the filestystem. However, if you have tried this on a Ubuntu server, you realize that this does not work. Since I am not a Linux guru I cannot tell you why this is the case. However, I have made a simple process by which you can do the exact same thing but with more steps.

1. Change directories to /etc (cd /etc)
2. Create a new document titled passwd-s3fs (sudo passwd-s3fs)
3. Change the permissions for the file to all user access (probably overkill, but it works: sudo chmod o+wx passwd-s3fs)
4. Add the Access Key and Access Secret (sudo echo [Access Key]:[Access Secret] > passwd-s3fs)
5. Change the permissions for the file to a restricted access (sudo chmod 640 passwd-s3fs)

Mounting a Bucket

Once the s3fs is installed, you are ready to add a bucket to the filesystem. Note that each bucket needs to be in lowercase letters in S3 as s3fs does not support uppercase letters. To add a bucket, do the following:

1. Change directories to the directory you want to mount the bucket. I prefer the /home directory as it makes it simple to access all the buckets from a relatively open directory, but you can use whatever directory you want. (cd /home)
2. Create a directory to place the bucket. (sudo mkdir myBucket)
3. Enter the following command: s3fs -o allow_other -o use_cache=/tmp [bucket name] /home/myBucket
1. Note that it is important to add the allow_other flag as not adding it will effectively hide the files from you and it will be almost like you didn't add the filesystem
4. If you have directories in the bucket, you must add the directory inside the myBucket directory for them to show. Note that the directory must match the name of the directory in the bucket and it must also be lowercase. (sudo mkdir bucketDirectory). 
1. This only needs to be done if you can't see the bucket. It is more of a troubleshooting trick.
2. Note that a new file will be added to your s3 bucket with the name of the directories. Don't delete these as they are used by s3fs to keep track of the directories in the bucket.

You should now be able to see your bucket! Congratulations!

Miscellaneous

This section covers a few miscellaneous notes about s3fs and other Ubuntu facts that I have learned that may be useful:

1. Often times the error messages thrown are very unhelpful. The best option is to copy the message directly and Google for the solution.
2. More often than not I have found that my biggest issue is that I forgot a step or didn't add sudo (super user access, which is like admin access on Windows) to the command. Be sure that you are doing everything correctly.
3. Some distributions for some reason do not come with the linux command make installed. If this is the case you will find that running make on an install won't work. I can't remember if this throws a helpful error or not. If you are attempting to make the install and everything has been completed and it still doesn't work, make sure that the make command is installed.
4. If you have buckets that you need to access that have uppercase characters, use Cloudberry to transfer files from one bucket to another. It is free and easy to use. To do this, simply open a bucket on the left pane and another on the right. It works much like a typical FTP client.

Updating Java - A More Excellent Way

If you have ever attempted to update Java to the newest edition (Java 7) you will have noticed that it is not as slick as some other Linux updates. Now that Oracle requires you to accept the Terms of Use Agreement it has made apt-get methods unusable. However, there is a way write by a group called web upd8 that runs a script on the linux machine that makes it as slick as apt-get. You can find out more about how to use it here. I have used it several times and it is a nice script that does most of the work for you.