Come for the Ride!

While reviewing our Amazon S3 account earlier this week, I came across an old unused bucket that had apparently sat dormant for an embarrassingly long time. As great as cloud hosting may be, paying for excess stuff you don’t use or need is simply idiotic – hence my determination to rid this problem in 5 minutes.

Unfortunately, I found out S3 buckets cannot be deleted when there are files inside it. Also, if you have a few hundred thousand files in a bucket, trying to list those files will crash most if not all UI based S3 tools that have a limit of listing only 1,000 files at one time.

I tried accessing the buckets using Bucket Explorer, S3Hub, S3Fox, and even some command line tools but all of them crashed while trying to retrieve the files. That was certainly NOT good news. After doing some more digging, I found a script called S3Nuke – and this ladies and gentlemen, is the magic broom that can help sweep away the waste.

With the help from a friend, we setup S3Nuke and ran 50 concurrent threads to clear out the unused bucket. It took about 3-4 hours on a wifi connection to list and delete roughly 3 million files – yeah, that number was a surprise to us as well. Considering other methods would have probably taken days or weeks if they worked without crashing, S3Nuke did an AMAZING job.

Although S3 is really cheap and it’s one of the best options out there for cloud storage, every little bit does add up over time. Amazon makes its money from holding onto these files and they certainly have no plans to make it easy for you to delete them. Whether you use S3 for remote backup or for your e-commerce site, make sure you keep your usage in check so you don’t end up paying the stupid tax. (Like we did)

Related posts:

1. The Amazon Startup Challenge
2. Amazon AWS Follow-up: Start-Nots.