I have an S3 bucket that contains several hundred files in a folder. I needed to copy those files into a different folder in another bucket. Sounds simple enough? but was unable to find a simple way to do this through the AWS Console. I found a number of stack overflow articles that talked about using Sync, or downloading the files and re-uploading them. None of which sounded particularly appealing.

In the end I just wrote this bash one liner (which I can probably optimise further by not repeating the sourcebucket / sourcefolder three times):

This just uses s3cmd to list all the files in the bucket/folder I wish to copy from. The output of that is piped to awk which I use to extract the s3 url of each file. I then use tail to remove the first line which I don’t need. I then use sed to build up a ‘s3cmd cp’ command which copies the file from its original location to my new location.

If anyone can suggest a better way that doesnt require me having to download the source files … I’d love to hear it.
If you can’t see the embedded Gist above then you can view it here.