simple Boto examples with S3

Requirements

all you need is python, boto and filechunkio for easy multipart-upload

Creating connection

import boto
import boto.s3.connection
access_key = 'put your acces key here'
secret_key = 'put your secret key here'
con = boto.connect_s3(access_key,secret_key)

you can create ~/.boto file and add your credentials on it :

[Credentials]
aws_access_key_id = put_your_acces_key-here
aws_secret_access_key = put_your_secret_key_here

so now you can create a connection object just by

con = boto.connect_s3()

Listing buckets

for bucket in con.get_all_buckets():
  print bucket.name

Creating bucket

bucket = con.create_bucket('new_bucket')

Listing all objects in a bucket

for key in bucket.list()
  print key.name

Deleting Bucket

bucket must be empty before deleting (no force method implemented in boto)

bucket.delete()

Creating an object from String

key = bucket.new_key("test.txt")
key.set_content_from_string("hello world")

Creating an object from File (alternative key usage)

from boto.s3.key import Key
key = Key(bucket)
key.key = "file"
key.set_contents_from_filename("path_to_file")

Retrieving object as String

key.get_contents_to_string()

Retrieving object in File

key.get_contents_to_filename()

S3 Multipart-Upload for big files (with retries)

import math, os
from filechunkio import FileChunkIO
chunk_size = 5*1024*1024 #5mo
file_name = "my_big_file"
file_size = os.stat(file_name).st_size
chunk_number = int (math.ceil(file_size/chunk_size))+1
mp = bucket.initiate_multipart_upload(os.path.basename(file_name))
numberOfRetries = 10
for i in range(chunk_number):
    offset = chunk_size*i
    bytes = min(chunk_size, file_size - offset)
    with FileChunkIO(file_name, "r", offset=offset, bytes=bytes) as fp :
        try:
            mp.upload_part_from_file(fp, part_num=i+1)
        except Exception, ex:
            if numberOfRetries <= 0:
                raise ex
            else:
                numberOfRetries -= 1
                mp.upload_part_from_file(fp, part_num=i+1)
mp.complete_upload()

Delete an object

bucket.delete_key('file_to_delete')

Change object ACL

key.set_canned_acl('public-read')

Generate signed url valid for 30 seconds

signed_url = key.generate_url(30)