In boto 2, you can write to an S3 object using these methods:

• Key.set_contents_from_string()
• Key.set_contents_from_file()
• Key.set_contents_from_filename()
• Key.set_contents_from_stream()

Is there a boto 3 equivalent? What is the boto3 method for saving data to an object stored on S3?

8 Answers

In boto 3, the 'Key.set_contents_from_' methods were replaced by

• Object.put()

• Client.put_object()

For example:

import boto3 some_binary_data = b'Here we have some data' more_binary_data = b'Here we have some more data' # Method 1: Object.put() s3 = boto3.resource('s3') object = s3.Object('my_bucket_name', 'my/key/including/filename.txt') object.put(Body=some_binary_data) # Method 2: Client.put_object() client = boto3.client('s3') client.put_object(Body=more_binary_data, Bucket='my_bucket_name', Key='my/key/including/anotherfilename.txt') 

Alternatively, the binary data can come from reading a file, as described in the official docs comparing boto 2 and boto 3:

Storing Data

Storing data from a file, stream, or string is easy:

# Boto 2.x from boto.s3.key import Key key = Key('hello.txt') key.set_contents_from_file('/tmp/hello.txt') # Boto 3 s3.Object('mybucket', 'hello.txt').put(Body=open('/tmp/hello.txt', 'rb')) 

boto3 also has a method for uploading a file directly:

s3 = boto3.resource('s3') s3.Bucket('bucketname').upload_file('/local/file/here.txt','folder/sub/path/to/s3key') 

You no longer have to convert the contents to binary before writing to the file in S3. The following example creates a new text file (called newfile.txt) in an S3 bucket with string contents:

import boto3 s3 = boto3.resource( 's3', region_name='us-east-1', aws_access_key_id=KEY_ID, aws_secret_access_key=ACCESS_KEY ) content="String content to write to a new S3 file" s3.Object('my-bucket-name', 'newfile.txt').put(Body=content) 

Here's a nice trick to read JSON from s3:

import json, boto3 s3 = boto3.resource("s3").Bucket("bucket") json.load_s3 = lambda f: json.load(s3.Object(key=f).get()["Body"]) json.dump_s3 = lambda obj, f: s3.Object(key=f).put(Body=json.dumps(obj)) 

Now you can use json.load_s3 and json.dump_s3 with the same API as load and dump

data = {"test":0} json.dump_s3(data, "key") # saves json to s3://bucket/key data = json.load_s3("key") # read json from s3://bucket/key 

A cleaner and concise version which I use to upload files on the fly to a given S3 bucket and sub-folder-

import boto3 BUCKET_NAME = 'sample_bucket_name' PREFIX = 'sub-folder/' s3 = boto3.resource('s3') # Creating an empty file called "_DONE" and putting it in the S3 bucket s3.Object(BUCKET_NAME, PREFIX + '_DONE').put(Body="") 

Note: You should ALWAYS put your AWS credentials (aws_access_key_id and aws_secret_access_key) in a separate file, for example- ~/.aws/credentials

it is worth mentioning smart-open that uses boto3 as a back-end.

smart-open is a drop-in replacement for python's open that can open files from s3, as well as ftp, http and many other protocols.

for example

from smart_open import open import json with open("s3://your_bucket/your_key.json", 'r') as f: data = json.load(f) 

The aws credentials are loaded via boto3 credentials, usually a file in the ~/.aws/ dir or an environment variable.

After some research, I found this. It can be achieved using a simple csv writer. It is to write a dictionary to CSV directly to S3 bucket.

eg: data_dict = [{"Key1": "value1", "Key2": "value2"}, {"Key1": "value4", "Key2": "value3"}] assuming that the keys in all the dictionary are uniform.

import csv import boto3 # Sample input dictionary data_dict = [{"Key1": "value1", "Key2": "value2"}, {"Key1": "value4", "Key2": "value3"}] data_dict_keys = data_dict[0].keys() # creating a file buffer file_buff = StringIO() # writing csv data to file buffer writer = csv.DictWriter(file_buff, fieldnames=data_dict_keys) writer.writeheader() for data in data_dict: writer.writerow(data) # creating s3 client connection client = boto3.client('s3') # placing file to S3, file_buff.getvalue() is the CSV body for the file client.put_object(Body=file_buff.getvalue(), Bucket='my_bucket_name', Key='my/key/including/anotherfilename.txt') 

You may use the below code to write, for example an image to S3 in 2019. To be able to connect to S3 you will have to install AWS CLI using command pip install awscli, then enter few credentials using command aws configure:

import urllib3 import uuid from pathlib import Path from io import BytesIO from errors import custom_exceptions as cex BUCKET_NAME = "xxx.yyy.zzz" POSTERS_BASE_PATH = "assets/wallcontent" CLOUDFRONT_BASE_URL = "" class S3(object): def __init__(self): self.client = boto3.client('s3') self.bucket_name = BUCKET_NAME self.posters_base_path = POSTERS_BASE_PATH def __download_image(self, url): manager = urllib3.PoolManager() try: res = manager.request('GET', url) except Exception: print("Could not download the image from URL: ", url) raise cex.ImageDownloadFailed return BytesIO(res.data) # any file-like object that implements read() def upload_image(self, url): try: image_file = self.__download_image(url) except cex.ImageDownloadFailed: raise cex.ImageUploadFailed extension = Path(url).suffix id = uuid.uuid1().hex + extension final_path = self.posters_base_path + "/" + id try: self.client.upload_fileobj(image_file, self.bucket_name, final_path ) except Exception: print("Image Upload Error for URL: ", url) raise cex.ImageUploadFailed return CLOUDFRONT_BASE_URL + id