I am trying to figure out what is the best way to read a JSON file from a specific endpoint and then save/post such object to AWS S3. I have created a mocked endpoint with a mocked response via and I would like to know what is the best way to 'POST' it to a S3 bucket. New JSON files will be available weekly and I was thinking that perhaps a way to go about would be to use Lambda AWS and the API Gateway. Is this a viable way? I also would like to explore the possibility of enable an event trigger way to pull data or a scheduler. What would you recommend? I know that AWS SQS is an option, but how do you send the fetched JSON files to the queue?
Thank you, any resource or suggestion is more than welcome. I am looking for potential approaches.
2 Answers
There are quite a lot of different ways you could approach this, but if I understand correctly you want to retrieve a JSON response once a week from a fixed endpoint (which you have set up?), and then write that JSON to a file, or sequence of files, you store on S3.
If that is correct, then all you really need is Cloudwatch Events (to set up a weekly scheduled recurring event in cron format) which triggers a lambda function that makes the request and then writes it to S3. You can also use the same lambda function (or write another which is triggered by the same CloudWatch Event) to post a message to SQS with the JSON.
Depending on what language you are most comfortable writing it in, you can use the SDK to do all the things you want to do. Personally I like the python library boto3, and combined with a little file IO to get the JSON to a text file of some kind, and the requests library to make the actual HTTP request to your endpoint, you should be able to do all you need. Helpful functions in boto3 will be sending a SQS message and writing to S3.
I'm not sure why you'd necessarily need API Gateway to do anything here, unless instead of triggering the lambda via a scheduled event, you wanted to do it by making a separate HTTP request, but then you may as well just make the request to your original API!
Please consider using Lambda with NodeJS code to do GET from endpoint for invoking the lambda function use cloudwatch event
1
