Stack Overflow Asked by xbeta on December 23, 2021
I am fairly new to AWS ecosystem, especially with the data side.
I have a project that requires me to run a query against a table in Redshift every 24hrs automatically, and perhaps remove a few columns in that query results and use RESTful API to hit some endpoints at a 3rd-party site for further checking.
I have a few questions on this.
Thanks,
Sam.
Normally you will find many AWS tools are able to solve the same problem. The right choice depends of your priorities. What are you looking for lowest cost? efficiency? convinience?
I answer your questions below:
Is it a good usage pattern to use AWS Lambda (Python) and Redshift for such task? Yes it's ok. Redshift is normally a very expensive service, are you sure you need Redshift here?
Should I choose Java vs Python vs NodeJS for AWS Lambda? Which one has a better support for querying Redshift?
Java will require an event bridge call every 5 minutes or so if you want to avoid cold starts. Apart from that it really is up to you.
Both Lambda and Redshift would be in the same VPC, and using the same private subnets for egress NAT gateway, is this secured setup? It's ok but again NAT Gateways are expensive. Depending on the problem you are trying to solve there might be some work around.
Any sample code to share on this setup? https://aws.amazon.com/blogs/big-data/building-an-event-driven-application-with-aws-lambda-and-the-amazon-redshift-data-api/
Does AWS Lambda has a regular scheduler to trigger every 24hrs? Or is it simply based on events? Yes you can use cron or similar to program the lambda trigger using event bridge.
Since application database is in DynamoDB, is it more efficient and easier to setup for AWS Lambda to query DynamoDB for similar data instead? I'm a bit confused with this last question, but normally is very easy to query Dynamo fom lambda.
Edit: Typo
Answered by Jose Mendez on December 23, 2021
I'll try to answer your question with the best intentions:
Answered by tpschmidt on December 23, 2021
Get help from others!
Recent Answers
Recent Questions
© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP