Just wanted to share some real-world use cases or challenges that we faced while working on a project
We used to run the app in containers, which used to pull the message from SQS queue and then process that message. However, there were so many challenges that we faced with this approach, some of them are listed below
Challenge # 1: Containers having issues pulling the message from SQS Queue
Solution: We moved the app from containers to the Elastic Beanstalk worker environment. This partially solved the message pulling issues from FIFO SQS, however, introduced another problem.
Challenge # 2: Single EC2 instance was unable to handle thousands of messages at a time.
Solution: As per AWS documentation. Latency from an Amazon EC2-based client to Amazon SQS in the same region is about 20 msec, which introduces additional overhead in each call to SQS. However, in this scenario, we need to read thousand of messages from SQS queue as quick as possible (without delay) while keeping the cost as low as possible
After doing some POC, we decided to move the app to Lambda, as Lambda has tight integration with SQS and has event source mapping. So Lambda will be auto-triggered every time there’s a message in SQS, and after processing the message, lambda will itself go to SQS and delete the message from the queue. Plus lambda support Auto-Scaling too, so the function can scale up to 1000 within seconds 🆒. This way Lambda functions quickly expand and read the message in batches from SQS and process them in parallel
Challenge # 3: Request sent from Lambda to third party API was getting blocked
Solution: As our workflow, Lambda gets the messages from the queue, processes that message, and sends a request to a third-party API that was getting blocked. After doing some troubleshooting, I found that Lambda was using an IP address range for an outbound call, which was getting blocked by third-party API. So to fix that issue, I associated the lambda function to a VPC in a private subnet and configure NAT Gateway in a public Subnet. I also configured the routing table to route the traffic, originating from Lambda to NAT Gateway, to use static Outbound IP address (attached with NAT Gateway) and then whitelist that IP address in third party company firewall. 😏
Although it took some time to sort out the issues and explore AWS documentation, however, It help me to understand and get familiar with the lambda function. Please hit the clap 👏 icon if you like this article and leave your comments if you have any questions or ideas to improve the flow.
