There are many use cases where we wanted to monitor the VPC Flow Logs to view the data going IN / OUT into our VPC. These network traces helps us to troubleshoot many network-related issues.
We do have a choice in AWS to save VPC Flow Log either into
- AWS CloudWatch or
- AWS S3 Buckets.
However, both of these solutions don’t provide a good user-friendly view and can become cumbersome when trying to find a specific IP Address or Port.
Well, thanks to Kinesis Data Firehose to provide us with pretty much options to cope with this situation. We can ingest the data from many possible AWS services into Kinesis Data Firehose and send that to 3rd party monitoring solution to create some AWSome custom Dashboard and monitor the logs.
I will walk you through step by step to configure this solution in this blog. We can divide this into 3 parts
1. Create Kinesis Data Firehose
2. Create the VPC Flow Logs
3. Transform the Log using the Lambda function (Optional)
4. Send the Logs to NewRelic Monitoring Solution
## Create Kinesis Data Firehose
Create a Kinesis Data firehouse and select Source as **Direct PUT** and Destination as **New Relic**. Please note that Kinesis Data Firehose is near a real-time solution but not the real-time solution as Kinesis.
Under Destination Setting — Select HTTP Endpoint URL as **NEW Relic Log — US**. Enter the API KEY (Copy the API Key from New Relic)
Click on the following URL, this will land you on the NewRelic API-Key screen, as shown below
https://one.newrelic.com/admin-portal/api-keys/home?
## Create VPC Flow Log
Go to VPC → Action and click on Create flow log
Under **Filter**, select whether you only want to monitor the ACCEPTED Traffic, REJECTED Traffic, or ALL Traffic.
Under **Destination**, select **Send to Kinesis Data Firehose in the same account** and select the Kinesis Data Firehose
This will take a few seconds and then you will start seeing the data on the NewRelic platform
If you like this article then don’t forget to hit the clap icon 👏and share it with others. 😏
