How To Monitor Your Services Hosted On AWS EC2 Instances
A secure way to implement a Lambda function that monitors all the services hosted on EC2 instances using Secrets Manager
Problem
AWS provides many tools to monitor its own services. You can add alarms on various CloudWatch events. As you host more and more services on AWS it becomes difficult and important to make sure these services are running. You can use CloudWatch to find the state of your services by creating new Metrics. This process becomes expensive and difficult to manage after a point.
Solution
A Lambda function monitors all the services running on EC2 instances securely. Here is the proposed architecture:
Architecture
The primary idea behind this architecture is to securely access EC2 instances over SSH and monitor the services. The SSH keys are rotated periodically.
Entities involved in the setup:
- EC2 Instances: These are the instances on which your services are running.
- Monitoring Lambda: This Lambda function is responsible for securely logging in using SSH to various instances, find the state of the services, and raise alarms if something is broken.
- AWS Security Manager: An AWS service used to store the SSH keys (both Private and Public). Keys are rotated by the Key Rotation Lambda function periodically.
- Key Rotation Lambda: Lambda function rotates the SSH key and adds the public key in the ~.ssh/authorized_keys file in the instances having a specific tag. It also adds the public key in an object in AWS S3.
- AWS Systems Manager: Rotation Lambda Function uses Systems Manager to run the script to add ssh public key in ~.ssh/authorized_keys.
Steps
Create a Key Rotation Lambda using CloudFormation
In this step, we use CloudFormation to create a Lambda Function that rotates the SSH key and adds the public key in the ~.ssh/authorized_keys file in the instances. The public key is put to an object in AWS S3.
Download the Lambda code from this repository
$ wget https://github.com/amritsingh/rotate-ssh-aws/archive/master.zip -O rotate_ssh.zip
$ unzip rotate_ssh.zip
Zip the contents of the repository
$ cd rotate-ssh-aws-master/
$ zip -r rotate-ssh-lambda.zip ./*
Upload rotate-ssh-lambda.zip to S3.
cloudformation.yaml is a Cloud Formation script to create the Lambda Function in AWS. In this file, change a few of the parameters (look for FIXME) based on your setup before providing the script to AWS Cloud Formation.
cloudformation.yaml needs to be configured with
- S3 Bucket where Public key will be stored
- S3 object where Public key will be stored
- Usernames for the Linux User that is used to log into the EC2 Instances
- S3 URI of Lambda code uploaded by you
- ARN of S3 Bucket where Public key will be stored, Security Group IDs and Subset IDs.
Goto CloudFormation in AWS console. Click on “Create Stack”.
Select the option to “Upload a template file”, choose cloudformation.yaml from your machine which we altered in the previous step, and click “Next”.
Enter the “Stack name” and change the parameters if required, and click “Next”
In Step 3 and Step 4 go with default options and create the stack.
Wait for a while till the stack creation is complete.
Create a secret in AWS Secrets Manager
On the AWS Secrets Manager console select “Store a New Secret”.
Select “Other type of secrets”, select the Plaintext tab and enter {}. Change the default encryption key to Customer Master Key.
In Step 2: Name and Description: Enter the name and description of the key. The name would be like /monitoring/ssh. Click on Next after adding a description.
In Step 3: Configure rotation, choose Enable automatic rotation. Change the rotation interval based on your requirements, and select the lambda which we created for rotating the SSH keys. Select Next.
In Step4, Review and select Store.
Give access to the S3 Bucket where the SSH public key resides to the IAM Roles you use for your EC2 instances
Alter the deployment of EC2 instances to pull the public key from the S3 bucket on a periodic basis
We created a cron job that pulls the SSH public key from S3 at the period of 15 minutes.
Bash Script which is run by the crontab looks like thishttps://levelup.gitconnected.com/media/9c8c2f771959387767dbdda794a4cce0
Setup a Lambda function that connects to EC2 instances over SSH and monitors the services
Here comes the last part of the complete setup. This is the code that you will have to alter whenever you want to monitor a new service.
I recommend using Python for this purpose. The Lambda function should have access to the Private key in Secrets Manager. It performs the following tasks:
- Download the Private Key from Secrets Manager.
- Use Paramiko to log in to EC2 machines.
- Monitor the state of the services and send AWS SNS alerts if required.
Wrapping it Up
Finally, after all these steps, you have your own custom monitor setup. This monitor is easy to scale to all your services. Each time you add a service to your backend you need to add the service in the monitoring Lambda. All the other parts of the setup remain the same.