Secure AWS EC2 Instances With MFA and SSH Key Rotation

November 11, 2020 Amrit

Idea

The primary idea behind this architecture is to securely access EC2 instances over SSH. There are two parts to this:

Secure SSH access to Bastion host using MFA. This process is explained in this AWS blog.
Public and Private SSH keys to access EC2 instances from the Bastion host are rotated periodically. An AWS Blog (https://aws.amazon.com/blogs/security/how-to-use-aws-secrets-manager-securely-store-rotate-ssh-key-pairs/) was a great help to understand this setup. It has Cloudformation scripts to setup the Bastion Host and other EC2 instances. We made few changes in the code to make it work with our existing setup in AWS.

Entities involved in the setup:

Bastion Host: Bastion is the hop before the other instances. It is placed in a (public) subnet that has an inbound rule of type SSH in the security group of other instances, with the source being either the IP or the security group of the bastion host. I recommend having MFA enabled on Bastion host as an extra level of security.
AWS Security Manager: An AWS service used to store the SSH keys (both Private and Public). Keys are rotated by a Lambda function periodically.
Rotation Lambda Function: Lambda function rotates the SSH key and adds the public key in the ~.ssh/authorized_keys file in the instances having a specific tag. It also adds the public key in an object in AWS S3.
AWS Systems Manager: Rotation Lambda Function uses Systems Manager to run the script to add ssh public key in ~.ssh/authorized_keys.

Lambda Function

Lambda function is in this repository is at this link

VPC

If your Lambda runs in VPC, there could be an issue with connecting to Secrets Manager. For the Lambda function to access the Secrets Manager endpoint add a NAT gateway to your VPC or use VPC Endpoint.
Please refer to https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotation-network-rqmts.html

Deployment Procedure

Steps:

Download the Lambda code from the repository

wget https://github.com/amritsingh/rotate-ssh-aws/archive/master.zip -O rotate_ssh.zipunzip rotate_ssh.zip

Zip the contents of the repository

cd rotate-ssh-aws-master/zip -r rotate-ssh-lambda.zip ./*

Upload rotate-ssh-lambda.zip to S3.
Create a secret in AWS Secrets Manager
- On AWS Secrets Manager console select Store New Secret.
- Select Other type of secrets, select the Plaintext tab and enter {}.
- Change the default encryption key to Customer Master Key.
- In Step 2: Name and Description: Enter the name and description of the key. The name would be like /dev/ssh. For various environments, you can create keys like “/beta/ssh” or “/production/ssh”. Click on Next after adding a description.
- In Step 3: Configure rotation, choose Enable automatic rotation. Change the rotation interval based on your requirements, and select the lambda which we created for rotating the SSH keys. Select Next.
- Review and select Store.

IAM Permissions

We need to set appropriate IAM permissions for the EC2 instances to get the public key from S3 and for the Bastion host to get the private key from the AWS Secrets Manager.

Bastion host IAM policy:

Other EC2 Instances IAM Policy:

SSH to EC2 Instances from Bastion host:

To SSH, Bastion host needs to get the private key from AWS Secrets Manager, use the key to connect to the EC2 Instance.

Give execute access to the script

sudo chmod +x connect.sh

Connecting to an EC2 instance

./connect.sh <<username>> <<ip address>>
# e.g. ./connect.sh ubuntu 10.2.2.21

Adding SSH Public key to Authorized keys on EC2 Instances

Add the following in the home directory of the user.