AWS CLI

Purpose

To manage AWS resources remotely with scripts.

Initial use case is to create and manage snapshots of EC2 instances. I want to create new snapshots then delete old ones. This could be managed internally with AWS, but it looks like that would cost more and I don't mind learning something new.

Process

Listing EC2 Instances

I need to list the instances and parse that list so I know what I am working with. Though not needed for creating snapshots, this was helpful in learning how aws-cli functions. I started with a serverfault post and broke down the steps contained within. The example provided was

aws ec2 describe-instances --filters Name=vpc-id,Values=vpc-e2f17e8b --query 'Reservations[].Instances[].Tags[?Key==`Name`].Value[]'

But the query statement didn't make much sense to me. At this point I started to replicate this in my lab. The key here is that you want to look at the output of the standard describe-instances command

{
    "Reservations": [
        {
            "Instances": [
                {
                    "Monitoring": {
                        "State": "disabled"
                    }, 
                    "PublicDnsName": "ec2-xxx-xxx-xxx-xxx.us-east-2.compute.amazonaws.com", 
                    "StateReason": {
                        "Message": "Client.UserInitiatedShutdown: User initiated shutdown", 
                        "Code": "Client.UserInitiatedShutdown"
                    }, 
                    "State": {
                        "Code": 80, 
                        "Name": "stopped"
...
                    "InstanceId": "i-xxxxxxxxxxad6183c",
...

                    "Tags": [
                        {
                            "Value": "My-VM", 
                            "Key": "Name"
                        }
                    ], 

Since the information I need is nested, I will need to drill down. Starting with Reservations, then Instances, I can then select the information I need.
NOTE: This is case sensitive

[root@aws-cli ~]# aws ec2 describe-instances --query 'Reservations[].Instances[].{Instance_name:Tags[?Key==`Name`].Value,ID:InstanceId,State:State.Name,Volume:BlockDeviceMappings[].Ebs.VolumeId}'
[
    {
        "Instance_name": [
            "My-VM"
        ],
        "Volume": [
            "vol-xxxxxxxxxxxxxx142"
        ], 
        "State": "stopped", 
        "ID": "i-xxxxxxxxxxad6183c"
    }
] 

To break this down:

• "Reservations[]" This will query ALL reservations.
• "Instances[]" This will query ALL instances
• "{}" This is creating an array, since we want multiple values found inside of Instances.
• "Instance_name" is an arbitrary name, you can put anything you want here without spaces. There might be a way to use spaces, but you shouldn't use them anyway.
• ":Tags[?Key==`Name`].Value" I do not fully understand this as of yet. However I needed it to parse the human readable name I gave the instance.
• "ID" is an arbitrary name.
• ":InstanceId" will pull the instance ID.
• "State" is an arbitrary name.
• ":State.Name" will pull the human readable state of the instance. In this case "Stopped".
• "Volume" is an arbitrary name.
• ":BlockDeviceMappings[].Ebs.VolumeId" Will grab the VolumeIDs that we will need later.

To only list Instance IDs for processing

[root@aws-cli ~]# aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text | sed -e 's/\s\+/\n/g'
i-xxxxxxxxxxad6183c
i-xxxxxxxxxxad6345c

To grab the associated volume IDs

[root@aws-cli ~]# aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text | sed -e 's/\s\+/\n/g' | while read line; do aws ec2 describe-instances --instance-ids "$line" --query 'Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId' --output text| sed -e 's/\s\+/\n/g' ; done
vol-xxxxxxxxxxxxxx142
vol-xxxxxxxxxxxxxxc0f

Creating Snapshots

According to Amazon, you want to stop the instance before taking a snapshot to ensure the state is clean. We will see about that. If you have time for that go for it! Otherwise we will look at making snapshots of live root volumes.

The other thing we will be doing is creating snapshots based on tags. I don't want to snapshot all machines, just the critical ones I get yelled at if they are not working.

#!/bin/bash
now=$(date +%s)
tag="My_Tag"
aws ec2 describe-instances --filter "Name=tag-value,Values=$tag" --query 'Reservations[].Instances[].{ID:InstanceId}' --output text | while read line;
do id=$line;
vol=$(aws ec2 describe-instances --instance-ids "$id" --query 'Reservations[].Instances[].{Volume:BlockDeviceMappings[].Ebs.VolumeId}' --output text | awk '{print $2}');
name=$(aws ec2 describe-instances --instance-ids "$id" --query 'Reservations[].Instances[].{Instance_name:Tags[?Key==`Name`].Value}' --output text | awk '{print $2}');
snapid=$(aws ec2 create-snapshot --description "$name $id $now" --volume-id $vol | grep -oE snap-[0-9a-z]+)
aws ec2 create-tags --resources $snapid --tags Key=function,Value=$tag Key=source,Value=$id Key=creation_date,Value=$now;
done

Deleting Snapshots

Since you have to pay for the storage of these snapshots, you probably want to delete old ones. We want to search through the snapshots using tags, then delete ones older than a set time. In my case I want to delete any older than two weeks. All my time stamps are in seconds making math easy.

#!/bin/bash
now=$(date +%s)
tag="My_Tag"
aws ec2 describe-snapshots --filters Name=tag-value,Values=$tag --query 'Snapshots[].{ID:SnapshotId}' --output text | while read line;
do snapid=$line;
cdate=$(aws ec2 describe-snapshots --snapshot-id $snapid --query 'Snapshots[].Tags[?Key==`creation_date`]' --output text | awk '{print $2}');
diff=$(($now - $cdate));
if [ "$diff" > "1209600" ];
then aws ec2 delete-snapshot --snapshot-id $snapid;
fi;
done;

Restoring Snapshots

The process for restoring a snapshot for a live system is

• Create a volume from the snapshot.
• Attach the volume to a new instance, the same type as the one being replaced.
• Boot the new image, and ensure it is responding correctly.
• Re-assign the elastic IP to the new instance.

Easy enough using the web gui, but that could be prone to error and it takes longer than it should. Since I have daily snapshots taken the following will pull from the last snapshot and restore. Alternatively an option can be added to select from which day to restore.

Syntax as follows

[root@aws-cli ~]# siterestore.sh yoursite.tld

The script being built is as follows.

#!/bin/bash

##Grab the site name from standard input.
sitename=$1
##Run query for the site data.
sitequery=$(aws ec2 describe-instances --filter "Name=tag-value,Values=$1" --query 'Reservations[].Instances[]' --output text)
if [ -z "$sitequery" ]; then
	echo "$1 not found. Check for typos"
else
	values=$(aws ec2 describe-instances --filter "Name=tag-value,Values=$1" --query 'Reservations[].Instances[].{IP:PublicIpAddress,VPC:VpcId,ID:InstanceId,Type:InstanceType,AZ:Placement.AvailabilityZone,Group:SecurityGroups[].GroupName,VOLID:BlockDeviceMappings[].Ebs.{ID:VolumeId}}')
	ipaddress=$(grep -oE "\"IP\":\ \"[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\"" <<<$values | awk 'BEGIN {FS="\""}{print $4}')
	type=$(grep -Eo "\"Type\": \"[a-z][0-9].[a-z]+\"" <<<$values | awk 'BEGIN {FS="\""}{print $4}')
	az=$(grep -Eo "\"AZ\": \"[a-z0-9-]+\"" <<<$values| awk 'BEGIN {FS="\""}{print $4}')
	security=$(grep -Eo "\"Group\":\ \[\ \"[A-Za-z]+\"\ \]" <<<$values | awk 'BEGIN {FS="\""}{print $4}')
	id=$(grep -Eo "\"ID\":\ \"i-[0-9a-z]+\"" <<<$values | awk 'BEGIN {FS="\""}{print $4}')
	volid=$(grep -Eo "\"VOLID\":\ \[\ \{\ \"ID\":\ \"vol-[0-9a-z]+\"" <<<$values | awk 'BEGIN {FS="\""}{print $6}')


#Search for last snapshot created for this site.
	dayago=$(date +%Y-%m-%d --date=yesterday)
	volvalues=$(aws ec2 describe-snapshots --filter "Name=volume-id,Values=$volid" "Name=start-time,Values=$dayago*" --query 'Snapshots[].{State:State,CreationDate:StartTime,SnapID:SnapshotId}')
	snapid=$(grep -Eo "\"SnapID\":\ \"snap-[0-9a-z]+" <<<$volvalues | awk 'BEGIN {FS="\""}{print $4}')

#Create volume from snapshot
	volvalues=$(aws ec2 create-volume --availability-zone $az --snapshot-id $snapid --volume-type gp2 --tag-specifications "ResourceType=volume,Tags=[{Key=Name,Value=$1}]")
	volidnew=$(grep -Eo "\"VolumeId\":\ \"vol-[0-9a-z]+\"" <<<$volvalues | awk 'BEGIN {FS="\""}{print $4}')

#Check that volume is created
	volstatus=$(aws ec2 describe-volumes --filters Name=volume-id,Values=$volidnew --query 'Volumes[].State' | grep -Eo [a-z]+)
	if [ -z "$volstatus" ]; then
		echo "Can not determine volume creation status. Stopping."
	else
		until [ "$volstatus" = "available" ]; do
			volstatus=$(aws ec2 describe-volumes --filters Name=volume-id,Values=$volidnew --query 'Volumes[].State' | grep -Eo [a-z]+)
			if [ -z "$volstatus" ]; then
				echo "Can not determine volume creation status. Stopping."
				exit 1
			fi
			sleep=3
		done
	fi

#Create instance for the new volume to use



#Clear variables just to make sure we don't break something.
unset ipaddress
unset type
unset az
unset security
unset id
unset volid
unset snapid
unset volvalues
unset volidnew
unset volstatus
fi