AWS CLI

Purpose

To manage AWS resources remotely with scripts.

Initial use case is to create and manage snapshots of EC2 instances. I want to create new snapshots then delete old ones. This could be managed internally with AWS, but it looks like that would cost more and I don't mind learning something new.

Process

Listing EC2 Instances

I need to list the instances and parse that list so I know what I am working with. Though not needed for creating snapshots, this was helpful in learning how aws-cli functions. I started with a serverfault post and broke down the steps contained within. The example provided was

aws ec2 describe-instances --filters Name=vpc-id,Values=vpc-e2f17e8b --query 'Reservations[].Instances[].Tags[?Key==`Name`].Value[]'

But the query statement didn't make much sense to me. At this point I started to replicate this in my lab. The key here is that you want to look at the output of the standard describe-instances command

{
    "Reservations": [
        {
            "Instances": [
                {
                    "Monitoring": {
                        "State": "disabled"
                    }, 
                    "PublicDnsName": "ec2-xxx-xxx-xxx-xxx.us-east-2.compute.amazonaws.com", 
                    "StateReason": {
                        "Message": "Client.UserInitiatedShutdown: User initiated shutdown", 
                        "Code": "Client.UserInitiatedShutdown"
                    }, 
                    "State": {
                        "Code": 80, 
                        "Name": "stopped"
...
                    "InstanceId": "i-xxxxxxxxxxad6183c",
...

                    "Tags": [
                        {
                            "Value": "My-VM", 
                            "Key": "Name"
                        }
                    ], 

Since the information I need is nested, I will need to drill down. Starting with Reservations, then Instances, I can then select the information I need.
NOTE: This is case sensitive

[root@aws-cli ~]# aws ec2 describe-instances --query 'Reservations[].Instances[].{Instance_name:Tags[?Key==`Name`].Value,ID:InstanceId,State:State.Name,Volume:BlockDeviceMappings[].Ebs.VolumeId}'
[
    {
        "Instance_name": [
            "My-VM"
        ],
        "Volume": [
            "vol-xxxxxxxxxxxxxx142"
        ], 
        "State": "stopped", 
        "ID": "i-xxxxxxxxxxad6183c"
    }
] 

To break this down:

• "Reservations[]" This will query ALL reservations.
• "Instances[]" This will query ALL instances
• "{}" This is creating an array, since we want multiple values found inside of Instances.
• "Instance_name" is an arbitrary name, you can put anything you want here without spaces. There might be a way to use spaces, but you shouldn't use them anyway.
• ":Tags[?Key==`Name`].Value" I do not fully understand this as of yet. However I needed it to parse the human readable name I gave the instance.
• "ID" is an arbitrary name.
• ":InstanceId" will pull the instance ID.
• "State" is an arbitrary name.
• ":State.Name" will pull the human readable state of the instance. In this case "Stopped".
• "Volume" is an arbitrary name.
• ":BlockDeviceMappings[].Ebs.VolumeId" Will grab the VolumeIDs that we will need later.

To only list Instance IDs for processing

[root@aws-cli ~]# aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text | sed -e 's/\s\+/\n/g'
i-xxxxxxxxxxad6183c
i-xxxxxxxxxxad6345c

To grab the associated volume IDs

[root@aws-cli ~]# aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text | sed -e 's/\s\+/\n/g' | while read line; do aws ec2 describe-instances --instance-ids "$line" --query 'Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId' --output text| sed -e 's/\s\+/\n/g' ; done
vol-xxxxxxxxxxxxxx142
vol-xxxxxxxxxxxxxxc0f

Creating Snapshots

According to Amazon, you want to stop the instance before taking a snapshot to ensure the state is clean. We will see about that. If you have time for that go for it! Otherwise we will look at making snapshots of live root volumes.

The other thing we will be doing is creating snapshots based on tags. I don't want to snapshot all machines, just the critical ones I get yelled at if they are not working.

#!/bin/bash
now=$(date +%s)
aws ec2 describe-instances --filter "Name=tag-value,Values=My_Tag" --query 'Reservations[].Instances[].{ID:InstanceId}' --output text | while read line;
do id=$line;
vol=$(aws ec2 describe-instances --instance-ids "$id" --query 'Reservations[].Instances[].{Volume:BlockDeviceMappings[].Ebs.VolumeId}' --output text | awk '{print $2}');
name=$(aws ec2 describe-instances --instance-ids "$id" --query 'Reservations[].Instances[].{Instance_name:Tags[?Key==`Name`].Value}' --output text | awk '{print $2}');
snapid=$(aws ec2 create-snapshot --description "$name $id $now" --volume-id $vol | grep -oE snap-[0-9a-z]+)
aws ec2 create-tags --resources $snapid --tags Key=function,Value=web Key=source,Value=$id Key=creation_date,Value=$now;
done

Deleting Snapshots

Since you have to pay for the storage of these snapshots, you probably want to delete old ones. We want to search through the snapshots using tags, then delete ones older than a set time. In my case I want to delete any older than two weeks. All my time stamps are in seconds making math easy.

#!/bin/bash
now=$(date +%s)
tag="My_Tag"
aws ec2 describe-snapshots --filters Name=tag-value,Values=$tag --query 'Snapshots[].{ID:SnapshotId}' --output text | while read line;
do snapid=$line;
cdate=$(aws ec2 describe-snapshots --snapshot-id $snapid --query 'Snapshots[].Tags[?Key==`creation_date`]' --output text | awk '{print $2}');
diff=$(($now - $cdate));
if [ "$diff" > "1209600" ];
then aws ec2 delete-snapshot --snapshot-id $snapid;
fi;
done;