EC2 Guide: Backing up and clean up (7 / 7)

Logs / Log rotation

Because we changed the paths to our mail logs we need to update our logrotate files. Only update those that have changed:

sudo nano /etc/logrotate.d/rsyslog 
/var/log/postfix/mail.info
/var/log/postfix/mail.warn
/var/log/postfix/mail.err
/var/log/postfix/mail.log
/var/log/dovecot/log
/var/log/dovecot/info
/var/log/dovecot/lda
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript
}

Clean up after ourselves:

sudo rm /etc/logrotate.d/rsyslog~ 

Installing a snapshotting tool (cloud only)

One of the great features of EC2 is the ability to take a snapshot of an entire EBS volume (irrespective of size) in about 3-10 seconds. Automating these calls gives some excellent backup potential. I recommend a tool called ec2-consistent-snapshot by Eric Hammond to assist with backups. This is very easy to get set up:

sudo add-apt-repository ppa:alestic 
sudo aptitude update 
sudo aptitude install ec2-consistent-snapshot 

Configure snapshotting (cloud only)

To make our commands simpler and avoid having config files all over the place we can set some environmental variables about our attached EBS volumes. To do this we need to obtain our attached volume IDs from Elasticfox:

Back on the server we will set these values into the environmental variables:

sudo nano /etc/bash.bashrc 
[...]
export VOLUME_ID_CORE=vol-XXXXXXXX
export VOLUME_ID_WEBSERVER=vol-XXXXXXXX
export VOLUME_ID_EMAIL=vol-XXXXXXXX
export VOLUME_ID_DATABASE=vol-XXXXXXXX
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
sudo nano /etc/profile 
[...]
export VOLUME_ID_CORE=vol-XXXXXXXX
export VOLUME_ID_WEBSERVER=vol-XXXXXXXX
export VOLUME_ID_EMAIL=vol-XXXXXXXX
export VOLUME_ID_DATABASE=vol-XXXXXXXX
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

Reload our shell variables:

source /etc/bash.bashrc 
source /etc/profile 
source ~/.bashrc 

We need to make bash script files to run our commands. Firstly we can backup the database volume, which needs to have the MySQL details passed so the snapshot can be taken safely (without losing data).

sudo mkdir /home/ubuntu/snapshotting 
sudo nano /home/ubuntu/snapshotting/database.sh 
#!/bin/bash -l

ec2-consistent-snapshot --description "Database Backup $(date +'%Y-%m-%d %H:%M:%S')" --region "$EC2_REGION" --mysql --mysql-host "localhost" --mysql-socket "/var/run/mysqld/mysqld.sock" --mysql-username "root" --mysql-password "root_db_password" --xfs-filesystem /database $VOLUME_ID_DATABASE

The webserver backup is similar, but does not require MySQL to be stopped:

sudo nano /home/ubuntu/snapshotting/webserver.sh 
#!/bin/bash -l

ec2-consistent-snapshot --description "Webserver Backup $(date +'%Y-%m-%d %H:%M:%S')" --region $EC2_REGION --xfs-filesystem /webserver $VOLUME_ID_WEBSERVER

The e-mail backup is almost identical to the webserver and does not require MySQL to be stopped:

sudo nano /home/ubuntu/snapshotting/email.sh 
#!/bin/bash -l

ec2-consistent-snapshot --description "E-mail Backup $(date +'%Y-%m-%d %H:%M:%S')" --region $EC2_REGION --xfs-filesystem /email $VOLUME_ID_EMAIL

These files need to be executable, and we should clean up any temporary files our editor may have left.

sudo chmod +x /home/ubuntu/snapshotting/* 
sudo rm /home/ubuntu/snapshotting/*~ 

In order to freeze / unfreeze the XFS file-system the actual commands need to run as root so we will add our commands to root's crontab.

su root 
crontab -e 
0 6 * * * /home/ubuntu/snapshotting/database.sh >> /home/ubuntu/backup.log 2>&1
1 6 * * * /home/ubuntu/snapshotting/webserver.sh >> /home/ubuntu/backup.log 2>&1
2 6 * * * /home/ubuntu/snapshotting/email.sh >> /home/ubuntu/backup.log 2>&1

Leave the root user.

exit 

Cleaning up old snapshots (cloud only)

Amazon don't seem to have a tool to do this neatly, and with a bit of research I found several built in PHP (a language I know) so I decided to roll my own based on a few of these. It's only a rough draft, but meets my current needs.

sudo nano /var/www/remove_old_snapshots.php 
#!/usr/bin/php
<?php
# Options may be hard coded here or passed in at run time
$ec2_private_key	= null; # '/home/ubuntu/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem';
$ec2_cert			= null; # '/home/ubuntu/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem';
$regions			= null; # array('eu-west-1', 'us-east-1'); # OR # 'eu-west-1';
$volumes			= null; # array('vol-XXXXXXXX', 'vol-XXXXXXXX'); # OR # 'vol-XXXXXXXX';
$number_to_keep		= null; # array(7, 1); # OR # 7;
$min_number_to_keep	= 1;	# Prevent any accidents

# Optional list of snapshot IDs to omit from this purge process
$omit = array(); # array('snap-XXXXXXXX', 'snap-XXXXXXXX'); # OR # 'snap-XXXXXXXX';

$options = getopt('K:C:v:n:r:o:sd:ed:');
 
if(!isset($options['v']) OR is_null($options['v']))
{
	if(is_null($volumes))
	{
		echo "ERROR: Please provide at least one volume to check (-v option)\n";
		exit;
	}
	else
	{
		$options['v'] = $volumes;
	}
}
 
if(!isset($options['n']) OR is_null($options['n']))
{
	if(is_null($number_to_keep))
	{
		echo "ERROR: Please provide number of snapshots to keep (-n option)\n";
		exit;
	}
	else
	{
		$options['n'] = $number_to_keep;
	}
}
 
if(!isset($options['r']) OR is_null($options['r']))
{
	if(is_null($regions))
	{
		echo "ERROR: Please provide a region for these volumes (-n option)\n";
		exit;
	}
	else
	{
		$options['r'] = $regions;
	}
}
 
if(!isset($options['K']) OR is_null($options['K']))
{
	if(is_null($ec2_private_key))
	{
		echo "ERROR: Please provide an EC2 key file location for these volumes (-K option)\n";
		exit;
	}
	else
	{
		$options['K'] = $ec2_private_key;
	}
}
 
if(!isset($options['C']) OR is_null($options['C']))
{
	if(is_null($ec2_cert))
	{
		echo "ERROR: Please provide an EC2 Cert file location for these volumes (-C option)\n";
		exit;
	}
	else
	{
		$options['C'] = $ec2_cert;
	}
}
 
if(!isset($options['o']) OR is_null($options['o']))
{
	$options['o'] = $omit;
}
 
if(!is_array($options['v']))
{
	$volumes = array($options['v']);
}
else
{
	$volumes = $options['v'];
}
 
if(!is_array($options['n']))
{
	$number_to_keep = array($options['n']);
}
else
{
	$number_to_keep = $options['n'];
}
 
if(!is_array($options['r']))
{
	$regions = array($options['r']);
}
else
{
	$regions = $options['r'];
}
 
if(!is_array($options['K']))
{
	$ec2_private_key = $options['K'];
}
else
{
	echo "ERROR: Please provide a SINGLE EC2 key file location for these volumes (-K option)\n";
	exit;
}
 
if(!is_array($options['C']))
{
	$ec2_cert = $options['C'];
}
else
{
	echo "ERROR: Please provide a SINGLE EC2 Cert file location for these volumes (-C option)\n";
	exit;
}
 
if(!is_array($options['o']))
{
	$omit = array($options['o']);
}
else
{
	$omit = $options['o'];
}
 
foreach($number_to_keep as $key => $value)
{
	if($value < $min_number_to_keep)
	{
		$number_to_keep[$key] = $value;
	}
}
 
$total_volumes	= count($volumes);
$total_kept		= count($number_to_keep);
$total_regions	= count($regions);
 
if($total_volumes != $total_kept)
{
	if($total_volumes > $total_kept AND $total_kept == 1)
	{
		# Use the number to keep as a global value
		foreach($volumes as $key => $value)
		{
			$number_to_keep[$key] = $number_to_keep[0];
		}
	}
	else
	{
		# An error is likely, we should stop
		echo "ERROR: Please check the values entered, there is an argument mismatch (-v vs. -n)\n";
		exit;
	}
}
 
if($total_volumes != $total_regions)
{
	if($total_volumes > $total_regions AND $total_regions == 1)
	{
		# Use the region as a global value
		foreach($volumes as $key => $value)
		{
			$regions[$key] = $regions[0];
		}
	}
	else
	{
		# An error is likely, we should stop
		echo "ERROR: Please check the values entered, there is an argument mismatch (-v vs. -r)\n";
		exit;
	}
}
 
foreach($regions as $key => $region)
{
	$process[$region][$volumes[$key]] = $number_to_keep[$key];	
}
unset($number_to_keep, $regions);
 
$raw = array();
foreach($process as $region => $details)
{
	$tmp = array();
	$cmd = 'ec2-describe-snapshots -K '.$ec2_private_key.' -C '.$ec2_cert.' --region '.$region;
	exec($cmd, $tmp);
 
	if(is_array($tmp))
	{
		foreach($tmp as $key => $value)
		{
			$tmp[$key] .= "\t".$ec2_private_key;
			$tmp[$key] .= "\t".$ec2_cert;
			$tmp[$key] .= "\t".$region;
		}
 
		$raw = array_merge($raw, $tmp);
	}
}
 
$found = array();
foreach($raw as $snapshot)
{
	$tmp = split("\t", $snapshot);
	if(in_array($tmp[2], $volumes) AND !in_array($tmp[1], $omit))
	{
		$tmp[] = $process[$tmp[11]][$tmp[2]];
		$found[$tmp[2]][strtotime($tmp[4])] = $tmp;
	}
}
 
foreach($found as $volume_id => $snapshots)
{
	krsort($found[$volume_id]);
}
 
unset($volumes, $process);
 
foreach($found as $volume_id => $snapshots)
{
	$total = 0;
	foreach($snapshots as $timestamp => $details)
	{
		if($total > $details[12] - 1)
		{
			echo 'Deleting Snapshot: '.$details[1]."\n";
			$out = array();
			$cmd = 'ec2-delete-snapshot -K '.$details[9].' -C '.$details[10].' --region '.$details[11].' '.$details[1];
			exec($cmd, $out);
 
			if(count($out) == 1 AND strcmp($out[0], 'SNAPSHOT	'.$details[1]) == 0)
			{
				echo "COMPLETE!\n";
			}
			else
			{
				echo "ERROR:\n";
				foreach($out as $line)
				{
					echo $line."\n";
				}
			}
		}
 
		$total++;
	}
}

We need to trigger this as a CLI process so we will wrap its call in a bash wrapper just like the other scripts.

sudo nano /home/ubuntu/snapshotting/remove-old.sh 
#!/bin/bash -l

php /var/www/remove_old_snapshots.php -v $VOLUME_ID_WEBSERVER -v $VOLUME_ID_EMAIL -v $VOLUME_ID_DATABASE -n 7 -r $EC2_REGION -K $EC2_PRIVATE_KEY -C $EC2_CERT

The -n value is the number of snapshots of each volume to keep. If you would like to keep different numbers for each specify -n 3 times (one for each). Again this needs to be executable and our editor may have left a temporary file to clean up:

sudo chmod +x /home/ubuntu/snapshotting/* 
sudo rm /home/ubuntu/snapshotting/*~ 

To remain consistent we will add our commands to root's crontab.

su root 
crontab -e 
[...]
3 6 * * * /home/ubuntu/snapshotting/remove-old.sh >> /home/ubuntu/backup.log 2>&1

Leave the root user.

exit 

Hopefully that is everything you will need to do to set up a secure server, so I wish you the best of luck with yours! All comments, criticisms and suggestions welcome, but please comment on the relevant pages to make them as useful as possible, thanks!

Download this guide

To make this guide more useful I've added a feature to allow you to save it offline in a simple HTML format. If you have not customised this guide to your own values you may wish to do so here before you download it. There are a few options here:

Guide contents

  1. Hosting a website on Amazon EC2 - The goals and assumptions of this guide
  2. Preparing required tools - Create an AWS account, configure Elastic Fox and add an SSH tool
  3. Customise this guide - Allow all commands to be tailored to you (optional)
  4. Core software installation - Install some common software to the server image
  5. Depending upon your chosen configuration there is a choice here:
    1. Create and attach new EBS volumes - New server that you may want to split in future
    2. Attach existing EBS volumes - If you have used this guide before and have EBS volumes
    3. No attached EBS volumes - If you are not using the cloud or don't want to use them
  6. Depending upon your chosen configuration there is another choice here:
    1. Software Configuration - Set up the system to work as a multi-function server (from 5a or 5c)
    2. Software Configuration from existing EBS volumes - Use settings from EBS volumes (from 5b)
  7. Backing up and clean up - Configure Crons, log rotation etc