Ubuntu 10.04 EC2 Guide (Apache / MySQL / PHP / Postfix / Dovecot)

EC2 Guide: Hosting a website on Amazon EC2 (1 / 7)

The Amazon Elastic Compute Cloud (Amazon EC2) is a web-service allowing companies or individuals to create / run / manage virtual servers (called instances) on their hardware infrastructure. It allows a choice of most operating systems and a range of virtual hardware configurations, so if you wanted lots of processing power you could choose a High CPU instance or for a database server you may choose a High Memory instance. You can of course change instance type with ease. To make this better it allows you to only pay for what you use by the hour and allows creation / destruction of new instances in seconds. It also offers extreme scalability, data storage and fault tolerance. In short it's a pretty good service!

Until relatively recently (Dec 2009) the instances were not persistent, that is any data you required on a server had to be (re-)created (by scripts) each time a machine was created / rebooted or crashed. This was, and still is, fine for compute intensive batch jobs such as photo processing or number crunching where you create a machine instance, pass it some data and it will give you back the output. That was (essentially) the end of the server's life. Obviously for a website this kind of arrangement is far from ideal. Thankfully Amazon addressed this issue and created a product called Elastic Block Store (EBS) for this task. A year on and many gremlins have been ironed out and this is now suitable for production website deployment.

Aims of this guide

This guide is not solely dedicated to creating an EBS backed Amazon EC2 instance, that would take a few seconds and wouldn't really help many people. It is instead designed to guide a relative novice through all the steps required to configure a production Ubuntu 10.04 Lucid Lynx (LTS) server running in the cloud environment. Much of this guide will work on other Linux flavours with certain changes, but I have only tested it on Ubuntu 10.04. Debian based systems will require relatively minor tweaks while RPM / Gentoo / Slackware based distros would require significant code updates (although conceptually it is similar). Likewise it is not required that your server is running in the cloud, but there is some cloud specific content in the guide.

Server Configuration and Scalability

With a small website it is likely that you will only need one server and the size of that will be dictated by your site traffic / server load. As a site grows and more demand is put on the server there eventually becomes a point where you have to make a change. There are a couple of options here:

Even if you have a site with low traffic it's always prudent to plan for the best and have an idea of what to do when the day comes where your server can no longer cope. Amazon EC2 allows an upgrade to a larger instance very quickly, but ultimately you run out of larger instances to use when you hit their Extra Large Instance and become stuck with migration difficulties.

Again Amazon EC2 offers a unique solution to this issue, their EBS storage. It doesn't just have to be for machine images, EBS volumes can be used essentially as detachable hard disks. This means that you can separate all core (config and data) files for the web-server (e.g. Apache / PHP, site data, vhosts etc), mail-server (e.g. Postfix / Dovecot, e-mails etc) and a database server (e.g. config and data) onto separate "drives" and if the situation ever arises where the server cannot cope you simply detach the required EBS volume from one instance and attach it to another making minimal configuration changes. You can of course copy EBS volumes quickly too and Amazon offers a load balancer if multiple web-servers are used.

Obviously I don't want to force this choice on anyone (because it does cost a few dollars a month extra) so I have set this up in such a way as it is completely optional to to this. Just follow the relevant section of the guide to your needs.

Requirements

EC2 Guide: Preparing required tools (2 / 7)

If you are using Amazon EC2 the following steps are necessary, but if not (i.e. you are just following this guide to set up a server) please skip this page and install Ubuntu 10.04 Lucid Lynx (LTS) from a DVD.

Register with Amazon AWS

I'm not going to go into detail for the actual registration of services, but you need a credit / debit card, a valid e-mail address and a phone(!). It takes a while to get all of it done / activated, and there are several services you can use, but for now only EC2 needs configuring.

There are several pieces of information that are required to manage your instances and they are configured / obtained in the Security Credentials section of your Amazon account. You will need to create and write down / save:

Install Firefox / Elasticfox

If you don't use Firefox then please download the latest version. Because Amazon haven't maintained Elasticfox for quite some time I no longer use the official one, instead I recommend you install this enhancement of Elasticfox which is a simple and bug free tool to manage all things AWS. To add it to Firefox:

Configure Elastic Fox

Launch Elasticfox by opening Firefox and selecting Tools -> Elasticfox - N.B. it is not under the Ad-ons section, it has it's own button. You should be prompted to enter your AWS credentials. You can enter new details or modify existing ones by clicking on the "Credentials" button in the top centre of the Elasticfox window:

You now need to create a Key Pair:

Download PuTTY / PuTTYgen (Windows only)

To manage your instance you will need and SSH tool and for Windows systems that basically means PuTTY, so please grab a copy, it's free (and legal in most countries - please check yours!).

The Key Pair file that you created earlier (mykey.pem) is sadly not compatible with PuTTY, but it does have a tool to convert it into a usable format called PuTTYgen. Download this too, run it, load in your key file (you might need to select "All Files" under the "Files of Type" dropdown) and press Generate. Save this new key (mykey.ppk) with your original (mykey.pem) - don't delete either!

Configure Elasticfox SSH

Launch Elasticfox and visit the Tools panel in the top right of the Elasticfox window.

Windows

Update the following:

Linux / OSX

Update the following:

Choose a Region and Availability Zone

These should be a reasonably easy choice, but still one that needs making. There are currently 15 main regions each with several availability zones:

Now, if you were using this in an enterprise environment you would probably want multiple servers in at least two of these regions so that if one malfunctioned the other could take over sole operation. However, given we are interested in a single server set up for now, it's best just to choose a region / zone nearest to you / your audience and stick with it. In Elasticfox just choose your option from the top left drop down menu.

Define Security Group(s)

A Security Group is a set of rules applied to an instance defining what ports are available to who. In general this boils down to 5 options per port, it is either: closed, open to the internal Amazon network, open to an IP of your choice, open to everyone or open to a combination of the last 3 options. It is easiest to understand with a couple of examples:

  1. Apache (the web-server) allows people to view web pages on port 80 (http://) or port 443 (https://) by default. It is likely that you want both of these options available to everybody, so you would set unrestricted access (0.0.0.0/0) to these two ports.
  2. SSH (the secure shell) allows you to connect to your server to manage it directly on port 22 by default. It is likely that you want this option only available to you on your IP only, so you would restrict access to only your IP address (18.232.99.123/32) on this port
  3. If you had a separate MySQL database server it would accept requests on port 3306 by default. You would need to let your web-server connect to it, but in this instance it would be wise to restrict access to only internal cloud machines (10.0.0.0/8). You could also allow your IP (18.232.99.123/32) direct access or of course allow everyone (0.0.0.0/0) if you had a non-cloud based server requiring access.

You only need one security group, but as mentioned previously there may come a time when you would like to separate your server into different parts. So I created 5 security groups which would allow a combo server set up (add all 5 groups) or individualised servers (with only the necessary groups added). My groups were:

Web-server

Protocol From Port / ICMP Type To Port / ICMP Code Source User: Group Source CIDR
tcp 80 80 0.0.0.0/0
tcp 443 443 0.0.0.0/0

Database Server

Protocol From Port / ICMP Type To Port / ICMP Code Source User: Group Source CIDR
tcp 3306 3306 10.0.0.0/8

E-mail Server

Protocol From Port / ICMP Type To Port / ICMP Code Source User: Group Source CIDR
tcp 25 25 0.0.0.0/0
tcp 110 110 0.0.0.0/0
tcp 143 143 0.0.0.0/0
tcp 465 465 0.0.0.0/0
tcp 993 993 0.0.0.0/0
tcp 995 995 0.0.0.0/0

SSH / SFTP / RDP

Protocol From Port / ICMP Type To Port / ICMP Code Source User: Group Source CIDR
tcp 22 22 18.232.99.123/32
tcp 3389 3389 18.232.99.123/32

FTP (optional)

Protocol From Port / ICMP Type To Port / ICMP Code Source User: Group Source CIDR
tcp 21 22 18.232.99.123/32

You can then load these in any combination you like to create servers fit for most purposes.

Assign and Whitelist an Elastic IP

Your machine will need an ip address (X.X.X.X) so that you can configure your zone files and point your domain to it. This is very easy to do in Elasticfox:

Until relatively recently Amazon had real trouble with ip blacklisting (especially bad for e-mails), but they now have a process in place which whitelists a new IP for you within a few days. For this to happen you need to fill out a use case request form to remove sending restrictions. It only takes a minute or two to inform them of your intentions and get your IP whitelisted.

Choose an Instance Size and AMI file

Before you can launch an instance you must first decide which type you need, there are many available and all have different costs associated with them so it's best not to be greedy! One key matter to consider is that some are 32 bit and some are 64 bit. This will dictate which AMI you choose to run.

Once you have made your choice you can visit Ubuntu's AMI list and choose the right one for you. Choose the correct (either 64 bit or 32 bit) EBS backed version for your chosen region and copy the AMI reference down.

Launch an AMI

Launching an AMI file is quite straightforward. In Elasticfox:

Associate your Elastic IP

Connect to your Instance

You should now see a terminal window open and be logged in to your server!

EC2 Guide: Core Software Installation (4 / 7)

The image that we are starting with is extremely minimal, and while it is configured to allow a rapid setup for a certain server type (e.g. a web server / mail server), it is often better to install exactly what you need on a machine and nothing more. This process will install everything needed to create a web-server / database server / e-mail server combo box. Of course you could install only the parts applicable to your server if it is a sole purpose box.

Install some basic apps/utilities

We need the following very basic utilities for a multitude of tasks so they go on first.

sudo aptitude install ssh openssh-server nano 

Set a Root User Password

By default the root account has no password, but because it is sometimes advantageous to run as root (not just sudo) it's a good idea to set one. Just don't forget it! (I use my IronKey for this kind of information storage)

sudo passwd 

Set an Ubuntu User Password

The default system user is called "ubuntu" and has a random password assigned to it. There are arguments for and against changing this, but since I use it as my default user and don't always want to count on having my AWS key file handy, I set this. Again, write it down somewhere safe.

sudo passwd ubuntu 

Create an FTP / SFTP User

It's often convenient to be able to move data onto a server (especially for a web server), so we'll add a user just for this.

sudo useradd ftp-user -d /home/ftp-user 
sudo passwd ftp-user 
sudo adduser ftp-user www-data 

Disable root SSH / allow SSH/SFTP with a password

There's no reason to allow the root user to log in via SSH (even if it is IP locked). We also want our other users to be able to log in with a password.

sudo nano /etc/ssh/sshd_config 

Uncomment / update the necessary lines:

[...]
PermitRootLogin no
[...]
PasswordAuthentication yes
[...]

Apply the changes:

sudo /etc/init.d/ssh restart 

Configure aptitude (package manager)

Remove the CD as a source and enable main, universe and multiverse support.

sudo nano /etc/apt/sources.list 
deb http://eu-west-2.ec2.archive.ubuntu.com/ubuntu/ lucid main universe multiverse
deb-src http://eu-west-2.ec2.archive.ubuntu.com/ubuntu/ lucid main universe multiverse
deb http://eu-west-2.ec2.archive.ubuntu.com/ubuntu/ lucid-updates main universe multiverse
deb-src http://eu-west-2.ec2.archive.ubuntu.com/ubuntu/ lucid-updates main universe multiverse
deb http://security.ubuntu.com/ubuntu lucid-security main universe multiverse
deb-src http://security.ubuntu.com/ubuntu lucid-security main universe multiverse
deb http://archive.canonical.com/ lucid partner

Then update the aptitude database:

sudo aptitude update 

Update all installed apps/utilities

sudo aptitude safe-upgrade 

IF a new Kernel has been added a reboot will be required.

sudo reboot 

Configure the network

We need to tell our machine what IP and domain we are using. Update your hosts file to look like this:

sudo nano /etc/hosts 
127.0.0.1       localhost.domain.com   localhost
X.X.X.X   subdomain.domain.com   subdomain

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
sudo hostname subdomain 
sudo nano /etc/hostname 
subdomain

Restart the service:

sudo /etc/init.d/hostname restart 

There are some errors with this service on Ubuntu 10.04 so don't pay too much attention to any message you receive. To check it has worked run:

sudo hostname 
sudo hostname -f 

First should read subdomain and the latter subdomain.domain.com.

Set timezone and synchronize the system clock

Always a good idea to set the server to your preferred time zone and have it synchronise itself. There are perhaps neater ways to do this, but this is reliable!

sudo rm -f /etc/localtime 
sudo ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime 

Make sure our server keeps itself up to date.

sudo aptitude install ntp ntpdate 

Disable AppArmor

This is a security application similar to SELINUX but is often the cause of a lot of painful troubleshooting. It is not required to construct a secure system, anyone who tells you otherwise is wrong!

sudo service apparmor stop 
sudo update-rc.d -f apparmor remove 
sudo aptitude remove apparmor apparmor-utils 

Install MySQL

sudo aptitude install mysql-client mysql-server 
New password for the MySQL "root" user: (choose)
Repeat password for the MySQL "root" user: (repeat)

Check that it is working:

sudo netstat -tap | grep mysql 
tcp 0 0 localhost:mysql *:* LISTEN 7753/mysqld

Install Postfix, Dovecot and Saslauthd

These are all required for our e-mail server (along with MySQL above).

sudo aptitude install postfix postfix-mysql postfix-doc dovecot-common dovecot-postfix dovecot-imapd dovecot-pop3d libsasl2-2 libsasl2-modules libsasl2-modules-sql sasl2-bin libpam-mysql openssl telnet mailutils 

This should also create the root mysql user and set their password. This is for both Local and external.

General type of mail configuration: (internet site)
System mail name: (domain.com)

Install Amavisd-new, SpamAssassin, ClamAv (and related trimmings)

While not essential for an e-mail server, these anti-spam / anti-virus tools are a very, very good idea!

sudo aptitude install amavisd-new spamassassin clamav clamav-daemon zoo unzip bzip2 arj nomarch lzop cabextract apt-listchanges libnet-ldap-perl libauthen-sasl-perl clamav-docs daemon libio-string-perl libio-socket-ssl-perl libnet-ident-perl zip libnet-dns-perl libmail-spf-query-perl pyzor razor arj bzip2 cabextract cpio file gzip lha nomarch pax rar unrar unzip zip zoo rpm2cpio p7zip-full unrar-free ripole

Install PHP / Apache

These form the basis of most unix based webservers.

sudo aptitude install apache2 apache2.2-common apache2-doc apache2-mpm-prefork apache2-utils libexpat1 ssl-cert libapache2-mod-php5 php5 php5-common php5-gd php5-mysql php5-imap php5-cli php5-cgi libapache2-mod-fcgid php-pear php-auth php5-mcrypt mcrypt php5-curl php5-pspell php5-imagick imagemagick flite 

There are some errors in the following package comments (the wrong comment identifiers have been used)! Open each and update any comments starting with # (hash / pound) to begin with a ; (semi-colon).

sudo nano /etc/php5/cli/conf.d/imagick.ini 
sudo nano /etc/php5/cli/conf.d/imap.ini 
sudo nano /etc/php5/cli/conf.d/mcrypt.ini 

Configure the basic mods that Apache will use.

sudo a2enmod rewrite ssl actions include dav_fs dav auth_digest headers expires 
sudo a2dismod autoindex 

Apply these changes:

sudo service apache2 restart 

Install PHP Extensions (from PECL) (optional)

If you need to add any PECL extensions there are a few other requirements before installation can take place:

sudo aptitude install php5-dev make 

Once they are available to the system any PECL extensions can be added as follows (I'm using Mailparse and Sphinx as my example, you can use any you like!):

sudo pecl install mailparse, sphinx 

To enable them a new configuration file is required for each.

sudo nano /etc/php5/conf.d/mailparse.ini 
; configuration for php Mailparse module
extension=mailparse.so
sudo aptitude install sphinxsearch 
sudo nano /etc/php5/conf.d/sphinx.ini 
; configuration for php Sphinx module
extension=sphinx.so

And then restart Apache to apply the changes:

sudo service apache2 restart 

Update the locate database

The locate command can be extremely useful. (usage example: locate master.cf)

sudo updatedb

Install the EC2 API command line tools (cloud only)

In order to control the ec2 instances / volumes from the server itself (e.g. for backup crons) we need to add a few tools. Also if you are going to add any EBS volumes to the main EBS backed instance the tools for creation and management of the XFS filesystem are required.

sudo aptitude install sun-java6-jre ec2-api-tools xfslibs-dev xfsdump xfsprogs dmapi 

To use these tools the keys that are required must be available to the system. Create a directory to store these X.509 keys. As management will likely be through the ubuntu (default) user we'll store the keys in their directory.

mkdir /home/ubuntu/ec2-keys 
nano /home/ubuntu/ec2-keys/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem 

Add your certificate data (copy / paste) to this file and save it.

nano /home/ubuntu/ec2-keys/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem 

Add your private key data (copy / paste) to this file and save it.

We can also add these keys (and other useful constants) to the global shell configuration files so that they are available to the terminal and login shells (called by crons).

sudo nano /etc/bash.bashrc 
[...]
# Basic convenience
alias ..='cd ..'
alias ...='cd ...'
alias cd..='cd ..'
alias cd-='cd -'
alias ls='ls -lh --color=auto'
alias dir='ls -lh --color=auto'
alias df="df -h"
alias h=history
alias targz="tar cvfz"
alias untargz="tar xvfz"
alias tarbz2="tar cvfj"
alias untarbz2="tar xvfj"
alias unrar="unrar x"
[...]
export EDITOR=nano
export EC2_CERT=/home/ubuntu/ec2-keys/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
export EC2_PRIVATE_KEY=/home/ubuntu/ec2-keys/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
export EC2_PRIVATE_IP="`wget -q -O - http://169.254.169.254/latest/meta-data/local-ipv4`"
export EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`"
export EC2_AVAIL_ZONE="`wget -q -O - http://169.254.169.254/latest/meta-data/placement/availability-zone`"
export EC2_REGION="`echo \"$EC2_AVAIL_ZONE\" | sed -e 's:\([0-9][0-9]*\)[a-z]*\$:\\1:'`"
export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
sudo nano /etc/profile 
[...]
# Basic convenience
alias ..='cd ..'
alias ...='cd ...'
alias cd..='cd ..'
alias cd-='cd -'
alias ls='ls -lh --color=auto'
alias dir='ls -lh --color=auto'
alias df="df -h"
alias h=history
alias targz="tar cvfz"
alias untargz="tar xvfz"
alias tarbz2="tar cvfj"
alias untarbz2="tar xvfj"
alias unrar="unrar x"
[...]
export EDITOR=nano
export EC2_CERT=/home/ubuntu/ec2-keys/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
export EC2_PRIVATE_KEY=/home/ubuntu/ec2-keys/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
export EC2_PRIVATE_IP="`wget -q -O - http://169.254.169.254/latest/meta-data/local-ipv4`"
export EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id`"
export EC2_AVAIL_ZONE="`wget -q -O - http://169.254.169.254/latest/meta-data/placement/availability-zone`"
export EC2_REGION="`echo \"$EC2_AVAIL_ZONE\" | sed -e 's:\([0-9][0-9]*\)[a-z]*\$:\\1:'`"
export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXXXXXXXX
export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Apply these changes.

rm /home/ubuntu/ec2-keys/*.pem~ 
source /etc/bash.bashrc 
source /etc/profile 
source ~/.bashrc 

Update the permissions on these files:

sudo chmod 0600 /home/ubuntu/ec2-keys/* 

At this point it is likely a good idea to disable your ability to (accidentally) terminate your server!

ec2-modify-instance-attribute $EC2_INSTANCE_ID --disable-api-termination true --region $EC2_REGION 

It should tell you that this has been actioned. If you wish to terminate your instance in the future you need to update this setting to false.

Install Subversion (optional)

I like to use Subversion as my version control system because there's just me, but you may prefer Git, Mercurial, Bazaar or many of the other alternatives. This is not required for the system to run!

sudo aptitude install subversion libapache2-svn 
sudo service apache2 restart 

Add a Remote Desktop system (optional / large)

This isn't something that I would bother with, but if it is absolutely required then FreeNX is excellent.

export DEBIAN_FRONTEND=noninteractive 
sudo aptitude install ubuntu-desktop 
sudo add-apt-repository ppa:freenx-team 
sudo aptitude update 
sudo aptitude install freenx 
cd /usr/lib/nx 
sudo wget http://launchpadlibrarian.net/47959420/nxsetup.tar.gz 
sudo tar xzvf nxsetup.tar.gz 
sudo ./nxsetup --install 

Download and launch the NX client and connect to the server via SSH using a user with a password login.

EC2 Guide: Create and attach new EBS volumes (5a / 7)

The EBS volumes that we are about to create will function as external hard drives to our instance. They can only be attached to one instance at a time. If you don't want to use these please make the few changes that are needed to keep this guide compatible here. If you already have these volumes with data on them as described below you need to go here.

Create the EBS volumes

Using Elasticfox you will need to create 3 EBS volumes (Database, Web-server and Mail-server):

Attaching the EBS volumes

Using Elasticfox:

Formatting the EBS volumes

Back to the server, where your new volumes are now attached we need to create a file-system on them. XFS is generally used for this:

sudo mkfs.xfs /dev/sdc 
sudo mkfs.xfs /dev/sdd 
sudo mkfs.xfs /dev/sde 

Mounting the EBS volumes in useful locations

Choose descriptive path names for each!

echo "/dev/sdc /webserver xfs noatime 0 0" | sudo tee -a /etc/fstab 
sudo mkdir -m 000 /webserver 
sudo mount /webserver 

echo "/dev/sdd /database xfs noatime 0 0" | sudo tee -a /etc/fstab 
sudo mkdir -m 000 /database 
sudo mount /database 

echo "/dev/sde /email xfs noatime 0 0" | sudo tee -a /etc/fstab 
sudo mkdir -m 000 /email 
sudo mount /email 

Move MySQL to an EBS volume

This assumes we have an EBS volume with a mount point called "database" attached.

sudo service mysql stop 

Move the existing MySQL parts to run on the EBS volume, but keep their 'place-holders' populated through mount points.

sudo mkdir /database/etc /database/log /database/data 

sudo chown mysql:mysql /database/data 
sudo chmod 0700 /database/data 
sudo chown mysql:adm /database/log
sudo chmod 0750 /database/log 
sudo chmod g+s /database/log

sudo cp -R /etc/mysql /etc/mysql.ORIG 
sudo cp -R /var/lib/mysql /var/lib/mysql.ORIG 
sudo cp -R /var/log/mysql /var/log/mysql.ORIG 

sudo mv /etc/mysql /database/etc/ 
sudo mv /var/lib/mysql/* /database/data/ 
sudo mv /var/log/mysql/* /database/log/ 

sudo mkdir /etc/mysql

echo "/database/etc/mysql /etc/mysql none bind" | sudo tee -a /etc/fstab 
echo "/database/data /var/lib/mysql none bind" | sudo tee -a /etc/fstab 
echo "/database/log /var/log/mysql none bind" | sudo tee -a /etc/fstab 

sudo mount -a 
sudo service mysql start 

Move Apache / PHP / sites to an EBS volume

This assumes we have an EBS volume with a mount point called "webserver" attached.

sudo service apache2 stop 

Move some Apache parts to run on the EBS volume, but keep the 'place-holders' populated through mount points.

sudo cp -R /etc/apache2 /etc/apache2.ORIG 
sudo cp -R /var/log/apache2 /var/log/apache2.ORIG 
sudo cp -R /var/www /var/www.ORIG 
sudo cp -R /etc/php5 /etc/php5.ORIG 

sudo mv /etc/apache2 /webserver/ 
sudo mv /var/log/apache2/ /webserver/log/ 
sudo mv /var/www/ /webserver/sites/ 
sudo mv /etc/php5 /webserver/ 

sudo mkdir /webserver/vhosts /webserver/apache2/vhosts /etc/apache2 /etc/apache2/vhosts /var/log/apache2 /var/www /etc/php5 

echo "/webserver/apache2 /etc/apache2 none bind" | sudo tee -a /etc/fstab 
echo "/webserver/vhosts /etc/apache2/vhosts none bind" | sudo tee -a /etc/fstab 
echo "/webserver/log /var/log/apache2 none bind" | sudo tee -a /etc/fstab 
echo "/webserver/sites /var/www none bind" | sudo tee -a /etc/fstab 
echo "/webserver/php5 /etc/php5 none bind" | sudo tee -a /etc/fstab 

sudo mount -a 
sudo service apache2 start 

Move E-mail / Postfix / Dovecot to an EBS volume

This assumes we have an EBS volume with a mount point called "email" attached.

sudo service postfix stop 
sudo service dovecot stop 

Move some E-mail related parts to run on the EBS volume, but keep the 'place-holders' populated through mount points. Because we are going to use a MySQL controlled, virtual mail set-up we also need to create a new user and set some permissions.

sudo mkdir /email/vmail /email/log /email/log/postfix /email/log/dovecot /email/spool /email/spool/postfix/ /email/spool/vmail/ /email/smtp-certificates /home/ubuntu/smtp-certificates /var/log/dovecot /var/log/postfix /var/spool/vmail /home/vmail

sudo groupadd -g 5000 vmail 
sudo useradd -g vmail -u 5000 vmail -d /home/vmail 
sudo chown vmail:vmail /home/vmail 

sudo chown vmail:adm /var/log/dovecot 
sudo chown syslog:adm /var/log/postfix 
sudo chown vmail:mail /var/spool/vmail
sudo chmod 0775 /var/spool/vmail 

sudo chown vmail:adm /email/log/dovecot 
sudo chown syslog:adm /email/log/postfix 
sudo chown vmail:vmail /email/vmail 
sudo chown vmail:vmail /email/log 
sudo chown vmail:vmail /email/spool/postfix 
sudo chown vmail:mail /email/spool/vmail 
sudo chmod 0775 /email/spool/vmail 

sudo cp -R /etc/postfix /etc/postfix.ORIG 
sudo cp -R /etc/dovecot /etc/dovecot.ORIG 
sudo cp -R /var/spool/postfix /var/spool/postfix.ORIG 

sudo mv /etc/postfix/ /email/postfix 
sudo mv /etc/dovecot/ /email/dovecot 
sudo mv /var/spool/postfix/* /email/spool/postfix 

sudo mkdir /etc/postfix /etc/dovecot /etc/postfix/virtual

echo "/email/postfix /etc/postfix none bind" | sudo tee -a /etc/fstab 
echo "/email/dovecot /etc/dovecot none bind" | sudo tee -a /etc/fstab 
echo "/email/spool/postfix /var/spool/postfix none bind" | sudo tee -a /etc/fstab 
echo "/email/spool/vmail /var/spool/vmail none bind" | sudo tee -a /etc/fstab 
echo "/email/vmail /home/vmail none bind" | sudo tee -a /etc/fstab 
echo "/email/smtp-certificates /home/ubuntu/smtp-certificates none bind" | sudo tee -a /etc/fstab 
echo "/email/log/dovecot /var/log/dovecot none bind" | sudo tee -a /etc/fstab 
echo "/email/log/postfix /var/log/postfix none bind" | sudo tee -a /etc/fstab 

sudo mount -a 
sudo service postfix start 
sudo service dovecot start 

Move Subversion to an EBS Volume (optional)

This assumes we have an EBS volume with a mount point called "webserver" attached.

We have no subversion repository location yet, so we can just create one:

sudo mkdir /webserver/svn /svn-repositories 

echo "/webserver/svn /svn-repositories none bind" | sudo tee -a /etc/fstab 

sudo mount -a 

EC2 Guide: Software Configuration (6a / 7)

Configure ClamAV

ClamAV will be our anti-virus system and is very easy to set-up! We just need to update some permissions:

sudo adduser clamav amavis 
sudo adduser amavis clamav 

Configure SpamAssassin

SpamAssassin's function is given away by it's name. We will use this for spam filtering our e-mails.

sudo nano /etc/default/spamassassin 
[...]
ENABLED=1
[...]
CRON=1
[...]

We need to create a razor config file for it to work:

sudo razor-admin -create 

Start the service:

sudo service spamassassin start 

Configure Amavisd-New

Amavisd-New will be our interface between our mail transport program (Postfix) and our spam and virus filters (SpamAssassin / ClamAV).

sudo nano /etc/amavis/conf.d/15-content_filter_mode 

Uncomment the spam and anti-virus lines.

use strict;

# You can modify this file to re-enable SPAM checking through spamassassin
# and to re-enable antivirus checking.

#
# Default antivirus checking mode
# Uncomment the two lines below to enable it
#

@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);

#
# Default SPAM checking mode
# Uncomment the two lines below to enable it
#

@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);

1;  # insure a defined return

Some fairly key definitions are missing because they are not "free" - thanks Ubuntu. Enable / uncomment them!

sudo nano /etc/amavis/conf.d/01-debian 
[...]
$unfreeze   = ['unfreeze', 'freeze -d', 'melt', 'fcat'];
[...]
$lha    = 'lha';
[...]

Amavis needs permissions here:

sudo chmod -R 775 /var/lib/amavis/tmp 
sudo service amavis restart 

Configure Mail Logging

Because we mounted parts of our file system through EBS volumes we need to update the mail log locations.

sudo nano /etc/rsyslog.d/50-default.conf 
[...]
mail.*                          -/var/log/postfix/mail.log
[...]
mail.info                       -/var/log/postfix/mail.info
mail.warn                       -/var/log/postfix/mail.warn
mail.err                        /var/log/postfix/mail.err
[..]
sudo service rsyslog restart 

Configure MySQL

MySQL set-up largely depends upon your use case and virtual hardware configuration. Shown here is an example, working config.

sudo nano /etc/mysql/my.cnf 
[client]
port                  = 3306
socket                = /var/run/mysqld/mysqld.sock
default-character-set = utf8

[mysqld_safe]
socket                = /var/run/mysqld/mysqld.sock
nice                  = 0

[mysqld]
user                  = mysql
socket                = /var/run/mysqld/mysqld.sock
port                  = 3306
basedir               = /usr
datadir               = /var/lib/mysql
tmpdir                = /tmp

skip-external-locking
default-character-set = utf8
collation_server      = utf8_general_ci
character_set_server  = utf8
bind-address          = 127.0.0.1

key_buffer            = 512M
max_allowed_packet    = 16M
thread_stack          = 192K
thread_cache_size     = 8
myisam-recover        = BACKUP
max_connections       = 200
table_cache           = 1024
thread_concurrency    = 10

query-cache-type      = 1 
query_cache_limit     = 8M
query_cache_size      = 128M

log_error             = /var/log/mysql/error.log

server-id             = 1
expire_logs_days      = 10
max_binlog_size       = 100M

[mysqldump]
quick
quote-names
max_allowed_packet    = 16M

[mysql]
no-auto-rehash
default-character-set = utf8

[isamchk]
key_buffer            = 64M
sort_buffer           = 64M
read_buffer           = 16M
write_buffer          = 16M

[myisamchk]
key_buffer            = 64M
sort_buffer           = 64M
read_buffer           = 16M
write_buffer          = 16M

!includedir /etc/mysql/conf.d/

If you require MySQL to be available from other locations you will need to comment out the "bind-address = 127.0.0.1" line.

sudo service mysql restart 

Configure PHP

The default installation of PHP on Ubuntu 10.04 is good. There isn't much wrong with it, but there are always changes that need making. Most notably is the session clean up (which should be exclusively by cron jobs due to potential security risks of giving the web user access) and the configuration of either iconv or mb_string. There are 3 php.ini files, one for Apache (which is the only one I will show here), one for CLI and one for CGI. You should probably make pretty similar changes to all of them.

sudo nano /etc/php5/apache2/php.ini 
engine					= On
short_open_tag				= Off
asp_tags				= Off
precision				= 14
y2k_compliance				= On
output_buffering			= 4096
zlib.output_compression			= Off
implicit_flush				= Off
unserialize_callback_func		=
serialize_precision			= 100
allow_call_time_pass_reference		= Off

safe_mode				= Off
safe_mode_gid				= Off
safe_mode_include_dir			=
safe_mode_exec_dir			=
safe_mode_allowed_env_vars		= PHP_
safe_mode_protected_env_vars		= LD_LIBRARY_PATH

disable_functions			=
disable_classes				=
expose_php				= On
max_execution_time			= 330
max_input_time				= 360
memory_limit				= 128M
error_reporting				= E_ALL & ~E_DEPRECATED
display_errors				= Off
display_startup_errors			= Off
log_errors				= On
log_errors_max_len			= 1024
ignore_repeated_errors			= Off
ignore_repeated_source			= Off
report_memleaks				= On
track_errors				= Off
html_errors				= Off
variables_order				= "GPCS"
request_order				= "GP"
register_globals			= Off
register_long_arrays			= Off
register_argc_argv			= Off
auto_globals_jit			= On
post_max_size				= 18M
magic_quotes_gpc			= Off
magic_quotes_runtime			= Off
magic_quotes_sybase			= Off
auto_prepend_file			=
auto_append_file			=
default_mimetype			= "text/html"
default_charset				= UTF-8
doc_root				=
user_dir				=
enable_dl				= Off
file_uploads				= On
upload_max_filesize			= 12M
max_file_uploads			= 20
allow_url_fopen				= On
allow_url_include			= Off
default_socket_timeout			= 60

pdo_mysql.cache_size			= 2000
pdo_mysql.default_socket		=

define_syslog_variables			= Off

SMTP					= localhost
smtp_port				= 25
mail.add_x_header			= On

sql.safe_mode				= Off

odbc.allow_persistent			= On
odbc.check_persistent			= On
odbc.max_persistent			= -1
odbc.max_links				= -1
odbc.defaultlrl				= 4096
odbc.defaultbinmode			= 1

ibase.allow_persistent			= 1
ibase.max_persistent			= -1
ibase.max_links				= -1
ibase.timestampformat			= "%Y-%m-%d %H:%M:%S"
ibase.dateformat			= "%Y-%m-%d"
ibase.timeformat			= "%H:%M:%S"

mysql.allow_local_infile		= On
mysql.allow_persistent			= On
mysql.cache_size			= 2000
mysql.max_persistent			= -1

mysql.max_links				= -1
mysql.default_port			=
mysql.default_socket			=
mysql.default_host			=
mysql.default_user			=
mysql.default_password			=
mysql.connect_timeout			= 60
mysql.trace_mode			= Off

mysqli.max_persistent			= -1
mysqli.allow_persistent			= On
mysqli.max_links			= -1
mysqli.cache_size			= 2000
mysqli.default_port			= 3306
mysqli.default_socket			=
mysqli.default_host			=
mysqli.default_user			=
mysqli.default_pw			=
mysqli.reconnect			= Off

mysqlnd.collect_statistics		= On
mysqlnd.collect_memory_statistics	= Off

pgsql.allow_persistent			= On
pgsql.auto_reset_persistent		= Off
pgsql.max_persistent			= -1
pgsql.max_links				= -1
pgsql.ignore_notice			= 0
pgsql.log_notice			= 0

sybct.allow_persistent			= On
sybct.max_persistent			= -1
sybct.max_links				= -1
sybct.min_server_severity		= 10
sybct.min_client_severity		= 10

bcmath.scale				= 0

session.save_handler			= files
session.use_cookies			= 1
session.use_only_cookies		= 1
session.name				= PHPSESSID
session.auto_start			= 0
session.cookie_lifetime			= 0
session.cookie_path			= /
session.cookie_domain			=
session.cookie_httponly			=
session.serialize_handler		= php
session.gc_probability			= 0
session.gc_divisor			= 1000
session.gc_maxlifetime			= 2700
session.bug_compat_42			= Off
session.bug_compat_warn			= Off
session.referer_check			=
session.entropy_length			= 0
session.entropy_file			=
session.cache_limiter			= nocache
session.cache_expire			= 180
session.use_trans_sid			= 0
session.hash_function			= 1
session.hash_bits_per_character		= 5

url_rewriter.tags			= "a=href,area=href,frame=src,input=src,form=fakeentry"

mssql.allow_persistent			= On
mssql.max_persistent			= -1
mssql.max_links				= -1
mssql.min_error_severity		= 10
mssql.min_message_severity		= 10
mssql.secure_connection			= Off

mbstring.language			= Neutral
mbstring.internal_encoding		= UTF-8
mbstring.http_input			= auto
mbstring.http_output			= UTF-8
mbstring.encoding_translation		= On
mbstring.detect_order			= auto
mbstring.substitute_character		= none;
mbstring.func_overload			= 0

tidy.clean_output			= Off

soap.wsdl_cache_enabled			= 1
soap.wsdl_cache_dir			= "/tmp"
soap.wsdl_cache_ttl			= 86400
soap.wsdl_cache_limit			= 5

ldap.max_links				= -1
sudo service apache2 graceful  

Configure Apache and a test site

Apache / PHP configuration again largely depends upon your use case. The following will set up a working system and configure a site at "http://www.domain.com" for you.

Create some directories for vhosts and sites first:

sudo mkdir /etc/apache2/vhosts/db-maintenance/ /etc/apache2/vhosts/catch-all/ /etc/apache2/vhosts/redirects/ /var/www/default /var/www/www.domain.com /var/www/db-maintenance.domain.com 

Technically you are not supposed to edit the main apache2.conf file file, but I found some parts of it not to my liking.

sudo nano /etc/apache2/apache2.conf 
ServerRoot "/etc/apache2"
LockFile /var/lock/apache2/accept.lock
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15

<IfModule mpm_prefork_module>
    StartServers          8
    MinSpareServers       5
    MaxSpareServers      20
    MaxClients          256
    MaxRequestsPerChild 4000
</IfModule>

<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

<IfModule mpm_event_module>
    StartServers          2
    MaxClients          150
    MinSpareThreads      25
    MaxSpareThreads      75
    ThreadLimit          64
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

AccessFileName .htaccess
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy all
</Files>

DefaultType text/plain
HostnameLookups Off
ErrorLog /var/log/apache2/error.log
LogLevel warn

Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf
Include /etc/apache2/httpd.conf
Include /etc/apache2/ports.conf

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog /var/log/apache2/access.log vhost_combined

Include /etc/apache2/conf.d/

#Include /etc/apache2/vhosts/db-maintenance/*.conf
Include /etc/apache2/vhosts/*.conf
Include /etc/apache2/vhosts/redirects/*.conf
Include /etc/apache2/vhosts/catch-all/*.conf

This is the user editable file which we will use to set a couple of options and define a default directory.

sudo nano /etc/apache2/httpd.conf 
DocumentRoot "/var/www/default"

<Directory />
    Options FollowSymLinks
    AllowOverride None
</Directory>

<Directory "/var/www/default">
            Options Includes FollowSymLinks MultiViews -Indexes
            AllowOverride AuthConfig
            Order allow,deny
            Allow from all
</Directory>

DirectoryIndex index.php index.html index.htm
AddDefaultCharset UTF-8

Next we will create some default site pages so you can check everything is working. The main (www) site first:

sudo nano /var/www/www.domain.com/index.php 
<?php phpinfo();

Then the database maintenance site:

sudo nano /var/www/db-maintenance.domain.com/index.php
<?php 
echo 'Sorry, this site is currently undergoing scheduled maintenance. 
Please check back soon!';

And a default page to catch incorrect sub-domains:

sudo nano /var/www/default/index.php
<?php 
echo 'We think that you may have mis-typed! 
Were you looking for: <a href="http://www.domain.com">www.domain.com</a>?';

The premise of these files will be that if you visit "http://www.domain.com" you will receive a PHP info screen, but if you visit a none-existent sub-domain (e.g. "http://totally-random.domain.com") you will end up at the default location (assuming you have set your DNS to a catch all, *, setting in your zone files). If you were to enable the db-maintenance vhosts in the apache2.conf file the idea will be your users will go there instead of your main site. None of this will work yet because we have not set up any vhost files.

The first vhost will be the default (catch-all):

sudo nano /etc/apache2/vhosts/default.conf  
<VirtualHost *:80>
        DocumentRoot /var/www/default/
        ServerName localhost
        <Directory "/var/www/default/">
                Options -Indexes
                AllowOverride AuthConfig
                Order allow,deny
                Allow from all
        </Directory>
</VirtualHost>

The only key points to note are the choice of document root (you may want the trailing slash, you may not), the exclusion of indexing (i.e. you don't want anyone able to view all folder contents) and the ability to over-ride the settings using an .htaccess file should you wish to (I prefer to make those changes at this level). The next vhost will be the main site:

sudo nano /etc/apache2/vhosts/www.domain.com.conf  
<VirtualHost *:80>
        ServerAdmin webmaster@domain.com
        DocumentRoot /var/www/www.domain.com
        ServerName www.domain.com
        <Directory "/var/www/www.domain.com/">
                Options -Indexes FollowSymLinks
                AllowOverride AuthConfig
                Order allow,deny
                Allow from all
        </Directory>
</VirtualHost>  

Hopefully everything should be fairly obvious and you will note we have used a unique "ServerName" value. Lastly we need the database maintenance vhost:

sudo nano /etc/apache2/vhosts/db-maintenance/www.domain.com.conf  
<VirtualHost *:80>
        ServerAdmin webmaster@domain.com
        DocumentRoot /var/www/db-maintenance.domain.com
        ServerName www.domain.com
        <Directory "/var/www/db-maintenance.domain.com/">
                Options -Indexes FollowSymLinks
                AllowOverride AuthConfig
                Order allow,deny
                Allow from all
        </Directory>
        <IfModule mod_rewrite.c>
                RewriteEngine On
                RewriteCond %{REQUEST_FILENAME} !-d
                RewriteCond %{REQUEST_FILENAME} !-f
                RewriteRule ^(.*)$ /index.php [L,QSA]
        </IfModule>
</VirtualHost>  

This time we have also included some re-write rules which mean that every address a user tries when maintenance is being carried out (e.g. http://www.domain.com/one-of-your-pages.php) will be handled by just one file, index.php, so you only need to create a single maintenance file. This technique is commonly used in MVC patterns.

Apache needs a restart / reload before these changes are applied:

sudo service apache2 graceful 

Now if you test your site using "http://www.domain.com" you should receive a PHP info screen, but if you visit a none-existent sub-domain (e.g. "http://totally-random.domain.com) you should receive the default message. If you enable database maintenance (by un-commenting the relevant line in the apache2.conf and restarting Apache) and visit your site using "http://www.domain.com/anything.php" you should see the maintenance message. If at this point you don't see these pages you should check your zone settings with your domain registrar or DNS manager.

Obviously it would be better that all incorrectly typed (non-existing) domains were forwarded on to an actual domain of your choosing. This isn't hard to achieve with a catch-all vhost:

sudo nano /etc/apache2/vhosts/catch-all/domain.com.conf  
<VirtualHost *:80>
        ServerAdmin webmaster@domain.com
        ServerName unknown.domain.com
        ServerAlias *.domain.com
        KeepAlive Off
        RewriteEngine On
        RewriteRule ^/(.*)$ http://www.domain.com/$1 [R=301,L]
</VirtualHost>

This time we are using a "ServerAlias" in addition to an arbitrary "ServerName" with a re-write rule to forward everything (including a page name and GET data) to the correct location on the actual site.

It would also be better if the domain without a sub-domain (i.e. http://domain.com) redirected to your main site. Again this isn't hard to achieve with another vhost:

sudo nano /etc/apache2/vhosts/redirects/domain.com.conf  
<VirtualHost *:80>
        ServerAdmin webmaster@domain.com
        ServerName domain.com
        KeepAlive Off
        RewriteEngine On
        RewriteRule ^/(.*)$ http://www.domain.com/$1 [R=301,L]
</VirtualHost>

Nothing new in this one, just another re-write rule. Once again we need to reload our Apche settings:

sudo service apache2 graceful 

Now if you test your site using a none-existent sub-domain (e.g. "http://totally-random.domain.com) or the raw domain "http://domain.com" you should be 301 redirected to "http://www.domain.com."

Install PhpMyAdmin (optional)

While we are configuring sites it is a good idea to install this very convenient tool to manage MySQL databases. It will need its own vhost:

sudo nano /etc/apache2/vhosts/database.domain.com.conf 
<VirtualHost *:80>
        ServerAdmin webmaster@domain.com
        DocumentRoot /var/www/database.domain.com
        ServerName database.domain.com
        <Directory "/var/www/database.domain.com/">
                Options -Indexes FollowSymLinks
                AllowOverride AuthConfig
                Order allow,deny
                Allow from all
        </Directory>
</VirtualHost>

Get the actual program, extract it and edit its settings. You will probably need to customise this link I'm afraid because it changes often!

cd /var/www 
sudo wget "http://downloads.sourceforge.net/project/phpmyadmin/phpMyAdmin/4.6.5.2/phpMyAdmin-4.6.5.2-english.tar.gz" -O /var/www/phpmyadmin.tar.gz 
sudo tar xvfz /var/www/phpmyadmin.tar.gz 
sudo rm /var/www/phpmyadmin.tar.gz 
sudo mv /var/www/phpMyAdmin-4.6.5.2-english /var/www/database.domain.com 
sudo mv /var/www/database.domain.com/config.sample.inc.php /var/www/database.domain.com/config.inc.php 
sudo nano /var/www/database.domain.com/config.inc.php 
[...]
$cfg['blowfish_secret'] = 'choose a suitably random string here';
[...]
$cfg['Servers'][$i]['extension'] = 'mysqli';
[...]

Clean up any temporary files left by our editor:

sudo rm /var/www/database.domain.com/config.inc.php~ 

It's also a good idea to password this directory so we'll create a general password file for use with our restricted sites:

sudo htpasswd -c /var/www/.htpasswd user 

And if you want to add another user:

sudo htpasswd /var/www/.htpasswd another_user 

Now we need to tell the phpMyAdmin directory to use this extra authentication:

sudo nano /var/www/database.domain.com/.htaccess 
AuthUserFile /var/www/.htpasswd
AuthName "Database Management"
AuthType Basic
require valid-user

Restart Apache to check that everything is working:

sudo service apache2 graceful 

If you visit "http://database.domain.com" you should be able to log in and then use your MySQL username (root) and password (root_db_password) to manage your MySQL databases.

Install PostfixAdmin

This is the tool that we will use to manage our e-mail domains / users and it must be installed at this stage because it dictates the structure of the database that Postfix and Dovecot will use for e-mail delivery.

We need to create a MySQL database and user for this. This can either be done through phpMyAdmin or via the MySQL command prompt, either way the 4 commands are the same.

mysql -uroot -proot_db_password
CREATE USER 'vmail'@'%' IDENTIFIED BY 'vmail_db_password'; 
GRANT USAGE ON * . * TO 'vmail'@'%' IDENTIFIED BY 'vmail_db_password' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0; 
CREATE DATABASE IF NOT EXISTS `vmail` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci; 
GRANT ALL PRIVILEGES ON `vmail` . * TO 'vmail'@'%'; 
exit;

This new site needs a vhost too:

sudo nano /etc/apache2/vhosts/mailadmin.domain.com.conf 
<VirtualHost *:80>
        ServerAdmin webmaster@domain.com
        DocumentRoot /var/www/mailadmin.domain.com
        ServerName mailadmin.domain.com
        <Directory "/var/www/mailadmin.domain.com/">
                Options -Indexes FollowSymLinks
                AllowOverride AuthConfig
                Order allow,deny
                Allow from all
        </Directory>
</VirtualHost>

Get the actual program, extract it and edit its settings. You may need to customise this link I'm afraid because it changes occasionally!

cd /var/www 
sudo wget "http://downloads.sourceforge.net/project/postfixadmin/postfixadmin/postfixadmin-3.0/postfixadmin-3.0.tar.gz" -O /var/www/postfixadmin.tar.gz 
sudo tar xvfz /var/www/postfixadmin.tar.gz 
sudo rm /var/www/postfixadmin.tar.gz 
sudo mv /var/www/postfixadmin-3.0 /var/www/mailadmin.domain.com 
sudo nano /var/www/mailadmin.domain.com/config.inc.php 
[...]
$CONF['configured'] = true;
[...]
$CONF['database_type'] = 'mysqli';
$CONF['database_host'] = 'localhost';
$CONF['database_user'] = 'vmail';
$CONF['database_password'] = 'vmail_db_password';
$CONF['database_name'] = 'vmail';
$CONF['database_prefix'] = '';
[...]
$CONF['admin_email'] = 'postmaster@domain.com';
[...]
$CONF['default_aliases'] = array (
    'abuse' => 'abuse@domain.com',
    'hostmaster' => 'hostmaster@domain.com',
    'postmaster' => 'postmaster@domain.com',
    'webmaster' => 'webmaster@domain.com'
);
[...]

Now we need to tell the PostfixAdmin directory to use our general .htpasswd file:

sudo nano /var/www/mailadmin.domain.com/.htaccess 
AuthUserFile /var/www/.htpasswd
AuthName "E-mail Management"
AuthType Basic
require valid-user

Restart Apache to apply the changes:

sudo service apache2 graceful 

You now need to visit "http://mailadmin.domain.com/setup.php" to install the database tables, then enter a password to generate a hash. Please copy this value, but do not proceed with any further prompts. Instead return to the terminal window:

sudo nano /var/www/mailadmin.domain.com/config.inc.php 
[...]
$CONF['setup_password'] = 'the password hash';
[...]

Clean up any temporary files left by our editor:

sudo rm /var/www/database.domain.com/config.inc.php~ 

Now return to "http://mailadmin.domain.com/setup.php", but don't refresh, and configure a system admin user using the un-hashed set-up password and whatever details you wish for the account. Then visit "http://mailadmin.domain.com/" and login as your admin user.

Visit the domain list, add domain.com as a new domain and create any e-mail addresses that you want here (e.g. info@domain.com, your_name@domain.com etc). You should also create the four default accounts (webmaster@domain.com, postmaster@domain.com, abuse@domain.com and hostmaster@domain.com) for it too. These can just be aliases to another account if you wish, but it's a good idea to have them set up.

Create SSL certificates (for Postfix / Dovecot)

In order to accept secure connections we need to generate some self signed certificates for e-mail use. If you wish you can use official certificates, but that is outside the scope of this guide.

cd /home/ubuntu/smtp-certificates 
sudo openssl req -new -outform PEM -out smtpd.cert -newkey rsa:2048 -nodes -keyout smtpd.key -keyform PEM -days 3650 -x509 
Country Name (2 letter code) [AU]: UK
State or Province Name (full name) [Some-State]: Greater London
Locality Name (eg, city) []: London
Organization Name (eg, company) [Internet Widgits Pty Ltd]: Company Name
Organizational Unit Name (eg, section) []: Trading Name
Common Name (eg, YOUR name) []: subdomain.domain.com
Email Address []: postmaster@domain.com

Create a self-singed root certificate.

sudo openssl req -new -x509 -extensions v3_ca -keyout cakey.pem -out cacert.pem -days 3650 
sudo chmod 0640 /home/ubuntu/smtp-certificates/smtpd.key 
sudo chmod 0640 /home/ubuntu/smtp-certificates/cakey.pem 

For far more detail on this process please visit Ubuntu's own guide on the subject.

Configure Postfix

Postfix will be our mail server, replacing the ancient Sendmail. Configuration is fairly short, but getting it right is a bit of a black art. To keep our database configuration files contained we'll put them in a single folder that Postfix owns:

sudo mkdir /etc/postfix/sql 
sudo chown postfix:postfix /etc/postfix/sql 
sudo chmod 0750 /etc/postfix/sql 

What we are essentially doing is using MySQL to determine what happens to the mail. There are 8 different possibilities for this. First up is the relay domains rule, which tells Postfix which domains are to have their mail passed on to another server.

sudo nano /etc/postfix/sql/relay_domains.cf 
user		= vmail
password	= vmail_db_password
hosts		= localhost
dbname		= vmail
query		= SELECT domain FROM domain WHERE domain = '%s' AND backupmx = '1'

We then define a rule for domain forwarding catch-all, e.g. anything@my-domain.com -> anything@mydomain.com:

sudo nano /etc/postfix/sql/virtual_alias_domain_catchall_maps.cf 
user		= vmail
password	= vmail_db_password
hosts		= localhost
dbname		= vmail
query		= SELECT goto FROM alias, alias_domain WHERE alias_domain = '%d' AND address = CONCAT('@', target_domain) AND alias.active = '1' AND alias_domain.active = '1'

We can also define a rule for account forwarding for certain addresses within a domain, e.g. userX@my-domain.com may be forwarded to userX@mydomain.com, but userY@my-domain.com may or may not be forwarded. This query selects the correct mailbox directory for that scenario:

sudo nano /etc/postfix/sql/virtual_alias_domain_mailbox_maps.cf 
user		= vmail
password	= vmail_db_password
hosts		= localhost
dbname		= vmail
query		= SELECT CONCAT(domain, '/', maildir) FROM mailbox, alias_domain WHERE alias_domain = '%d' AND username = CONCAT('%u', '@', alias_domain.target_domain) AND mailbox.active = '1' AND alias_domain.active = '1'

Continuing from the previous rule we also need a query to select the account that a user on an aliased domain will be sent to:

sudo nano /etc/postfix/sql/virtual_alias_domain_maps.cf 
user		= vmail
password	= vmail_db_password
hosts		= localhost
dbname		= vmail
query		= SELECT goto FROM alias, alias_domain WHERE alias_domain = '%d' AND address = CONCAT('%u', '@', target_domain) AND alias.active = '1' AND alias_domain.active = '1'

We can also just define simple aliases on a per address basis, e.g. userX@my-domain.com could be forwarded to anything@yahoo.com, or in fact multiple recipients. This query handles those.

sudo nano /etc/postfix/sql/virtual_alias_maps.cf 
user		= vmail
password	= vmail_db_password
hosts		= localhost
dbname		= vmail
query		= SELECT goto FROM alias WHERE address = '%s' AND active = '1'

We need a simple query to select the mailbox for a certain domain:

sudo nano /etc/postfix/sql/virtual_mailbox_domains.cf 
user		= vmail
password	= vmail_db_password
hosts		= localhost
dbname		= vmail
query		= SELECT domain FROM domain WHERE domain = '%s' AND active = '1'

If the mailboxes are defined with disk space quotas we need to be able to access them:

sudo nano /etc/postfix/sql/virtual_mailbox_limit_maps.cf 
user		= vmail
password	= vmail_db_password
hosts		= localhost
dbname		= vmail
query		= SELECT quota FROM mailbox WHERE username = '%s' AND active = '1'

And finally just get a mailbox location for a certain user on a domain:

sudo nano /etc/postfix/sql/virtual_mailbox_maps.cf 
user		= vmail
password	= vmail_db_password
hosts		= localhost
dbname		= vmail
query		= SELECT CONCAT(domain, '/', maildir) FROM mailbox WHERE username = '%s' AND active = '1'

We can then configure the two main Postfix files:

sudo nano /etc/postfix/main.cf 
smtpd_banner				= $myhostname ESMTP $mail_name
biff					= no
append_dot_mydomain			= no
myhostname				= subdomain.domain.com
alias_maps				= hash:/etc/aliases
alias_database				= hash:/etc/aliases
myorigin				= /etc/mailname
mydestination				= subdomain.domain.com, localhost.domain.com, localhost
relayhost				=
mynetworks				= 127.0.0.0/8
mailbox_size_limit			= 0
recipient_delimiter			= +
inet_interfaces				= all
message_size_limit			= 50000000

# Virtual mail set-up
virtual_mailbox_base			= /home/vmail
virtual_uid_maps			= static:5000
virtual_gid_maps			= static:5000
virtual_transport			= dovecot
dovecot_destination_recipient_limit	= 1
virtual_create_maildirsize		= yes
virtual_mailbox_extended		= yes
relay_domains				= proxy:mysql:/etc/postfix/sql/relay_domains.cf
virtual_mailbox_domains			= proxy:mysql:/etc/postfix/sql/virtual_mailbox_domains.cf
virtual_alias_maps			= proxy:mysql:/etc/postfix/sql/virtual_alias_maps.cf,
					  proxy:mysql:/etc/postfix/sql/virtual_alias_domain_maps.cf,
					  proxy:mysql:/etc/postfix/sql/virtual_alias_domain_catchall_maps.cf
virtual_mailbox_maps			= proxy:mysql:/etc/postfix/sql/virtual_mailbox_maps.cf,
					  proxy:mysql:/etc/postfix/sql/virtual_alias_domain_mailbox_maps.cf
virtual_mailbox_limit_maps		= proxy:mysql:/etc/postfix/sql/virtual_mailbox_limit_maps.cf
virtual_mailbox_limit_override		= yes
virtual_maildir_limit_message		= Sorry, that user is over their e-mail quota. Please try again later.
virtual_overquota_bounce		= yes

# SASL Related
smtpd_sasl_type				= dovecot
smtpd_sasl_path				= private/auth
smtpd_sasl_local_domain			=
smtpd_sasl_auth_enable			= yes
smtpd_sasl_security_options		= noanonymous
broken_sasl_auth_clients		= yes
smtpd_sasl2_auth_enable			= yes
smtpd_sasl_authenticated_header		= yes
smtpd_sasl_tls_security_options		= noanonymous
#smtpd_sasl_exceptions_networks		= $mynetworks

# TLS Commands
smtpd_tls_auth_only			= no
smtp_tls_security_level			= may
smtpd_tls_security_level		= may
smtp_tls_note_starttls_offer		= yes
smtpd_tls_cert_file			= /home/ubuntu/smtp-certificates/smtpd.cert
smtpd_tls_key_file			= /home/ubuntu/smtp-certificates/smtpd.key
smtpd_tls_CAfile			= /home/ubuntu/smtp-certificates/cacert.pem
smtpd_tls_loglevel			= 1
smtpd_tls_received_header		= yes
smtpd_tls_session_cache_timeout		= 3600s
tls_random_source			= dev:/dev/urandom

# Include Spam settings
disable_vrfy_command			= yes
smtpd_delay_reject			= yes
smtpd_helo_required			= yes
smtpd_helo_restrictions			= permit_mynetworks,
					  reject_non_fqdn_hostname,
					  reject_invalid_hostname,
					  permit

smtpd_recipient_restrictions		= reject_unauth_pipelining,
					  permit_mynetworks,
					  permit_sasl_authenticated,
					  reject_unauth_destination,
					  reject_invalid_hostname,
					  reject_non_fqdn_hostname,
					  reject_non_fqdn_sender,
					  reject_non_fqdn_recipient,
					  reject_unknown_sender_domain,
					  reject_unknown_recipient_domain,
					  reject_rbl_client zen.spamhaus.org,
					  reject_rbl_client dnsbl.sorbs.net,
					  reject_rbl_client dsn.rfc-ignorant.org,
					  reject_rbl_client bl.spamcop.net,
					  permit

smtpd_error_sleep_time			= 1s
smtpd_soft_error_limit			= 10
smtpd_hard_error_limit			= 20

content_filter				= smtp-amavis:[127.0.0.1]:10024
sudo nano /etc/postfix/master.cf 
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
smtp      inet  n       -       n       -       -       smtpd
smtps     inet  n       -       -       -       -       smtpd
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o broken_sasl_auth_clients=yes
pickup    fifo  n       -       -       60      1       pickup
  -o content_filter=
  -o receive_override_options=no_header_body_checks
cleanup   unix  n       -       -       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       -       -       -       trivial-rewrite
bounce    unix  -       -       -       -       0       bounce
defer     unix  -       -       -       -       0       bounce
trace     unix  -       -       -       -       0       bounce
verify    unix  -       -       -       -       1       verify
flush     unix  n       -       -       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       -       -       -       smtp
relay     unix  -       -       -       -       -       smtp
	-o smtp_fallback_relay=
showq     unix  n       -       -       -       -       showq
error     unix  -       -       -       -       -       error
retry     unix  -       -       -       -       -       error
discard   unix  -       -       -       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       -       -       -       lmtp
anvil     unix  -       -       -       -       1       anvil
scache    unix  -       -       -       -       1       scache
maildrop  unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
uucp      unix  -       n       n       -       -       pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix	-	n	n	-	2	pipe
  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman   unix  -       n       n       -       -       pipe
  flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
  ${nexthop} ${user}
dovecot   unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -d ${recipient}
smtp-amavis     unix    -       -       -       -       2       smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
        -o max_use=20
127.0.0.1:10025 inet    n       -       -       -       -       smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_delay_reject=no
        -o smtpd_client_restrictions=permit_mynetworks,reject
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o smtpd_data_restrictions=reject_unauth_pipelining
        -o smtpd_end_of_data_restrictions=
        -o mynetworks=127.0.0.0/8
        -o smtpd_error_sleep_time=0
        -o smtpd_soft_error_limit=1001
        -o smtpd_hard_error_limit=1000
        -o smtpd_client_connection_count_limit=0
        -o smtpd_client_connection_rate_limit=0
        -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

That is all for Postfix, but before we can use the mail system we need to tweak Dovecot.

Configure Dovecot

Dovecot will be our POP3(S) / IMAP(S) server and handle our SASL authentication. Again, there's not too much to set up, but if you get a setting or two wrong, it simply won't work properly!

We are using Dovecot to handle our authentication, and again we will do this from our MySQL tables to ease management:

sudo nano /etc/dovecot/dovecot-sql.conf 
driver			= mysql
connect			= host=127.0.0.1 dbname=vmail user=vmail password=vmail_db_password
user_query		= SELECT concat('/home/vmail/', domain, '/', maildir) as home, concat('maildir:/home/vmail/', domain, '/', maildir) as mail, 5000 AS uid, 5000 AS gid, concat('maildir:storage=', quota) AS quota FROM mailbox WHERE username = '%u' AND active = '1'
default_pass_scheme	= MD5
password_query		= SELECT username as user, password FROM mailbox WHERE username = '%u' AND active = '1'

And then we need to configure Dovecot's main configuration file:

sudo nano /etc/dovecot/dovecot.conf 
protocols		= pop3 pop3s imap imaps
log_path		= /var/log/dovecot/log
info_log_path		= /var/log/dovecot/info
log_timestamp		= "%Y-%m-%d %H:%M:%S "
ssl			= yes
ssl_cert_file		= /home/ubuntu/smtp-certificates/smtpd.cert
ssl_key_file		= /home/ubuntu/smtp-certificates/smtpd.key
disable_plaintext_auth	= no
login_dir		= /var/run/dovecot/login
login_chroot		= yes
login_user		= dovecot
login_greeting		= Dovecot ready.
mail_location		= maildir:/home/vmail/%d/%u
mail_debug		= no

auth_executable		= /usr/lib/dovecot/dovecot-auth
auth_verbose		= no
auth_debug		= no
auth_debug_passwords	= no

protocol imap {
  login_executable	= /usr/lib/dovecot/imap-login
  mail_executable	= /usr/lib/dovecot/imap
  imap_max_line_length	= 65536
}

protocol pop3 {
  login_executable	= /usr/lib/dovecot/pop3-login
  mail_executable	= /usr/lib/dovecot/pop3
  pop3_uidl_format	= %08Xu%08Xv
}

protocol lda {
  log_path		= /var/log/dovecot/lda
  auth_socket_path	= /var/run/dovecot/auth-master
  postmaster_address	= postmaster@domain.com
}

auth default {
  mechanisms		= plain login

  passdb sql {
    args		= /etc/dovecot/dovecot-sql.conf
  }

  userdb sql {
    args		= /etc/dovecot/dovecot-sql.conf
  }

  socket listen {
    master {
      path		= /var/run/dovecot/auth-master
      mode		= 0600
      user		= vmail
      group		= vmail
   }

    client {
       path		= /var/spool/postfix/private/auth
       mode		= 0660
       user		= postfix
       group		= postfix
    }
  }
}

There are some files designed to be automatically included. They shouldn't be, but just in case:

sudo mv /etc/dovecot/conf.d/01-dovecot-postfix.conf 01-dovecot-postfix.conf.bak 
sudo mv /etc/dovecot/auth.d/01-dovecot-postfix.auth 01-dovecot-postfix.auth.bak 

Before we can use the mail system the new settings need to be loaded in.

sudo service dovecot restart 
sudo service postfix restart 

Configure Squirrelmail (optional)

Squirrelmail is a convenient web-mail client. It will need a vhost file:

sudo nano /etc/apache2/vhosts/mail.domain.com.conf 
<VirtualHost *:80>
        ServerAdmin webmaster@domain.com
        DocumentRoot /var/www/mail.domain.com
        ServerName mail.domain.com
        <Directory "/var/www/mail.domain.com/">
                Options -Indexes FollowSymLinks
                AllowOverride AuthConfig
                Order allow,deny
                Allow from all
        </Directory>
</VirtualHost>

Get the actual program, extract it and edit its settings. You may need to check this link I'm afraid because it may change.

cd /var/www 
sudo wget "http://downloads.sourceforge.net/project/squirrelmail/stable/1.4.22/squirrelmail-webmail-1.4.22.tar.gz" -O /var/www/squirrelmail.tar.gz 
sudo tar xvfz /var/www/squirrelmail.tar.gz 
sudo rm /var/www/squirrelmail.tar.gz 
sudo mv /var/www/squirrelmail-1.4.22 /var/www/mail.domain.com 

Squirrelmail needs a private folder, so we need to create and secure it:

sudo mkdir /var/www/private 
sudo mkdir /var/www/private/mail.domain.com 
sudo mkdir /var/www/private/mail.domain.com/data 
sudo mkdir /var/www/private/mail.domain.com/attachments 
sudo chown -R root:www-data /var/www/private/mail.domain.com  
sudo chmod -R 0730 /var/www/private/mail.domain.com 

We can now configure it. There is a Perl script to do this, but I find it faster just to edit the settings manually.

sudo mv /var/www/mail.domain.com/config/config_default.php /var/www/mail.domain.com/config/config.php 
sudo nano /var/www/mail.domain.com/config/config.php 
[...]
$org_name = 'Your Website Name';
[...]
$domain = 'domain.com';
[...]
$smtp_auth_mech = 'plain';
[...]
$data_dir = '/var/www/private/mail.domain.com/data/';
[...]
$attachment_dir = '/var/www/private/mail.domain.com/attachments/';
[...]
$default_charset = 'utf-8';
[...]

Clean up any temporary files left by our editor:

sudo rm /var/www/mail.domain.com/config/config.php~ 

To apply the changes Apache needs to have the data reloaded:

sudo service apache2 graceful 

If you visit "http://mail.domain.com" you will be able to send and receive e-mails. Of course this is no replacement for a mail client, but it's useful to have on occasion!

Configure Subversion (optional)

We will bring your website (www.domain.com) under version control. This step will create the repositories and set their permissions to allow the web user access.

It's unlikely that you'll want anyone to be able to access your files so we can generate a password file for SVN access:

sudo htpasswd -c /svn-repositories/passwd svn-username 

We can then create a folder, a repository within the folder and import our files:

sudo mkdir /svn-repositories/www.domain.com 
sudo svnadmin create /svn-repositories/www.domain.com 
sudo svn import /var/www/www.domain.com file:////svn-repositories/www.domain.com 

This repository needs to be owned by the web user and be granted certain permissions:

sudo chown -R www-data:www-data /svn-repositories/www.domain.com 
sudo chmod -R g+rws /svn-repositories/www.domain.com 

We can then remove our existing site directory and recreate it from our repository:

cd /var/www 
sudo rm -rf /var/www/www.domain.com/ 
sudo svn co file:///svn-repositories/www.domain.com 
sudo chown -R www-data:www-data /var/www/www.domain.com/ 

We can also create a trigger so that every time you commit a change it goes live. This probably wants some modification for production use, i.e. Local -> Staging -> Production with the later step conducted manually.

sudo nano /svn-repositories/www.domain.com/hooks/post-commit 
#!/bin/bash

cd /var/www/www.domain.com
/usr/bin/svn update

This also needs the correct permissions to run:

sudo chown www-data:www-data /svn-repositories/www.domain.com/hooks/post-commit 
sudo chmod +x /svn-repositories/www.domain.com/hooks/post-commit 

We also need to add this repository to a vhost file so that you can check it out. I would suggest a new one for this purpose. Multiple locations can be added, but for now we just have one.

sudo nano /etc/apache2/vhosts/svn.domain.com.conf 
<VirtualHost *:80>
        ServerAdmin webmaster@domain.com
        DocumentRoot /var/www/default/
        ServerName svn.domain.com
        <Directory "/var/www/default/">
                Options -Indexes FollowSymLinks
                AllowOverride FileInfo
                Order allow,deny
                Allow from all
        </Directory>
        <Location /svn/www.domain.com>
        	DAV svn
        	SVNPath /svn-repositories/www.domain.com
        	AuthType Basic
        	AuthName "www.domain.com subversion repository"
        	AuthUserFile /svn-repositories/passwd
        	Require valid-user
        </Location>
</VirtualHost>

Before this can be tested Apache needs to be reloaded:

sudo service apache2 graceful 

You can now check out the repository from "http://svn.domain.com/svn/www.domain.com" using the command line or client (Linux / OSX) or a client such as Tortoise SVN on Windows. You will need to validate using the username / password that you created in this step. If you would rather use the .htpasswd file used for the database and mail admin just update the vhost file with that location.

EC2 Guide: Backing up and clean up (7 / 7)

Logs / Log rotation

Because we changed the paths to our mail logs we need to update our logrotate files. Only update those that have changed:

sudo nano /etc/logrotate.d/rsyslog 
/var/log/postfix/mail.info
/var/log/postfix/mail.warn
/var/log/postfix/mail.err
/var/log/postfix/mail.log
/var/log/dovecot/log
/var/log/dovecot/info
/var/log/dovecot/lda
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                reload rsyslog >/dev/null 2>&1 || true
        endscript
}

Clean up after ourselves:

sudo rm /etc/logrotate.d/rsyslog~ 

Installing a snapshotting tool (cloud only)

One of the great features of EC2 is the ability to take a snapshot of an entire EBS volume (irrespective of size) in about 3-10 seconds. Automating these calls gives some excellent backup potential. I recommend a tool called ec2-consistent-snapshot by Eric Hammond to assist with backups. This is very easy to get set up:

sudo add-apt-repository ppa:alestic 
sudo aptitude update 
sudo aptitude install ec2-consistent-snapshot 

Configure snapshotting (cloud only)

To make our commands simpler and avoid having config files all over the place we can set some environmental variables about our attached EBS volumes. To do this we need to obtain our attached volume IDs from Elasticfox:

Back on the server we will set these values into the environmental variables:

sudo nano /etc/bash.bashrc 
[...]
export VOLUME_ID_CORE=vol-XXXXXXXX
export VOLUME_ID_WEBSERVER=vol-XXXXXXXX
export VOLUME_ID_EMAIL=vol-XXXXXXXX
export VOLUME_ID_DATABASE=vol-XXXXXXXX
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
sudo nano /etc/profile 
[...]
export VOLUME_ID_CORE=vol-XXXXXXXX
export VOLUME_ID_WEBSERVER=vol-XXXXXXXX
export VOLUME_ID_EMAIL=vol-XXXXXXXX
export VOLUME_ID_DATABASE=vol-XXXXXXXX
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

Reload our shell variables:

source /etc/bash.bashrc 
source /etc/profile 
source ~/.bashrc 

We need to make bash script files to run our commands. Firstly we can backup the database volume, which needs to have the MySQL details passed so the snapshot can be taken safely (without losing data).

sudo mkdir /home/ubuntu/snapshotting 
sudo nano /home/ubuntu/snapshotting/database.sh 
#!/bin/bash -l

ec2-consistent-snapshot --description "Database Backup $(date +'%Y-%m-%d %H:%M:%S')" --region "$EC2_REGION" --mysql --mysql-host "localhost" --mysql-socket "/var/run/mysqld/mysqld.sock" --mysql-username "root" --mysql-password "root_db_password" --xfs-filesystem /database $VOLUME_ID_DATABASE

The webserver backup is similar, but does not require MySQL to be stopped:

sudo nano /home/ubuntu/snapshotting/webserver.sh 
#!/bin/bash -l

ec2-consistent-snapshot --description "Webserver Backup $(date +'%Y-%m-%d %H:%M:%S')" --region $EC2_REGION --xfs-filesystem /webserver $VOLUME_ID_WEBSERVER

The e-mail backup is almost identical to the webserver and does not require MySQL to be stopped:

sudo nano /home/ubuntu/snapshotting/email.sh 
#!/bin/bash -l

ec2-consistent-snapshot --description "E-mail Backup $(date +'%Y-%m-%d %H:%M:%S')" --region $EC2_REGION --xfs-filesystem /email $VOLUME_ID_EMAIL

These files need to be executable, and we should clean up any temporary files our editor may have left.

sudo chmod +x /home/ubuntu/snapshotting/* 
sudo rm /home/ubuntu/snapshotting/*~ 

In order to freeze / unfreeze the XFS file-system the actual commands need to run as root so we will add our commands to root's crontab.

su root 
crontab -e 
0 6 * * * /home/ubuntu/snapshotting/database.sh >> /home/ubuntu/backup.log 2>&1
1 6 * * * /home/ubuntu/snapshotting/webserver.sh >> /home/ubuntu/backup.log 2>&1
2 6 * * * /home/ubuntu/snapshotting/email.sh >> /home/ubuntu/backup.log 2>&1

Leave the root user.

exit 

Cleaning up old snapshots (cloud only)

Amazon don't seem to have a tool to do this neatly, and with a bit of research I found several built in PHP (a language I know) so I decided to roll my own based on a few of these. It's only a rough draft, but meets my current needs.

sudo nano /var/www/remove_old_snapshots.php 
#!/usr/bin/php
<?php
# Options may be hard coded here or passed in at run time
$ec2_private_key	= null; # '/home/ubuntu/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem';
$ec2_cert			= null; # '/home/ubuntu/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem';
$regions			= null; # array('eu-west-1', 'us-east-1'); # OR # 'eu-west-1';
$volumes			= null; # array('vol-XXXXXXXX', 'vol-XXXXXXXX'); # OR # 'vol-XXXXXXXX';
$number_to_keep		= null; # array(7, 1); # OR # 7;
$min_number_to_keep	= 1;	# Prevent any accidents

# Optional list of snapshot IDs to omit from this purge process
$omit = array(); # array('snap-XXXXXXXX', 'snap-XXXXXXXX'); # OR # 'snap-XXXXXXXX';

$options = getopt('K:C:v:n:r:o:sd:ed:');
 
if(!isset($options['v']) OR is_null($options['v']))
{
	if(is_null($volumes))
	{
		echo "ERROR: Please provide at least one volume to check (-v option)\n";
		exit;
	}
	else
	{
		$options['v'] = $volumes;
	}
}
 
if(!isset($options['n']) OR is_null($options['n']))
{
	if(is_null($number_to_keep))
	{
		echo "ERROR: Please provide number of snapshots to keep (-n option)\n";
		exit;
	}
	else
	{
		$options['n'] = $number_to_keep;
	}
}
 
if(!isset($options['r']) OR is_null($options['r']))
{
	if(is_null($regions))
	{
		echo "ERROR: Please provide a region for these volumes (-n option)\n";
		exit;
	}
	else
	{
		$options['r'] = $regions;
	}
}
 
if(!isset($options['K']) OR is_null($options['K']))
{
	if(is_null($ec2_private_key))
	{
		echo "ERROR: Please provide an EC2 key file location for these volumes (-K option)\n";
		exit;
	}
	else
	{
		$options['K'] = $ec2_private_key;
	}
}
 
if(!isset($options['C']) OR is_null($options['C']))
{
	if(is_null($ec2_cert))
	{
		echo "ERROR: Please provide an EC2 Cert file location for these volumes (-C option)\n";
		exit;
	}
	else
	{
		$options['C'] = $ec2_cert;
	}
}
 
if(!isset($options['o']) OR is_null($options['o']))
{
	$options['o'] = $omit;
}
 
if(!is_array($options['v']))
{
	$volumes = array($options['v']);
}
else
{
	$volumes = $options['v'];
}
 
if(!is_array($options['n']))
{
	$number_to_keep = array($options['n']);
}
else
{
	$number_to_keep = $options['n'];
}
 
if(!is_array($options['r']))
{
	$regions = array($options['r']);
}
else
{
	$regions = $options['r'];
}
 
if(!is_array($options['K']))
{
	$ec2_private_key = $options['K'];
}
else
{
	echo "ERROR: Please provide a SINGLE EC2 key file location for these volumes (-K option)\n";
	exit;
}
 
if(!is_array($options['C']))
{
	$ec2_cert = $options['C'];
}
else
{
	echo "ERROR: Please provide a SINGLE EC2 Cert file location for these volumes (-C option)\n";
	exit;
}
 
if(!is_array($options['o']))
{
	$omit = array($options['o']);
}
else
{
	$omit = $options['o'];
}
 
foreach($number_to_keep as $key => $value)
{
	if($value < $min_number_to_keep)
	{
		$number_to_keep[$key] = $value;
	}
}
 
$total_volumes	= count($volumes);
$total_kept		= count($number_to_keep);
$total_regions	= count($regions);
 
if($total_volumes != $total_kept)
{
	if($total_volumes > $total_kept AND $total_kept == 1)
	{
		# Use the number to keep as a global value
		foreach($volumes as $key => $value)
		{
			$number_to_keep[$key] = $number_to_keep[0];
		}
	}
	else
	{
		# An error is likely, we should stop
		echo "ERROR: Please check the values entered, there is an argument mismatch (-v vs. -n)\n";
		exit;
	}
}
 
if($total_volumes != $total_regions)
{
	if($total_volumes > $total_regions AND $total_regions == 1)
	{
		# Use the region as a global value
		foreach($volumes as $key => $value)
		{
			$regions[$key] = $regions[0];
		}
	}
	else
	{
		# An error is likely, we should stop
		echo "ERROR: Please check the values entered, there is an argument mismatch (-v vs. -r)\n";
		exit;
	}
}
 
foreach($regions as $key => $region)
{
	$process[$region][$volumes[$key]] = $number_to_keep[$key];	
}
unset($number_to_keep, $regions);
 
$raw = array();
foreach($process as $region => $details)
{
	$tmp = array();
	$cmd = 'ec2-describe-snapshots -K '.$ec2_private_key.' -C '.$ec2_cert.' --region '.$region;
	exec($cmd, $tmp);
 
	if(is_array($tmp))
	{
		foreach($tmp as $key => $value)
		{
			$tmp[$key] .= "\t".$ec2_private_key;
			$tmp[$key] .= "\t".$ec2_cert;
			$tmp[$key] .= "\t".$region;
		}
 
		$raw = array_merge($raw, $tmp);
	}
}
 
$found = array();
foreach($raw as $snapshot)
{
	$tmp = split("\t", $snapshot);
	if(in_array($tmp[2], $volumes) AND !in_array($tmp[1], $omit))
	{
		$tmp[] = $process[$tmp[11]][$tmp[2]];
		$found[$tmp[2]][strtotime($tmp[4])] = $tmp;
	}
}
 
foreach($found as $volume_id => $snapshots)
{
	krsort($found[$volume_id]);
}
 
unset($volumes, $process);
 
foreach($found as $volume_id => $snapshots)
{
	$total = 0;
	foreach($snapshots as $timestamp => $details)
	{
		if($total > $details[12] - 1)
		{
			echo 'Deleting Snapshot: '.$details[1]."\n";
			$out = array();
			$cmd = 'ec2-delete-snapshot -K '.$details[9].' -C '.$details[10].' --region '.$details[11].' '.$details[1];
			exec($cmd, $out);
 
			if(count($out) == 1 AND strcmp($out[0], 'SNAPSHOT	'.$details[1]) == 0)
			{
				echo "COMPLETE!\n";
			}
			else
			{
				echo "ERROR:\n";
				foreach($out as $line)
				{
					echo $line."\n";
				}
			}
		}
 
		$total++;
	}
}

We need to trigger this as a CLI process so we will wrap its call in a bash wrapper just like the other scripts.

sudo nano /home/ubuntu/snapshotting/remove-old.sh 
#!/bin/bash -l

php /var/www/remove_old_snapshots.php -v $VOLUME_ID_WEBSERVER -v $VOLUME_ID_EMAIL -v $VOLUME_ID_DATABASE -n 7 -r $EC2_REGION -K $EC2_PRIVATE_KEY -C $EC2_CERT

The -n value is the number of snapshots of each volume to keep. If you would like to keep different numbers for each specify -n 3 times (one for each). Again this needs to be executable and our editor may have left a temporary file to clean up:

sudo chmod +x /home/ubuntu/snapshotting/* 
sudo rm /home/ubuntu/snapshotting/*~ 

To remain consistent we will add our commands to root's crontab.

su root 
crontab -e 
[...]
3 6 * * * /home/ubuntu/snapshotting/remove-old.sh >> /home/ubuntu/backup.log 2>&1

Leave the root user.

exit 

Hopefully that is everything you will need to do to set up a secure server, so I wish you the best of luck with yours! All comments, criticisms and suggestions welcome, but please comment on the relevant pages to make them as useful as possible, thanks!