Weird rocketry

Imagine, that your system is pretty complicated and consists of a number of components, each deployed into separate machine.  For development & testing needs it’s always too costly to keep up & running all these machines.  It’s not that as easy to make modification into your system structure, i.e. adding new machines with different roles, etc.

One solution is to virtualize all your stuff & isolate environment of each component.  An easy way would be to buy a hardware & setup hardware virtualization such as XEN for instance, but there is more popular approach — using Amazon EC2 cloud services.

In this post I’m going to show you how to setup OpenVZ virtualization on Amazon EC2 node (which, actually itself is virtualized XEN node)

We’re going to use as a base a CentOS 5.6 Amazon image with pv-grub enabled (this is required to be able to boot into custom kernel)

Ok, from the AWS console start CentOS 5.6/x64  (centos-5.6-64bit-ebs-pvgrub / ami-a14f1ce4), obtain public DNS name & make sure that default ssh 22 port is enabled:

ssh -i ~/aws_ssh.key root@ec2-50-18-60-65.us-west-1.compute.amazonaws.com

Install OpenVZ & some dependencies

yum -y install gcc.x86_64 gcc-c++.x86_64 java-1.6.0-openjdk iptables openssl-devel zlib-devel pkgconfig glib2-devel

#Open VZ, Install XEN-compatible kernel!
wget -O /etc/yum.repos.d/openvz.repo http://download.openvz.org/openvz.repo
rpm --import http://download.openvz.org/RPM-GPG-Key-OpenVZ
yum --enablerepo=openvz-kernel-rhel5 -y install ovzkernel-xen.x86_64 ovzkernel-xen-devel.x86_64 vzctl.x86_64 vzquota.x86_64

Override some kernel parameters

echo "net.ipv4.ip_forward = 1
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.rp_filter = 1
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.conf.default.forwarding=1
net.ipv4.conf.default.accept_source_route = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296" > /etc/sysctl.conf
# reload from disk
sysctl -p

#Substitute default kernel by OpenVZ
sed -i 's/kernel/ovzkernel/' /etc/sysconfig/kernel

Install ntpd

This will help to sync time between all virtual machines.

yum install -y ntp.x86_64
/usr/sbin/ntpdate 0.rhel.pool.ntp.org europe.pool.ntp.org
# start on next boot
chkconfig ntpd on

Enable rpmforge repo

Chances are pretty high that you’ll want something from this repo (latest git for example), so you’d better install it now.

rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
wget -O /tmp/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
rpm -Uvh /tmp/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
yum -y install --enablerepo=rpmforge git.x86_64

Setup firewall

#Setup firewall
echo "*filter" > /etc/sysconfig/iptables
echo "COMMIT" >> /etc/sysconfig/iptables

Install OpenVZ kernel to boot by default

For some reason, yum install process does not add OpenVZ kernel into grub bootlist, therefore we’re doing it manually:

#!/bin/bash

#detect newly installed OpenVZ kernel
VMLINUZ_XEN=`ls /boot/ | grep -i "^vmlinuz" | grep -i "stab"`
INITRD_XEN=`ls /boot/ | grep -i "^initrd" | grep -i "stab"`

if [ $VMLINUZ_XEN ]; then
echo "
title CentOS-openvz-xen
  root (hd0)
  kernel /boot/$VMLINUZ_XEN root=/dev/sda1
  initrd /boot/$INITRD_XEN" >> /boot/grub/menu.lst 

#Make sure we'll boot into newly installed XEN kernel, which have INDEX=1
sed -i 's/default=0/default=1/' /boot/grub/menu.lst
fi

Check your /boot/grub/menu.lst, it should look like following:

default=1
title centos5.6
  root (hd0)
  kernel /boot/vmlinuz-2.6.18-238.5.1.el5xen root=/dev/sda1
  initrd /boot/initrd-2.6.18-238.5.1.el5xen.img

title CentOS-openvz-xen
  root (hd0)
  kernel /boot/vmlinuz-2.6.18-274.3.1.el5.028stab094.3xen root=/dev/sda1
  initrd /boot/initrd-2.6.18-274.3.1.el5.028stab094.3xen.img

Reboot and then login again to check that you’re on OpenVZ kernel:

ssh -i ~/aws_ssh.key root@ec2-204-236-152-176.us-west-1.compute.amazonaws.com
Last login: Tue Nov 29 05:34:50 2011 from 94.45.135.130
[root@ip-10-176-46-67 ~]# uname -a
Linux ip-10-176-46-67 2.6.18-274.7.1.el5.028stab095.1xen #1 SMP Mon Oct 24 21:00:35 MSD 2011 x86_64 x86_64 x86_64 GNU/Linux

Download & install OpenVZ CentOS5 template:

wget http://download.openvz.org/template/precreated/centos-5-x86_64.tar.gz

#Symlink OpenVZ CentOS template
[ -d /vz/template/cache ] || mkdir -p /vz/template/cache
ln -s /root/centos-5-x86_64.tar.gz /vz/template/cache

Create your first OpenVZ container(VM)

vzctl create 101 --ostemplate centos-5-x86_64 > /dev/null
vzctl set 101 --ipadd 10.0.102.101 --save > /dev/null
vzctl set 101 --nameserver 8.8.8.8 --save > /dev/null
vzctl set 101 --name "APP1" --save > /dev/null
vzctl set 101 --onboot yes --save > /dev/null

Try to login into APP1 shell:

vzctl enter 101

At this point, if you was able to create APP1 container, you can setup other components of your system — Database, Cache server, whatever you want.