Installing OpenVZ on Amazon EC2
Thursday, December 1st, 2011 03:49 pm GMT +2

Imagine, that your system is pretty complicated and consists of a number of components, each deployed into separate machine.  For development & testing needs it’s always too costly to keep up & running all these machines.  It’s not that as easy to make modification into your system structure, i.e. adding new machines with different roles, etc.

One solution is to virtualize all your stuff & isolate environment of each component.  An easy way would be to buy a hardware & setup hardware virtualization such as XEN for instance, but there is more popular approach — using Amazon EC2 cloud services.

In this post I’m going to show you how to setup OpenVZ virtualization on Amazon EC2 node (which, actually itself is virtualized XEN node)

We’re going to use as a base a CentOS 5.6 Amazon image with pv-grub enabled (this is required to be able to boot into custom kernel)

Ok, from the AWS console start CentOS 5.6/x64  (centos-5.6-64bit-ebs-pvgrub / ami-a14f1ce4), obtain public DNS name & make sure that default ssh 22 port is enabled:


ssh -i ~/aws_ssh.key root@ec2-50-18-60-65.us-west-1.compute.amazonaws.com

Install OpenVZ & some dependencies


yum -y install gcc.x86_64 gcc-c++.x86_64 java-1.6.0-openjdk iptables openssl-devel zlib-devel pkgconfig glib2-devel

#Open VZ, Install XEN-compatible kernel!
wget -O /etc/yum.repos.d/openvz.repo http://download.openvz.org/openvz.repo
rpm --import http://download.openvz.org/RPM-GPG-Key-OpenVZ
yum --enablerepo=openvz-kernel-rhel5 -y install ovzkernel-xen.x86_64 ovzkernel-xen-devel.x86_64 vzctl.x86_64 vzquota.x86_64

Override some kernel parameters


echo "net.ipv4.ip_forward = 1
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.all.rp_filter = 1
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts=1
net.ipv4.conf.default.forwarding=1
net.ipv4.conf.default.accept_source_route = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296" > /etc/sysctl.conf
# reload from disk
sysctl -p

#Substitute default kernel by OpenVZ
sed -i 's/kernel/ovzkernel/' /etc/sysconfig/kernel

Install ntpd

This will help to sync time between all virtual machines.

yum install -y ntp.x86_64
/usr/sbin/ntpdate 0.rhel.pool.ntp.org europe.pool.ntp.org
# start on next boot
chkconfig ntpd on

Enable rpmforge repo

Chances are pretty high that you’ll want something from this repo (latest git for example), so you’d better install it now.

rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
wget -O /tmp/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
rpm -Uvh /tmp/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm
yum -y install --enablerepo=rpmforge git.x86_64

Setup firewall

#Setup firewall
echo "*filter" > /etc/sysconfig/iptables
echo "COMMIT" >> /etc/sysconfig/iptables

Install OpenVZ kernel to boot by default

For some reason, yum install process does not add OpenVZ kernel into grub bootlist, therefore we’re doing it manually:

#!/bin/bash

#detect newly installed OpenVZ kernel
VMLINUZ_XEN=`ls /boot/ | grep -i "^vmlinuz" | grep -i "stab"`
INITRD_XEN=`ls /boot/ | grep -i "^initrd" | grep -i "stab"`

if [ $VMLINUZ_XEN ]; then
echo "
title CentOS-openvz-xen
  root (hd0)
  kernel /boot/$VMLINUZ_XEN root=/dev/sda1
  initrd /boot/$INITRD_XEN" >> /boot/grub/menu.lst 

#Make sure we'll boot into newly installed XEN kernel, which have INDEX=1
sed -i 's/default=0/default=1/' /boot/grub/menu.lst
fi

Check your /boot/grub/menu.lst, it should look like following:

default=1
title centos5.6
  root (hd0)
  kernel /boot/vmlinuz-2.6.18-238.5.1.el5xen root=/dev/sda1
  initrd /boot/initrd-2.6.18-238.5.1.el5xen.img

title CentOS-openvz-xen
  root (hd0)
  kernel /boot/vmlinuz-2.6.18-274.3.1.el5.028stab094.3xen root=/dev/sda1
  initrd /boot/initrd-2.6.18-274.3.1.el5.028stab094.3xen.img

Reboot and then login again to check that you’re on OpenVZ kernel:


ssh -i ~/aws_ssh.key root@ec2-204-236-152-176.us-west-1.compute.amazonaws.com
Last login: Tue Nov 29 05:34:50 2011 from 94.45.135.130
[root@ip-10-176-46-67 ~]# uname -a
Linux ip-10-176-46-67 2.6.18-274.7.1.el5.028stab095.1xen #1 SMP Mon Oct 24 21:00:35 MSD 2011 x86_64 x86_64 x86_64 GNU/Linux

Download & install OpenVZ CentOS5 template:

wget http://download.openvz.org/template/precreated/centos-5-x86_64.tar.gz

#Symlink OpenVZ CentOS template
[ -d /vz/template/cache ] || mkdir -p /vz/template/cache
ln -s /root/centos-5-x86_64.tar.gz /vz/template/cache

Create your first OpenVZ container(VM)

vzctl create 101 --ostemplate centos-5-x86_64 > /dev/null
vzctl set 101 --ipadd 10.0.102.101 --save > /dev/null
vzctl set 101 --nameserver 8.8.8.8 --save > /dev/null
vzctl set 101 --name "APP1" --save > /dev/null
vzctl set 101 --onboot yes --save > /dev/null

Try to login into APP1 shell:

vzctl enter 101

At this point, if you was able to create APP1 container, you can setup other components of your system — Database, Cache server, whatever you want.

  • Raj

    Curios , how many ip addresses do we get from AWS ?

    • Anonymous

      Did you mean external(world visible) IP addresses? 

  • Sam

    Excellent work, thanks for this write-up. I was able to copy-paste the instructions exactly and get this working. 

    Note that you will have to “vzctl start 101″ before “vzctl enter 101″.

    Also, this works only with CentOS 5.6 with PV Grub – I couldn’t get it running on a CentOS 6 image.

    • Anonymous

      Thanks for update! Actually, we’ve moved to CentOS 6.0 because of zero swap in older OpenVZ kernels. Pretty heavy stuff like MongoDB/elasticsearch just failed to run on 5.6 no matter how hard we played with container configuration. New RHEL6 based OpenVZ kernel has new memory model, and our infrastructure works like a charm. 

      http://wiki.openvz.org/VSwap

      Configuration on CentOS6 is even simpler, because you don’t have to install special kernel with XEN enabled, as of RHEL6 it’s enabled by default, it looks something like this

      wget -O /etc/yum.repos.d/openvz.repo http://download.openvz.org/openvz.reporpm
      rpm –import http://download.openvz.org/RPM-GPG-Key-OpenVZ 
      yum –enablerepo=openvz-kernel-rhel6 -y install vzkernel.x86_64 vzkernel-devel.x86_64 vzctl.x86_64 vzquota.x86_64

      Another tip is that you should make sure that installer will add a valid record into /boot/grub/menu.lst  because it fails to do so sometimes

  • si

    Hi, Would you tell us approximately how much you are paying to run this configuration on EC2? I’ve thought of running something similar, yet when I look into the EC2 pricing the costs run up too high. Especially if I include use of bandwidth.