Project

General

Profile

Actions

BGBM Xen server documentation


This document describes the general setup Xen servers hosting virtual machines (VMs) at BGBM. Currently, the following servers are configured as Xen servers:

  • wp5demo.e-taxonomy.eu (160.45.63.20)

  • bgbm45.bgbm.fu-berlin.de (160.45.63.135)

All Xen installations base on the Linux distribution Debian 5 (Lenny) for the 64 bit architecture (amd64). Debian Lenny provides packages for Xen version 3.2.1. That way, virtual machines may run on 64 bit or 32 bit architectures alternatively.

Regarding partition management, all Xen servers use Logical Volume Manager (LVM2) LVM permits flexible and individual storage attribution to virtual machines.

The setup of all Xen servers at BGBM should be as similar as possible. That way, improvements on one Xen server can easily be transmitted to any others. Synchronising these different hosts will be done through the version management tool Subversion For reasons of precaution, some adoptions to the usual file system must be done to enable version management with different repository locations.

The following sections shall serve as documentation and setup guide for the installation of similar Xen server hosts.

Debian System Installation

Please follow the instructions of the Debian Linux Installation Guide.

Xen Installation

Based on a core Debian System 64 bit (amd64) installation, use apt-get to install the required software packages as follows:

apt-get install xen-linux-system-2.6.26-2-xen-amd64 xen-utils-3.2-1 xenstore-utils xenwatch

All Xen servers are configured for Xen network bridging, which allows direct internet access and individual IP address assignment to all virtual machines hosted on those servers. To configure Xen network bridging, ensure the following settings in /etc/xen/xend-config.sxp:

(network-script network-bridge)
...
(vif-script vif-bridge)
...
#(network-script network-dummy)

Now we can reboot the system to use the xen kernel!

reboot

If the reboot was succesful, entering the following command should produce the subsequent output:

# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     3922     2 r-----      5.9

Xen-Shell Installation

Xen-Shell provides an easy to use xen administration shell for virtual machines managed by dedicated users. These users can access the Xen server host using SSH only. Therefore, beside xen-shell, also software package sudo is required to enable those users for running the corresponding xen commands with root user rights.

apt-get install xen-shell sudo

Next, we must use the command visudo to edit the configuration of sudo.

If you like to change root's system editor e.g. to vi, run the following command and select your favorite editor.

update-alternatives --config editor

Now, running visudo will open /etc/sudoers with your favorite editor. Adding the following lines would enable the users listed in the user alias XENUSERS (e.g. vmadmin1 and vmadmin2) to run the Xen commands specified with XEN and XENIMG without entering a password.

###########################
#### Xen-Shell entries ####
###########################
User_Alias   XENUSERS = vmadmin1,vmadmin2
Cmnd_Alias   XEN      = /usr/sbin/xm
Cmnd_Alias   XENIMG   = /usr/bin/xen-create-image
XENUSERS     ALL      = NOPASSWD: XEN,XENIMG

Next, add any users listed in XENUSERS to your system, and make xen-shell their login shell

adduser vmadmin1
chsh -s /usr/bin/xen-login-shell vmadmin1

After that, ask the xen-shell users for their SSH public keys and repeat the following steps for each newly created user.

First, create a subdirectory .ssh and the file .ssh/authorized_keys. Edit authorized_keys and paste the content of the submitted SSH public key file into. Take care that the line with the public key is not splitted, i.e., the public key must be in a single line.

mkdir -p /home/vmadmin1/.ssh
vi /home/vmadmin1/.ssh/authorized_keys
Cut&Paste the public Key:
ssh-rsa AAAAB3NzaC1yc2EAGwMwjHTaOpXBBT9FMMNTJtqYIULkxhthzxl1/f4Zo0K4sMOmtivUVUSYZPpnBtfHGX9AkGOs8My5f8/KdRCTqimw1lYz1QOPdb9nnmIC4NqfEA++OlQ92Jz9tft3CFncL+8v61BTygjulV/1a4Vx+KGD2LsKoBxLKxFVRq+Py8TqR5uvx6lN/KdRCTqimw1lYz1QOPdb9nVblQ5YDVKR0YCRuW2GO4xG2umFEQok7Pw8L8Ma/Zog4dWMeT41X0pzoOT5Ecf3YIFXyTw== vmadmin1

Then, set appropriate file and directory access rights to the home and .ssh directory of these users

chown -R vmadmin1:vmadmin1 /home/vmadmin1
chmod 700 /home/vmadmin1/.ssh
chmod 600 /home/vmadmin1/.ssh/authorized_keys

Ok, having added the users, the last step is to restart the sudo daemon

/etc/init.d/sudo restart

Now, if a users connects to the Xen server host via SSH, xen-shell should attempt to start.

Note: xen-shell works in cooperation with Xen's virtual machines configuration files. That means, for each virtual machine running on your system, a line denoting the user names enabled for managing the given virtual machine must be added to the relating xxx.cfg file in the /etc/xen/ directory.

xen_shell = 'vmadmin1,vmadmin2'

If the connecting user had not been added to any existing virtual maching configuration file, xen-shell would simply abort and close the connection!

Xen-Tools Installation

Xen-tools is a script collection allowing to easily create new Xen virtual machines. It can be installed to Xen server hosts running the following command

apt-get install xen-tools

Extended Xen-tools installation

For the BGBM Xen server hosts, xen-tools has been extended and adopted to our individual needs. For instance, more scripts and script libraries have been added to create, backup and manage virtual machines as well as for automatically setting up machines for specific roles (e.g. Shibboleth IdP and SP, Drupal hosts etc.).

Therefore, the BGBM Xen server hosts use a dedicated partitions for xen-tools, which is mounted to the file system at /xen. Here is how it was created.

First, a logical volume "xen" was created and formatted with an ext3 file system. Next, create the directory "/xen" and mount the logical volume "xen" to that directory

Note: Adopt the volume group name "vg" and the size of the logical volume ( "-L"parameter of lvcreate command ).

lvcreate -L 20G -n xen vg
mkfs.ext3 /dev/vg/xen

mkdir /xen
mount /dev/vg/xen /xen

In order to mount the "xen" logical volume at boot time to the /xen directory add the following lines to the file /etc/fstab:

/dev/vg/xen    /xen            ext3    defaults        1       2

The most recent revision of the extended xen-tools can be retrieved from the following subversion repository http://dev.e-taxonomy.eu/svn/trunk/security/xen.

You can install subversion with the following command to your host:

apt-get install subversion

In order to get a fresh copy of the system, the following command gets the latest revision from the subversion repository and copies it below the /xen directory

svn checkout http://dev.e-taxonomy.eu/svn/trunk/security/xen /xen

Now, the following directory hierarchie should have been created below the "/xen" directory:

| backup | stores vm backups by vm hostnames |
| boot | stores Linux Kernels needed for booting by distribution and system architecture |
| etc/xen-tools | stores anything related to run xen-tools |
| etc/xen-tools/hooks | stores script libraries |
| etc/xen-tools/lib | stores script libraries |
| etc/xen-tools/partitions.d | stores script libraries |
| etc/xen-tools/resources | stores resources already downloaded from the web |
| etc/xen-tools/role.d | stores resources already downloaded from the web |
| etc/xen-tools/skel | stores skeletons for vm configurations |
| images | stores basic images of vm prototypes |
| mnt | provides mount points for individual vm file systems |
| sbin | stores extended scripts for xen management |
| skel | stores skeletons for the Xen server host |

Next we touch the original xen-tools installation and redirected any necessary files to our xen file hierarchie on /xen.

First, xen-tools comes with some scripts called hooks, which will be executed whenever a new virtual machine is being created. As we want to control this ourselves, we need to copy some of them and create some links pointing to the file hierachie on "/xen".

For the Debian Lenny distribution, create a lenny.d directory in our hooks directory, copy the original file into and relink the original lenny.d to /xen/etc/xen-tools/hooks/lenny.d.

mkdir -p /xen/etc/xen-tools/hooks/lenny.d
cp -av /usr/lib/xen-tools/debian.d/* /usr/lib/xen-tools/lenny.d/
unlink /usr/lib/xen-tools/lenny.d
ln -s /xen/etc/xen-tools/hooks/lenny.d/ /usr/lib/xen-tools/

Since, we have some other requirements, the following modifications were made to the hook scripts.

| 35-setup-users | deactivated ( "chmod -x /xen/etc/xen-tools/hooks/lenny.d/35-setup-user" |
| 60-copy-host-files | modified, copying of Xen server sudo configuration deactivated |
| 65-copy-user-files | deactivated (chmod -x /xen/etc/xen-tools/hooks/lenny.d/65-copy-user-files") |
| 95-configure-locales | deactivated (chmod -x /xen/etc/xen-tools/hooks/lenny.d/95-configure-locales"), locale setup in minimal role |

Any of the newly introduced xen scripts can be adopted to individual system configurations (e.g. LVM volume group name) by environment variables. The usual place for those system wide configurations is /etc/profile. For better distinguishing system relevant configurations and global xen configurations, any global configurations necessary for our xen-tools setup will be located below /xen/skel/host. Symbolic links will be created in the system pointing at the corresponding locations within the xen host skeleton. System relevant configuration will be added or modified in the original system configuration files on the Xen server host (e.g. /etc).

The main intention of these configuration issues is for version management. That way, individual system configurations and global xen configurations can be strictly separated and managed within different repositories. Also, modifications to our xen-tools development can be easily upgraded using subversion and does not bother individual system configurations.

Here are the most relevant modifications:

/etc/profile

This file must be extended by the following lines:

. /etc/profile.lvm
. /etc/profile.xen

/etc/profile.lvm

defines environment variables for system individual LVM settings. The LVM_VG variable must be set to the local volume group used for xen virtual machines.

export LVM_PROFILE=/etc/profile.lvm
export LVM_VG='vg1'

/etc/profile.xen

includes any required environment variables for the extended xen-tools. Herein, only the following variables may be subject of modification

# base path of the xen script environment
export XENBASE_PATH='/xen'

# path to the original xen-tools distribution's configuration directory
export DIST_XENTOOLS_PATH=/etc/xen-tools

Anyway, only /etc/profile and /etc/profile.xen should be linked to the Xen server host skeleton. Rename, copy or delete the file before linking!

ln -s /xen/skel/host/etc/profile /etc/
ln -s /xen/skel/host/etc/profile.xen /etc/

Also, the following links to the xen system should be created

ln -s /xen/skel/host/etc/hosts /etc/
ln -s /xen/skel/host/etc/ssh/sshd_config /etc/ssh
ln -s /xen/skel/host/root/.bashrc /root

Please find more documentation on how to use the extended Xen-tools here.

Firewall setup (TODO)

Using Xen's NAT network configuration, connecting our virtual machines e.g. via the ssh protocol, we have to configure the firewall of our dom0 to forward e.g. port 110 to port 22 of our virtual machine. Therefore, store the following script in /etc/network/if-up.d/iptables and make it executable (chmod 755).

#!/bin/sh
### Clean up rules ###
/sbin/iptables -F
### Nothing in ###
/sbin/iptables -P INPUT DROP
### All out ###
/sbin/iptables -P OUTPUT ACCEPT
### Accept all localhost traffic ###
/sbin/iptables -A INPUT -j ACCEPT -i lo
### Accept all icmp traffic ###
/sbin/iptables -A INPUT -j ACCEPT -p icmp
### Accept all responses ###
/sbin/iptables -A INPUT -j ACCEPT -m conntrack --ctstate ESTABLISHED,RELATE
### Accept Incoming Ports ###
/sbin/iptables -A INPUT -j ACCEPT -p tcp --dport ssh
/sbin/iptables -A INPUT -j ACCEPT -p tcp --dport http
### Accept SSH-Port for base.wp5demo.org ###
/sbin/iptables -A INPUT -j ACCEPT -p tcp --dport 110
### Forwarding Ports ###
### Forward SSH-Port for base.wp5demo.org ###
/sbin/iptables -A PREROUTING -t nat -p tcp -i eth0 --dport 110 -j DNAT --to 10.0.0.10:22

Now you should be able to login to the base virtual machine using ssh on port 110.

Updated by Andreas Müller almost 2 years ago · 43 revisions