Chapter 1: Introduction
This document is for the administrator of oolong, a Dell PowerEdge 1950 (abbr is PE1950) rack server that host Hypermedia Lab Game Research Group's website and web services. The document describes the hardware specifications, operating systems, virtualization tools, services, and related configurations running on oolong.
Chapter 2: Server Specifications
This chapter includes the hardware specifications and system architectures of oolong.
2.1 Hardware Specifications
All hardwares are originally come with PE1950, there are no extra parts.
Item | Specification | Quantity |
---|---|---|
Model | Dell PowerEdge 1950 1U rack server | 1 |
CPU | Intel Xeon Quad Core | 1 |
Hard Disk Drive | 143GB SAS HDD | 2 |
RAM | 2GB | 2 |
Power | 2 | |
RAID Controller | Dell PERC 5/i SAS/SATA/SCSI RAID Controller (LSI MegaRAID) | 1 |
Network | Broadcom NetXtreme II Gigabit Ethernet | 2 |
Support | 3 Year, 4 hour 24/7 onsite sSilver upport | N/A |
Operating System | None | None |
2.2 System Architecture
The system uses Xen virtualization software to run three virtual systems. Xen is an open source virtualization solution which supports a wide range of virtual operating systems including Windows, Linux, Solaris, and various versions of the BSD operating systems. It is theorically possible to run all mentioned operating systems on Xen, but due to the host OS and hardware, it is simply more difficult to get some operating systems working well than others, for example, I could not run FreeBSD well on a Debian-based Xen box. But there are more success stories about running Windows and Linux on Xen.
The base operating system (called Dom0 in Xen) is Debian GNU/Linux 4.0 (a.k.a Debian Etch), the virtual systems (called DomU in Xen) running on it are two Ubuntu 6.06 LTS and one Debian Etch. The respective operating systems are listed in the table below:
Dom0/DomU | Operating System |
---|---|
Dom0 | Debian Etch |
DomU | Ubuntu 6.06 LTS |
DomU | Debian Etch |
DomU | Ubuntu 6.06 LTS |
2.2.1 Network Interfaces, IPs and hostnames
There are six static IPs available for use, two are for the physical ethernet network interface on Dom0 with specified Mac address, four are for DomU virtual machines, listed as below:
Note: because the server will be moved to university's computer center, all the IP addresses will change after moving.
NIC/DomU | IP | Hostname |
---|---|---|
NIC (mac: 00:19:B9:EB:BE:85) | 153.1.17.22 | oolong.uta.fi |
NIC (mac: 00:19:B9:EB:BE:87) | 153.1.17.26 | oolong1.uta.fi |
DomU | 153.1.17.86 | oolong2.uta.fi |
DomU | 153.1.17.87 | oolong3.uta.fi |
DomU | 153.1.17.88 | oolong4.uta.fi |
DomU | 153.1.17.89 | oolong5.uta.fi |
Chapter 3: System Installation
This chapter describes the system-level configurations and how to install required operating systems on oolong.
3.1 RAID System
oolong uses RAID 1 (one-to-one mirror) backup, so the actually available HDD space is 143 GB, the other HDD is for mirroring. To modify the RAID configuration, press Ctrl+R to enter the PERC 5/i controller utility at booting.
3.2 Debian Etch Installation
To install Debian Etch, download the CD/DVD image from http://ftp.funet.fi/pub/linux/mirrors/debian-cdimage/. No special tweaks are required during the installation process, network card and RAID controller are automatically detected and activated. But the network driver has to be upgraded and the network configuration has to be modified after installation.
3.2.1 Network Interface Card Configuration
Dell PE1950 present their onboard network cards as Gb1 and Gb2 in BIOS, but these map to eth1 and eth0 respectively ("backwards") under Linux. This issue is fixed in kernel 2.6.19-rc3 and higher. But the current kernel of oolong is kernel 2.6.18-5, so manually changing the network card configurations is necessary. In our case, the default dhcp configuration will assign IP addresses and Hostnames to each network card as following:
Mac address | Card name | IP | Hostname |
---|---|---|---|
00:19:B9:EB:BE:85 | eth1 | 153.1.17.26 | oolong.uta.fi |
00:19:B9:EB:BE:87 | eth0 | 153.1.17.22 | oolong1.uta.fi |
This is different from our expectation: eth0 as oolong.uta.fi and eth1 as oolong1.uta.fi. The solution is to switch the network mac address, make 00:19:B9:EB:BE:85 become eth0's mac address and 00:19:B9:EB:BE:87 become eth1s'. To change mac address information, edit /etc/udev/rules.d/z25_persistent-net.rules. The modified file is as below:
# This file was automatically generated by the /lib/udev/write_net_rules # program, probably run by the persistent-net-generator.rules rules file. # # You can modify it, as long as you keep each rule on a single line. # MAC addresses must be written in lowercase. # PCI device 0x14e4:0x164c (bnx2) SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:19:b9:eb:be:87", NAME="eth1" # PCI device 0x14e4:0x164c (bnx2) SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:19:b9:eb:be:85", NAME="eth0"
In this way, we can guarantee that dhcp will assign correct IP to each ethernet network card. /etc/udev/rules.d/z25_persistent-net.rules is automatically generated at first time startup, after that, it will be modified if there are new interfaces hardware added to the system, but the existing content will remain the same. So unless you reinstall the system or manually modify the existing content, otherwise, the configuration file will not be changed. More information about /etc/udev/rules.d/z25_persistent-net.rules can be found at Debian Wiki udev.
3.2.2 Add Ethernet Network Card
Only one network card is detected and activated at installation, so we have to manually add and configure another ethernet card. For a detail document on how to configure network on Debian, refer to Debian Wiki - NetworkConfiguration. The network interface configuration file is /etc/network/interfaces, the current content is as following:
# This file describes the network interfaces available on your system # and how to activate them. For more information, see interfaces(5). # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 allow-hotplug eth0 # iface eth0 inet dhcp iface eth0 inet static address 153.1.17.22 netmask 255.255.255.0 gateway 153.1.17.5 auto eth1 allow-hotplug eth1 # iface eth1 inet dhcp iface eth1 inet static address 153.1.17.26 netmask 255.255.255.0 gateway 153.1.17.5 up /sbin/iptables-restore /etc/network/iptable
Although as mentioned in 3.2.1 that dhcp works, I personally perfer specifying network configuration information manually, just a double check. Besides the default eth0, eth1 and its configurations are added to the file. The last line up /sbin/iptables-restore /etc/network/iptable activates the firewall. After changing /etc/network/interfaces, it is required to restart the network to make the new configuration working. To relaunch the network, run /etc/init.d/networking restart with root permission.
3.2.3 Upgrade Network Driver
The original Broadcom NetXtreme II network card driver (bnx2 v1.4.44) that comes with Debian Etch can not work properly on Xen, this issue is solved in version 1.5.10c, so the network card driver has to be upgraded to at least or higher than 1.5.10c. To download the latest driver, visit http://www.broadcom.com/support/ethernet_nic/netxtremeii.php, or there is a copy of bnx2 1.5.10c driver at //arnika/laitos$/hyper/Projektit/Gamelab/Server_documents/PE1950. Refer to the xensource mailing list for the detail instructions. The complete email content is copied as below:
I have a new x3655 IBM and whenever the /etc/xen/scripts/network-bridge script starts, the ethernet would no longer work. This post isn't about the cause, but the fix. I did a lot of tcpdumps and Googling, but I'll spare you that. All they do is prove that yes, there is an issue. :) I tried Debian 4.0 i386/amd64 and Ubuntu 7.04 Server i386/amd64 and they all exhibit the problem. Upgrading the bnx2 driver is the key. I came to find bnx2 is included in the initrd. Debian comes with 1.4.44 and Ubuntu 1.4.45 of the driver. As of this writing 1.5.10c is the latest from Broadcom and is what I ended up using. Summary: * Install kernel source, make config/prepare the distro's config file * Download bnx2 source, compile and install * Update initrd Here are step by step instructions for clean install of Debian 4.0 amd64: aptitude install linux-source unzip cd /usr/src tar jxf linux-source-2.6.18.tar.bz2 ln -s linux-source-2.6.18 linux cd linux cp /boot/config-2.6.18-5-xen-amd64 ./.config make oldconfig (through all the prompts to take defaults) make prepare cd /tmp wget http://www.broadcom.com/docs/driver_download/NXII/linux-1.5.10c.zip unzip linux-1.5.10c.zip cp Server/Linux/Driver/bnx2-1.5.10c.tar.gz /usr/src cd /usr/src tar zxf bnx2-1.5.10c.tar.gz cd bnx2-1.5.10c/src make make install mv /boot/initrd.img-2.6.18-5-xen-amd64 \ /boot/initrd.img-2.6.18-5-xen-amd64.orig update-initramfs -k `uname -r` -c reboot Verify: zcat /boot/initrd.img-2.6.18-5-xen-amd64.orig|strings|grep "bnx2 v" Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.4.44 (August 10, 2006) zcat /boot/initrd.img-2.6.18-5-xen-amd64|strings|grep "bnx2 v" Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.5.10c (May 4, 2007) Other keywords: Dell DELL -- Kris S. Amundson Founder, CIO GPG Key: D6D39F2C OpenSourcery, LLC. http://www.opensourcery.com/
3.3 Xen Installation
To install Xen, follow the The Perfect Xen 3.1.0 Setup For Debian Etch (i386) on Howtoforge. My experience with installing Xen from the binary package and from source all went smoothly, so the installation should not be a problem. The major difference between binary package and source is the ability to compile kernel for the virtual system. If you prefer customizing kernels for different virtual systems, you should install Xen from source. That is what I did to oolong. In The Perfect Xen 3.1.0 Setup For Debian Etch (i386), Page 5 and Page 6 decribe how to install Xen 3.1 from source step-by-step. Be sure to double check the driver version of the network card after Xen is installed, it must be higher than bnx2 1.5.10c.
If you plan to compile a new kernel for virtual system, besides enabling quota and iptables as modules in the kernel configuration menu (as mentioned in Page 5), you also have to enable SMB and CIFS network file systems support, otherwise, the system can not connect to our network hard disk drive. In the kernel configuration menu, you can find SMB and CIFS options at:
File Systems ---> Network File Systems ---> [M]SMB [M]CIFS
Chapter 4: Xen Configuration
All Xen-related configuration and scripting files are placed in /etc/xen, xend-config.sxp is the core Xend configuration file.
4.1 Base system (Dom) Configurations
As a host of all other virtual machines, Dom0 itself does not run any particular services, only OpenSSL server so we can ssh to it. We need to configure the network so that Dom0 can provide network for all DomU and keep itself safe from unknown connections at the same time.
A basic introduction to Xen network can be found at http://wiki.xensource.com/xenwiki/XenNetworking.4.1.1 Network Configuration
Xen provides two types of networking: bridging and routing, oolong uses bridging. For security reason, Dom0 does not share ethernet network card with DomUs. There are two ethernet network cards on oolong, eth0 is for Dom0, eth1 is for DomUs. The figure is illustrated as following:

To achieve above configuration, we do not use the default xend network configuration in xend-config.sxp. Instead, we create our own network bridge script and add it to xend-config.sxp. Open /etc/xen/xend-config.sxp, find the line (network-script oolong-network), oolong-network is the network bridge script created for our needs, you can replace this script with any other scripts you create for oolong.
All xen-related scripts are placed in /etc/xen/scripts, in /etc/xen/scripts, there is a file called oolong-network, the content is very simple:
dir=$(dirname "$0") "$dir/network-bridge" "$@" vifnum=0 netdev=eth1 bridge=xenbr1
It merely calls the network-bridge script with vifnum=0 netdev=eth1 bridge=xenbr1 as parameters, then network-bridge will do the magic for us.
With those parameters, we specify eth1 as the network interface for DomU, and specify the name of bridge as xenbr1. A detailed example article can be found at http://www.debian-administration.org/articles/470.4.1.2 Firewall Configuration
Besides the firewall settings in the university firewall, there are also firewall running on Dom0. The policy is: Dom0 is only accessible via eth0 (153.1.17.22/oolong.uta.fi), access to Dom0 via eth1 (153.1.17.26/oolong1.uta.fi) is not allowed. It is valid connecting to DomUs from Dom0, but connecting to Dom0 from DomUs is not allowed.
By now, only 153.1.17.87/oolong3.uta.fi is open for external http connection requests (port 80 and 443), other IPs are not open to the network outside our university, so they are only accessible from within the university network.
The firewall tool running on Dom0 is iptables, iptables is a firewall tool and is also a command to manage iptables rules in Linux. Use command man iptables to learn how to use iptables. After changing iptables rules, you can save the rules by using iptables-save. The iptables rules will be saved to /etc/network/iptable, and they will be automatically loaded when boot, so that you don't have to set up iptables rules every time. If you plan to change iptables rules, do not change /etc/network/iptable directly, instead, you should use iptables command, after you are sure the iptables rules work as expected, use iptables-save to save the rules. Before saving new iptables rules to /etc/network/iptable, you should back up the current one. The current iptables rules on oolong is as below:
# Generated by iptables-save v1.3.6 on Wed Jan 23 17:33:16 2008 *nat :PREROUTING ACCEPT [2660:485548] :POSTROUTING ACCEPT [1166:193315] :OUTPUT ACCEPT [57:3666] COMMIT # Completed on Wed Jan 23 17:33:16 2008 # Generated by iptables-save v1.3.6 on Wed Jan 23 17:33:16 2008 *filter :INPUT DROP [821:130868] :FORWARD ACCEPT [72814:100421356] :OUTPUT ACCEPT [613:72279] -A INPUT -d 153.1.17.22 -i eth0 -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -i eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT COMMIT # Completed on Wed Jan 23 17:33:16 2008
What those rules do are:
- The default INPUT policy is DROP.
- If the input connection request is received from eth0 and the destination is to 153.1.17.22, the connectio request is accepted.
- If the input connection request is a reponse to a related or established output connections to eth1, the connection request is accepted.
- Any input connection requests other than the above two, are dropped by default.
The iptables rules are automatically applied through /etc/network/interfaces at booting. To stop this, remove or comment /sbin/iptables-restore /etc/network/iptable in /etc/network/interfaces.
4.2 Virtual system (DomU) Configurations
xen-tool is used to create DomU images, a global configuration file (xen-tools.conf) is places in /etc/xen-tools/. Values can be set in this file so they don't need to be specified everytime upon the command line. The command to create a DomU image is xen-create-image, take oolong3.uta.fi for example, the command is:
xen-create-image --hostname=oolong3.uta.fi --ip=153.1.17.87 --dist=etch --mirror=http://ftp.fi.debian.org/debian/ --size=30GbOther parameters required for creating DomU images are specified in /etc/xen-tools/xen-tools.conf. You can also overwrite the values by specifying them upon the command line. xen-create-image creates the DomU image and configuration file for every created DomU image.
The configuration file of every DomU image are stored at /etc/xen/*.cfg. To make the network working, one line has to be modified for every .cfg file. In each .cfg file, there is a block like:
# # Networking # vif = [ 'ip=,bridge=xenbr0' ]
Make sure the IP address is correct, then change the bridge=xenbr0 to bridge=xenbr1. In our configurations, the name of the bridge is xenbr1 not xenbr0.
This table lists the major configuration parameters of every DomU:
hostname | ip | dist | mirror | size | memory | swap |
---|---|---|---|---|---|---|
oolong2.uta.fi | 153.1.17.86 | dapper | http://gb.archive.ubuntu.com/ubuntu/ | 45Gb | 1.5Gb | 256Mb |
oolong3.uta.fi | 153.1.17.87 | etch | http://ftp.fi.debian.org/debian/ | 30Gb | 512Mb | 256Mb |
oolong4.uta.fi | 153.1.17.88 | dapper | http://gb.archive.ubuntu.com/ubuntu/ | 20Gb | 256Mb | 256Mb |
Chapter 5: oolong2.uta.fi
oolong2.uta.fi hosts the Gamelab's website, the administrator is Jussi Kuittinen.
Chapter 6: oolong3.uta.fi
oolong3.uta.fi hosts the web services related to Gamelab's researches, including a movabletype blog system (http://www.pelitutkimus.fi) and a mediawiki wiki system (http://www.gamescultures.org) so far. This chapter describes the web services running on oolong3.uta.fi, their related configuration, maintance and backup policy.
The web services on oolong3.uta.fi are built upon: Apache web server, MySQL database system, Movabletype, and Mediawiki. The versions are listed as below:
Software | version |
---|---|
Apache | 2.2.3 |
MySQL | 5.0 |
Movabletype | MTOS 4.1 |
Mediawiki | 1.11.1 |
6.1 Database Management
Database management can be done with web interface, the URL is http://oolong3.uta.fi/phpmyadmin/. The database name and user name are listed below:
user | database | password |
---|---|---|
root | root can access and manipulate all databases | |
pelitutkimus | mt | to be filled |
gamescultures | mediawiki | to be filled |
6.2 Apache Configuration
We use Apache virtual host feature to host different website with different domain names, each website has its own configuration. The configuration files are place at /etc/apache2/sites-available, pelitutkimus is for pelitutkimus.fi and gamescultures is for gamescultures.org.
Debian has its special directory design for virtual host managment. To add a new website:
- create a virtual host configuration file for that site and put the file (preferably) in /etc/apache2/sites-available.
- In /etc/apache2/sites-enable, create a symbolic link to the created configuration file.
- Restart apache, I usually use apache2ctl graceful.
6.3 Backup Policy
The backup directory on oolong3 is /backup. There are two backup scripts doing daily backup: /etc/cron.daily/automysqlbackup and ~/root/back/back.pl. /etc/cron.daily/automysqlbackup creates daily, weekly and monthly database backup and stores them in /backup/db, ~/root/backup/backup.pl backups /backup/db, /etc/apache2, /var/www/pelitutkimus.fi, and /var/www/gamescultures.org, and copies the generated tar.gz file to the network hard disk drive //arnika/laitos$/hyper/Projektit/oolong3_Backup/ as remote backup.
On oolong3, the network hard disk drive arnika is auto mounted at booting to /mnt/arnika, where /mnt/arnika/hyper/Projektit/oolong3_Backup/ is a hidden directory that is only visible to Frans, Mikael and Chienting, any permission changes has to be configured by Mikael. To change the auto mount configuration, edit the file /etc/fstab and /etc/arnikapw. By now, the username and password used for connecting //arnika is Chienting's username and password, they will have to be changed in the future.
6.4 Pelitutkimus.fi
pelitutkimus.fi is a website providing information about game, game research and game design in Finnish. It is built by Movable Type Open Source 4.1 (MTOS4.1), with Mollio as its design template.
The configuration file of Movable Type is /var/www/pelitutkimus.fi/mt/mt-config.cgi, it is automatically generated when installing Movable Type, it can also be manually modified to further meet our requirement, just remember to backup before making any changes to the file. Be sure to keep this file safe, because it also contains the database username and password used by Movable Type. Information on all configuration options can be found at http://www.movabletype.org/documentation/config. To manage pelitutkimus.fi, connect to log in page, the administrator username and password are: admin / (to be filled).
The Movable Type User and Administration Manual is stored at //arnika/hyper/Projektit/Gamelab/Server_documents/MovableType, more documentation can be found at movabletype.org/documentation.
6.5 Gamescultures.org
gamescultures.fi is a website for creating knowledge about games cultures. It is built by Mediawiki 1.11.1. The configuration file of Mediawiki is /var/www/gamescultures.org/wiki/LocalSettings.php, it is automatically generated when installing Mediawiki, it can also be manually modified to further meet our requirement, just remember to backup before making any changes to the file. Be sure to keep this file safe, because it also contains the database username and password used by Mediawiki. Information on all configuration options can be found at http://www.mediawiki.org/wiki/Manual:Configuration_settings. To manage gamescultures.org, connecto to log in page, the administrator username and password are: admin / (to be filled).
In addition to the basic Mediawiki, there are extra plugins installed and mofications made to improve the functions.
Short URL turns the original long URL http://www.gamescultures.org/index.php/Main_Page to a shorter one, http://www.gamescultures.org/Main_Page. To achieve this, follow the instructions on mediawiki website. For oolong3, there are two files to modify, /etc/apache2/sites-available/gamescultures and /var/www/gamescultures.org/wiki/LocalSettings.php. In /etc/apache2/sites-available/gamescultures, added the following content:
RewriteEngine On # RewriteBase /wiki RewriteCond %{REQUEST_URI} !^/(stylesheets|images|skins)/ RewriteCond %{REQUEST_URI} !^/(redirect|texvc|index).php RewriteCond %{REQUEST_URI} !^/error/(40(1|3|4)|500).html RewriteCond %{REQUEST_URI} !^/favicon.ico RewriteCond %{REQUEST_URI} !^/robots.txt # Rewrite http://wiki.domain.tld/article properly, this is the main rule RewriteRule ^/(.*)$ /index.php/?title=$1 [L,QSA]
Then added the following content to /var/www/gamescultures.org/wiki/LocalSettings.php:
## For more information on customizing the URLs please see: ## http://www.mediawiki.org/wiki/Manual:Short_URL ## For short URL ## short urls $wgArticlePath = "/$1"; $wgUsePathInfo = false;
File upload allows users to upload files to gamescultures.org. To enable this feature, add or modify the following options in /var/www/gamescultures.org/wiki/LocalSettings.php:
## To enable image uploads, make sure the 'images' directory ## is writable, then set this to true: $wgEnableUploads = false;
If you want to have a preview of the uploaded image, you have to first install ImageMagick, then add or modify the folloing options in /var/www/gamescultures.org/wiki/LocalSettings.php:
$wgUseImageMagick = true; $wgImageMagickConvertCommand = "/usr/bin/convert";
Extra Edit buttons help users edit content on gamescultures.org, so they don't have to memorise too many complex wikipedia syntax. There are more than one way to do it, the way gamescultures.org does it to add the javascript of extra edit buttons to http://www.gamescultures.org/MediaWiki:Common.js. In order to do so, you must login as an administrator.
6.6 Domain Name management
The registrar of both pelitutkimus.fi and Gamescultures.org is Nebula, any domain name related changes (ex. @ record, CNAME record, MX record) is done via Nebula customer service emai: yritystuki@nebula.fi.
Both domain names use Google applications for you domain to provide email service. To manage Google applications for you domain service, login from Google applications for you domain page, usernames and passwords are as below:
Domain | Username | Password |
---|---|---|
pelitutkimus.fi | admin@pelitutkimus.fi | Frans knows it |
gamescultures.org | admin@gamescultures.org | To be filled |
Chapter 7: oolong4.uta.fi
oolong4.uta.fi hosts the web services related to the Social Media research group, it is a mediawiki wiki site at this moment. The system administrator is Jani.