Server 101: Keep server apps running during a crash or maintenance

By on
Server 101: Keep server apps running during a crash or maintenance

Leigh Dyer looks at how to get failover redundancy on a shoestring budget

 

Today, people rely on IT services around the clock, but how do you keep them running despite hardware faults and software upgrades? The answer lies, surprisingly, in a line from the movie Contact: "why build one when you can have two for twice the price?". By adding a redundant server, you can boost your uptime dramatically, and with some neat Linux features, you don't even need to double your costs.

Whether you're running a web farm with a thousand servers or a home file server on a couple of old desktop boxes, the idea is the same: running your applications on redundant servers helps you ensure service to your users despite server crashes or scheduled maintenance.

The main drawback to redundancy is cost: the extra server hardware costs money, and the added complexity increases maintenance requirements. It's not all bad, though - compared to a single high-end server and its expensive rapid-response support contract, a pair of cheaper redundant servers might not cost much more overall.

The biggest potential cost is in storage hardware. There are some tasks that don't need to store data: serving static web pages, for instance, is fine as long as each server has its own up-to-date copy of the pages.

Most other applications need some kind of storage though, and if you don't want your redundant servers to go to waste, you need an equally reliable shared storage system. Traditional solutions can cost tens of thousands of dollars, but Linux has a trick up its sleeve to bring that cost right down.

Sharing the love

The 'enterprise' solution for storage is a Storage Area Network device, or SAN, which puts a bunch of hard drives in a highly redundant enclosure, with high-speed Fibre Channel links to your servers. Even a low-end SAN can cost around $10,000 though, and cheap SANs often have lousy performance.

The low-cost Linux alternative is the Distributed Replicated Block Device, or DRBD (www.drbd.org). When set up on a pair of servers, it sits between the filesystem and hard drives on one of them (the 'primary' server), automatically replicating any disk changes over a network link to the other (the 'secondary' server). Only the primary server can access the data on the drives, but if it fails, the secondary can take over the primary role and gain access in its place.

The beauty of DRBD is that it doesn't require any expensive hardware - the only thing you really want is a direct, dedicated network link between the two servers to run the DRBD replication. Gigabit ethernet adapters with a cross-over cable are a perfect cheap solution: most machines have gigabit ethernet as standard these days, and if not, a pair of adapters can cost less than $100.

Set up DRBD

Once the hardware is in place, follow these steps to set up DRBD:

1. Configure the underlying storage on each system. My systems run LVM, so I just created a new LVM volume called "drbd" on each system, but a partition or dedicated hard drive would work just as well. They'll need to be the same size, though, for obvious reasons.

2. Install the DRBD software, which consists of a kernel module and the user-space configuration tools, on each system. Ubuntu 8.04 ships with DRBD as part of the standard kernel, so I just had to install the tools:

sudo aptitude install drbd8-utils

3. Create the "/etc/drbd.conf" config file and configure your DRBD device. This file must be identical on both systems:

common {

  protocol C;

}

resource mydrbddrive {

  syncer {

    rate 50M;

  }

  on server1 {

    device    /dev/drbd1;

    disk      /dev/ubuntu/drbdtest;

    address   10.197.10.1:7789;

    meta-disk internal;

  }

  on server2 {

    device    /dev/drbd1;

    disk      /dev/ubuntu/drbdtest;

    address   10.197.10.2:7789;

    meta-disk internal;

  }

}

You'll have to change the server names, disk devices, and IP addresses to match your own configuration. If you're running a dedicated network for DRBD, make sure you use those IP addresses here.

 

4. Initialise the DRBD device on each system. This essentially "formats" the device, setting up the metadata that DRBD uses to track disk changes in the future:

sudo drbdadm create-md mydrbddrive

5. Enable the DRBD device on each system:

sudo drbdadm up mydrbddrive

6. On one of the systems, run the following command to set that system as the "primary" system, and commence the initial drive synchronisation:

sudo drbdadm -- --overwrite-data-of-peer primary mydrbddrive

If you check "/proc/drbd" now, the "st" section should show "Primary/Secondary", which means that the current server is now the primary server. It will also show that the synchronisation is underway - it'll take some time to complete, but you can start to use the device in the meantime.

7. On the same system, format and mount the DRBD device:

sudo mkfs.xfs /dev/drbd1

sudo mkdir /mnt/mydrbddrive

sudo mount /dev/drbd1 /mnt/mydrbddrive

Once the initial synchronisation is complete, any changes you make on your new drive on the primary server will automatically be mirrored over to the secondary server. If the secondary goes down at this point, the primary will continue to operate as usual, while also keeping track of which disk blocks have changed. When the secondary comes back up and reconnects, it has to "catch up" by resynchronising those changed blocks.

click to view full size image
Format and mount your devices just like any other drive

click to view full size image
A DRDB device performing its initial synchronisation


If the primary fails, you can promote the secondary server to the primary role with the following command:

sudo drbdadm primary mydrbddrive

You can then mount the drive on the new primary server and access your data. When the old primary server is brought back up, it'll automatically reconnect in the secondary role. Once it has resynchronised, you can promote it to the primary role as above, but you first must demote the current primary back to its secondary role:

sudo drbdadm secondary mydrbddrive

Automating the failover

In a real disaster, you want your secondary server to promote itself automatically, and the tool for that is Heartbeat. The name comes from the method it uses to detect failed servers - each server sends little "heartbeat" network packets to the other once a second or so. If those packets stop arriving, it knows that a server has failed, and can automatically start your services on the other server.

In our case, we want Heartbeat to manage a virtual IP address, a copy of Samba running on that IP address, and the underlying filesystem and DRBD device. Here's how to set it up:

1. Move your Samba password databases over to your DRBD drive:

cd /var/lib

sudo mv samba /mnt/mydrbddrive/

sudo ln -s /mnt/mydrbddrive/samba samba

2. Unmount your DRBD drive and it and set both servers to "secondary" mode. Also, shut down Samba, and ensure it doesn't start automatically on boot by deleting the "/etc/rc2.d/S20samba" file. These resources will all be managed by Heartbeat, and it will get confused if they're running independently.

3. Install Heartbeat on both servers:

sudo aptitude install heartbeat

4. Create the "/etc/ha.d/ha.cf" file, which defines the basic Heartbeat setup. This file must be identical on both servers:

autojoin none

bcast eth0

warntime 5

deadtime 15

initdead 60

keepalive 2

auto_failback on

node server1

node server2

Change the "node" lines to match your own server host names.

5. Create the "/etc/ha.d/authkeys" file, which contains a shared secret used to authenticate Heartbeat's communications:

auth 1

1 sha1 mysecret

The "mysecret" string can be anything you like, as long as it's the same on both servers. You also need to make sure the file is only readable by root:

sudo chmod 600 /etc/ha.d/authkeys

6. Create the "/etc/ha.d/haresources" file, which lists the resources that Heartbeat will manage:

server1 drbddisk::mydrbddrive Filesystem::/dev/drbd1::/mnt/mydrbddrive::xfs 192.168.1.27 samba

The resources are, in order: the DRBD device itself, the XFS filesystem on top of that device, a virtual IP address, and the Samba server, which will be started using the standard "/etc/init.d/samba" initscript. These services will be started on "server1' by default. Again, this file must be identical on both servers.

7. Configure your Samba shares, using your DRBD drive for storage. You'll also need to tell Samba to bind only to your virtual IP address, and to use a consistent name for the server, rather than using the host machine's hostname, since it'll change as the Samba service moves between your two servers. Add these lines to the "[global]" section of your smb.conf:

netbios name = DRBDSAMBA

interfaces = 192.168.1.27

bind interfaces only = yes

Make sure you copy your smb.conf file over to the other server.

8. Start Heartbeat on both servers:

sudo /etc/init.d/heartbeat start

That's it! It may take a minute to get going, but eventually everything will be running on the server you listed in the "haresources" file. Try rebooting that server to see what happens - if everything works as expected, the second server should detect the shutdown and automatically take over, promoting and mounting the DRBD drive, assuming the IP address, and starting Samba.

With the same data available, and even the same server IP address, the only clue your users will have that anything happened is a few seconds of downtime, and perhaps an interrupted file transfer.

DRBD performance

As you'd expect, DRBD does impose a performance penalty, but you can make tradeoffs between performance and data integrity. As I've configured it here, DRBD is extremely paranoid - it doesn't report a write operation as complete until the data is physically on disk on both the primary and secondary servers. This ensures data integrity even in the very unlikely case that both servers crash or lose power at the same time, and one of them is completely destroyed in the process.

click to view full size image


For typical home or office work, where you can deal with a small data loss under such unlikely scenarios, you can enable write caching by adding the following lines inside the "resource" section of your "/etc/drbd.conf" file:

disk {
  no-disk-flushes;
  no-md-flushes;
}

Even in this mode, you won't lose data if one server dies - you'll only lose data if you lose power on both servers simultaneously, and even then it's limited to the few writes that were cached in RAM but hadn't yet been saved to disk.

It's also worth remembering that DRBD only affects write performance. Data reads hit the local disks only, just like they would normally, so their performance is unaffected, no matter how paranoid your DRBD configuration is.

Multi page
Tags:

Most Read Articles

Poll

What would you like to see more of on BiT?
News
Reviews
Features
How To's
Lollies
Photo Galleries
Videos
Opinion
View poll archive

Log In

Email:
Password:
  |  Forgot your password?