Server 101: Keep server apps running during a crash or maintenance

By on
Server 101: Keep server apps running during a crash or maintenance
Page 2 of 3  |  Single page

4. Initialise the DRBD device on each system. This essentially "formats" the device, setting up the metadata that DRBD uses to track disk changes in the future:

sudo drbdadm create-md mydrbddrive

5. Enable the DRBD device on each system:

sudo drbdadm up mydrbddrive

6. On one of the systems, run the following command to set that system as the "primary" system, and commence the initial drive synchronisation:

sudo drbdadm -- --overwrite-data-of-peer primary mydrbddrive

If you check "/proc/drbd" now, the "st" section should show "Primary/Secondary", which means that the current server is now the primary server. It will also show that the synchronisation is underway - it'll take some time to complete, but you can start to use the device in the meantime.

7. On the same system, format and mount the DRBD device:

sudo mkfs.xfs /dev/drbd1

sudo mkdir /mnt/mydrbddrive

sudo mount /dev/drbd1 /mnt/mydrbddrive

Once the initial synchronisation is complete, any changes you make on your new drive on the primary server will automatically be mirrored over to the secondary server. If the secondary goes down at this point, the primary will continue to operate as usual, while also keeping track of which disk blocks have changed. When the secondary comes back up and reconnects, it has to "catch up" by resynchronising those changed blocks.

click to view full size image
Format and mount your devices just like any other drive

click to view full size image
A DRDB device performing its initial synchronisation

If the primary fails, you can promote the secondary server to the primary role with the following command:

sudo drbdadm primary mydrbddrive

You can then mount the drive on the new primary server and access your data. When the old primary server is brought back up, it'll automatically reconnect in the secondary role. Once it has resynchronised, you can promote it to the primary role as above, but you first must demote the current primary back to its secondary role:

sudo drbdadm secondary mydrbddrive

Automating the failover

In a real disaster, you want your secondary server to promote itself automatically, and the tool for that is Heartbeat. The name comes from the method it uses to detect failed servers - each server sends little "heartbeat" network packets to the other once a second or so. If those packets stop arriving, it knows that a server has failed, and can automatically start your services on the other server.

In our case, we want Heartbeat to manage a virtual IP address, a copy of Samba running on that IP address, and the underlying filesystem and DRBD device. Here's how to set it up:

1. Move your Samba password databases over to your DRBD drive:

cd /var/lib

sudo mv samba /mnt/mydrbddrive/

sudo ln -s /mnt/mydrbddrive/samba samba

2. Unmount your DRBD drive and it and set both servers to "secondary" mode. Also, shut down Samba, and ensure it doesn't start automatically on boot by deleting the "/etc/rc2.d/S20samba" file. These resources will all be managed by Heartbeat, and it will get confused if they're running independently.

3. Install Heartbeat on both servers:

sudo aptitude install heartbeat

4. Create the "/etc/ha.d/" file, which defines the basic Heartbeat setup. This file must be identical on both servers:

autojoin none

bcast eth0

warntime 5

deadtime 15

initdead 60

keepalive 2

auto_failback on

node server1

node server2

Change the "node" lines to match your own server host names.

5. Create the "/etc/ha.d/authkeys" file, which contains a shared secret used to authenticate Heartbeat's communications:

auth 1

1 sha1 mysecret

The "mysecret" string can be anything you like, as long as it's the same on both servers. You also need to make sure the file is only readable by root:

sudo chmod 600 /etc/ha.d/authkeys

6. Create the "/etc/ha.d/haresources" file, which lists the resources that Heartbeat will manage:

server1 drbddisk::mydrbddrive Filesystem::/dev/drbd1::/mnt/mydrbddrive::xfs samba

The resources are, in order: the DRBD device itself, the XFS filesystem on top of that device, a virtual IP address, and the Samba server, which will be started using the standard "/etc/init.d/samba" initscript. These services will be started on "server1' by default. Again, this file must be identical on both servers.

7. Configure your Samba shares, using your DRBD drive for storage. You'll also need to tell Samba to bind only to your virtual IP address, and to use a consistent name for the server, rather than using the host machine's hostname, since it'll change as the Samba service moves between your two servers. Add these lines to the "[global]" section of your smb.conf:

netbios name = DRBDSAMBA

interfaces =

bind interfaces only = yes

Make sure you copy your smb.conf file over to the other server.

8. Start Heartbeat on both servers:

sudo /etc/init.d/heartbeat start

That's it! It may take a minute to get going, but eventually everything will be running on the server you listed in the "haresources" file. Try rebooting that server to see what happens - if everything works as expected, the second server should detect the shutdown and automatically take over, promoting and mounting the DRBD drive, assuming the IP address, and starting Samba.

With the same data available, and even the same server IP address, the only clue your users will have that anything happened is a few seconds of downtime, and perhaps an interrupted file transfer.

Previous PageNext Page
1 2 3 Single page

Most Read Articles


What would you like to see more of on BiT?
How To's
Photo Galleries
View poll archive

Log In

  |  Forgot your password?