Yesterday’s outage of the Geograph website was brought about by all three webservers rendering their network interfaces unusable due to a failed network driver. Although there are many references to similar failures, I thought it would be useful to write about it if only to give a little more Google-juice to the problem.

Geograph’s three webservers are running Ubuntu 6.06 LTS, regularly updated. The eth0 NIC is a Marvell Technology Group Ltd. 88E8050 Gigabit Ethernet Controller (rev 17), driven by the sky2 driver.

Each of those NICs failed at some point on Sunday, but the servers themselves kept on trucking, eventually writing entries like this to syslog

Feb 24 16:30:30 scone kernel: [35337220.416000] NETDEV WATCHDOG: eth0: transmit timed out
Feb 24 16:30:30 scone kernel: [35337220.416000] sky2 eth0: tx timeout
Feb 24 16:30:30 scone kernel: [35337220.420000] sky2 eth0: transmit ring 112 .. 89 report=112 done=112
Feb 24 16:30:30 scone kernel: [35337220.420000] sky2 hardware hung? flushing


There are many reports of similar failures. One suggested fix is replacement of the sky2 driver with sk98lin, but as our remote KVM is also down, we’re limited to actions we can reliably take over a network connection (in the short term at least).

So, for some short term protection against reoccurence, I’ve written a simple watchdog script called by cron every 5 minutes. It performs some network connectivity tests, and if they all fail, increments a counter. If the script is called and has failed for the 4th successive time, it will attempt to reload the sky2 module, and if that doesn’t work, trigger an immediate reboot. This should mean that a server will enter “radio silence” for around 15 minutes and recover. That’s a tolerable delay for a cluster of three servers.

Fortunately, we’ve found the second NIC on the machine uses an Intel Corporation 82541GI/PI Gigabit Ethernet, driven by the e1000 driver. By all accounts, this should be much more stable. So longer term we’ll be switching the cabling over to the second NIC.

So, the moral of the story is, don’t build a server which uses the sky2 driver!

Well, here they are, racked up and ready to roll. This setup will breath some fresh life into Geograph which is struggling to cope with its popularity at the moment. Here’s a few factoids about the setup, if anyone wants to know more, just ask and I’ll write about it.

• The 2U unit is called “Jam” and has 6 400GB SATA drives in a RAID5 array with a hot spare, providing 1.6TB of storage. It also has dual 3GHz Xeon CPUs and 4GB RAM. This machine provides photo storage and the database
• There are 3 1U units called “Toast”, “Scone” and “Crumpet” again with dual 3GHz Xeon CPUs and 4GB RAM. These are the main webservers
• There is a 1U unit called “Tea” with a single 3Ghz Pentium 4 with 4GB RAM, this is primarily a load balancer
• We also have a remote power switch and a remote KVM switch, allowing us to perform most maintainance remotely
• All the servers run Ubuntu 6.06 LTS, with the exception of “Jam” where we had problems booting from the RAID array after installation. Jam runs Debian Sarge instead.
• We are moving away from SourceForge for the code and bug tracking, using our own installation of Trac and Subversion instead.
• The load balancing will be carried out by HAProxy, which will allow us to carry out a very smooth changeover. We’ll simply proxy the old server right up until we’re ready to go live, at which point we have a short period of downtime while we synchronise the databases.

Here’s a few more pictures. Exciting stuff – click for full size goodness.

This is the front. Woooo!

…and here is the back. Check out that neat cabling job!

Soon as we’re ready for the big switchover, we’ll announce it on the site….almost ready now!