After Samba proved too problematic for Geograph we switched to NFS and have enjoyed a good period of stability ever since. The webservers are barely breaking a sweat, but the load on our database is a little higher that I hoped. By no means maxed out, but definitely a lot more than it needs to be. The next week or so will be spent distributing some of that load to the webservers (like using memcached for session storage) and also looking at some of the heavyweight queries and tasks to see how they could be performed more efficiently….
Category Archives: Geograph
New Geograph server cluster goes live
Can’t write too much as I’ve been on holiday for a week and I’ve lots to catch up on, but last night the new Geograph servers went live! At the appointed time we sync’d the database and images onto the new servers, then simply tweaked the haproxy configuration, opening the floodgates!
We’ve a couple of minor teething problems but everything seems to be holding up well!
The only major issue is every now and then the smbiod process on one of the webservers consumes 100% of the CPU, taking it out of the cluster. Not seen this issue before, will investigate it during the week. I thought Samba would be a low-maintenance solution for file sharing between servers, as I’ve never had a good time with NFS….we shall see!
Geograph servers coming to life
Well, here they are, racked up and ready to roll. This setup will breath some fresh life into Geograph which is struggling to cope with its popularity at the moment. Here’s a few factoids about the setup, if anyone wants to know more, just ask and I’ll write about it.
- The 2U unit is called “Jam” and has 6 400GB SATA drives in a RAID5 array with a hot spare, providing 1.6TB of storage. It also has dual 3GHz Xeon CPUs and 4GB RAM. This machine provides photo storage and the database
- There are 3 1U units called “Toast”, “Scone” and “Crumpet” again with dual 3GHz Xeon CPUs and 4GB RAM. These are the main webservers
- There is a 1U unit called “Tea” with a single 3Ghz Pentium 4 with 4GB RAM, this is primarily a load balancer
- We also have a remote power switch and a remote KVM switch, allowing us to perform most maintainance remotely
- All the servers run Ubuntu 6.06 LTS, with the exception of “Jam” where we had problems booting from the RAID array after installation. Jam runs Debian Sarge instead.
- We are moving away from SourceForge for the code and bug tracking, using our own installation of Trac and Subversion instead.
- The load balancing will be carried out by HAProxy, which will allow us to carry out a very smooth changeover. We’ll simply proxy the old server right up until we’re ready to go live, at which point we have a short period of downtime while we synchronise the databases.
Here’s a few more pictures. Exciting stuff – click for full size goodness.
This is the front. Woooo!
…and here is the back. Check out that neat cabling job!
Soon as we’re ready for the big switchover, we’ll announce it on the site….almost ready now!