Geograph has three redundant webservers – from around 6am they’ve all been down. I’m working to restore them now.
Edit: 7:50am – servers not responding to power cycles, this could be an extended outage. Made the load balancer redirect to a page explaining what it going on. Seems a trip to Heathrow is in order….
Edit: 8:22am – data centre staff are attending shortly, hopefully resolve this without a road trip…
Edit: 9:05am – looks like we’re back. Phew!
Big thankyou due to Mark at Fubra for his assistance on what was already a frantic Sunday morning for him!
Good Luck!
Nice work getting them back up.
What caused all 3 to fail together?
They failed at different times. Crumpet was already down, as we had suspected hardware failure and left it offline until we could get to Heathrow to remove it. Scone failed Saturday afternoon, Toast handled the load until finally succumbing on Sunday morning.
More details on what happened are in a followup post here