Sorry for the downtime, folk. It's not cheapo hosting, honest. It's a dedicated server (and an expensive one at that) with a lot of paid options.
Some of you know about the problems at the host. But that's no excuse for several days of being offline. More later. Just wanted to offer my sincere apologies and to say welcome back.
There may be a bit of instability over the next day or two. Please bear with me.
They managed to turn off the water sprinklers then?
I've had hosting go down on me before but never on this scale where the site's been out for three whole days.
For those interested in the background - apparently a non-employee at the DC testing the fire safety systems set off the Inergen Fire Extinguishers. These work by pumping a cocktail of gases into the DC at high pressure to reduce the percentage of oxygen and thus put out the fire. The high pressure damaged hundreds of hard disks (that bit is plausible given my hardware knowledge of hard disks). From Westhost's status page:
On Saturday, February 20 2010, at approximately 2:20 p.m. MST during an annual fire system inspection of our data center the fire suppression system was inadvertently triggered and discharged into the data center environment. The discharge of the fire system has affected the hardware and operations of our servers. Not all servers have been affected by the discharge. We are working to restore servers and services as quickly as possible. Due to the nature of this interruption we will be experiencing extended outages.
The CEO of UK2Group, the company that owns Westhost, says that the DC should have had safety values to release the pressure. That the hard disks went down suggests that maybe they didn't... or they weren't working properly.
Many other customers are still waiting to get their sites online and some could be waiting till Saturday - a whole week! My sympathies if any of you are in there.
Wow, just wow. I can hardly believe that happened.
I was worried about you Clinton. I don't know you very well yet, but I was upset the site was gone, and thought about trying to email you. Figured if the site was down the email would be also.
If you want, lets talk about disaster recovery. We have all of our company sites backing up in a way that we could have moved to a new host in a matter of hours. If you need some help with something like that I would be glad to share my experience with you.
Glad your back!
You're very kind, Shane, thanks so much for your concern and your offer of help. I'm very tied up with other stuff right now and don't have time to change my disaster recovery measures but I may take you up on that later.
At present I rely on automated nightly backups. The last copy I had was from Friday night. But I didn't bother setting up new hosting and changing name servers because I didn't think this outage would have us down for three whole days. I expected the site to come back on Sunday based on Westhost's reassurances. If Westhost had told me on Saturday itself that we'd be down for three days and there was a chance the site could be down for a week I would have moved.
If email is ever down again, you can always reach me via the contact on the experienced-people.co.uk site (it's with a different host).
I would be interested to hear about your setup as well, if you are willing to share it publicly. I have managed to avoid too many hosting disasters because I always have servers at multiple hosts in different DCs and make regular backups, but new ideas for improvement are always helpful.Originally Posted by Shane
Posting Permissions
Reply With Quote
Bookmarks