I’ve just dealt, for the first time ever, with complete system failure. I don’t ever want to deal with it again.

For about a year I’ve been running my own dedicated server, hosted by iWeb. For about a year, everything has been fine.

Last week, as I was moving a client over to my server, the good people at iWeb noticed that one of my mirror drives was showing signs of failure. “Let’s swap it out”, they said. “Should only take about 15 minutes of down time”, they said.

At around 0430 on April 5th, I received word that the second mirrored hard drive had failed, but not to worry, iWeb had swapped in another drive and was running a synchronization. At 02h30 on April 9th, I received word that the RAID synchronization had completed. All this time, my server remained up. I was so happy I had professional people who know about servers working on this.

Then…

At 02h00 on April 10th, they shut down my server, took out the bad drive, put in the new drive and tried to boot the server. Nothing.

At 08h00 on April 10th, they sent me a message:

That’s what I woke up to that day.

Ummm…what?

As far as I understood it, they essentially had three hard drives on hand with mirror copies of my server. Two of them at least were good (one had failed, thus requiring the swap out). Why they had to do a full re-install of the server, I still don’t fully understand. But, they said they couldn’t get the thing to boot, so a re-installing away they went. Server re-installation started at 19h00 on April 10th. They quoted four hours. Eight hours later I had a brand new fresh server online. None of the websites were there yet though.

Did I mention that one of my clients was hosting their app on my server for their event that started on April 10th? All promo had my web address on it and I didn’t even have a working server.

I sure as hell mentioned it to iWeb, who said that they would plug in the old hard drives via USB to the server so site migration could happen right away. I went to bed around 03h00 being given the impression that by morning, the sites should be back up and going.

Oh no… ’twas not to be.

The next morning around 09h00, I was informed:

So…ha ha ha, good one iWeb. You’ve misplaced the two hard drives that contain exact replicas of my server.

At this point I began Operation “Welcome to Fuck Land Trevor”.

I immediately secured a second dedicated server, from iWeb, fully paid in advance for a month because the sales rep I spoke to had no sympathy for the fact that their company had totally fucked me. This second server was ready in two hours.

I then installed WordPress.

…multisite.

…Plugins activated.

And got the critical app back up (this link works in Webkit browsers only).

That took around ten minutes.

I think at that point, I smoked something. And landed on the doorstep of Mitch Podolak and called it a day.