Last week on Tuesday 10th November at 19:58 we saw a complete outage of our main site.

This was caused by a corrupt file system. From Tuesday through to Thursday the linux box has been FSCKing, unfortunatelly this took over 3 days to complete.

The server seemd to take approx 5 seconds to check 100 Inodes. It’s upto around Inode 70740000 – the majority are being checked/fixed. Want to know what an inode is?

The problem we had is then it got to inode 98337000 it should have been close to finishing, but it dropped back down to inode: 9207789.

A few hours later, we started to see results and managed to recover all our files. We then .Zip’ed the whole site up and moved it to another server, in fear of having a repeat performance.

To ensure this does not happen we have taken measures to ensure, we have a second site which should come online should one fail.

Usally this should have taken only a few hours, but this turned into days.

Gaming, Dedicated, Voice, Web-Hosting services were not affected.

This entry was posted on Sunday, November 15th, 2009 at 3:47 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.