studio’s IT infrastructure failure

loocas | hardware,miscellaneous,technical | Thursday, April 28th, 2011

Unfortunately, on tuesday, this week, the primary file server acting also as a domain controller at the studio, failed. The issue was caused by an error on the primary RAID array where the OS was installed. This lead to a complete loss of the OS and the entire studio setup.

I don’t know what caused the problem, but when I returned to the studio, all my machines were restarted, running, but restarted. The Internet Router I used for external access was also down. I had to manually start up some services and buy a new router. Then, when Windows Update popped up on the file server and I hit “Install Updates”, since I figured when it was all down anyways, I may as well update the system. That was the biggest mistake! The server crashed into a BSOD and I had to restart it manually which resulted in the failed RAID array and the complete loss of the OS.

The other issue was with the MD1000 DAS with all the company and projects data on. Since I configured it as a software RAID I lost the complete array with the OS. The data is still on the drives, but I can’t access them, since the OS is down.

So, right now, the entire studio is offline and I’m working on putting it all together before we start rendering the current projects. However, I’ll also make the best out of this disaster and config the entire IT infrastructure a bit differently. I’ll use my license server (a SuperMicro SuperServer I bought about half a year ago) and make it a primary file server as well as the domain controller. I just need to upgrade its RAM and possibly add another CPU. Once all done, I’ll need to install everything from scratch, config the domain, all the software etc… Then the studio will not only be a bit faster, but also a bit leaner as I’ll be able to lose the old PowerEdge 2600 server that has been the core of the studio’s pipeline. :)

So, the bad news is, we’re currently offline. The good news is, I’ll be able to build the IT from ground up on a more powerful and less memory hungry hardware. I’ll also upgrade the PowerVault to sport all 15 500GB drives (currently there are only 12 500GB drives) and I’ll make everything more streamlined and a bit faster than before.

Hopefully we’ll be up and ready within a week (waiting for the HW parts for the SuperServer).

No Comments »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a comment

Powered by WordPress | Theme by Roy Tanck