Windows 2008 server Hyper-V host and corrupted windows install

I had a bit of a worrying time last week when a colleague upgraded an old (but live) Dell PowerEdge R310 server running Hyper-V with some new RAM.
He powered down the old server, upgraded the RAM and booted the server back up.
A few minutes later he was looking at the screen loading windows, when it failed. He put the DVD and tried to repair windows but when he got to the  System recovery window it didn’t show any installations of Windows.
He then looked to restore from backup to another server to get the Hyper-V clients running while we fix this server.

Yes, you know what’ coming here, the latest backup we had was over a year old. Our client had not swapped out the drives for that length of time and something had corrupted the one drive that was plugged in.  Windows Backup was reporting successful backups, but when you try to restore, you cannot see any dates to select from. An investigation for another day I think !

The big lesson here, one which I think my colleague will be doing every time now, is to test the backups before performing work on a server.

I can’t quite image the conversation between my colleague and our boss, or indeed the client, but this is when I got a call to step in and assist.

Now, although windows would not boot, I was fairly confident the data was still on the drives, as windows started loading. I ran through the start up recovery procedure, and again it showed no windows installations. Now at this point, you need to get the Dell RAID drivers from their web site and extract to a USB drive and loaded in to the recover console. This allowed me to see the failing windows install. JOY !
I then ran through the Startup Repair, and got the message: “Windows failed to repair the start up process”.

As we can  now see the windows installation, I figured I’d plug a large USB drive, and copy the required vhd to it. Using the load driver button, I navigated to the vhd and copied it. I then navigated to the USB drive and pasted. This was a bit of a nervous time, as you don’t get a status bar for the copy, just a spinning wheel. I decided to wait.
Eventually, about an hour later, (110Gb vhd file), the spinning wheel disappeared and the vhd was on the USB drive. I gave this to my colleague to go off and rebuild a Hyper-V client.

Having recovered the virtual hard drive, I then thought I’d see about fixing the windows install. I ran through the Startup Repair again, and got the same message – “Windows failed to repair the start up process”.
Looking at the log file it generated, it showed the acpi.sys was corrupted.
I went back to the recover console and fired up the command prompt. I copied the acpi.sys from the temporary windows install the boot from DVD created, to the failing windows install. I rebooted the server and it all came backup, Hyper-V clients and all.

In hindsight I think repairing the windows install would be best & quickest solution without copying the vhd off first, but under the pressure and panic to get the vhd file back, we copied it onto the USB drive. It may also have had a different outcome if the windows repair wasn’t so quick.

Anyway, we got it recovered in the end, with no data lost, and a refocus on backups, instead of complacency that just because Windows backup is not reporting any errors, doesn’t mean your data is safe.

If you’ve had a similar day, I’d like to hear, or if this has helped you please let me know in comments below,

Thanks

Ian