Windows Server 2008 and the Black Screen of Waiting
Last Week, one of our x3650 which serve as a primary file and print server crashed during Backup.
This wasn’t nice, especially because i couldn’t start the machine using the RSA Adapter. After going on-site (at around 23:00), i unplugged the power, reconnected it, and the machine booted. And crashed. And booted.
I booted it in safe mode, disabled all DFS replication task by disabling the DFS service, and the machine was finally able to boot. As soon as something IO intensive happened, it crashed. I opened a call with IBM, IBM replaced the system board.
This worked again for about a day. Then the machine crashed again, this time with “Planar Voltage Channel Fault”. This was at around 20:00. A few hours later, IBM arrived on-site and replaced the system board again.
The ServeRAID Controller then started AutoSynchronization of the array, which made the machine unbearably slow. And then it crashed again. I disabled the Automatic Server Reboot service, and set the synchronization priority to low. I also upgraded the hard disk firmware to version 1.03.
After about a day, the synchronization completed. Performance was comparably normal, but the machine still required about 20 minutes to boot, which was unacceptable. Looking at the eventlog, i saw some log entries before the waiting time, and some after. All of them normal, none of them looking like errors.
IBM came on site two times, replacing the ServeRAID memory and replacing the CPU and HD backplane power connector. However, neither of that fixed the issue. I began to suspect a software issue.
Today, IBM was on site again. They brought a replacement server. After testing our server with one of their hard disks (which was fast, as usual), it seemed clear that this was a software issue.
I had an idea, mostly related to the fact that the first crash was during a backup. I looked at the device manager, set it to show all hidden devices, and found about 100 shadow copies.
I looked at vssadmin list shadows, which also showed a lot of shadow copies. I removed all shadow copies on one of the drives which contained shadow copies, but there were still some there. I deleted them using vssadmin delete shadows. The reboot was fast again: Instead of 20 minutes, it only took around 5 (including the BIOS).
So, if you have Windows Server 2008 hanging at the Black Screen of Death err Waiting, it’s probably a good idea to look at your shadow copies. They may be culprit.
Also, big thanks to the techs from IBM, and for taking this issue seriously.
Windows Server 2008 und der Schwarze Schirm des Wartens - MCSEboard.de MCSE Forum:
[...] Windows Server 2008 und der Schwarze Schirm des Wartens Ich hatte die letzte Tage hier mit unserem internen Fileserver (IBM x3650) relativ heftig zu kämpfen. Und zwar lief dieser unter Server 2008 x64, und zeigte nach einem Tausch des Mainboards ein sehr merkwürdiges Bootproblem: Die Maschine blieb ca. 15 Minuten lang einfach auf einem Schwarzen Schirm hängen – im Eventlog sah man auch schön das die Maschine gestartet ist, dann 15 Minuten lang nix getan hat, und dann weitergemacht hat wie sonst nix. Die Maschine performte dann nachher auch normal. Nach einigem Hin- und Her stellte sich heraus das es irgendein Problem im Zusammenhang mit den Shadow Copies war. Das löschen sämtlicher Volume-Shadow Copies (für die Clients) und anderen (mit vssadmin delete shadows) sorgte nachher wieder für normale Performance beim booten. Also, Black Screen of Death ist nicht wirklich einer (zumindest in dem Fall). 30 Minuten warten, und wenn das hilft sollte man mal im Device Manager die Hidden Devices anschauen, und mit vssadmin list shadows gucken ob es eventuell viele Shadow Copies hat. In Englisch: Windows Server 2008 and the Black Screen of Waiting Lukas Beeler’s IT Blog Blog Archive [...]
3. March, 2009, 17:27Hyper-V Backups and spurious entries in the plug and play database » Lukas Beeler’s IT Blog » Blog Archive:
[...] several months, i’ve had a problem on a Hyper-V host described WS08 and the black screen of waiting. Basically, the machine boots up, hangs 50 minutes being completely unresponsive, and then goes on [...]
31. May, 2009, 17:52