Another power outage at Layer One
At about 14:00 today, i received an SMS from our Nagios instance, telling me that our switch at Layer One was down. I investigated quickly, and found that other machines located at Layer One were down too.
A quick call to our bandwidth provider Init7 confirmed that there was another power outage. If i counted right, this is the third power outage in the ~2 years that Sylon is located at Layer One. It happened because of power maintenance in another floor. Both power circuits went down, because they were plugged into the same UPS. That makes perfect sense.
Fortunately, there were few problems. Unfortunately, one of the problems we had required attention on site. One of the new DL360G5 decided to stay asleep. iLO wasn’t responding either. On site, i just power toggled the machine, and it worked fine. Strange.
There were a few (obvious) problems with customers still using MyISAM tables for data storage – and where myisamchk didn’t fix it, i’ve just restored the most recent backup made with bontmia.
UPDATE:
Power failed again last night for about 20 minutes. It didn’t effect all Racks (DATALINE’s machines went down, but Sylon’s didn’t). They didn’t know why this happened.
Mike:
The thing with two circuits on a single UPS is more common as you may think.
21. February, 2007, 18:25Greetings to our Service Provide and his whole Citrix-Farm :-)
Lukas Beeler:
I’ve seen customers who had their server plugged into the normal wall socket, with a fully functional UPS standing next to the server for years.
But a Housing provider should act in a much more professional manner.
21. February, 2007, 20:56