This weekend's downtime

Moderator: The Merrimack Volunteer Moderators

This weekend's downtime

Postby Nat Fairbanks » Mon Jun 02, 2008 7:50 am

Everything looks like it's back to normal now. If you can see the forum (and this post) you should be all set.

-Nat (admin)

Here's some technical detail (this evening was Saturday evening)
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.

I am sorry that this accident has occurred and I apologize for the impact.
Nat Fairbanks
Posts: 1020
Joined: Wed Nov 20, 2002 1:05 am
Location: Merrimack NH

Re: This weekend's downtime

Postby MickAv8r » Tue Jun 03, 2008 10:13 am

I was wondering if this site was hosted at the same host as some other sites I'm aware of. Apparently the explosion was quite uhhh impressive. The CEO was thankful data center employees weren't killed :shock:

I do believe that the servers are currently on backup generators. They have to literally rewire the power to the datacenter and thus don't be surprised if the servers go down again for a shorter period of time while the switch over from the generators to the new power grid.
Posts: 40
Joined: Sun Apr 27, 2008 10:46 am

Re: This weekend's downtime

Postby Brian McCarthy » Tue Jun 03, 2008 3:00 pm

I'm waiting for the dust to settle before I ask more questions.

The site is listed as being in IL - so a hit in TX should not have effected it - but, like you said, they are still working on getting things back to normal so I'll ask, but now is not the time.

The Data Center is known as 'The Planet' and has both avid supporters and people who hate the place.

Brian McCarthy
Posts: 1347
Joined: Thu Nov 21, 2002 8:12 am
Location: Merrimack, NH

Return to Forum Software Information

Who is online

Users browsing this forum: No registered users and 1 guest