Media Temple Issue

This evening I arrived home at a relatively early 5:30 PM and promptly fired up a web browser.  Yes, I’ll admit that after spending all day at the office sitting in front of a computer, the first thing I do when I arrive home is… sit down in front of another computer.  Having started the snnyc blog four weeks ago, I’d set it set as my home page in Firefox.  This time, however, the blog wouldn’t load.

I immediately loaded a couple well-known web sites to confirm that I still had Internet access.  I did.  Then I logged into my WordPress hosting provider, Media Temple, to see if anything was going on.  It was.

“We are currently experiencing intermittent connectivity issues on (gs) Grid-Service Cluster.08. (mt) Media Temple engineers are working to resolve this issue as quickly as possible. We appreciate your patience.”

The issue was time-stamped April 18, 2011 1:48 pm PDT, or 4:48 PM EDT, meaning that it had been going on for close to 45 minutes by the time I learned of it.

Next I fired up Twitter to see if others were discussing @mediatemple.  They were.  And Media Temple was proactively responding to people mentioning issues related to this service outage.  Some users were taking it in stride, while others, uh, weren’t.  User @lithiumcorp tweeted, “We switched to @mediatemple for ‘spike ready’ hosting. Today we got that spike. We’re offline. As soon (mt) is back online we’re gone.”

By 3:24 pm PDT, 6:24 PM EDT, or 96 minutes after acknowledging the problem, Media Temple provided an update indicating that their engineers had identified and corrected a problem and that the load and services were returning to normal.  Indeed, the snnyc blog was back online too.

That lasted until about 7:40 PM Eastern, when Twitter users began mentioning Media Temple again.  The snnyc blog fluctuated between unavailable, able to partially load the page, and able to load the full content, albeit slowly.

[Update: Media Temple subsequently stated that the outage was caused by a Distributed Denial of Service attack.  The incident required them to put additional filtering rules in place to stop the attack.]

Lessons Learned
As an IT Professional, I’m certainly able to empathize with any person or organization charged with the task of maintaining technology infrastructure and services for a wide user population.  Things happen.  And I’m not ready to give up on a platform at the first sign of trouble.  Instead, I’m more interested in timely communication and a competent response when the a problem does occur.

Media Temple communicated the problem quickly, using both this RSS feed and proactively via Twitter using @mediatemple and @mt_monitor.  They acknowledged to one person that they monitor Twitter for any mention of Media Temple.  Ideally MT might have e-mailed affected customers, but at least the RSS feed allows us to maintain our own alerts going forward.

Naturally I’d like Media Temple to resolve this issue and any others as quickly as possible.  Ninety-six minutes seems a little on the long side, and then lingering instability near the three hour mark is a bit frustrating.  I’ll certainly be attuned to any further outages in the weeks and months ahead, and share that experience here.

[I noted no additional Media Temple issues affecting the snnyc blog through the remainder of 2011 or 2012.]

Tweet about this on TwitterShare on Google+Share on LinkedInShare on FacebookShare on RedditEmail this to someone

Comments

  1. Thanks for posting your adventures in DDoS.
    I would have never thought to blog about such a thing, but, when someone else does, I learn a whole lot about new ways to handle the situation. The way you mined twitter was especially intriguing to me.

Speak Your Mind

*

CAPTCHA Image

Reload Image