Gmail was down and everyone was losing it

Update 3 (Jan. 27, 2014 – 5:07pm ET):

Google’s press team responded to an email request for comment with a link to a blog post, written by Ben Traynor, its vice-president of engineering. The post was dated Friday, Jan. 24:

“At 10:55 a.m. PST this morning, an internal system that generates configurations—essentially, information that tells other systems how to behave—encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors. Users began seeing these errors on affected services at 11:02 a.m., and at that time our internal monitoring alerted Google’s Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time. By 11:30 a.m. the correct configuration was live everywhere and almost all users’ service was restored.

With services once again working normally, our work is now focused on (a) removing the source of failure that caused today’s outage, and (b) speeding up recovery when a problem does occur. We’ll be taking the following steps in the next few days:
1. Correcting the bug in the configuration generator to prevent recurrence, and auditing all other critical configuration generation systems to ensure they do not contain a similar bug.
2. Adding additional input validation checks for configurations, so that a bad configuration generated in the future will not result in service disruption.
3. Adding additional targeted monitoring to more quickly detect and diagnose the cause of service failure.”

You can read Google’s full statement here.

Update 2 (Jan. 24, 2014 – 3:47pm ET):

Google’s press team has issued two statements via email, the first one saying it is “investigating reports of an issue with some Google services.” The next statement, sent about half an hour later, said: “Most of the issues w/ Google services have been resolved. Check the Apps Status Dashboard for updates: http://goo.gl/tYnNak”

Hang in there—most of the issues w/ Google services have been resolved. Check the Apps Status Dashboard for updates: http://t.co/p9ChqMyEUn

— A Googler (@google) January 24, 2014

Update 1 (Jan. 24, 2013 – 3:20pm ET): It looks as though it’s not just Gmail that is experiencing service disruptions. A number of other services, like Google Drive and Google Docs, seem to be having issues, according to Google’s Apps Status Dashboard.

Google Inc.’s email service, Gmail, experienced a service outage on Friday afternoon, with users in Europe, the U.S., Canada, and India reporting they were unable to access the service. At the time of this writing, the service appears to be up and running again, though it appears as though there will still be service disruptions and slower-than-normal response times. There also has been no word from Google as to why the disruption occurred, with no dial tone coming from the search giant’s media relations line. And as Gmail is down, it probably isn’t surprising there has been no response as of yet to an email request for more information. Still, the outage may be related to a strange glitch sending emails to a Hotmail account, according to TechCrunch. If a user Googles “Gmail” and then goes to click “Gmail” next to the description “Email – 10+ GB of storage, less spam, and mobile access,” a new Gmail compose window will randomly pop up. The “To” field will automatically add [email protected] as the message’s recipient. We’ve been able to duplicate the bug – if you try it out, though, maybe you should avoid actually sending anything. TechCrunch is reporting the owner of the email address, David S. Peck, is saying his inbox has been flooded with 500 emails every hour. Also worth noting – right around the time of the outage, Google’s Site Reliability Engineering team took to Reddit to do an “Ask Me Anything” session, where users fire off questions and the team answers in a live chat, Q & A format. Gmail’s official Twitter account has not tweeted since Jan. 22, nor has it posted anything on its Facebook page since that date. But with no access to email, many people took to Twitter to express their feelings of loss and confusion.

Like millions of others, our productivity at work took a total nosedive today. #gmail #gmaildown pic.twitter.com/NuLawHl0Xb

— Phil Marshall MD MPH (@philmarshallmd) January 24, 2014

Gmail Down = Adult Snow Day

— Kirstine Stewart (@kirstinestewart) January 24, 2014

PANIC AND FREAK OUT #gmail

— andrew hyde (@andrewhyde) January 24, 2014

Gmail’s down and then Tweetdeck crashed. It’s like the world has been put on hold. I feel so alone.

— Mike Masnick (@mmasnick) January 24, 2014

For all those affected by the #gmail outage. Please contact the #NSA for a copy of your e-mail communications until service is restored.

— Jason Humeniuk (@VanMortgage) January 24, 2014

I’m worried I’m going to have to actually speak to someone if this keeps up much longer. I hope my phone has an app for that. #gmail

— Steve Ladurantaye (@sladurantaye) January 24, 2014