Millions of users and businesses hit by Gmail outage
GOOGLE'S GMAIL, the whole web-based shebang, crashed yesterday at lunchtime PDT, leaving tens of millions of Gmail Pro and non-Pro users gmail-less, bereft and miserable.
True, as a free service, you shouldn't expect blissful perfection, but Gmail for Business was also affected. A very large number of businesses - about 1.75 million according to Google's own figures - rely on Gmail to do their work as part of Google Apps, the $50-per-user-per-year subscription web-based application suite.
For a rather long 100 minutes - Google's number - Gmail was out for the count, leaving companies mail-less and users tweeting furiously on, er... Twitter, which seems to have become the impromptu soapbox for the disillusioned masses.
The crash was bad enough to get Google's PR people up off their tushies and doing a rain dance online.
Just a few hours later, the company had diagnosed the problem, fixed it, and set everything back in order.
Google's Site Reliability Czar, Ben Treynor, owned up to the problem saying, "Today's outage was a Big Deal, and we're treating it as such."
Treynor added that the outage had been caused by a miscalculation, as a few servers were taken offline for upgrades and that caused a disturbance in the force. The resulting overload sent routers crashing, so people trying to read their personal emails over lunch break were royally stuck.
Google promised it would never do it again.
Apparently IMAP/POP-based users were unaffected by the outage, which would have been nice to know at the time because it would have given businesses some options to sort out their problems.
One COO sighed, "The Google outage yesterday was an inconvenience for my business as it meant we had to configure alternate mail clients like Outlook, which took time, and then we were each working independently rather than all accessing a single web interface to share work."
This raises the issue that far too many businesses have naively migrated their essential systems to online, cloud-based resources without considering business impacts if those go down for any reason, and therefore don't have backups in place to continue business operations in the event of such an outage. µ
Share this:
ShareBusiness Intelligence Software
Customer Relationship Management (CRM)
Enterprise Accounting Software
Enterprise Resource Planning (ERP)
Enterprise Systems Management
As reliable as Gmail has been for me, even with the downtime it has had, it is no worse than any other company ran mail system. If your panicking over a few hours of downtime you may need to get out of the basement more often.
</2cents
~Dave
That's not exactly the same thing though, is it? When a company's mail system goes down, it's just one company that's out.
When Gmail goes down, every company paying to use the service is out, and so are all the free users too. It's a much bigger user base that is unable to use email in that situation.
Considering the whole thing died rather than just parts of it falling over with redundancies kicking in, the question of resilience and the number of single points of failure in the system has to be asked.
The total downtime per year at Google is still way less then most companies who have hosted their own email solutions.
Not had unscheduled downtime for over 8 years now and scheduled has been in the middle of the night not the middle of the day.
When you have your own email server you are in control. This recent outage shows a major issue with using a system which was designed to be free in a business environment.
Why would any real business rely on web mail anyway?
i lost gmail most of yesterday and it's still down now from early this morning ... i cannot find ANY information on the current outage ... WHAT IS GOING ON? when will it be back? why such poor communication?
@Photoboy
I agree with Dave. Uptime is most likely similar or worse for most company run mail systems... and many companies do not have the money to implement and support their own. Does it matter if you go down along with everyone else if the amount of yearly down time is the same? I can't see the difference.
@OldTimer
Do you typically notify all 25 of your users before scheduled downtime? :P
I've never seen or heard of a system that never goes down... maybe I'm not OLD enough yet...
I remember a year or two ago when the Blackberry network went down. Single point of failure with all traffic flowing through London, Ont (?). I thought, what a pile of rubbish, they'll surely take a beating in the market..
Then the unthinkable happened... Their stock went up... A pal of mine suggested it showed the investors how critical the services was - how many depended on it.
I can see the motivational posters rolling off the presses now:
"Servive Impacting Outage: A good [calculated]outage every now and then is a good way to show your investors, just how important your really are - which translates directly to how much you are really worth ..."
Too big to fail!
When the POTS blows up - try http://en.wikipedia.org/wiki/Downtime#Famous_outages - everyone fails to get work done equally. And as long as all the world relies on Google, or on the ILoveYou/MyDoom/NIMDA-susceptible OS, everyone fails to get work done equally in a Major Event. Everyone enjoys a free holiday, right?
Silicon Valley's big innovation has been to disclaim any Service Level Agreements.
"Inconvenience" my ass.
Affected businesses got an hour or two of decent productivity out of people who were working instead of dorking with email. Unless they were commenting on Inq articles instead.
@Dave and @b
I worked at my last job for 10 years. In that time we grew from about 1500 users to over 6000. We also moved from Exchange 5.5 up to Exchange 2003 (when I left).
Email was never down for any significant time period during business hours. It was slow sometimes and overwhelmed by SPAM or stupid mass-mailing viruses; but it always worked and never[1] went down.
Maintenance was off-hours (meaning you can *start* at 10:00 PM).
For some businesses, an email outage is unacceptable. Clearly, GMail isn't ready to fill the roll for those companies.
With redundant servers, disaster recovery / business continuity locations and proper clustering, you can have zero outages.
GMail's problem wasn't even with the mail servers, it was with networking equipment.
Anyone who thinks no downtime is a laughable goal has never worked at a large corporation with real SLAs and OLAs.
[1] Five nines: 99.999% uptime
"few servers were taken offline for upgrades and that caused a disturbance in the force"
Hmm, so the other servers could not withstand the extra load. How maxed out are the servers I wonder.
Well, just going by Googles own number of businesses 1.75 million who were down for 100 minutes, that's a total combined downtime of 175 million minutes or nearly 333 years of downtime. Adding the tens of millions of private users who were affected as well.. at least it kept the google spam out of my hotmail inbox.
I can't believe so many people seem unable to last 100 minutes without their precious dose of Spam, ESPECIALLY in business environment...
In a serious business setting, one is used to take a plane every now and then, and that can take longer than a meager two hours.
In a serious business setting, one does not brag about their 6k company and how they never had e-mail outages, except for planned after-working-hours maintenance. Well guess what? I've been working for a major global company with 50k+ employees, and seen my share of outages. They tried scheduling maintenances for Saturdays or Sundays, but that still affected those few countries where it's just another working day...
Gmail is doing an amazing job, not the least due to the fact that one can count on fingers the number of outages during their 5 year tenure... Oh, and it's fast too, and you don't need to pull your hair if your PC crashes (or worse) with your precious e-mails and their drafts.
Being a serious e-mail user, one can only appreciate the productivity boost Gmail provides, not to mention the cost savings. Plus, if everyone used Gmail (as they definitely should), one would never have to worry about outages again, since during a rare event of such, no-one would have an edge over your sorry ass.
And to repeat myself, if you can't live without e-mail for a duration of a good lunch, e-mail is the least of your problems...
When you get 100+ Legitimate emails a day that you have to respond to, 100 minutes of downtime is unaccepctable. We've been in situations at work if the power goes out for longer than 10 minutes we all get out of there to the nearest coffee shop, VPN, and start doing email again.
Some businesses live by email, and it should never be down, and certaintly not be managed by a company that really in the end does not care about weather or not your company succeeds.
I access Gmail via IMAP on Outlook, Thunderbird and my Nokia phone. I saw no break in service whatsoever. POP3 access was also apparently unaffected as well. If companies use Gmail, they're bonkers if they depend on the flaky browser interface.
When register.com went down at the beginning of April, our email server was "down", our website was "down", and we couldn't do anything about it. We lived. If you depend on something 100% to make money, then you should have a back-up plan. Real businesses don't depend on any one thing to make money. That way, if something goes wrong, it isn't the end of the world. And if you do depend on any one thing to make your company operate, you should have failsafes in place that will protect you from failures. Everything fails, it is just a matter of when. This may teach people that you can't depend on a 3rd party entirely to make money.
It sounds like you have it all figured out Jason. Has google or microsoft called yet? I mean 6,000 users, 10 years and no down time?! When I posted earlier, I didn't realize someone who worked for a "large corporation" would have solved such a complex problem as unexpected down time... I just wish my electric company would hire you... I know it's not as "unacceptable" as an email outage though :P
http://gmailblog.blogspot.com/2009/01/new-in-labs-offline-gmail.html
Offline Gmail ftw
or a freaking mail client. Problem solved.
If someone, or a corporation, approaches me and promotes the use of their free product based on features such as speed, reliability and etc., I have the right to complain when they can't deliver at their stated feature level. If the product keeps under-performing and I accept less 100% THEN I need to either shut up or get out.
"Free" has never meant lack of responsibility, it means nothing more than a lack of immediate cost.
Will GMail become beta again?