How upgrading to Google Apps for Business killed my company's email for 6 hours(floriancornu.blogspot.com)
floriancornu.blogspot.com
How upgrading to Google Apps for Business killed my company's email for 6 hours
http://floriancornu.blogspot.com/2013/05/how-upgrading-to-google-apps-for.html
20 comments
Or $19/month get a Mailgun.net account (also run by Rackspace) and use it as a redirection service into individual free gmail (or google apps) accounts.
I do the same thing for free with my domain registration company and then pop all my messages off gmail using fetchmail for good measure. I then send through an external smtp server. All the fun of gmail, no reliance on Google. I actually also tried to pay them before setting that all up and they couldn't sort it out.
Similarly, if you use DNSimple you can enable email-forwarding on your domains [1]. That's the solution I use for my SaaS product so far.
[1] http://support.dnsimple.com/articles/email-forwarding
[1] http://support.dnsimple.com/articles/email-forwarding
No reason not to use a 3rd party IMAP/POP and still use the Gmail front-end.
Best of both worlds.
Best of both worlds.
There were multiple reasons right in his post. Including the actual support options like being able to call someone at 3am. And the service level agreement that states it has to stay up or you don't pay for it. And the regularly scheduled backups that you know are occurring. Gmail has none of that.
If you're really only using the frontend you don't care, you have an alternative if it stops working.
Using Google Apps, all factors considered, seems to be less reliable than the old free Gmail. (Since Google can't pull the plug on you like it does to its paying customers, like this case)
Using Google Apps, all factors considered, seems to be less reliable than the old free Gmail. (Since Google can't pull the plug on you like it does to its paying customers, like this case)
The thing is, it's not just the frontend. Gmail's backend has outtages, too. Often in conjunction with the frontend. Sometimes just the IMAP/POP goes down. In any of those situations, it'll be fixed eventually, but you have no support available to you to let you know what's up or to report issues to if its something only affecting your account.
Google can easily pull the plug on any Gmail account they want. It says so right in the terms of service. You have no guarantee of service or access to your data. Even higher profile reporters have had access to all their Google data (gmail, calendar, contacts, documents, google drive, etc) disabled for having what appeared to be client payment data in a single Google doc. They were only able to restore it after a few days because they had some connections.
Google can easily pull the plug on any Gmail account they want. It says so right in the terms of service. You have no guarantee of service or access to your data. Even higher profile reporters have had access to all their Google data (gmail, calendar, contacts, documents, google drive, etc) disabled for having what appeared to be client payment data in a single Google doc. They were only able to restore it after a few days because they had some connections.
You're just adding more points of failure.
Not really if you can also use the other account directly if Gmail were to fail.
I thought that a single point of failure was a bad thing. Sure you might want to only have a single place where things can fail, but if that failure is catastrophic...
"Single point of failure" means a point at which a failure effectively breaks the entire system or workflow. A system may have many SPOFs, which is worse than a single SPOF, which is, in turn, worse than no SPOFs (that is, the desired case is no single failure kills forward progress).
This is true, but sometimes be introducing a new risk (point of failure) to the system, you get the ability to route around another. In this instance, separating POP and Gmail gives you the ability to route around Gmail when it's down.
Yes, extra points of failure may include:
- Having webmail which is not protected with two factor authentication
- Enabling POP/IMAP which also does not have two factor authentication
- Having webmail which is not protected with two factor authentication
- Enabling POP/IMAP which also does not have two factor authentication
No, adding more points of failure doesn't make the first potential point of failure more fail-proof, or anything like that.
If adding a new layer (gmail) creates a new point of failure without creating any new resilience against failure, you're putting yourself into a bad position.
Think about failure like resistance. Points of failure in parallel gives less overall risk of failure. Points of failure in serial (like adding a front-end) gives greater overall risk of failure.
Think about failure like resistance. Points of failure in parallel gives less overall risk of failure. Points of failure in serial (like adding a front-end) gives greater overall risk of failure.
The polling interval for POP (there is no IMAP) can be up to an hour, which I find sufficient reason not to be happy with that approach.
Why not forward from Rackspace, that's what I do.
This
Forward to your firstname.lastname@gmail
Configure SMTP
Configure filters so you have a folder for company email
problem solved
Forward to your firstname.lastname@gmail
Configure SMTP
Configure filters so you have a folder for company email
problem solved
With, perhaps, the disadvantage that now the NSA can choose the easiest target to mine all your email from - if Google pushes back on a request, they can just go to RackSpace instead.
It's a difficult compromise between availability and security/privacy. I want to be able to get at my mail from all my devices via any Internet connection, but. I still want privacy from dragnet intercept-and-record-everything surveillance.
My current plan - a mail server in the least US influenced jurisdiction I can manage, automatically encrypting all (unencrypted) email to a public key of mine, the forward the encrypted email to a gmail account. I can decrypt mail at the client on my phone/tablet/laptop/desktop, but I lose a lot of mail searchability this way.
It's a difficult compromise between availability and security/privacy. I want to be able to get at my mail from all my devices via any Internet connection, but. I still want privacy from dragnet intercept-and-record-everything surveillance.
My current plan - a mail server in the least US influenced jurisdiction I can manage, automatically encrypting all (unencrypted) email to a public key of mine, the forward the encrypted email to a gmail account. I can decrypt mail at the client on my phone/tablet/laptop/desktop, but I lose a lot of mail searchability this way.
Considering that they have an open line to Gmail, if you have stuff in Gmail, the NSA isn't going to need to worry about going to Rackspace.
One minor annoyance with that is that it breaks Gmail's SPF lookups as Rackspace does not do SRS [1] (last time I checked).
[1] http://www.openspf.org/SRS
[1] http://www.openspf.org/SRS
Why not set up your own SMTP server that only accepts connections from rackspace. Then forward to aaa.bbb.ccc.ddd:55525.
there is no IMAP
What do you mean?
What do you mean?
The Gmail mail fetcher that allows you to use non Gmail accounts in the Gmail interface is POP only
https://support.google.com/mail/answer/21289?rd=1
https://support.google.com/mail/answer/21289?rd=1
I've been using Rackspace mail for a few years now, flawless service and shockingly good customer support - helpful, competent people available instantly, cannot recommend enough. Given our email accounts our the skeleton keys of our digital lives relying on a free service, or even a paid service where support is something the company sees as a numbers game seems madness.
My only gripe is the lack of support for two-factor authentication, apparently this is under development currently.
My only gripe is the lack of support for two-factor authentication, apparently this is under development currently.
My only gripe with Rackspace is how long it takes for them to push out a new version of Exchange when they're released... Was something like a year and a half after Exchange 2010 was released before it appeared on Rackspace? Curious to see how long before 2013 appears...
Otherwise I love them.
Otherwise I love them.
I've recently converted to Fastmail.fm, which seems great and they have a nice web interface too. They're now owned by Opera, so I assume their data retention policies must be consistent with EU law, though I think they have servers based in the US so I'm not sure what that means in practice.
What I'd like is a service that kept my emails encrypted on-disk with my user password. Not because I think emails are private -- they're not in the sense that I fully expect someone could intercept them in transit, even if that would be illegal -- but just because it would give reassurances that my emails can't be data-mined for advertising purposes à la gmail, or if the mailserver had a major security vulnerability, a hacker couldn't copy all my emails wholesale.
But I don't know of any service that provides this, nor of a way to do it myself easily (any mailserver I set up myself would probably have far more vulnerabilities than one I rent from someone else!).
What I'd like is a service that kept my emails encrypted on-disk with my user password. Not because I think emails are private -- they're not in the sense that I fully expect someone could intercept them in transit, even if that would be illegal -- but just because it would give reassurances that my emails can't be data-mined for advertising purposes à la gmail, or if the mailserver had a major security vulnerability, a hacker couldn't copy all my emails wholesale.
But I don't know of any service that provides this, nor of a way to do it myself easily (any mailserver I set up myself would probably have far more vulnerabilities than one I rent from someone else!).
Unfortunately, most folks would not pay the extra fees that would be necessary for implementing a service like this. And the NSA wouldn't like it as they couldn't just ping Google's server and read all your mail from Gmail.
I'm paying for Fastmail anyway, so I don't think it's that unreasonable for a paid service.
Or a $10/mo Digital Ocean VM running Zimbra 8 Community Edition.. https://s3.amazonaws.com/uploads.blog.zimbra.com/wp-content/...
Hows the Rackspace web mail interface?
That's great - I love worthy and proper competition combatting nonchalant/tyrant attitude of incumbent giants. It is so annoying to deal with customer service reps that act like they are your boss and you're inconveniencing them through your questions. The stronger the company, the more of those reps you'll get.
Thanks for posting this!
Thanks for posting this!
SLA is sort of a red herring here.
If my email goes down, getting a partial or full refund on my $2/month mailbox fee is the least of my concerns.
If my email goes down, getting a partial or full refund on my $2/month mailbox fee is the least of my concerns.
The point of the SLA here isn't so much to save you the $2, but to make it cost them $2 * N accounts. This gives them an economic incentive to avoid problems.
Yeah, fair enough, but regardless of any SLA, I wouldn't expect to be charged full price for inadequate service. Not if you want my continued business anyway.
I used to love rackspace... but had a really bad experience recently. I was using Rackspace Files as a CDN for my ecommerce site. One of my customers emailed me saying that the graphics were missing from the site. Turned out 2 of the 3 zones were down, according to status.rackspace.com. They fixed it about 12 hours after it started IIRC. They never sent an email notifying me of the down time.
Then -- and this is where they destroyed my trust in their service -- I took a look at the status panel the following day, and they cleared the history of the downtime (it shows 30 days of issues). Instead of showing the issue, they marked it as green and deleted the messages describing the issue.
That day, I stopped using Rackspace for everything. (This was about a month ago)
Then -- and this is where they destroyed my trust in their service -- I took a look at the status panel the following day, and they cleared the history of the downtime (it shows 30 days of issues). Instead of showing the issue, they marked it as green and deleted the messages describing the issue.
That day, I stopped using Rackspace for everything. (This was about a month ago)
Interesting.. I just looked at their status panel, and the downtime is there, including all of the messages. They were definitely missing the day after it happened.
Still disappointed they don't send out emails when there's downtime..
Would probably use them again the future. (This was my one negative experience with them, btw)
Still disappointed they don't send out emails when there's downtime..
Would probably use them again the future. (This was my one negative experience with them, btw)
tldr; Author experienced an 6-hour outage in Google Apps service, after upgrading to a paid Google Apps account. This behavior is not documented.
Is "outage" the right word? That suggests that it's an error, or down for everyone.
They really screwed the pooch, here.
They really screwed the pooch, here.
>Is "outage" the right word? That suggests that it's an error, or down for everyone.
As someone who has been in IT for 20 years, I can tell you're in IT. This type of semantics drives business users crazy.
Business User: Hey my system is down!
IT Person: No, it's not down.
Business User: I can't log in to it.
IT Person: We are doing scheduled maintenance so the system is unavailable, but it's planned maintenance so it's not down.
As someone who has been in IT for 20 years, I can tell you're in IT. This type of semantics drives business users crazy.
Business User: Hey my system is down!
IT Person: No, it's not down.
Business User: I can't log in to it.
IT Person: We are doing scheduled maintenance so the system is unavailable, but it's planned maintenance so it's not down.
Wrong direction. This is not failure-apologetics.
Outage suggests temporary and regrettable.
This is more like Google just straight up fucking people over via neglect. I was suggesting that stronger negative terms be used.
Outage suggests temporary and regrettable.
This is more like Google just straight up fucking people over via neglect. I was suggesting that stronger negative terms be used.
It's still an "outage" if even just one user is affected. In fact, it's still an "outage" if the service is unavailable (which includes being temporarily unusable) for some period of time, and no users are actively affected by it.
The same thing happened to us as http://www.thinkful.com/, only it took 18 hours to come back! Customer support was hilariously useless as well: "Yes, happens to everyone." "No, there's nothing we can do about it."
For us, accounts that were forwarding mail someplace else continued to work, but anyone that had to log into Thinkful's gmail account didn't.
For us, accounts that were forwarding mail someplace else continued to work, but anyone that had to log into Thinkful's gmail account didn't.
Customer support was hilariously useless as well: "Yes, happens to everyone." "No, there's nothing we can do about it."
The time delay is ridiculous and unacceptable, but it sounds like customer support did the most they can possibly do in that case: Engineering had implemented a (questionable) migration policy, and there are no toggles or switches that customer service can fiddle with to make it happen more quickly.
The time delay is ridiculous and unacceptable, but it sounds like customer support did the most they can possibly do in that case: Engineering had implemented a (questionable) migration policy, and there are no toggles or switches that customer service can fiddle with to make it happen more quickly.
Failure to communicate up front and in advance the likelihood and nature of this... "situation", is in some ways even more unacceptable.
Many/most parties could probably cope with it reasonably well -- if they were told in advance that it would occur.
Upgrade over the weekend. Let customers know your systems will be experiencing some planned downtime. Etc.
Come on, Google. You claim to be supporting businesses. Well, act like it! Among other things, this means communicate, before the sh-t hits the fan.
----
P.S. This situation isn't even asking for a dialog. Just Google's communicating some information -- what's really gonna happen -- up front.
Sometimes, your marketing needs to take a bit of a hit, for the sake of your customer.
I've little doubt that most of the engineers there respect this. We have to deal with such situations every day.
If I were particularly direct, I might suggest one of them go over to marketing with his/her "stomping" boots on, until they come to understand this point.
Many/most parties could probably cope with it reasonably well -- if they were told in advance that it would occur.
Upgrade over the weekend. Let customers know your systems will be experiencing some planned downtime. Etc.
Come on, Google. You claim to be supporting businesses. Well, act like it! Among other things, this means communicate, before the sh-t hits the fan.
----
P.S. This situation isn't even asking for a dialog. Just Google's communicating some information -- what's really gonna happen -- up front.
Sometimes, your marketing needs to take a bit of a hit, for the sake of your customer.
I've little doubt that most of the engineers there respect this. We have to deal with such situations every day.
If I were particularly direct, I might suggest one of them go over to marketing with his/her "stomping" boots on, until they come to understand this point.
It is bad, yes. But compared to the experiences you get the an ISP and the crappy email services they supply. I recently complained to my ISP about slow services, and they disconnected us and turned off all services I had from the for 3 weeks (whilst trying to fix it). No emails and no phone calls. Fortunately I don't use my ISPs provided email address for anything that matters (for this reason), it was the lack of Internet that was worst. This experience cost us quite a bit of money and a mass off time. 18 hours would have been nice. Orcon. New Zealand.
I disagree – customer support doesn't start with the person on the phone with the customer, it ends there. When a company designs a process where the person on the phone isn't empowered to effect change that person is guaranteed to be useless.
That this was happening to Google, and that it occurred exactly when we tried to _pay them money_ definitely makes the situation hilarious.
That this was happening to Google, and that it occurred exactly when we tried to _pay them money_ definitely makes the situation hilarious.
Disclaimer: I work for Google
Google's support is a complete joke. The only way to get things fixed in a timely manner (if at all) is to ask someone who works for Google to escalate internally. Then you stand a chance. Otherwise you're SOL.
On top of that the Gmail team routinely ignores internal feedback before rolling out new features or UI changes claiming the Googlers do not represent average users.
I would recommend the OP to contact a Googler friend if they have one.
Google's arrogance and disrespect for its users will be its undoing.
Google's support is a complete joke. The only way to get things fixed in a timely manner (if at all) is to ask someone who works for Google to escalate internally. Then you stand a chance. Otherwise you're SOL.
On top of that the Gmail team routinely ignores internal feedback before rolling out new features or UI changes claiming the Googlers do not represent average users.
I would recommend the OP to contact a Googler friend if they have one.
Google's arrogance and disrespect for its users will be its undoing.
The solution to this is for Google to explicitly tell the end user that the upgrade involves a migration to bigger and better infrastructure, during which their accounts will be inaccessible for a number of hours. Then let the user schedule the migration to minimize the inconvenience. During the migration, show the account status as "Migrating..." instead of empty. Problem solved.
But that would involve Google adding documentation to their system. Which, from my perspective, seems very difficult for them to manage.
The solution to this is for Google to explicitly tell the end user that the upgrade involves a migration to bigger and better infrastructure, during which their accounts will be inaccessible for a number of hours. [...] Problem solved.
Sorry, but that isn't even close to a solution. The problem here is that a business-critical facility was off-line for pretty much an entire business day, at who-knows-what cost to the business.
Many small businesses that use facilities like this are trading internationally or otherwise need 24-hour functionality, so the idea that you could just schedule something to "minimize the inconvenience" doesn't really help them at all.
Sorry, but that isn't even close to a solution. The problem here is that a business-critical facility was off-line for pretty much an entire business day, at who-knows-what cost to the business.
Many small businesses that use facilities like this are trading internationally or otherwise need 24-hour functionality, so the idea that you could just schedule something to "minimize the inconvenience" doesn't really help them at all.
I have some clients who are paying MessageLabs something like two orders of magnitude more than Google charges per seat. They would, undertstandably, scream blue murder if their entire corporate email went down for 24hrs.
I have other clients who wouldn't dream of paying $5 per user per month, because they get free email forwarding with their domain name or cheapo web hosting.
I have _many many_ clients, who if I gave then sufficient notice and said "we're upgrading some infrastructure, we need to take the mail server down for up to 24 hrs, does after close of business the Friday after next suit you, or would you prefer we did it Sat night/Sun morning?", would be _perfectly_ happy. (with the possible exception along the lines of "oh, we're at a trade show that week, can you move it forward or wait till a week later?")
I wonder how big the crossover is between companies who really absolutely _have_ to have 24/7/365 email uptime at better tha 3 nines, but who haven't already got something more robust in place than a handful of personal gmail accounts?
I can't help but think they're "doin it wrong"…
I have other clients who wouldn't dream of paying $5 per user per month, because they get free email forwarding with their domain name or cheapo web hosting.
I have _many many_ clients, who if I gave then sufficient notice and said "we're upgrading some infrastructure, we need to take the mail server down for up to 24 hrs, does after close of business the Friday after next suit you, or would you prefer we did it Sat night/Sun morning?", would be _perfectly_ happy. (with the possible exception along the lines of "oh, we're at a trade show that week, can you move it forward or wait till a week later?")
I wonder how big the crossover is between companies who really absolutely _have_ to have 24/7/365 email uptime at better tha 3 nines, but who haven't already got something more robust in place than a handful of personal gmail accounts?
I can't help but think they're "doin it wrong"…
I wonder how big the crossover is between companies who really absolutely _have_ to have 24/7/365 email uptime at better tha 3 nines, but who haven't already got something more robust in place than a handful of personal gmail accounts?
The problem here appears to be that Google's pathway to get to such functionality -- assuming a service branded "Google Apps for Business" does in fact offer a higher standard of service than three 9s and isn't pure marketing spin -- is broken in a way that could undermine an organisation that wanted to make that change.
Obviously that won't affect all businesses, but if you're in that boat then it seems Google can't even help you get to their better services without risking exactly the kind of interruption that a more serious/professional service is supposed to avoid. That seems bizarre to me, and doesn't speak well of how they run their business-level services. Combined with their recent trend of killing off even moderately popular services altogether, it doesn't exactly fill you with confidence about trusting Google with anything your business really depends on.
The problem here appears to be that Google's pathway to get to such functionality -- assuming a service branded "Google Apps for Business" does in fact offer a higher standard of service than three 9s and isn't pure marketing spin -- is broken in a way that could undermine an organisation that wanted to make that change.
Obviously that won't affect all businesses, but if you're in that boat then it seems Google can't even help you get to their better services without risking exactly the kind of interruption that a more serious/professional service is supposed to avoid. That seems bizarre to me, and doesn't speak well of how they run their business-level services. Combined with their recent trend of killing off even moderately popular services altogether, it doesn't exactly fill you with confidence about trusting Google with anything your business really depends on.
It wasn't business-critical before, since they were using the free version. Businesses don't use free services with no SLA for business-critical things. Unless they don't know how to do business.
Don't make assumptions on what's business critical and what's not. Not every business has the same level of free capital, circumstances or domain knowledge to handle everything theoretically correctly. There is implicit trust in working with companies like Google; you expect everything to work properly. They don't have to, of course, but when they don't, articles like this chip away at the confidence we all have in Google.
Businesses don't use free services with no SLA for business-critical things.
As far as I can see it's precisely the upgrade path to get them off that free service and onto a higher level that is broken here.
As far as I can see it's precisely the upgrade path to get them off that free service and onto a higher level that is broken here.
>It wasn't business-critical before, since they were using the free version.
The free version was available for a year before the paid version was made available.
The free version was available for a year before the paid version was made available.
Just because it doesn't work for everyone does not mean that a scheduled outage would not be a significant improvement for large percentages of customers. Most businesses can find a 6 hour window.
I had a similar situation where I switched a 50 person charity to Google Apps. They had a 30 day trial period and during that time the CEO's credit card was stolen and replaced. I changed the information but the Apps interface said 'No valid payment information stored' so I called their support and they assured me the new card was stored and everything would be fine.
Surprise, surprise on the day the account totally shut down. I called in a panic and after a lot of fiddling about (and reassuring me that they had recorded that I thought there might be a problem) they got it back up with everyone's mail after almost a whole day of downtime.
To add insult to injury Google don't run a charity program in the UK despite the non-profit laws being a lot stricter than the US and it being easy to validate a charity - so they're paying full rate for Google Apps for Business.
Surprise, surprise on the day the account totally shut down. I called in a panic and after a lot of fiddling about (and reassuring me that they had recorded that I thought there might be a problem) they got it back up with everyone's mail after almost a whole day of downtime.
To add insult to injury Google don't run a charity program in the UK despite the non-profit laws being a lot stricter than the US and it being easy to validate a charity - so they're paying full rate for Google Apps for Business.
That last part is not entirely accurate. I did some IT volunteer work for a charity very recently, and they definitely had a free Google Apps for Business setup. This is in addition to a reasonably generous grant in the form of AdWords credit.
Google Apps for Nonprofits page states it's only available in the US:
https://support.google.com/nonprofits/answer/1614602?hl=en&t...
If you could ask someone involved in that charity how they got free Google Apps I'd appreciate it as I work for a number of charities and the costs of Google Apps are a significant admin overhead. I've tried calling the sales people and they say it's coming soon but they've been saying that for over 5 years now.
https://support.google.com/nonprofits/answer/1614602?hl=en&t...
If you could ask someone involved in that charity how they got free Google Apps I'd appreciate it as I work for a number of charities and the costs of Google Apps are a significant admin overhead. I've tried calling the sales people and they say it's coming soon but they've been saying that for over 5 years now.
Would you have any idea who we're able to contact about this? I'm involved in a registered UK charity and we were looking to use Google Apps before, but can't justify the monthly cost of $5 per account.
Thanks!
Thanks!
Sorry for the misleading information, chaps.
I asked the organisation, and found out that it's not an official programme in the UK. The only one they have here is Google Grants for AdWords - http://www.google.co.uk/grants/.
Basically, an enterprising IT volunteer had set them up with vanilla Google Apps back when it was still free for small business and had a 50 user limit, and they just got grandfathered in to the new Apps for Business. They've never had any problems with the user or space limit since they just use the setup for email, analytics and AdWords campaigns and only have about 4-6 people "working" there.
Apologies again.
I asked the organisation, and found out that it's not an official programme in the UK. The only one they have here is Google Grants for AdWords - http://www.google.co.uk/grants/.
Basically, an enterprising IT volunteer had set them up with vanilla Google Apps back when it was still free for small business and had a 50 user limit, and they just got grandfathered in to the new Apps for Business. They've never had any problems with the user or space limit since they just use the setup for email, analytics and AdWords campaigns and only have about 4-6 people "working" there.
Apologies again.
Ah. The offering used to be called Google Apps for Domains back when users could still sign up for free 100/50/10-user-limit accounts. Google Apps for Business, in some of our minds, is the post-December 2012 paid service.
Here's a link comparing the free edition with Google Apps for Business: http://support.google.com/a/bin/answer.py?hl=en&answer=17512...
Thanks for getting back to us though!
Here's a link comparing the free edition with Google Apps for Business: http://support.google.com/a/bin/answer.py?hl=en&answer=17512...
Thanks for getting back to us though!
We had another issue, where the catch-all address started bouncing. Apparently they have changed the way they handle email routing. So if you upgrade to the 'new google apps', you better be very careful about your email routing rules. Went through a very similar experience with google customer care. Again turned out to be a bug in their system. But what sucks is for 3 days our emails were bouncing! And we're a paid google apps customer...Would be happy to pay 5X, just want some sort of customer service + acceptable SLAs on issues. And yes, please have a test team go over important features like enterprise email! I know it's Google policy to not have features go through a testing team, but maybe they should consider it for Google Apps.
/end{rant}
/end{rant}
If you are relying on a third-party cloud email service, please keep a regular backup of your data. From random hacking attempts leading to a account lockout to incorrectly being marked for spamming violations, anything could go wrong. The easiest way is to install Thunderbird and let it automatically sync and backup your emails to your computer.
From past experience at an email service provider, admittedly a small one, and quite a few years ago, the most likely failure mode related to internet outages knocking out the customer's internal email unlike a locally hosted email server which customers could never seem to understand, and DNS issues (failure to renew, got scammed / domain stolen because of noobs, etc). Probably the single biggest support problem was people (on either side) getting on spammer lists so genuine business communication can no longer flow.
Generally speaking the web interface being down, or the imap server being bonkers, was very rarely if ever a problem.
Generally speaking the web interface being down, or the imap server being bonkers, was very rarely if ever a problem.
For whatever it is worth, here is my experience from a few hours ago: my google apps trial ended this morning. Google emailed me, I clicked the billing link in the email and entered my info. After submitting, my account was immediately reactivated and I received a few emails that had arrived while the account was suspended. No downtime or anything... Maybe it has to do with how much data your accounts contain when you switch over to paying for access?
For a trial, I'd expect you started on the paid infrastructure from day 1, even though you hadn't paid yet. That would mean no migration and no downtime.
So the first thing about Google Apps is that you still need to back it up. Because it's (mostly) pretty easy to use it's easy to fall into the trap that it does everything right first time and that your data is safe when you make a change, unlike say running your own infrastructure. It isn't.
Different issue with Google Apps: We work with some big pharma companies. During the free evaluation, Google flagged two of our partners and one of our project managers as spammers and silently suspended their accounts. We hosted our email somewhere else.
I've known of a few cases where the Gmail account associated with Google Apps becomes unreachable. Rather than upgrading to the paid plan, most of the people involved just opened a support ticket which was resolved in a few hours.
I migrated a company with 60 franchises from a broken, old, 25 MB per mailbox mail system, to Google Apps last year.
The biggest suggestion I have for anyone wanting to use the paid Google Apps is to go through an authorized reseller. You will get better pricing, and they generally have faster access to higher level support than going directly through Google. I had an absolute nightmare of a time with client support (construction workers with little computer literacy) but any legitimate problems during the transition were quickly dealt with by Google via the reseller.
Also, good luck getting a response from the Google Apps sales team in the first place.
The biggest suggestion I have for anyone wanting to use the paid Google Apps is to go through an authorized reseller. You will get better pricing, and they generally have faster access to higher level support than going directly through Google. I had an absolute nightmare of a time with client support (construction workers with little computer literacy) but any legitimate problems during the transition were quickly dealt with by Google via the reseller.
Also, good luck getting a response from the Google Apps sales team in the first place.
I used to be IT manager of a company that wanted to save money and they had me switch them to google business. it was a smooth transition, but the company wanted it's employees (50ish non tech salespeople) to still use Outlook. It was ok. google had an app that allowed this. All was good. I left the company but my friend took my job from me when i left. He now has come across the problem of google not supporting the newest outlook. he gets new computers in and has to downgrade the outlook just to get email to work. headache
Killed it for 6 hours... not convenient, but probably shouldn't make big IT changes during business hours and without warning staff. Our company's move from GApps to Office365 + Outlook has been much more systemically affecting...
I think the issue is that this wasn't a big IT change. It was a simply addition of functionality to a cloud provider, which is SUPPOSED to be easy. Of course, they were using the free version of Gmail, which has no SLA and can be shut off at any time for any reason. Which means they didn't consider email that mission-critical.
I've been POPing my email since the 90s... am I doing it wrong? Yes, accessing email archives on multiple machines is a kinda-headache (I replicate my mailboxes), but never lost email.
[deleted](1)
tl;dr : Whenever you click on 'Upgrade Now' button just click it and go to sleep. Once you wake you everything should be fine.
Author was awake and saw some horrible things.
Author was awake and saw some horrible things.
And where is your backup? Oh wait you dont have a backup...
[deleted]
So the upgrade took a few hours to complete and it didn't happen instantly in defiance to the laws of physics. Next time play it safe and upgrade during the weekend.
The post is dated May 27, is Google planning to announce a new feature for Apps this week and this is some sort of a preemptive PR attack?
The post is dated May 27, is Google planning to announce a new feature for Apps this week and this is some sort of a preemptive PR attack?
6 hours... We run a 4 person company and 6 hours during the weekend would still be a problem for us.
Given a company larger than 10 might actually upgrade (small amount of shared email accounts) I could see it causing big problems for some.
Given a company larger than 10 might actually upgrade (small amount of shared email accounts) I could see it causing big problems for some.
How do you get 24x7x365 coverage with only 4 people? The "standard" is it takes about 5 to fill a position 24x7x365 over very long term on a larger average with absolutely no failure (unstaffed) tolerated. As in, 50 people can fill 10 positions absolutely positively all the time, but it doesn't downscale well at all to just 5 people and one desk. You can "fill" ten positions with a lot less than 50, but even at scale there's going to be a heck of a lot of time when there's only 8 or 9 people there hence the quotes around "fill". For instance you can "fill" 10 positions with 30 people but there's going to be a heck of a lot of time where the supposed 10 positions only have 7 (or even fewer) bodies actually on deck.
Depending on the business sector of course. If you're a stereotypical weekday business then a 6 hour outage at 9am on a weekday would be a disaster, but a 6 hour outage from midnight to 6am on Sunday morning wouldn't even be blinked at because no one cares.
Depending on the business sector of course. If you're a stereotypical weekday business then a 6 hour outage at 9am on a weekday would be a disaster, but a 6 hour outage from midnight to 6am on Sunday morning wouldn't even be blinked at because no one cares.
I agree with you midnight to 6am Sunday would probably be okay. Just would be a really bad time to get an email stating an issue with a payment gateway or marketplace listings or an issue with the website.
Really? Do you run a highly reliable email service yourself then? I certainly would not use gmail if that were the case. You should seriously look at why you are so reliant on email, as it is not really a reliable service, the mail servers at the other side could easily delay mail for 6 hours, it is only a best efforts eventually consistent protocol with very long delays allowed (you don't normally get bounces for days if the mail is queued for retry).
Well the world of business tends to rely pretty heavily on mail, not sure what other methods of reliable messaging there is?
Sure there is phone calls etc, but emails are used when you need an actual record of something. Also relying on mailservers to not bounce the emails and actually deliver in the end is laughable, when mailservers do go down this hardly ever happens properly.
Sure there is phone calls etc, but emails are used when you need an actual record of something. Also relying on mailservers to not bounce the emails and actually deliver in the end is laughable, when mailservers do go down this hardly ever happens properly.
Well, websites (or APIs) are generally much more reliable forms of messaging, because you can get immediate feedback on processing. The OP said it was business critical; most corporate email is not business critical, and missing business critical email in a channel with spam, high volume unimportant messages, and also relying on a human to answer stuff is all pretty unreliable. Sure you can feed email into a ticketing system, but at least there there is also a web interface (so you do not rely on email).
If you need a record of something, I do not think that unsigned emails are legally binding anyway, again you should probably submit signed contracts over the web not email.
If it is not time critical, email can work, but the OP said a six hour outage would be a problem, and I still think that is one they have created themselves and is a business risk.
If you need a record of something, I do not think that unsigned emails are legally binding anyway, again you should probably submit signed contracts over the web not email.
If it is not time critical, email can work, but the OP said a six hour outage would be a problem, and I still think that is one they have created themselves and is a business risk.
It sounds like email isn't critical for the corner of the universe that you operate in -- good for you.
For lots of people in different roles, email is an essential tool for getting work done. Not everyone has a role that can be readily translated into an API. Business Dev/Sales for example, depends on email to communicate with the various folks that they need to engage in. Whatever those folks do, it ends with the company getting a check, so it's important.
Generally speaking, it is pretty inexpensive to deliver a 99.9% available mailbox with a 100% guarantee of external mail delivery. The fact that Google bungles a conversion from free to paid service so poorly is a sad statement when they are supposed to be a shiny alternative to the traditional Micrsoft messaging stack.
For lots of people in different roles, email is an essential tool for getting work done. Not everyone has a role that can be readily translated into an API. Business Dev/Sales for example, depends on email to communicate with the various folks that they need to engage in. Whatever those folks do, it ends with the company getting a check, so it's important.
Generally speaking, it is pretty inexpensive to deliver a 99.9% available mailbox with a 100% guarantee of external mail delivery. The fact that Google bungles a conversion from free to paid service so poorly is a sad statement when they are supposed to be a shiny alternative to the traditional Micrsoft messaging stack.
Bus dev/sales can survive a day without email every now and again (in my experience, far more than a day without a phone system). Everything will resume the next day, sure it pays the bills but a six hour outage is not "mission critical", it wont stop the "check" from arriving (email is not a payment method after all).
I have known large (email dependent) businesses have 2 day outages on the traditional Microsoft mail stack too. There are coping strategies. It is annoying not mission critical.
I have known large (email dependent) businesses have 2 day outages on the traditional Microsoft mail stack too. There are coping strategies. It is annoying not mission critical.
Are you serious? Running my small business, I used email to communicate with my coworkers, potential hires, customers, prospects, partners, potential partners, reporters, accountants, lawyers, banks, web hosts, PayPal, UPS, various government departments etc etc. Being completely cut off from that for effectively an entire business day would be destroy my ability to get anything outside "solo hacker in basement" coding done, and there really isn't a sensible mitigation plan or alternative to e-mail for this (unless you'd like to convince my bank to start posting, say, notifications of incoming wire transfers to a web form of my own design?).
Your bank provides a website you can poll to find out about stuff. Not ideal, would be nice to have a proper API, but so far we only seem to be getting these for credit card payments but this is changing (eg see gocardless). For most of those things you list you could manage in an email outage, as you have phone numbers or other alternatives.
If it is that critical do you have 99.99%+ (52 minutes a year) uptime guarantees on your mail service? That is what you are asking for, and to actually deliver that (rather than an empty SLA promise) is something that very few businesses actually try to work to, especially small companies. Gmail certainly does not try to provide this.
If it is that critical do you have 99.99%+ (52 minutes a year) uptime guarantees on your mail service? That is what you are asking for, and to actually deliver that (rather than an empty SLA promise) is something that very few businesses actually try to work to, especially small companies. Gmail certainly does not try to provide this.
Google's enterprise Apps SLA is 99.9%, and last year they hit 99.98%.
To play devil's advocate:
1. There is no indication in the upgrade process that the upgrade would "take a few hours to complete". Quite the opposite, it indicated there would be no downtime.
2. The customer service reps seems to have little insight into what was happening with the account which is a bit scary. I'm always nervous of black hole "processing windows" where all you can do is wait and hope for the best. It's bad when it is customer facing, it's worse when it is customer service facing.
3. Upgrading on the weekend could potentially be more troubling given that customer service may not be staffed with the quantity/quality to fix issues if they occur. Not saying this is or is not the case for Google, but running into problems on a Saturday morning and getting a "call us back during normal business hours" is just as troublesome.
What should have happened in my opinion?
1. Google should document that there can be a temporary delay of x - x hours while the upgrade happens.
2. Florian should have scheduled the upgrade ahead of time with the team, letting them know that worse case there may be an outage of 'x' minutes/hours/days.
3. Google customer service should have better tools as to upgrade process for both the admin (customer) and customer service rep so that it is not a black hole of 'wait and hope'. Even a step 1 of x, or you are number # in queue, or estimated time, etc.
1. There is no indication in the upgrade process that the upgrade would "take a few hours to complete". Quite the opposite, it indicated there would be no downtime.
2. The customer service reps seems to have little insight into what was happening with the account which is a bit scary. I'm always nervous of black hole "processing windows" where all you can do is wait and hope for the best. It's bad when it is customer facing, it's worse when it is customer service facing.
3. Upgrading on the weekend could potentially be more troubling given that customer service may not be staffed with the quantity/quality to fix issues if they occur. Not saying this is or is not the case for Google, but running into problems on a Saturday morning and getting a "call us back during normal business hours" is just as troublesome.
What should have happened in my opinion?
1. Google should document that there can be a temporary delay of x - x hours while the upgrade happens.
2. Florian should have scheduled the upgrade ahead of time with the team, letting them know that worse case there may be an outage of 'x' minutes/hours/days.
3. Google customer service should have better tools as to upgrade process for both the admin (customer) and customer service rep so that it is not a black hole of 'wait and hope'. Even a step 1 of x, or you are number # in queue, or estimated time, etc.
So the upgrade took a few hours to complete and it didn't happen instantly in defiance to the laws of physics.
What law of physics necessitates that such an upgrade takes 6 hours?
What law of physics necessitates that such an upgrade takes 6 hours?
> What law of physics necessitates that such an upgrade takes 6 hours?
the one where the customer isn't paying enough money to have somebody working 24/7?
If your business depends on a particular service, you don't accept something vague - you get things signed in contract with legally enforcible SLA's. If you can't afford that, then you just have to live with shitty service.
the one where the customer isn't paying enough money to have somebody working 24/7?
If your business depends on a particular service, you don't accept something vague - you get things signed in contract with legally enforcible SLA's. If you can't afford that, then you just have to live with shitty service.
Have we just transitioned from the "It's free, you can't expect not to get screwed" trope to "It's inexpensive, you can't expect not to get screwed"?
It's Google. You never see the Google front page go down. There's an implied availability that Google uses to their advantage. People are comfortable with upgrading _because_ it's Google. Google wouldn't fuck you.
Except when they do. Lesson learned.
Except when they do. Lesson learned.
You do realize we live in a world of automation, and that Google... automates?
It wasn't that it took a long time. It's that the interface showed a state as if all the mail was deleted. That could have been devastating.
We recently upgraded to paid Google Apps for Business and we didn't get any downtime. This is how it is supposed to work.
Unfortunately we also had irritating problems after the upgrade. We upgraded because our customer support email address had run out of mail quota. Paid google apps has a 10x higher quota, but upgrading to premium didn't increase it.
After ringing customer service they refused to increase it immediately and said that it would be increased "in a couple of months time".
Meanwhile hundreds of our customers were going unanswered.
I suppose I can imagine a scenario in which they would want to wait until after credit card chargeback window before lifting a quota limit, but their support department should understand the paralysis of a business offer to solve the problem.
We had to create a temporary [email protected] email and manually port email across which wasn't fun and played hell with our ticketing system.
We recently upgraded to paid Google Apps for Business and we didn't get any downtime. This is how it is supposed to work.
Unfortunately we also had irritating problems after the upgrade. We upgraded because our customer support email address had run out of mail quota. Paid google apps has a 10x higher quota, but upgrading to premium didn't increase it.
After ringing customer service they refused to increase it immediately and said that it would be increased "in a couple of months time".
Meanwhile hundreds of our customers were going unanswered.
I suppose I can imagine a scenario in which they would want to wait until after credit card chargeback window before lifting a quota limit, but their support department should understand the paralysis of a business offer to solve the problem.
We had to create a temporary [email protected] email and manually port email across which wasn't fun and played hell with our ticketing system.
Why not pull down/delete old mail off your [email protected] email address?
I can't imagine you have 5 gigs (or however many they give you these days) of tickets that can't deal with a few days of cold storage.
I can't imagine you have 5 gigs (or however many they give you these days) of tickets that can't deal with a few days of cold storage.
Most people would assume that this is nothing more than an administrative change on the account, with no long drawn out six hour process. And if there is an upgrade time required, it absolutely should have in-your-face warnings given the importance of email 24/7 for many organizations.
Even if we buy that this is more than an administrative change and it somehow moves to better hardware, this is a problem that I would think that Google would have built to a mostly transparent process -- at most long term archives are unavailable after a very brief initial migration, etc.
Even if we buy that this is more than an administrative change and it somehow moves to better hardware, this is a problem that I would think that Google would have built to a mostly transparent process -- at most long term archives are unavailable after a very brief initial migration, etc.
That exactly the sort of assumption you can't afford to make if you've honestly got company-destroying problems if your email goes down for 6 or even 24hrs.
It's easy in hindsight, but I've seen that happen before, and I have no doubt that if you'd considered the risk and (perhaps ironically) googled for information, you too would have known about this.
On the other side of the coin (and perhaps the reason Google haven't cared enough to fix the problem), SMTP is nicely designed so as to not result in this sort of thing losing any mail - "well behaved" email systems will just queue and retry mail for 5 days if needed.
It's easy in hindsight, but I've seen that happen before, and I have no doubt that if you'd considered the risk and (perhaps ironically) googled for information, you too would have known about this.
On the other side of the coin (and perhaps the reason Google haven't cared enough to fix the problem), SMTP is nicely designed so as to not result in this sort of thing losing any mail - "well behaved" email systems will just queue and retry mail for 5 days if needed.
That exactly the sort of assumption you can't afford to make
Okay so what if the transition took five days? How about thirty days? In the absence of seemingly any warning information at all on this, how does one ever perform such a change?
Let's take it further -- what if adding a user took down your email for a month? That is just as rational as removing an artificial limit is ("Oh didn't you know? You pushed our global user database past the threshold so we had to migrate platforms"). How about if you send an email that you CC to ten people and that takes your email down for days?
If this wasn't an expected behavior, and is seemingly a mere administration change, there is no universe where it can be pinned on the user. Doubly obvious given the confused responses of customer service.
Okay so what if the transition took five days? How about thirty days? In the absence of seemingly any warning information at all on this, how does one ever perform such a change?
Let's take it further -- what if adding a user took down your email for a month? That is just as rational as removing an artificial limit is ("Oh didn't you know? You pushed our global user database past the threshold so we had to migrate platforms"). How about if you send an email that you CC to ten people and that takes your email down for days?
If this wasn't an expected behavior, and is seemingly a mere administration change, there is no universe where it can be pinned on the user. Doubly obvious given the confused responses of customer service.
Exactly my point. If you're making changes to mission-critical things, you need to have both reliable information on how long those changes will take, as well as rollback plans to cover the risk of things going wrong.
Randomly clicking things like "change my email system" buttons, then complaining afterwards that it didn't work how you expected is the sort of mistake most of us have made at least once.
Once you've suffered through those mistakes, you tend to view phrases like "expected behavior" and "seemingly a mere administration change" a lot more suspiciously.
If it's mission critical, don't "assume", don't "expect" - things are often not as they "seem". As they say "Trust, but verify." Yeah, Google (or Rackspace, or MessageLabs) will _presumably_ "get it right", but when the consequences of "presuming" are business-destroyingly-high, verify the presumptions first.
Randomly clicking things like "change my email system" buttons, then complaining afterwards that it didn't work how you expected is the sort of mistake most of us have made at least once.
Once you've suffered through those mistakes, you tend to view phrases like "expected behavior" and "seemingly a mere administration change" a lot more suspiciously.
If it's mission critical, don't "assume", don't "expect" - things are often not as they "seem". As they say "Trust, but verify." Yeah, Google (or Rackspace, or MessageLabs) will _presumably_ "get it right", but when the consequences of "presuming" are business-destroyingly-high, verify the presumptions first.
If you aren't fully addicted to the Gmail interface, there are better options for business mailbox hosting.