Google wreaks havoc on our company’s calendars: lost data and security breaches(elezea.com)
elezea.com
Google wreaks havoc on our company’s calendars: lost data and security breaches
http://www.elezea.com/2012/12/google-calendar-lost-data-security-breach/
22 comments
This sounds like a severe technical problem that has to be analyzed and fixed by real experts. I doubt that these kind of people do night and weekend shifts. And unless the problem is widespread, I don't think they're calling those experts in from their weekends for a single incident.
If it can happen once, it can happen again. Calling in those experts on the weekend is exactly the thing you should do if you're Google. Pay them double if you need to. But you need to catch this sort of thing before it escalates, not least for the PR damage it can cause.
I'm sure Google employs several people on call
Hopefully they can help solve this problem or at least raise the (SNAFU) alarm
Hopefully they can help solve this problem or at least raise the (SNAFU) alarm
24/7 support does not mean (to me, obviously) 24/7 cheerleading: "come on, we are going to solve this issue".
As far as I know, "support" means "tackling with the problem", not just saying "I'll do what I can expect a mail from me...".
As far as I know, "support" means "tackling with the problem", not just saying "I'll do what I can expect a mail from me...".
Sounds like a database migration [by Google that is] or dump/reimport that hasn't preserved the original primary/foreign keys and so items are now owned by other people. Oh the joys of referential integrity.
The ownership of things seems to have been kept within your company but not so for individual items in the calendar which could mean that random other people are seeing your calendar entries appearing in their calendars.
"Oops" is the operative word from Google.
The ownership of things seems to have been kept within your company but not so for individual items in the calendar which could mean that random other people are seeing your calendar entries appearing in their calendars.
"Oops" is the operative word from Google.
Are all of your employees using two-factor authentication? Are you 100% that this could not have happened via a normal security breach via a number of your employees' accounts?
GMail accounts get hacked regularly, often due to users using the same password for different services, or just getting keylogged somewhere.
I'm very curious too see how this pans out. Good luck with the recovery!
GMail accounts get hacked regularly, often due to users using the same password for different services, or just getting keylogged somewhere.
I'm very curious too see how this pans out. Good luck with the recovery!
It's not possible to move calendars around in this way. Plus it doesn't make sense to do something like this as a hack.
We're having a different issue on our companies' shared calendar -- all of our events are duplicated. Apparently Google know about it but there doesn't seem to be any fixes happening.
I've used Google's services from both an end-user at a company who uses Google Apps, and from a IT/Infrastructure Manager who moved his company to Google Apps.
For 95% of businesses who don't do a lot of intensive activities (be it scheduling, emailing, or anything else), google apps works fine.
If you're in that 5% who push the limits of your email/calendaring setup, then Google Apps (and their lackluster support) aren't the choice to make.
We had a support issue where the CEO couldn't send email to the company-wide mailing alias -- every time he did so, it was marked as 'bulk' email, and rejected. Google's support was non-existent (read this support page -- did that help? No? That's all I can do). Fast forward to a few weeks later, when users would randomly get non-delivery notifications when sending email internally. Google mentioned that they couldn't see the notifications come from their servers.
Another option was when a trial ran out, they'll lock EVERYONE out of email, calendars, and everything else until the trial is cancelled, or, until they pay. If an admin unknowingly signs up for something that will cost him a few extra bucks/user/month, and then forgets to update a credit card, every user in the organization will be locked out, and they can't call support (because you can't get a phone pin when you are locked out of a domain like that).
If you want minimal hassles, then Google Apps works great for small places (up to 20 users). Then, it's probably wise, from both a business risk and a infrastructure standpoint to just start running your own.
For 95% of businesses who don't do a lot of intensive activities (be it scheduling, emailing, or anything else), google apps works fine.
If you're in that 5% who push the limits of your email/calendaring setup, then Google Apps (and their lackluster support) aren't the choice to make.
We had a support issue where the CEO couldn't send email to the company-wide mailing alias -- every time he did so, it was marked as 'bulk' email, and rejected. Google's support was non-existent (read this support page -- did that help? No? That's all I can do). Fast forward to a few weeks later, when users would randomly get non-delivery notifications when sending email internally. Google mentioned that they couldn't see the notifications come from their servers.
Another option was when a trial ran out, they'll lock EVERYONE out of email, calendars, and everything else until the trial is cancelled, or, until they pay. If an admin unknowingly signs up for something that will cost him a few extra bucks/user/month, and then forgets to update a credit card, every user in the organization will be locked out, and they can't call support (because you can't get a phone pin when you are locked out of a domain like that).
If you want minimal hassles, then Google Apps works great for small places (up to 20 users). Then, it's probably wise, from both a business risk and a infrastructure standpoint to just start running your own.
Obviously having something like this happen sucks. If nothing else, you should be able to have a specified timetable for updates (every 4 hours, for example), even if it's just "we're still analyzing this issue" or "we are evaluating potential solutions".
That said, when you're dealing with a cloud-based application, fixing a problem like this is not going to be easy. First they need a clear enough understanding of how this happened to prevent it from happening again. Next you have to start coming up with scripts to correct the issue. After that you have to test the scripts to make sure you don't make things worse. Finally you can update production.
Most likely the support team has engaged an operations team and probably some developers as well. But even if they've been working through the weekend, there's no guarantee that they'll have a solution yet.
Or maybe Google doesn't have the necessary people on call over the weekend so nothing has actually been done. I can't say.
That said, when you're dealing with a cloud-based application, fixing a problem like this is not going to be easy. First they need a clear enough understanding of how this happened to prevent it from happening again. Next you have to start coming up with scripts to correct the issue. After that you have to test the scripts to make sure you don't make things worse. Finally you can update production.
Most likely the support team has engaged an operations team and probably some developers as well. But even if they've been working through the weekend, there's no guarantee that they'll have a solution yet.
Or maybe Google doesn't have the necessary people on call over the weekend so nothing has actually been done. I can't say.
I sync everything with the cloud and have a local backup which allows me to rollback any change to any file. So in the event that the cloud goes rogue I can just disconnect and restore from backup. But also, in the event that my office gets burnt down, I can still access all my data from the cloud. I personally would never have a system where the only point of access to mission critical data is through 'the cloud'.
You failed to perform due diligence before deciding to rely on this service. I wrote about this kind of thing over a year ago: http://lee-phillips.org/gcaldisaster/ If you just Google (hah!) "calendar privacy" today my article, and a few other relevant warnings, are on the first page.
Remember, according to Derek Sivers, everything is your fault.
Maybe they tested Google Spanner on Elezea's bit of data?
So it's being dealt with then. Not sure when wording an attention seeking post and submitting it to HN became standard operating procedure in these situations. Patience remains a virtue when this stuff happens.
>an attention seeking post
In fairness, I would like to know about issues like this before I sign up for this service. The fact it's being dealt with is really irrelevant (of course it's being dealt with). The fact it happened at all is the issue.
(Temporary) data loss and downtime is one thing. There's always something you don't predict, and you just have to live with it. But having calendars being assigned to the wrong people: that just shouldn't happen, and suggests there's some underlying architectural problem.
In fairness, I would like to know about issues like this before I sign up for this service. The fact it's being dealt with is really irrelevant (of course it's being dealt with). The fact it happened at all is the issue.
(Temporary) data loss and downtime is one thing. There's always something you don't predict, and you just have to live with it. But having calendars being assigned to the wrong people: that just shouldn't happen, and suggests there's some underlying architectural problem.
It's very unlikely that Google would have a bug that affects only this specific company, posts like these are valuable because they bring wider attention to the issue so other Google Calendar users can see if something is up with their calendars.
It's not valid to let other Google Apps users - current or potential - know about serious issues with the service?
That's a really easy decision for you to make when it's not your business that's being held hostage.
Personally, If I can't run something on my LAN, I don't want it. So, I love hearing about these issues; I'd like to someday see developers moving away from SAAS for every little function.
Personally, If I can't run something on my LAN, I don't want it. So, I love hearing about these issues; I'd like to someday see developers moving away from SAAS for every little function.
"That's a really easy decision for you to make when it's not your business that's being held hostage.
Personally, If I can't run something on my LAN, I don't want it. So, I love hearing about these issues; I'd like to someday see developers moving away from SAAS for every little function."
My former employer at a small business used to say the same thing, and gleefully pointed out when gmail went down for 3 hours 2 years ago, screaming about how dangerous it is to rely on an external entity for these services. The outcome: we had our own servers for our email, and experienced DAYS of unplanned outages a year, not counting the planned outages due to migrations. On top of that, he had to pay the salaries of sys admins who could have been devoting their skills to stuff that our business actually got paid for.
The justifications for insourcing IT vs. using a SaaS pretty much align with the justifications for keeping cash in a safe in your house vs. using a bank. Sure, they are there, but they all are much, much less likely than the scenarios which will make you wish you weren't handling it yourself (local server failures, local internet failure due to ice storm, hurricane, software update, idiot sys admin) or in the case of money in a safe(house fire, robbery, flood). My company had very competent sys admins, but I'll never forget losing email for 3 business days due to an ice storm 3 states away where the other office was hosting the servers. Then we decided to purchase servers in our other office to prevent this, and then we hired more people to run them. It was fucking stupid, and all due to a stupid mental paradigm of "If i'm not in direct control then it's not being done right." We had less uptime, less security, and a much higher operating cost. That's a high price to pay for illogical, emotionally based paranoia.
My former employer at a small business used to say the same thing, and gleefully pointed out when gmail went down for 3 hours 2 years ago, screaming about how dangerous it is to rely on an external entity for these services. The outcome: we had our own servers for our email, and experienced DAYS of unplanned outages a year, not counting the planned outages due to migrations. On top of that, he had to pay the salaries of sys admins who could have been devoting their skills to stuff that our business actually got paid for.
The justifications for insourcing IT vs. using a SaaS pretty much align with the justifications for keeping cash in a safe in your house vs. using a bank. Sure, they are there, but they all are much, much less likely than the scenarios which will make you wish you weren't handling it yourself (local server failures, local internet failure due to ice storm, hurricane, software update, idiot sys admin) or in the case of money in a safe(house fire, robbery, flood). My company had very competent sys admins, but I'll never forget losing email for 3 business days due to an ice storm 3 states away where the other office was hosting the servers. Then we decided to purchase servers in our other office to prevent this, and then we hired more people to run them. It was fucking stupid, and all due to a stupid mental paradigm of "If i'm not in direct control then it's not being done right." We had less uptime, less security, and a much higher operating cost. That's a high price to pay for illogical, emotionally based paranoia.
I'm not saying that running your own servers is always easier, nor is my stance one of emotional based paranoia.
I'm saying that "private cloud" products that you host yourself could be just as easy to setup and use as cloud-based products if developers wanted to build those products. Instead they want to nickel-and-dime everyone for every business function while retaining absolute control over access to the product. SAAS can absolutely be more expensive in the long run.
(EDIT: Coincidentally, gmail is down again today.)
I'm saying that "private cloud" products that you host yourself could be just as easy to setup and use as cloud-based products if developers wanted to build those products. Instead they want to nickel-and-dime everyone for every business function while retaining absolute control over access to the product. SAAS can absolutely be more expensive in the long run.
(EDIT: Coincidentally, gmail is down again today.)
Don't thrust the cloud...