Now is a good time to point out that the SLA of Google Cloud Storage only covers HTTP 500 errors: https://cloud.google.com/storage/sla. So if the servers are not responding at all then it's not covered by the SLA. I've brought this to their attention and they basically responded that their network is never down.
Doesn't surprise me. The value of their ad products is deteriorating rapidly. For instance just today we found that one of our UAC campaigns, which supposedly is super smart and will automatically find the best channel for you decided to switch to a new channel with 14x higher CPC than the other channel that was working very well. That new CPC was 2x our target CPI. Why on earth they don't put in limits to prevent stupid bids like that is beyond me.
Given the big mess they had when Google Drive migrated to Google Play subscriptions I wouldn't exactly say that they're good at it. http://imgur.com/a/gzkZ5
I'd be very surprised if Dropbox manages to IPO at $10B valuation. The growth simply isn't there. They're big on platforms and markets with no growth but they're getting killed in growth markets. E.g. my cloud storage startup Degoo recently surpassed Dropbox on the Android grossing rank in India (http://imgur.com/HsII1KK) and we're only 5 people.
1% of ~2 roughly billion users = 20 million users. With one request every 5 years that'd be roughly 10k support requests per day. Considering that a lot of the cases can be answered very quickly (or possibly automatically) you can easily handle that with a couple hundred employees. The cost of that would be roughly 1/1000 of Google's annual income.
Also on GCS, if you do a HEAD after DELETE on a bucket that is under lifecycle management it returns 200 instead of 404. Not really a consistency issue but it can really come and bite if you if you're not aware of it. GET returns 404 but HEAD returns 200.
I reported it as a bug but Google said it was by design. More specifically they said: "You are correct, if the versioning enabled in your bucket then the object metadata is saved as an archive object in the bucket [1].This is the reason you are getting 200 for your HEAD request."
We are also calling S3 from our servers. S3 is receiving more requests and has had 0 issues. This is the corresponding NewRelic data for S3: http://imgur.com/HjH4f0Q
I asked Google's support team the same thing regarding the SLA not including situations when the system would not be responding to any requests at all. This was there response: "please understand that these SLA's are meant to cover backend issues on our end. In your scenario, we would have no control over our DNS server getting hacked. I apologize if there was confusion caused."
So Google claims that it does not have control over its own DNS servers and is therefore not to blame if the DNS is pointing to the wrong IP. Not very reassuring.
This also occurs for small payloads (e.g. list operations). Google's support team has acknowledged that the problem was on their end. They sent us this message: "I wanted to let you know we have some more information regarding the root cause of the issue you faced. Further investigation with our engineering team confirmed that the issue was caused by a provisioning error in the internal Cloud Storage infrastructure that led to low performance and “Service Unavailable” errors when handling uploads to the US region."
Unfortunately the problem is still occurring after I got this message (although less frequently).
Thanks for the follow up. This has been an open support case since October 15th. Regardless of the reason of the issue I find it absurd that only HTTP 500 errors are covered by the SLA.
We use Multi-regional, Regional, DRA, Nearline and Cloudline. NewRelic doesn't differentiate between the buckets. It only provide an average across all request to storage.googleapis.com. However, since you now are promising sub-second access time for all storage classes it still wouldn't explain it.
Don't you agree that it's odd to only include HTTP 500 errors in the error rate? Let's say someone hacks your DNS servers and points storage.googleapis.com to 127.0.0.1. Then the entire service would be down completely but according to your SLA you'd have 100% up time.
If you consider to use Google Cloud Platform it's important to know that their SLA is practically useless. It only includes requests with HTTP Status code 500. If the system is not responding at all it's not covered by the SLA. See their definition of "Error rate": https://cloud.google.com/storage/sla
This is not just a theoretical issue. In the past week we've been doing a bit more than 5 request/sec to Google Cloud Storage and according to NewRelic the average response time was 8 seconds! I.e. the service has been down and not been responding at all for large periods of time. I've been in contact with their support team and they've refused to reimburse us anything.
Great that they solved it! My company (degoo.com) had to use CloudFront instead of CloudFlare for our binaries because of this bug. Our conversion rate went down about 10% whenever we used CloudFlare instead. Perhaps time to switch back.