Billing Incident Update(twilio.com)
twilio.com
Billing Incident Update
http://www.twilio.com/blog/2013/07/billing-incident-update.html
13 comments
Call me crazy, but I tend to think issues like this ultimately serve, if handled well, to improve a company's public standing.
I royally screwed up the embed script testing for our very first customer resulting in their entire site not functioning and a panicking customer on the other end. I ended up writing a 3 page post-martem and apology. We debated a bit internally to be as open as I'd been, essentially admitting basic sloppiness on my part and promising to learn from the mistake if given a chance.
Fast forward 7 months, the same customer(originally acquired from a cold call) recently introduced us--without asking--to one of the top seed funds locally.
I royally screwed up the embed script testing for our very first customer resulting in their entire site not functioning and a panicking customer on the other end. I ended up writing a 3 page post-martem and apology. We debated a bit internally to be as open as I'd been, essentially admitting basic sloppiness on my part and promising to learn from the mistake if given a chance.
Fast forward 7 months, the same customer(originally acquired from a cold call) recently introduced us--without asking--to one of the top seed funds locally.
Happened to me. I made an SQL update without specifying the conditions on a live database (project running, surprise surprise, late). Our team has sent some flowers and chocolates too. I am going to do the same thing next time I screw up, and I will.
These days it is too much about the electronic and not enough of the "physical" in my mind.
These days it is too much about the electronic and not enough of the "physical" in my mind.
This isn't crazy at all, in fact it's been proven to be true.
The Marriott hotel chain found that guests who had something bad happen during their stay but which the Marriott fixed were actually more likely to return than those who had nothing bad happen at all.
Source: www.cbsnews.com/8301-505125_162-48540174/how-customer-complaints-can-improve-business/
The Marriott hotel chain found that guests who had something bad happen during their stay but which the Marriott fixed were actually more likely to return than those who had nothing bad happen at all.
Source: www.cbsnews.com/8301-505125_162-48540174/how-customer-complaints-can-improve-business/
Reminds me a bit of people who say "Apple is the best! My phone died but they replaced it on the spot."
Thinking about it, it makes sense to me.
Everybody makes mistakes, and everybody experiences bad luck.
If you only experience good times with a company, then you have no idea how they respond to bad times. They might be great, or they might be awful. You just have no idea.
If you experience a problem with a company, then you know how they respond to problems. If they do poorly, you'll avoid them. But if they do well, you now know that they handle problems well.
The time then comes to choose a company for something. Which do you choose: an unknown, or a company that you know handles problems well?
Everybody makes mistakes, and everybody experiences bad luck.
If you only experience good times with a company, then you have no idea how they respond to bad times. They might be great, or they might be awful. You just have no idea.
If you experience a problem with a company, then you know how they respond to problems. If they do poorly, you'll avoid them. But if they do well, you now know that they handle problems well.
The time then comes to choose a company for something. Which do you choose: an unknown, or a company that you know handles problems well?
In 1982 some nutjob went around to supermarkets and pharmacies and put potassium cyanide in Tylenol bottles. Johnson & Johnson responded by issuing a recall for ALL Tylenol products nationwide. This incident is anecdotally credited with having directly led to those paper or foil seals on over-the-counter pill bottles.
https://en.wikipedia.org/wiki/Chicago_Tylenol_murders
https://en.wikipedia.org/wiki/Chicago_Tylenol_murders
https://en.wikipedia.org/wiki/Chicago_Tylenol_murders is the link in case anyone has not heard of the case.
Correct me if I'm wrong, but I believe that also lead to the seals we see today on a wide variety of products.
Replace the word "company" in "a company's public standing" with "well-established startup that's already known for good service" and I 100% agree.
If Twilio's overall service wasn't excellent, people wouldn't be nearly as willing to view this from the positive angle. If Twilio weren't a startup - if it were, say, SalesForce.com - this also would hurt more than it would help.
Fortunately, Twilio seems to know this. They're absolutely in no way resting on their existing positive reputation. Even though they've earned the trust and respect of so many customers, they're still treating each and every customer as if they must still continue to earn their trust and respect.
Word of advice to Twilio execs: when this is all done and over with give your PR people a bonus and/or a raise. They did a fantastic job here in a nearly impossible situation, and I'm sure the recruiters are circling like sharks.
If Twilio's overall service wasn't excellent, people wouldn't be nearly as willing to view this from the positive angle. If Twilio weren't a startup - if it were, say, SalesForce.com - this also would hurt more than it would help.
Fortunately, Twilio seems to know this. They're absolutely in no way resting on their existing positive reputation. Even though they've earned the trust and respect of so many customers, they're still treating each and every customer as if they must still continue to earn their trust and respect.
Word of advice to Twilio execs: when this is all done and over with give your PR people a bonus and/or a raise. They did a fantastic job here in a nearly impossible situation, and I'm sure the recruiters are circling like sharks.
Truthfully? Our PR guy is on his honeymoon. Not kidding.
Glad to hear you got the information you needed when you needed it - still a lot of work to do to resolve this to our customers' complete satisfaction.
We'll get it done.
Glad to hear you got the information you needed when you needed it - still a lot of work to do to resolve this to our customers' complete satisfaction.
We'll get it done.
After reading some of the comments posted on the blog, just wanted to give you a shout-out for your having to put up with douchey comments and responding in a humble fashion to them.
Can't be easy. :)
Can't be easy. :)
Ha! Maybe I should've said "give the people who are handling PR for this..."
You're not crazy. I've found that a small customer service interaction early on (regardless of fault) helps solidify a good first impression. Not that I intentionally create problems...
Thank you very much for that - our work is still not done. Through the night we have identified what we believe to be the root cause and have reconstructed the timeline that produced the incident.
We still owe you a full technical post mortem of the incident and we are responding to all the servicing fee reimbursement requests at [email protected].
Deeply appreciate your support - more to come.
We still owe you a full technical post mortem of the incident and we are responding to all the servicing fee reimbursement requests at [email protected].
Deeply appreciate your support - more to come.
It was surprising to read they will be reimbursing any overdraft fees caused by the problem. Good move.
Why are you surprised they are reimbursing overdraft fees? That is the bare minimum that I would expect from any company which made this mistake
Of course it is to be expected, but unfortunately one cannot expect it happen. If you get it?
In this case, I think one can. If they cause damages due to negligence (or malice), they are liable for those damages. Only the dumbest of dumb companies would refuse to reimburse overdraft fees in a case like this, because doing so will open them up to lawsuits from every single affected person, lawsuits which those people basically cannot lose.
At the end of the day aren't they likely liable?
I don't think they are, but it's the right thing to do. One would hope the banks would be understanding enough about the situation (haha, I know, right?) to reverse the fees. But even in that case Twilio is still causing you to, at a minimum, call your bank, wait on hold, spend 15 minutes getting transferred around to someone who has the ability to reverse the fees, etc. If they want minimal impact for their customers, they'll just reimburse the fees.
I played this game with Dreamhost and SunTrust back with their whole billing debacle in Jan 2008. Dreamhost didn't respond to my support requests to reimburse "non-sufficient funds fees", The bank manager at SunTrust said that regardless of the reason funds weren't available in my account when the other transactions posted, so the NSF fees that I was incurring were legitimate.
In the end I think I paid a little over $300 for their mistake.
I played this game with Dreamhost and SunTrust back with their whole billing debacle in Jan 2008. Dreamhost didn't respond to my support requests to reimburse "non-sufficient funds fees", The bank manager at SunTrust said that regardless of the reason funds weren't available in my account when the other transactions posted, so the NSF fees that I was incurring were legitimate.
In the end I think I paid a little over $300 for their mistake.
I would hope they have a business liability insurance policy, in which cases it should cover the costs associated with this unfortunate mess, including any reimbursed fees.
Think it is the least we could do.
Simple, honest, open and straight to the point. Nice to see that they recognise it's not just about the money, but the trust people had in Twilio. I'm looking forward to reading the promised post-mortem, because I can't recall reading one from them before.
It has been some time since we have had an incident this disruptive for our customers. Looking forward to delivering that investigation summary to you as soon as we feel we got it right.
[deleted]
This is about as good as you can handle an incident like this, PR wise.
Congrats to Twilio on a textbook example on how to handle a billing outage. While I hope to never to through this I do admire a team that can execute like this.
Cheers, from one telco nerd to your team of hackers.
Cheers, from one telco nerd to your team of hackers.
Always appreciate your support Josh. much work to do before this incident is resolved fully.
So many updates on their status page! One of the first companies I felt like I knew wtf was going on during the issue.
And you can see status from two years ago! Google won't even recognize that gmail is down unless 5% of requests are failing (!!!! Try getting your work done all day if 1/20 actions fails.)
Very glad you found the stashboard useful during the incident. This is actually one of my favorite open source projects of ours.
If you would like a similar service for your company, check it out here: http://www.stashboard.org
If you would like a similar service for your company, check it out here: http://www.stashboard.org
Interesting. I just came back from an event here at Lone Star Ruby Conference they sponsored (movie - they used Twilio powered service to pick the movie we watched at a local brew theater). Would have made for some interesting questions had more known about this problem :-)
Why not batch process billing every hour or 30 minutes? That way you can run simple checks to look for anomalies.
It's a good question. As our service is delivered through a utility pricing model, most of our API requests are billing events - each call, text message or recording ends up having a debit to the account balance associated with it. As you can imagine then, our auto-recharge feature could be called any moment to make sure the balance doesn't run out.
That said, we do have the monitoring you describe in place which is how we were alerted to the incident and limited the damage to 1.4% of the account base.
That said, we do have the monitoring you describe in place which is how we were alerted to the incident and limited the damage to 1.4% of the account base.
Twilio should really consider a lighter gray color for their text on the white background. Right now the text is simply too legible still! Why, I can almost read it, if I crane my neck and look at the screen at just the right angle. What could they have been thinking? Hopefully they will take this feedback to heart so that future urgent messages like these will be completely obscured.
I've been a fan of ContrastRebellion, though I've lately simply taken to modifying websites' CSS that I visit often enough that it annoys me. E.g., I've swapped the border and content background colors on Hacker News.
What the heck, here it is: http://stylebot.me/styles/2945
I use Stylebot, but Stylish or other CSS management plugins should work just fine, copy/paste the CSS source.
What the heck, here it is: http://stylebot.me/styles/2945
I use Stylebot, but Stylish or other CSS management plugins should work just fine, copy/paste the CSS source.
Kinda ironic that this comment shows up as light grey on a grey background. :)
Appreciate the feedback. Will get it fixed.
Nice response, bad text color. The gray on bright white background really strains my eyes. Ouch.
I use iReader extension whenever the website's color scheme strain my eyes too much. It's essentially a clone of "Reader" functionality on safari.
https://chrome.google.com/webstore/detail/ireader/ppelffpjgk...
https://chrome.google.com/webstore/detail/ireader/ppelffpjgk...
Good feedback. Thank you.
That's how it is done. Classy.
For anyone looking for multiple (redundant) SMS provisioning, I have had not perfect but pretty good experience with http://fortytwotele.com/ over a good few years. I am not connected to them at all, just throwing the option out there. As a bonus, they're European, which means you are better positioned for compliance with EU privacy laws for non-US clients.
Call me crazy, but I tend to think issues like this ultimately serve, if handled well, to improve a company's public standing.
Though I suppose that depends on how one defines "well"...