Posterous is being DDoS'd(twitter.com)
twitter.com
Posterous is being DDoS'd
http://twitter.com/posterous/status/21082191639
11 comments
Gigenet's service looks interesting. They don't provide pricing though. I'm guessing it's not cheap? Is it based on the bandwidth they'd have to use?
Relying on a DNS TTL of 5 minutes doesn't work everywhere. Some ISPs will forcefully reset any TTL below 24 hours to 24h.
AT&T/sbcglobal being one.
Absolutely true, and completely ridiculous. What, DNS bandwidth is too high? Please.
Well, not following the RFCs is bad, but I can see why they do it.
Most people using low TTLs probably don't know what they're doing, and if you're a big ISP having to constantly make recursive queries hurts page load times for your customers, who'll blame you.
Most people using low TTLs probably don't know what they're doing, and if you're a big ISP having to constantly make recursive queries hurts page load times for your customers, who'll blame you.
I really wish setting DNS TTL's to 5 minutes meant that browsers and everything else (OS caching) would respect that but sadly in my experience it doesn't.
Surprising that it works so well for you.
Surprising that it works so well for you.
[deleted]
This is hot off the heels of an email sent out last night:
As you’re no doubt aware, Posterous has had a rocky six days.
On Wednesday and Friday, our servers were hit by massive Denial of Service (DoS) attacks. We responded quickly and got back online within an hour, but it didn’t matter; the site went down and our users couldn’t post.
On Friday night, our team worked around the clock to move to new data centers, better capable of handling the onslaught. It wasn’t easy. Throughout the weekend we were fixing issues, optimizing the site, some things going smoothly, others less so.
Just at the moments we thought the worst was behind us, we’d run up against another challenge. It tested not only our technical abilities, but our stamina, patience, and we lost more than a few hairs in the process.
I’m happy to report Posterous is at 100% and better than ever. Switching to a new data center will help us avoid the type of attacks we saw last week, and the new, bigger, beefier servers will speed up the site and increase capacity. We were hit pretty hard, but we’ve come out stronger in the end.
While we were certainly frustrated, we know that no one was more frustrated than you. Your website was down, and I humbly apologize for that. Know that throughout these six days, restoring your site and your trust has been our number one priority.
At the same time, I have never been prouder of the Posterous team. Garry led the team in the server migration, Vince worked like a mad man until he passed out on his desk, and Jackson helped the community understand the impacts of some pretty technical issues. This week has been all hands on deck, and we had some pretty great hands.
We have great plans for the future of Posterous. Our team is stronger than ever, and we are laser focused on developing better ways for you to share your content online. Thank you for believing in us, thank you for trusting us, and thank you for sticking with us through the past six days.
--Sachin Agarwal Cofounder and CEO, Posterous
As you’re no doubt aware, Posterous has had a rocky six days.
On Wednesday and Friday, our servers were hit by massive Denial of Service (DoS) attacks. We responded quickly and got back online within an hour, but it didn’t matter; the site went down and our users couldn’t post.
On Friday night, our team worked around the clock to move to new data centers, better capable of handling the onslaught. It wasn’t easy. Throughout the weekend we were fixing issues, optimizing the site, some things going smoothly, others less so.
Just at the moments we thought the worst was behind us, we’d run up against another challenge. It tested not only our technical abilities, but our stamina, patience, and we lost more than a few hairs in the process.
I’m happy to report Posterous is at 100% and better than ever. Switching to a new data center will help us avoid the type of attacks we saw last week, and the new, bigger, beefier servers will speed up the site and increase capacity. We were hit pretty hard, but we’ve come out stronger in the end.
While we were certainly frustrated, we know that no one was more frustrated than you. Your website was down, and I humbly apologize for that. Know that throughout these six days, restoring your site and your trust has been our number one priority.
At the same time, I have never been prouder of the Posterous team. Garry led the team in the server migration, Vince worked like a mad man until he passed out on his desk, and Jackson helped the community understand the impacts of some pretty technical issues. This week has been all hands on deck, and we had some pretty great hands.
We have great plans for the future of Posterous. Our team is stronger than ever, and we are laser focused on developing better ways for you to share your content online. Thank you for believing in us, thank you for trusting us, and thank you for sticking with us through the past six days.
--Sachin Agarwal Cofounder and CEO, Posterous
This sort of crisis management is always a challenge. How do you communicate this sort of problem to customers without losing credibility later if there is a risk of lingering issues? Based on the tone of the email that you posted, as a customer, I would expect the problem solved, case closed. Certainly, it makes customers feel good that the team has it handled.
The new events, if not handled quickly, could create doubt in customer minds and the team, as the optimistic outlook may not hold. I hope they beat this soon.
The new events, if not handled quickly, could create doubt in customer minds and the team, as the optimistic outlook may not hold. I hope they beat this soon.
Simple rule about DDoSes: there are no guarantees.
Why DDOS posterous? What's there to gain from it?
This isn't like the ones who DDOS'ed MS out of their hatred for it or the ones who blackmailed & DDOS'ed a gambling website when they refused to pay up.
This isn't like the ones who DDOS'ed MS out of their hatred for it or the ones who blackmailed & DDOS'ed a gambling website when they refused to pay up.
I severely doubt this line of thought.. but they didn't make any friends with their "switch from X to us" campaign, and it only takes one bruised ego with a lack of ethics to stir up trouble.
That said, if it's like most other DDOS attacks I've seen reported, the target is probably a user of the service with Posterous merely being unlucky enough to host a particular site someone doesn't like. I vaguely recall another platform got DDOSed recently due to some Israel vs Iran type cyberwar.
That said, if it's like most other DDOS attacks I've seen reported, the target is probably a user of the service with Posterous merely being unlucky enough to host a particular site someone doesn't like. I vaguely recall another platform got DDOSed recently due to some Israel vs Iran type cyberwar.
Must be something that they really dislike.
Never underestimate the power of lulz when trying to understand a DDoS.
someone probably has an issue w/ the blog/content of someone who is using posterous rather than posterous themselves.
That's my guess as well.
Weebly said (I think in their YC Founders at Work interview) that they get multiple DDoSes a day, aimed not at them but at sites they host. Now they have good enough systems in place to deal with it that they said they often just don't notice.
I guess when you host enough people's content, it's only a matter of time before someone wants to DoS something you're hosting.
Weebly said (I think in their YC Founders at Work interview) that they get multiple DDoSes a day, aimed not at them but at sites they host. Now they have good enough systems in place to deal with it that they said they often just don't notice.
I guess when you host enough people's content, it's only a matter of time before someone wants to DoS something you're hosting.
[deleted]
Maybe, and take it with a grain of salt, some people got infuriated by their recent switch to posterous campaign. I know this may be a very remote option, but in my opinion it's very possible.
The botnet controller could be extorting websites directly... wire $5000 to my paypal account or your website will go down. Anyone from Posterous care to confirm/deny in this case?
This is very common with online gaming (== gambling) sites, and payment systems; anything where there is obviously money being processed.
Do you have any evidence that this has ever been done in the past?
It's not inconceivable, but it seems like a huge leap.
It's not inconceivable, but it seems like a huge leap.
Google seems to provide plenty of examples. Even if it didn't, I don't understand why you see this extortion as a stretch.
It's not uncommon. There was an article a long time ago telling the story of a DDOS attack on a gambling website, and how a guy who was a philosophy major (iirc) figured out how to beat it, and then formed a company providing the same service. Forgot the url/title/etc, but it was good. Plenty more available with a search.
Could this have anything to do with Posterous' risqué post claiming superiority over Tumblr?
http://www.blogherald.com/2010/06/24/posterous-slanders-tumb...
http://www.blogherald.com/2010/06/24/posterous-slanders-tumb...
Who knew what the attackers are thinking. Not too long ago DnsMadeEasy was DDoS from Korea.
Maybe there's financial incentives?
Maybe there's financial incentives?
Our datacenter is experiencing heavy packet loss. We're on the line with Rackspace now.
I don't see where they stated it was a DDOS attack. Packet loss can occur due to a large number of different issues.
EDIT: They just clarified with the following: The DDoS attackers have returned and evolved their attack around our countermeasures. We expect to be back online ASAP w/ @gigenet antiddos
I don't see where they stated it was a DDOS attack. Packet loss can occur due to a large number of different issues.
EDIT: They just clarified with the following: The DDoS attackers have returned and evolved their attack around our countermeasures. We expect to be back online ASAP w/ @gigenet antiddos
It's interesting that they're basically saying Rackspace's network couldn't hold up and doesn't have an effective solution for DDOS victims on their network.
Gige's DDOS protection is basically a hosted redirect and filtering system that "leverage[s] the cost of DDoS mitigation amongst a large group of businesses, giving you access to the infrastructure that would normally be out of reach financially." http://www.gigenet.com/ddos-protection.html
But if Posterous is hosted on the Rackspace cloud, why isn't there a mechanism to do this through their existing host? After all, Rackspace has an equivalent service (maybe rebranded?) that would seem to offer protection without needing to go to a 3rd party vendor. http://www.rackspace.com/managed_hosting/services/security/d...
So it's interesting that Posterous needed to go around Rackspace to get a solution.
Gige's DDOS protection is basically a hosted redirect and filtering system that "leverage[s] the cost of DDoS mitigation amongst a large group of businesses, giving you access to the infrastructure that would normally be out of reach financially." http://www.gigenet.com/ddos-protection.html
But if Posterous is hosted on the Rackspace cloud, why isn't there a mechanism to do this through their existing host? After all, Rackspace has an equivalent service (maybe rebranded?) that would seem to offer protection without needing to go to a 3rd party vendor. http://www.rackspace.com/managed_hosting/services/security/d...
So it's interesting that Posterous needed to go around Rackspace to get a solution.
It's tough -- if someone is hitting you at > 1gbps, that exceeds a gigabit ethernet port -- typically on shared services you only pay for that much. Any more than that and it affects all the other customers on the shared network. So yes, Rackspace has the bandwidth for it, but at the same time if you were Rackspace, would you let our DDoS take down the other hosts in the datacenter?
Gigenet / Black Lotus / Prolexic comes in and says -- we have excess bandwidth, we'll take care of it. Then they proxy clean traffic to you. Unfortunately as we are learning, it's also very expensive.
Gigenet / Black Lotus / Prolexic comes in and says -- we have excess bandwidth, we'll take care of it. Then they proxy clean traffic to you. Unfortunately as we are learning, it's also very expensive.
I seriously doubt this is the reason but they insulted a lot of people with their "switch to us" targeted attacks.
That was classless.
That was classless.
It wasn't classless, it was marketing. No one wants their competitors poaching their clients but in business you deal with it. That said, the same thought went through my mind, I wonder if it's related.
That sounds very clear and professional. Unfortunately, it doesn't always work out that way.
Most of their competitors are in the same situation they are in.
Being the first one to torpedo the other ship and not expecting someone to get upset or God forbid do something awful in return is naive.
I think there is a difference in poaching your competitor's customers and claiming they are "no good".
Most of their competitors are in the same situation they are in.
Being the first one to torpedo the other ship and not expecting someone to get upset or God forbid do something awful in return is naive.
I think there is a difference in poaching your competitor's customers and claiming they are "no good".
Can someone estimate the cost of this attack? It shouldn't be cheap to lose all these bots for a massive 1gbps attack.
I could claim with nearly all certainty that this is a result of their campaign. The attacked several communities. Not just startups, but communities with developers (wordpress, tumblr etc). Their marketing was bold; but looking back they might have been more subtle.
So this is the 2nd DDOS attack. Can DDOS attack be fended off?
Usually it means using a hardware solution like Intruguard or going through a third party vendor that resells Intruguard and/or other proprietary anti-DDOS technology.
We've opted to use Gigenet, and they're holding the line right now quite well.
We've opted to use Gigenet, and they're holding the line right now quite well.
By crushing the enemy.
I hope Posterous does a write-up about this on Axon Flux when this is all over. Poor fellars.
I hope Posterous does a write-up about this on Axon Flux when this is all over. Poor fellars.
2nd DDOS in a matter of a week.
I wonder if you guys know who's doing it, and why.
[deleted]
Yes. It is getting DDOS'd now. I just posted a post. But can't access it.
TODAY, the attackers hopped IP's to our new IP immediately. So they appear to be learning. But then again, so are we.
Gigenet's anti-DDoS service has helped us a ton here and is now serving as our front door IP to block the synflood. They've been really responsive.