How Digg is Built(about.digg.com)

85 pointsby atularoravor 15 Jahren18 comments

about.digg.com

http://about.digg.com/blog/how-digg-is-built

22 comments

gumbovor 15 Jahren

Read this again and i find it great. so helpful for a young start-up to choose from the start the rigth tool and architechture.

Thanks again.

skimbrelvor 15 Jahren

Cassandra, Redis, MySQL, and Mogile as primary data stores? Good lord. And I thought my company had a complicated architecture.

dajobevor 15 Jahren

You think this diagram is complex? You should see the one that describes the full details. Things I missed out include the tiny Java bit. No Ruby though.

gumbovor 15 Jahren

Why are you using so much data stores? is it due to some legacy? I understand why you need MySQL (because some key algorithms are based on jointures), but why have at the same time Redis and Casandra? you also mention that you'll maybe replace MySQL by HBase? why? You also mention that the MapReduce jobs inputs are some flat logs (i suppose that those logs are stored in HDBS) so are you using any API to write in HDBS from RabbitMQ?

dajobevor 15 Jahren

Boy, a lot of questions. I already mentioned that the different stores have different features. I wouldn't see Redis or Cassandra replacing each other for anything. HBase is a potential replacement for bigger MySQL-like things such as big joins across all user data, actions. M-R inputs are logs in HDFS, copied from scribe. RabbitMQ isn't involved in logging those.

gumbovor 15 Jahren

Thanks for the answers. I know, a lot of questions: this is because we are architecturing our plateform and having a detailled help from a successful website is very helpfull.

Looking forward to see the next article.

rhizomevor 15 Jahren

It's not the diagram that's complex but the variety of its dependencies, particularly with respect to the variety of data storage methods.

dajobevor 15 Jahren

True, and it isn't the full story of system dependencies. Data dependencies aren't really shown here and they are also tricky, although Cassandra is the the primary ("source of truth") for most things. So for example, if a user does something on the site such as Digg a story, you mainly write Cassandra (several times) but in some cases have to write Redis, as well as flush some caches. Meanwhile async jobs can be writing to HDFS and ultimately MySQL.

u48998vor 15 Jahren

So basically users are happily swimming in the sea of advertisements with content they don't own or create.themselves. Nice business!

dajobevor 15 Jahren

I'm the author of this article which was about the technology we use. Your comment doesn't seem to be about tech at all.

u48998vor 15 Jahren

Agree.

[deleted]vor 15 Jahren

Toddvor 15 Jahren

Great overview of the technology choices and how they interact. I really appreciate when important companies reveal some of the inner workings of their sites. It gives something for the rest of us to go on. It can help inform our judgement when we make similar choices. This happens all too rarely. Many thanks!

dajobevor 15 Jahren

Thanks, we hope to go deeper in areas people are interested in. Let us know.

pkalervor 15 Jahren

I'd love to see you go deeper into devops. How are changes pushed? How often do you push changes? How are changes tested? How do you performance test at scale? How do you recover if something goes wrong?

cdurukvor 15 Jahren

Hi! We posted something that somehow answers that question.

http://about.digg.com/blog/continuous-deployment-code-review...

Some parts of it are a bit outdated --we don't use Selenium tests for pre-testing all commits-- but overall it should give some idea of how deployment works at Digg.

[deleted]vor 15 Jahren

gumbovor 15 Jahren

rhizomevor 15 Jahren

Very interesting. If that's world-class (which it is), I'm probably charging too little for my time.

sbhat7vor 15 Jahren

Nice article. It'd be great to have some numbers that indicate the size/scale of the system and its components.

dajobevor 15 Jahren

I'll see if we can include some figures in tech posts that follow.