How Discord Stores Billions of Messages (2017)(blog.discord.com)
blog.discord.com
How Discord Stores Billions of Messages (2017)
https://blog.discord.com/how-discord-stores-billions-of-messages-7fa6ec7ee4c7
368 comments
I've used cassandra quite a bit and even I had to go back and figure out what this primary key means:
They also cite the most common cassandra mistake, which is not understanding that your partition key has to limit partition size to less than 300MB, and no surprise: They had to craft the "bucket" column as a function of message date-time because that's usually the only way to prevent a partition from eventually growing too large. Anyhow, this is incredibly important if you don't want to suffer a catastrophic failure months/years after you thought everything was good to go.
They didn't mention this part: Oh, I have to include all partition key columns in every query's "where" clause, so... I have to run as many queries as are needed for the time period of data I want to see, and stitch the results together... ugh... Yeah it's a little messy.
((channel_id, bucket), message_id)
They also cite the most common cassandra mistake, which is not understanding that your partition key has to limit partition size to less than 300MB, and no surprise: They had to craft the "bucket" column as a function of message date-time because that's usually the only way to prevent a partition from eventually growing too large. Anyhow, this is incredibly important if you don't want to suffer a catastrophic failure months/years after you thought everything was good to go.
They didn't mention this part: Oh, I have to include all partition key columns in every query's "where" clause, so... I have to run as many queries as are needed for the time period of data I want to see, and stitch the results together... ugh... Yeah it's a little messy.
Discord had like $300M invested and they created unparalleled piece of software that ate whole market, damn.
One of the most impressive softwares that I've seen and use after years of using ventrilo/mumble/teamspeak.
One of the most impressive softwares that I've seen and use after years of using ventrilo/mumble/teamspeak.
Looks like they migrated (at least partially) to Scylla: "Discord Chooses Scylla as Its Core Storage Layer" (2020)
https://www.scylladb.com/press-release/discord-chooses-scyll...
https://www.scylladb.com/press-release/discord-chooses-scyll...
This is like a masterclass in how to answer system design questions. Maybe a bit verbose.
They cover requirements, how to answer those requirements, relevant tech for the problem, implementation, and techniques for maintenance
We took a big bet on Cassandra, and then on an opinionated wrapper around Cassandra at $PASTJOB. The use case was a text search engine for syslog-type stuff.
The product we built using Cassandra was widely known as our buggiest and least maintainable, and it died a merciful death after several years of being inflicted on customers.
We didn't have a good handle on the exact perf implications of different values of read/write replication. Writing product code to handle a range of eventual consistency scenarios is challenging. The memory consumption and duration of compactions and column/node repair jobs is hard to model and accommodate. It's hard to tell what the cluster is doing at any given moment. Our experience with support plans from Datastax was also pretty dismal.
Maybe the situation has changed since 2016. In my experience with several employers since then, it seems like every enterprise architect fell in love with Cassandra around 2014-2015 and then had a long, painful, protracted breakup.
The product we built using Cassandra was widely known as our buggiest and least maintainable, and it died a merciful death after several years of being inflicted on customers.
We didn't have a good handle on the exact perf implications of different values of read/write replication. Writing product code to handle a range of eventual consistency scenarios is challenging. The memory consumption and duration of compactions and column/node repair jobs is hard to model and accommodate. It's hard to tell what the cluster is doing at any given moment. Our experience with support plans from Datastax was also pretty dismal.
Maybe the situation has changed since 2016. In my experience with several employers since then, it seems like every enterprise architect fell in love with Cassandra around 2014-2015 and then had a long, painful, protracted breakup.
Discord is so good. I just can't imagine it can stay this good forever. My fear is that eventually it will be bought out and aggressively monetized.
TFA states:
> we knew we were not going to use MongoDB sharding because it is complicated to use and not known for stability
But then goes on to describe using Cassandra and overcoming sharding and stability issues. I.e., changing the key, changing TTL knobs, adding anti-entropy sweepers, and considering switching to a different cassandra impl entirely.
Are these issues significantly harder to solve in MongoDB than Cassandra?
> we knew we were not going to use MongoDB sharding because it is complicated to use and not known for stability
But then goes on to describe using Cassandra and overcoming sharding and stability issues. I.e., changing the key, changing TTL knobs, adding anti-entropy sweepers, and considering switching to a different cassandra impl entirely.
Are these issues significantly harder to solve in MongoDB than Cassandra?
KKV databases (Cassandra and DynamoDB are good examples) have a common problem with hotspots or "hot partitions". The most common mistake is to use a timestamp of any kind in the range (cluster) column. Then, whatever partition represents "today" or "this hour" ends up being the hot partition.
The article mentions hot partitions becomming a problem with max partition size, but they're also a problem with scalability. Say, if your writing a very high throughput of logs into the table (contrived example), then your bottlenecked by the rate at which you can write to one partition.
Adding the bucket id (say, the current day or hour), is a common solution, and solves the max partition size issue, but not the scalability issue of hot partitions.
The article mentions hot partitions becomming a problem with max partition size, but they're also a problem with scalability. Say, if your writing a very high throughput of logs into the table (contrived example), then your bottlenecked by the rate at which you can write to one partition.
Adding the bucket id (say, the current day or hour), is a common solution, and solves the max partition size issue, but not the scalability issue of hot partitions.
I'd first reach for Postgres to do this. Anyone have any idea how Postgres would stack up in a similar challenge?
They wouldn't need to store so many if they actually let people delete their messages on account deletion. Instead, they ban many people who attempt to do so via automated scripts.
I love Discord's tech blog. There are a few corporate tech blogs that are just fantastic. Fly.io is another one that has great writing and interesting topics.
Can anyone share experiences with using Discord as a communications tool in a workplace? We're currently on Google Chat because it comes with the package that we pay for anyway, but it's pretty lame. So from time to time we consider jumping to Slack. But then, why not Discord?
They should check out Scylla if they want even faster queries and - depending on their workload - fewer instances.
Edit: It seems they have moved to Scylla
Edit: It seems they have moved to Scylla
I would have used CockroachDB, it has all the requirements listed and you don't need to know in advance the queries you will perform when deciding the database schema.
I'm curious of the 2021 measure of total disk space that Discord consumes. Servers that I'm in share images every few minutes, which must add up pretty quick.
We use scylla for our IoT stream, bucket per day, with a date index for second resolution data. The current day is a hot spot of, but we throw that in redis. It's running one of the largest re insurance providers IoT deployments.
Nothing about any sort of encryption.
If you can, use Scylla over Cassandra. The performance difference is tremendous in my experience and replacing can be trivial (easier if you start with Scylla on day 0)
Partition Key selection aside, if you want a better Cassandra, look at ScyllaDB. Much, much better engine, imho.
/disclaimer/ I used to work at Scylla.
/disclaimer/ I used to work at Scylla.
Maybe I don't know enough about databases, but could you really not have done this with postgres?
Two horrible choices of databases... first they used MongoDB then they migrated to Cassandra. I've used tons of databases [1] in production and those two are the worst.
[1] I've used RethinkDB, Postgres, MongoDB, MySQL, Cassandra, CockroachDB, TimescaleDB, SSDB, and others
[1] I've used RethinkDB, Postgres, MongoDB, MySQL, Cassandra, CockroachDB, TimescaleDB, SSDB, and others
"Billions of messages"
that's a prime target for acquisition by big-tech / big-data companies
that's a prime target for acquisition by big-tech / big-data companies
do they ever explain what their "anti-entropy" processes are for and what they do?
I am excited for the upcoming launch of full text search on discord.
Could somebody help me understand the reasoning behind
`timestamp = snowflake_id >> 22`
Thanks :)
`timestamp = snowflake_id >> 22`
Thanks :)
plenty of discussion when this was news:
https://news.ycombinator.com/item?id=13439725
https://news.ycombinator.com/item?id=13439725
oh great, here come the mongodb haters.
I do use discord for a few groups, too bad they will not allow 3rd party clients because a discord terminal app similar to irssi would be awesome
I do use discord for a few groups, too bad they will not allow 3rd party clients because a discord terminal app similar to irssi would be awesome
Well, very good for them and their "business partners". That data and metadata all for them to snoop into...
Can't wait for OpenFeint 2.0 to have its scandal lawsuit too.
Can't wait for OpenFeint 2.0 to have its scandal lawsuit too.
Did they ever make it to ScyllaDB?
All this happens on the aforementioned MongoDB cluster and just two server nodes. And the two server nodes are really only for redundancy, a single node easily fits the load.
What I want to say is:
-- processing a hundred million simple transactions per day is nothing difficult on modern hardware.
-- modern servers have stupendous potential to process transactions which is 99.99% wasted by "modern" application stacks,
-- if you are willing to spend a little bit of learning effort, it is easily possible to run millions of non trivial transactions per second on a single server,
-- most databases (even as bad as MongoDB is) have a potential to handle much more load than people think they can. You just need to kind of understand how it works and what its strengths are and play into rather than against them.
And if you think we are running Rust on bare metal and some super large servers -- you would be wrong. It is a normal Java reactive application running on OpenJDK on an 8 core server with couple hundred GB of memory. And the last time I needed to look at the profiler was about a year ago.