10 petabytes - visualized(blog.backblaze.com)
blog.backblaze.com
10 petabytes - visualized
http://blog.backblaze.com/2011/01/05/10-petabytes-visualized/
52 comments
My 350GB of music is backed up on Backblaze (a wonderful service), but seeing those visualizations reminded me of the environmental impact of my digital packratting. That's a sizable data center keeping my data happy and backed up.
I wonder how many thousands of people are paying them to back up stuff they could easily redownload. Great business model if they de-dupe internally.
they encrypt all data and i dont think it would be wise of them analyzing their clients data for dupes...if somebody would get to know that their business would be gone.
ZFS supports encryption with deduplication, and Dropbox does dedupe accross customers (and why not? It's not a human digging through things).
http://blogs.sun.com/darren/entry/compress_encrypt_checksum_... - ZFS link
http://blogs.sun.com/darren/entry/compress_encrypt_checksum_... - ZFS link
When different customers use different key to encrypt the data, even if the source is the same, the outcome is different. So it is not feasible to do de-duplication across different customers' data, unless all of them use the same key.
If you're implying they are crazy for doing so, have you ever lost everything and tried to redownload it all? It's a huge timesink, I tell you.
As for de-duping, Dropbox does this. When you drop a Windows ISO or the Office installer into your folder, it hashes the file and realizes 'hey we already have this!' and you don't have to upload it.
As for de-duping, Dropbox does this. When you drop a Windows ISO or the Office installer into your folder, it hashes the file and realizes 'hey we already have this!' and you don't have to upload it.
Off topic, but what do you listen to that takes up 350GB? Is everything in FLAC?
A fair collection of (DJ) mixes (live or not) and sets can easily account for a few hundred gigabytes. Take Tiesto's "Club Life" and Armin Van Buurin's "A State of Trance" for example. With each set being around two hours or more of high quality mp3, with the former having over 300 episodes, and the latter having around a hundred. These things add up quickly.
I listen to a lot of music (from all genres). I only purchase/download full albums (even if most of the songs are not very good), and only listen to at least 196k MP3 (most are at least 256k or higher). Some are Apple Lossless (converted from FLAC). And throw in a few discographies in there and all of a sudden you have 350GB. I've discovered that I add about 50GB/6 months.
I had no idea Backblaze had gotten this popular, but I'm glad to see their business growing. If you happen to have the profile they've optimized for, it really is the best thing going.
I love #3, especially the Stay Puft Marshmallow Man!
Yeah, I think Stay Puft should be mandatory in all big-stack-of-stuff infographics.
Is there any reason why 10 PB was split in that particular combination of 2, 1.5 and 1TB drives?
EDIT: And, just a comparison: Google processes over 20PB of data per day!
EDIT: And, just a comparison: Google processes over 20PB of data per day!
It looks like that is their actual inventory of drives. At that scale, when 2TB drives become the most economical to buy that does not mean one immediately dives in and replaces all the 1.5TB drives. You have to do a bit more math than that.
It's also hard to tell whether they're actively decommissioning their 1TB drives or if they've just on a growth curve and they've done significantly more business while 1.5TB and 2TB drives were the more economical choice, but sooner or later the cost/benefit curves do cross and you do want to start decommissioning the smaller drives, it's only a matter of time.
It's also hard to tell whether they're actively decommissioning their 1TB drives or if they've just on a growth curve and they've done significantly more business while 1.5TB and 2TB drives were the more economical choice, but sooner or later the cost/benefit curves do cross and you do want to start decommissioning the smaller drives, it's only a matter of time.
There's a big difference between "storage" and "processing". But it's true, Google stores plenty of data too.
and why it takes more 2TB drives than 1TB drives...
That's the drives they actually have.
2280 * 2 + 3166 * 1.5 + 1 * 749 = 10,058 TB.
2280 * 2 + 3166 * 1.5 + 1 * 749 = 10,058 TB.
I am not american, nor do we use inches here in germany but when looking at #3 it should be noted that 5.75 inches is actually the drive length, not height as they state. Well depends on how you look at it but it confused me in the beginning.
I'll give you a much more compact way to envision 10 Pb:
10 Petabytes is 10,000,000,000,000,000 bytes or 80,000,000,000,000,000 bits, divided by 8,000,000,000 bits per full human genome (2 bits per base-pair) that's about 10,000,000 cells (not red blood cells because they don't contain DNA), or about 5 milliliters! (10 um diameter on average so about 500 cubic um, so 2 million or so per ml), and that includes all the stuff besides the DNA in the cells.
Hard disk storage certainly is impressively compact but it still has a long way to go before we beat mother nature.
-1 already eh? Downmodders please correct my math or say what you think is wrong with the comparison, I note the article ends on: "How would you visualize ten petabytes of storage?".
10 Petabytes is 10,000,000,000,000,000 bytes or 80,000,000,000,000,000 bits, divided by 8,000,000,000 bits per full human genome (2 bits per base-pair) that's about 10,000,000 cells (not red blood cells because they don't contain DNA), or about 5 milliliters! (10 um diameter on average so about 500 cubic um, so 2 million or so per ml), and that includes all the stuff besides the DNA in the cells.
Hard disk storage certainly is impressively compact but it still has a long way to go before we beat mother nature.
-1 already eh? Downmodders please correct my math or say what you think is wrong with the comparison, I note the article ends on: "How would you visualize ten petabytes of storage?".
I see many debatable points:
- Blood is far from being that dense, but I'll assume you mean to calculate a theoretical limit.
- If you use the whole of a cell's DNA to encode information, the cell will die. It can't be used to freely encode information like a hard disk.
- There is no way to retrieve information in this system. This is a bit like packing a large amount of extremely high-density magnetic platters in a box and calling this a storage system, which is very different from using actual hard drives with all the complex reading/writing system (heads, space between the platters, magnets...). Even if they're not connected, it still takes room.
- Reliability of hard disks is far, far better than this. Encoding information as a single copy in a single cell is wildly unsafe. Even using multiple copies, it's likely to degrade with mutations.
Edit: I was under the impression that a base pair could only encode one bit of information, since only T-C and A-G are valid combinations. Wikipedia appears to disagree with me but I see no source; does anyone know why Wiki says a base pair could encode 2 bits?
- Blood is far from being that dense, but I'll assume you mean to calculate a theoretical limit.
- If you use the whole of a cell's DNA to encode information, the cell will die. It can't be used to freely encode information like a hard disk.
- There is no way to retrieve information in this system. This is a bit like packing a large amount of extremely high-density magnetic platters in a box and calling this a storage system, which is very different from using actual hard drives with all the complex reading/writing system (heads, space between the platters, magnets...). Even if they're not connected, it still takes room.
- Reliability of hard disks is far, far better than this. Encoding information as a single copy in a single cell is wildly unsafe. Even using multiple copies, it's likely to degrade with mutations.
Edit: I was under the impression that a base pair could only encode one bit of information, since only T-C and A-G are valid combinations. Wikipedia appears to disagree with me but I see no source; does anyone know why Wiki says a base pair could encode 2 bits?
> - Blood is far from being that dense, but I'll assume you mean to calculate a theoretical limit.
There are more ways of having cells arranged, I explicitly mentioned red blood cells because they are exceptional in not containing any DNA, but any chunk of tissue with that volume would do.
> If you use the whole of a cell's DNA to encode information, the cell will die. It can't be used to freely encode information like a hard disk.
Yes, that's obvious, but a cell's DNA does hold that much information, it's just not our information.
> - There is no way to retrieve information in this system.
There actually is, the information retrieval mechanism that is used to 'express' the DNA (actually, the RNA, an 'unzipped' strand of DNA, but who's counting) is a wonderful little nano machine called a ribosome, it's probably the most amazing structure that I know of outside of the DNA itself.
They're in the volume quoted, the DNA only occupies about 25% of that volume iirc.
> Reliability of hard disks is far, far better than this.
The error correction mechanism that allows your cells to be copied through very large numbers of generations is actually pretty good, most 'mutations' are lethal and only very few actually result in viable copies passing their changes on to newer generations. Mutations are also pretty rare on the whole.
You are right that only TC and AG are valid, but those combinations can be attached 'in reverse' as well (CT / GA) so that makes for four possible combinations in all.
If it weren't for that the movie 'GATTACA' would have been unpronouncable :)
There are more ways of having cells arranged, I explicitly mentioned red blood cells because they are exceptional in not containing any DNA, but any chunk of tissue with that volume would do.
> If you use the whole of a cell's DNA to encode information, the cell will die. It can't be used to freely encode information like a hard disk.
Yes, that's obvious, but a cell's DNA does hold that much information, it's just not our information.
> - There is no way to retrieve information in this system.
There actually is, the information retrieval mechanism that is used to 'express' the DNA (actually, the RNA, an 'unzipped' strand of DNA, but who's counting) is a wonderful little nano machine called a ribosome, it's probably the most amazing structure that I know of outside of the DNA itself.
They're in the volume quoted, the DNA only occupies about 25% of that volume iirc.
> Reliability of hard disks is far, far better than this.
The error correction mechanism that allows your cells to be copied through very large numbers of generations is actually pretty good, most 'mutations' are lethal and only very few actually result in viable copies passing their changes on to newer generations. Mutations are also pretty rare on the whole.
You are right that only TC and AG are valid, but those combinations can be attached 'in reverse' as well (CT / GA) so that makes for four possible combinations in all.
If it weren't for that the movie 'GATTACA' would have been unpronouncable :)
> There actually is, the information retrieval mechanism that is used to 'express' the DNA (actually, the RNA, an 'unzipped' strand of DNA, but who's counting) is a wonderful little nano machine called a ribosome, it's probably the most amazing structure that I know of outside of the DNA itself.
Actually I was going to say that one way to think of using that information is to compute with it, in other words, an organism is simply the result of a computation on its DNA.
Actually I was going to say that one way to think of using that information is to compute with it, in other words, an organism is simply the result of a computation on its DNA.
- There is no way to retrieve information in this system.
DNA is natively a content addressable storage system, due to natural base-pairing. But to first address your question in your edit: think of DNA as a pair of singly-linked lists, each with an alphabet of four characters. Each singly-linked list is the "reverse complement" of the other: the reverse sequence with an A-T swap and a C-G swap. At each position there's 2 bits of information, and the other linked list allows for some redundancy.
To probe for information in a DNA database, you construct the reverse-complement of the desired bit of information, attach a marker to your probe (such as a fluorescent dye, biotin, magnetic bead), then physically mix it in to your DNA database. A couple cycles of melting and cooling, and your probe will eventually find it's target DNA.
Of course, the thermodynamics of a physical database like this aren't particularly great. I'm not sure of the asymptotic behavior; my intuitive guess is lg(N) just like in B-trees or what have you, but I've never run the numbers or heard of anyone else running it. Also, the constant in front may be just a few orders of magnitude larger than our current systems :)
Reading DNA is getting super cheap these days, and the pace of DNA sequencing technology makes Moore's law look positively wimpy. There are about 30 serious startups working on technologies that fall into a few broad categories, and some like PacBio had an IPO this year. Writing DNA is a much more difficult challenge, I don't know of many people looking into it yet. The market for writing DNA isn't nearly as obvious as it is for sequencing. Of course if it becomes feasible to write your own pets/plants/children instead of breeding them, the market may explode.
DNA is natively a content addressable storage system, due to natural base-pairing. But to first address your question in your edit: think of DNA as a pair of singly-linked lists, each with an alphabet of four characters. Each singly-linked list is the "reverse complement" of the other: the reverse sequence with an A-T swap and a C-G swap. At each position there's 2 bits of information, and the other linked list allows for some redundancy.
To probe for information in a DNA database, you construct the reverse-complement of the desired bit of information, attach a marker to your probe (such as a fluorescent dye, biotin, magnetic bead), then physically mix it in to your DNA database. A couple cycles of melting and cooling, and your probe will eventually find it's target DNA.
Of course, the thermodynamics of a physical database like this aren't particularly great. I'm not sure of the asymptotic behavior; my intuitive guess is lg(N) just like in B-trees or what have you, but I've never run the numbers or heard of anyone else running it. Also, the constant in front may be just a few orders of magnitude larger than our current systems :)
Reading DNA is getting super cheap these days, and the pace of DNA sequencing technology makes Moore's law look positively wimpy. There are about 30 serious startups working on technologies that fall into a few broad categories, and some like PacBio had an IPO this year. Writing DNA is a much more difficult challenge, I don't know of many people looking into it yet. The market for writing DNA isn't nearly as obvious as it is for sequencing. Of course if it becomes feasible to write your own pets/plants/children instead of breeding them, the market may explode.
There are several companies that will "write" arbitrary sequences of DNA, to order. One is http://www.mrgene.com , but at $0.39/bp it is still much more economical to isolate and amplify desired sequences using PCR.
> To probe for information in a DNA database, you construct the reverse-complement of the desired bit of information
To build the reverse-complement of the information, don't you need to have the information in the first place?
I know it's possible to store information in DNA through various means, but I don't believe it can be done at the density the OP calculated. If we're going to take into account only information storage while ignoring retrieval considerations, then we shouldn't compare naked cells with no DNA duplication to reliable whole hard drives.
Call it human pride, but I think we've beaten mother nature in several aspects ;)
To build the reverse-complement of the information, don't you need to have the information in the first place?
I know it's possible to store information in DNA through various means, but I don't believe it can be done at the density the OP calculated. If we're going to take into account only information storage while ignoring retrieval considerations, then we shouldn't compare naked cells with no DNA duplication to reliable whole hard drives.
Call it human pride, but I think we've beaten mother nature in several aspects ;)
Think of it as a massive key-value store: you construct your query off the key, use that to pull out the key-value pair, and when you sequence your key you continue to sequence more in order to pull out the value. If you prefer sequential addresses, your key could be just that.
And actually, this could be done at a much higher density than what the original poster described, as he's counting the full cell in the density calculation, and DNA is only a small fraction of the cellular volume. You could duplicate all the DNA 10-100 times in the same amount of space once you take out all the ribosomes, proteins and extra water. And as long as it's not stored in direct sunlight or next to your pile of plutonium, DNA is going to be much much more stable than aligning magnetic fields. We're still getting good DNA sequence out of bones that are tens of thousands of years old.
When you think of nanotechnology and miniaturization, think of biology, because that's where all the real nanotechnology is going on. We've not done any better than nature when it comes to making small machinery. Nature has already invented the commodity interchangeable parts (amino acids and nucleic acids) that can self-assemble into rather fantastic machines.
However, we have beaten mother nature on latency: as I alluded to, a DNA database like this would have latency on the order of days for a lookup. On the other hand, as much parallel access as you can imagine is built in, without additional volume. And this isn't a system that has been engineered at all, I'm just talking about the fundamental properties of a little puddle of DNA and water. If half the engineering that went into modern computer hardware were put into a DNA database, it could be quite competitive with our electronic systems.
And actually, this could be done at a much higher density than what the original poster described, as he's counting the full cell in the density calculation, and DNA is only a small fraction of the cellular volume. You could duplicate all the DNA 10-100 times in the same amount of space once you take out all the ribosomes, proteins and extra water. And as long as it's not stored in direct sunlight or next to your pile of plutonium, DNA is going to be much much more stable than aligning magnetic fields. We're still getting good DNA sequence out of bones that are tens of thousands of years old.
When you think of nanotechnology and miniaturization, think of biology, because that's where all the real nanotechnology is going on. We've not done any better than nature when it comes to making small machinery. Nature has already invented the commodity interchangeable parts (amino acids and nucleic acids) that can self-assemble into rather fantastic machines.
However, we have beaten mother nature on latency: as I alluded to, a DNA database like this would have latency on the order of days for a lookup. On the other hand, as much parallel access as you can imagine is built in, without additional volume. And this isn't a system that has been engineered at all, I'm just talking about the fundamental properties of a little puddle of DNA and water. If half the engineering that went into modern computer hardware were put into a DNA database, it could be quite competitive with our electronic systems.
Sure, a key-value store would work. My point is that the OP's system is not such a store. He just stores 10 PBs of raw data with no indexing and no duplication, so there is no way to retrieve data and comparison with hard disks is meaningless. My post was an answer to his "please correct my math".
The reason one base pair can encode 2 bits is that T-C and C-T are different valid base pairs. The same is true for A-G and G-A as well. A "single" strand (the famous double helix) of DNA actually contains 2 copies of genetic information.
If I had to guess, it's because the way you read DNA is only along one of the strands (starting at the 5' end, IIRC), in which case you've still got four possible "characters" to use in your representation- A-G is different from G-A, etc. People have thought up all sorts of clever ways to think of how one might encode data using DNA base pairs, but it's been too long since I did any bioinformatics for me to be able to remember exactly how they calculated it all out.
Re: Number of bits per pair: TC, AG, CT and GA? I.e. composition (1 bit) and direction (1 bit)? Not sure, but that seems to be the most logical way.
We can already write individual atoms on a surface with a atomic force microscope.
So at one bit per atom position we could encode 10Pb in about 0.1micromoles of xenon atoms on a silicon substrate.
Well under what nature can do.
Well under what nature can do.
the 'on a silicon substrate' is a bit problematic here, a substrate typically is a lot larger than just a few atoms, the neat thing about DNA is that it is a three dimensional encoding, you are talking about a two dimensional encoding scheme, not three dimensional (an atomic force microscope typically moves in two dimensions and uses an electrical charge to deposit the atoms).
You could stack the atoms three dimensionally but then there would be no easy way to read them out again.
You could stack the atoms three dimensionally but then there would be no easy way to read them out again.
First off, I didn't downvote you.
However, the comparison is wrong imho because disk storage is not simply about storing information. Disks also need fast read/write access, which DNA certainly doesn't have for us (for now?).
We are envisioning storage, a bunch of disconnected drives piled on top of each other is also not accessible for read/write access.
[deleted]
-1 already eh? [..]
If you took all of those drives apart and put the spindles in a pile you'd have a big pile of motors. The Petabox consumes 6kW so let's be generous and call it 60kW of servo motors.
60kW is roughly 80hp. (I know this is a specious comparison in several ways but bear with me.) 80hp is enough power a small but highway-capable motorcycle. A single 60kW brushless DC motor weighs over 100kg. That's heavier than a good many of us HN readers.
Imagine a motor like that bolted to one long spindle of large platters--say the 99cm platters from a 1961 hard drive.
If these platters were remade with the data density of modern 2TB drives, how tall would the spindle be if this giant hard drive had a 10PB capacity? How over- or under-sized would our motor be?
60kW is roughly 80hp. (I know this is a specious comparison in several ways but bear with me.) 80hp is enough power a small but highway-capable motorcycle. A single 60kW brushless DC motor weighs over 100kg. That's heavier than a good many of us HN readers.
Imagine a motor like that bolted to one long spindle of large platters--say the 99cm platters from a 1961 hard drive.
If these platters were remade with the data density of modern 2TB drives, how tall would the spindle be if this giant hard drive had a 10PB capacity? How over- or under-sized would our motor be?
it'd be rad if you could visualize the ratios to better illustrate the scale.
10 petabytes : 10000 terabytes :: 10000 gigabytes : 10 terabytes
10 petabytes : 10000 terabytes :: 10000 gigabytes : 10 terabytes
The visualization in storage pods is most helpful. Just put the drives, in their current size in a physical space you can compare to your office.
The skyscraper comparison is fun but misleading. The stack of drives would be very very thin. It would work better if drives were put in one box and that box was compared to a room, house, etc..
There can be other visualizations like electrical power consumption compared to a normal computer. How much time it would take to read/write that data. How many processors are needed to read/write that data in reasonable time, compare their surface area to the our unfolded brain, which is a large cloth napkin.
I think these comparisons would be more useful to people in the field than how many books would be needed to store that much data, the standard pop show example.
The skyscraper comparison is fun but misleading. The stack of drives would be very very thin. It would work better if drives were put in one box and that box was compared to a room, house, etc..
There can be other visualizations like electrical power consumption compared to a normal computer. How much time it would take to read/write that data. How many processors are needed to read/write that data in reasonable time, compare their surface area to the our unfolded brain, which is a large cloth napkin.
I think these comparisons would be more useful to people in the field than how many books would be needed to store that much data, the standard pop show example.
Anyone want to guess how much data they actually have stored?
Backblaze's pods are a data-loss nightmare -- lots of single points of failure which will wipe out many TB of data at a time -- and backblaze has stated that they replicate data across multiple pods. Given that the 10 PB seems to be the amount of raw storage backblaze has, I'm guessing that the amount of actual data stored is much less -- depending on what sort of erasure correction scheme they're using, of course. (They're still much bigger than Tarsnap, of course!)
Backblaze's pods are a data-loss nightmare -- lots of single points of failure which will wipe out many TB of data at a time -- and backblaze has stated that they replicate data across multiple pods. Given that the 10 PB seems to be the amount of raw storage backblaze has, I'm guessing that the amount of actual data stored is much less -- depending on what sort of erasure correction scheme they're using, of course. (They're still much bigger than Tarsnap, of course!)
At SpiderOak we get a 3x replication equivalent for about 35% overhead, using Reed-Solomon at the cluster level (on top of RAID6 at the machine level.) Not nearly as expensive as outright replication.
Agree those SATA port multiplies are worrisome. In the beginning, our prototype machines used them to squeeze as many drives into a single machine as possible. They have unusually low tolerance for electrical interference and make it possible for one badly malfunctioning drive to take an entire array offline until manually serviced. We've seen occasions where just touching a cable attached to a port multiplier caused the Linux kernel to emit "dazed and confused" NMI events. I am not brave enough to try them again, even in a redundant setup.
Agree those SATA port multiplies are worrisome. In the beginning, our prototype machines used them to squeeze as many drives into a single machine as possible. They have unusually low tolerance for electrical interference and make it possible for one badly malfunctioning drive to take an entire array offline until manually serviced. We've seen occasions where just touching a cable attached to a port multiplier caused the Linux kernel to emit "dazed and confused" NMI events. I am not brave enough to try them again, even in a redundant setup.
3x replication equivalent for about 35% overhead
How did you compute this "replication equivalent"?
How did you compute this "replication equivalent"?
Picked a number from thin air? Raid six requires 2 drives for back and is normally used in set's of 8 or 16 drives but looks like they are using 45 drives. So 45/43 = 4.65% overhead from using RAID.
Not if they lose 35% on top of that they are around 41% overhead. But, they are taking a huge it on write speeds, network traffic and reliability for doing so.
Edit: Looks like they have 10,058 TB before partitioning the drives so my guess is ~3-6TB of actual user data.
Not if they lose 35% on top of that they are around 41% overhead. But, they are taking a huge it on write speeds, network traffic and reliability for doing so.
Edit: Looks like they have 10,058 TB before partitioning the drives so my guess is ~3-6TB of actual user data.
Does SpiderOak only provide backup service? Erasure encoding is efficient for cold data. Do you use erasure encoding to distribute the hot data across clusters?
Amazing, a discussion of data without comparing the internationally accepted unit of "library of congress". I'm more interested in how many LoCs 10 Pb is? If we built a city of LoCs, how many acres would that city be?
These are the questions that keep me up late at night.
These are the questions that keep me up late at night.
That's a sizable operation and difficult to visualize. I'm guessing their failure points reduce potential damage to minimal levels. At AltDrive, we use ZFS RaidZ2 with a number of hot spares. Six drives would have to fail consecutively on a given machine before any data loss... we replace them as they occur. And we have inter-H/W duplication. ZFS is self healing and makes management easy. Additionally we periodically compare the integrity of the user's AES-256 CTR encrypted data against the database records and make corrections if necessary. From our customer's and white label provider's view, it just works.
http://altdrive.com
Or about 9 (American) football fields covered with 4GB DVDs: https://sites.google.com/site/10petabytes/viz