Facebook's new optical storage solution [video](facebook.com)
facebook.com
Facebook's new optical storage solution [video]
https://www.facebook.com/photo.php?v=10152128660097200
17 comments
Another benefit of this system is that the storage media is immune electromagnetic pulse. So, even in the case of a large-scale EMP event our cat pictures and network graphs will remain intact and recoverable. :)
True, but it's easy to build a storage container that doubles as a faraday cage when the tapes are in long-term storage.
Actually, I'd be very surprised if this wasn't the case (pun intended).
Actually, I'd be very surprised if this wasn't the case (pun intended).
I assume SSDs are screwed by EMPs, but how does magnetic media fare (the tape/disk media itself, not reader/writer electronics)?
Wiped.
Can anyone knowledgeable explain why they use Blu-Ray instead of tape backup? When would an organization use Blu-Ray, and when wouldn't then? Is this reflective of any kind of general trend?
I'm not knowledgeable, but some total guesses:
1) As blu-ray is a consumer product its probably manufactured in greater quantities than backup grade tapes, and therefore can be had lower in price, and there are more vendors to source them from. Similarly the drives themselves are cheaper.
2) Tapes aren't random access, therefore it's slow and cumbersome to retrieve single files.
1) As blu-ray is a consumer product its probably manufactured in greater quantities than backup grade tapes, and therefore can be had lower in price, and there are more vendors to source them from. Similarly the drives themselves are cheaper.
2) Tapes aren't random access, therefore it's slow and cumbersome to retrieve single files.
Re: 1) Looking at the prices on Newegg, 1.5TB(native) LTO5 tapes run about $30.00, so $20 per TB. Write-once BluRay looks to be about $32/TB, and rewriteable is $80/TB.
Re: 2) Although it would take an hour or so to search through an entire tape if all the files were written in a single session, normally backup software writes files in sessions containing a few GB. You can seek to a specific session in about 30 seconds average. So if you put about 4GB per session, it would take a max of approx. 30 seconds to read in that session to retrieve a file. So figure about a minute to retrieve any file. Also, LTO5 transfers about 140MB/sec, vs. 6x BDR disks (what I'm seeing for cheap right now) is only 27MB/sec.
Re: 2) Although it would take an hour or so to search through an entire tape if all the files were written in a single session, normally backup software writes files in sessions containing a few GB. You can seek to a specific session in about 30 seconds average. So if you put about 4GB per session, it would take a max of approx. 30 seconds to read in that session to retrieve a file. So figure about a minute to retrieve any file. Also, LTO5 transfers about 140MB/sec, vs. 6x BDR disks (what I'm seeing for cheap right now) is only 27MB/sec.
3) IIRC most tape storage technolgies are closed source. FB has been heavily pushing their open compute project. Using BDs get them past a lot of licensing problems.
LTO is an open standard: http://en.wikipedia.org/wiki/Linear_Tape_Open
Not knowledgeable either, but one difference is durability. LTO is rated 15-30 years while the video notes the blu-ray discs FB uses are "certified for 50 years of operation" and some go up to 1,000(!).
http://en.wikipedia.org/wiki/Linear_Tape_Open#Tape_durabilit...
http://en.wikipedia.org/wiki/Linear_Tape_Open#Tape_durabilit...
Any media is as durable as the availability of media reader. I seriously doubt, in 15-20 or 50 years, you will have the media reader available to read either of the media, whether tape or BD.
How many 8" or 5.25" floppy disk drives are around today and usable with current server hardware? Anyone who is considering cold storage on media for more than 10 years lifespan is fooling themselves.
A few years ago I had a customer who had to perform data migration from DLT to LTO-3. That was over 24 month project and customer begging the vendor to provide a few old DLT drives to read. On top of that the provided DLT drives were so finicky communicating with modern server hardware that they barely kept up for a few hours before going offline.
How many 8" or 5.25" floppy disk drives are around today and usable with current server hardware? Anyone who is considering cold storage on media for more than 10 years lifespan is fooling themselves.
A few years ago I had a customer who had to perform data migration from DLT to LTO-3. That was over 24 month project and customer begging the vendor to provide a few old DLT drives to read. On top of that the provided DLT drives were so finicky communicating with modern server hardware that they barely kept up for a few hours before going offline.
> I seriously doubt, in 15-20 or 50 years, you will have the media reader available to read either of the media, whether tape or BD.
Today's Blu-Ray drives are perfectly capable of playing an audio CD from the 1980s, and compatible players probably will continue to be available for another 10-20 years even after everyone stops buying content on physical media.
I wouldn't be surprised if I can plug in my USB 3.0 CD/DVD/Blu-Ray combo drive into a USB 7.5 port in 2030 and use it to play an audio CD from 1980, provided that the disc is in good condition. That's a lifespan of 50 years!
Even in the digital age, well-designed and highly forwards-compatible forms of physical media can last a long time.
Today's Blu-Ray drives are perfectly capable of playing an audio CD from the 1980s, and compatible players probably will continue to be available for another 10-20 years even after everyone stops buying content on physical media.
I wouldn't be surprised if I can plug in my USB 3.0 CD/DVD/Blu-Ray combo drive into a USB 7.5 port in 2030 and use it to play an audio CD from 1980, provided that the disc is in good condition. That's a lifespan of 50 years!
Even in the digital age, well-designed and highly forwards-compatible forms of physical media can last a long time.
You're completely right that the media solution will be gone before ten years pass.
That said, I'd prefer to have as much headroom possible, if data is to be stored reliably.
5 years before migrating, with a 15-20 year window vs 5 years before migrating, with a 50-100 year window
Probably a better recovery rate, at their scale.
That said, I'd prefer to have as much headroom possible, if data is to be stored reliably.
5 years before migrating, with a 15-20 year window vs 5 years before migrating, with a 50-100 year window
Probably a better recovery rate, at their scale.
15 years isn't that long of a time. I have CDs from much earlier than 1999 for sure, and the media reader is sitting right in my living room. And I'm not even running a data center.
The cost of cartridges are comparable to optical disks. However, LTO drives are non-trivial in terms of both mechanical complexity and cost. I can toss an optical tray into the recycle bin without batting an eye. Not true with LTO, and I suspect their MTBF is similar (trade the quality of the LTO drive for the mass production and cheapness of the optical tray).
Enemy #1 in a large data center is power. When he mentioned the entire rack pulls 1,000W for the density suspected in this thread (~500TB to ~1PB), that piqued my interest. Even if you can't push 2-3PB per rack, the top stat you're crunching forever and always is watt-per-gigabyte. Space is important but if it breaks into the top spot you've done something else wrong along the way.
Enemy #27 for cold(ish) storage might be, say, pressure per sq. in. on your floor. I'd be interested to see a weight comparison of a fully-loaded LTO/cartridge rack and this design.
Perhaps not what you were looking for in an answer, but I'm interested to look over the schematics.
Enemy #1 in a large data center is power. When he mentioned the entire rack pulls 1,000W for the density suspected in this thread (~500TB to ~1PB), that piqued my interest. Even if you can't push 2-3PB per rack, the top stat you're crunching forever and always is watt-per-gigabyte. Space is important but if it breaks into the top spot you've done something else wrong along the way.
Enemy #27 for cold(ish) storage might be, say, pressure per sq. in. on your floor. I'd be interested to see a weight comparison of a fully-loaded LTO/cartridge rack and this design.
Perhaps not what you were looking for in an answer, but I'm interested to look over the schematics.
Contrary to the other two commenters, my guess was going to be that they are optimizing TB/m^3.
My hopes were dashed when I realized that this wasn't a new type of optical hard disc, like holographic drive technology or something.
Maybe in a couple years: http://rt.com/news/5d-nanostructured-glass-optical-memory-94...
With Big Data Everything, we really need an Optical Storage System like Blu Ray but with 100x to 1000x the capacity.
No comment on the access time and having only one drive means reading only one disc at once. This puts this storage solution in the realm of "I don't need this data within an hour" so I presume the application this system acts as redundancy for failed disks.
The guy was very explicit that this is a cold storage solution. Probably "backups of backups."
The Facebook open computer definition for "cold storage":
"where data is stored on disk but almost never read again, like legal data or backups of third copies of data — keep on increasing dramatically, there are huge demands for developing some form of cold storage system with the highest capacity and the lowest cost."
source: http://www.opencompute.org/projects/storage/
"where data is stored on disk but almost never read again, like legal data or backups of third copies of data — keep on increasing dramatically, there are huge demands for developing some form of cold storage system with the highest capacity and the lowest cost."
source: http://www.opencompute.org/projects/storage/
Are these Blu-ray disks rewritable? If so, wouldn't this be a great tech for Amazon's Glacier service? I believe they currently depend of costly tape drives to store data.
I believe Glacier is an all-hard-drive solution. Tapes are troublesome because they're not "online" so it's harder to verify data integrity and notice failures.
The drives are spun much slower than a regular drive (I've heard 300rpm) and so they can be crammed into much higher density racks without worrying about heat.
Maybe data is eventually moved to tapes for Glacier; I am privy to no special information.
The drives are spun much slower than a regular drive (I've heard 300rpm) and so they can be crammed into much higher density racks without worrying about heat.
Maybe data is eventually moved to tapes for Glacier; I am privy to no special information.
Each LTO-6 tape cartridge stores 6TB of data. Each quadruple layer Blue-ray disk stores only 128GB. I am surprised that FB went with Optical Disk instead of Tape.
Optical jukeboxes/libraries for MO/DVD/CD used to be popular a decade or so ago and now considered antique. There is just not enough density with optical.
Optical jukeboxes/libraries for MO/DVD/CD used to be popular a decade or so ago and now considered antique. There is just not enough density with optical.
My guess:
1) Lower cost for drives (commodity drives).
2) The ability to (easily) seek to an arbitrary location on disk
3) Cost of BD burners are <$100; Cost per GB of disks is $0.025/GB. LTO-6 drives are $1000+, and cost per GB is around $0.012/GB. -- I think the cost works out actually, especially as the cost of blu-ray is still falling.
[deleted]
how reliable are these tapes?
Why bother with rewriteable disks? This is cold storage meaning that they will likely never throw this data away. I'd suspect Amazon's Glacier runs on something similar.
>Why bother with rewriteable disks?
Its okay for Facebook and other single entities for backups. However for Glacier, it would mean throwing away the disk whenever someone deletes their data. It would also means constant replacement of disks.
Its okay for Facebook and other single entities for backups. However for Glacier, it would mean throwing away the disk whenever someone deletes their data. It would also means constant replacement of disks.
> However for Glacier, it would mean throwing away the disk whenever someone deletes their data.
Probably not a huge concern, given the use case. Glacier is pretty fire-and-forget.
> It would also means constant replacement of disks.
They're already doing that with servers, why would disks be a problem?
Probably not a huge concern, given the use case. Glacier is pretty fire-and-forget.
> It would also means constant replacement of disks.
They're already doing that with servers, why would disks be a problem?
Do rewritable blurays last as long as write-once discs?
Does anyone know what data is Facebook trying to archive? Old user posts? But that doesn't fit this usecase because of the high latency.
Or maybe its just a nice content delivery mechanism to the NSA. Just ship a bunch of blue-rays instead of pesky optical fiber interception.
Or maybe its just a nice content delivery mechanism to the NSA. Just ship a bunch of blue-rays instead of pesky optical fiber interception.
FB is required by law to store all its credit card transactions that every singal FB ads account has for 6 years. (So are other US based companies that process credit cards)
Surely, credit-card info can't span 100s of terabytes, even if they record each and every transaction.
backups of images/videos/posts/everything
Facebook has built a novel implementation, but this isn't a new idea or product. Similar to how they used to buy servers and now build them to spec, they probably used to buy these and now design and build them.
http://www.disc-group.com/products/bd-series-benefits/
http://www.disc-group.com/products/bd-series-benefits/
After I saw this on James Hamilton's blog the other day I did a quick check of BD-R versus 4TB HDDs cost per GB. Using whatever I could find after about 5 minutes of looking around, I came up with $0.05/GB for both types of media.
If anyone can provide a link to BD media for sale at a price that make this economical I'd be interested to know.
If anyone can provide a link to BD media for sale at a price that make this economical I'd be interested to know.
I would think the media's longevity factors into the lifetime cost. If the optical media is certified for 50+ years, that essentially means they won't have a need to recopy data if for some reason they want to preserve it.. for a long time.. I don't know how long magnetic tape lasts, but I would imagine it would be a huge pain to need to keep backing up old tapes to new tapes every X years.
Any guess on the total storage capacity in that rack and comparison to a similar sized magnetic storage?
I talked to them at the Open Compute Summit and got the chance to look at it up close. They quoted me 100GB/disc and 10,000 discs in the cabinet for 1PB. They could have been rounding those numbers for simplicity, but that gets you in the neighborhood.
From the video: 24 magazines/rack * 36 cartridges/magazine * 12 discs/cartridge * 50 GB/disc = 518TB/rack. Based standard dual-layer discs. I cannot comment on storage of our traditional spinning-disk racks.
Backblaze Pods are about 180TB, so 1800TB per rack. And a whole lot of power consumption.
A comment on the video suggested 950 TB per rack.
> 1 Rack = 44 magazines; 1 magazine = 36 cartridges; 1 cartridge = 12 blueray discs. So, ~950TB.
> 1 Rack = 44 magazines; 1 magazine = 36 cartridges; 1 cartridge = 12 blueray discs. So, ~950TB.
If they use 128 GB Blu-Ray disks its about 1.26 PB per rack.
24 mag -> 36 cart -> 12 disks
10,368 Disks in total per rack
24 mag -> 36 cart -> 12 disks
10,368 Disks in total per rack
I was expecting some kind of optical fiber storage with reflecting crystals and shit.
This was a bit disappointing.
This was a bit disappointing.
I might be interested in cloud backup service which would backup my "keep forever" stuff, put them to optical disc and then mail the discs to me when they fill up.
Could be also an add-on service to existing online backup business.
Could be also an add-on service to existing online backup business.
What about bit rot though? Some of my dvdr are no longer readable after 5 yrs, stored in very good conditions. Hdd backup, esp. with Zfs can provide very good bit rot protection.
These are high grade guaranteed for 30+ years. There's also 1000 year grade blu rays out there.
I had failures of both "archival" quality DVDs and SD cards before, just after a year or two after last backup.
Do you have any links on the methodology used to measure the 30+ or 1000yr grade for a blu-ray ?
Do you have any links on the methodology used to measure the 30+ or 1000yr grade for a blu-ray ?
Blu-rays are but a few years old. It's preposterous for any one to claim that they are "guaranteed" for 30 years, let alone 1,000 years. Not to mention that that guarantee most likely has an asterisk that limits damages to the cost of the original blu-ray disk in case it actually fails.
FB: How can you get a consumer version of this so that you don't have to use Amazon Glacier for personal data archival?!
I did not know that they made disks that are certified for long term storage like that.
At 1:55 he says "FRID'd", he meant RFID'd, right?
Still magnetic storage however you read/write the data...
Flash required, so it won't play here.
Really, Facebook? It's 2014.
Really, Facebook? It's 2014.