Amazon Glacier(aws.amazon.com)
aws.amazon.com
Amazon Glacier
http://aws.amazon.com/glacier/
http://aws.amazon.com/glacier/
378 comments
I'm a long time user of backblaze, and I'm a big fan of the product - it does a great job of always making sure my working documents are backed up, particularly when I'm traveling overseas, and my laptop is more vulnerable to theft or damage.
With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,vud,vmdk,vmsn,vmsd,hdd,vdi,vmwarevm,nvram,vmx,vmem,iso,dmg,sparseimage,sys,cab,exe,msi,dll,dl_,wim,ost,o,log,m4v" files. They also don't want to backup your /applications, /library, /etc, and so on locations. They also make it clear that backing up a NAS is not the target case for their service.
I can live with that - because, honestly, it's $4/month, and my goal is to keep my working files backed up. System Image backups, I've been using Super Duper to a $50 external hard drive.
Glacier + a product like http://www.haystacksoftware.com/arq/ means I get the benefit of both worlds - Amazon will be fine with me dropping my entire 256 Gigabyte Drive onto Glacier (total cost - $2.56/month) and I get the benefit of off site backup.
The world is about to get a whole lot simpler (and inexpensive) for backups.
With that said - Backblaze is optimized for working documents - and the default "exclusion" list makes it clear they don't want to be backing up your "wab~,vmc,vhd,vo1,vo2,vsv,vud,vmdk,vmsn,vmsd,hdd,vdi,vmwarevm,nvram,vmx,vmem,iso,dmg,sparseimage,sys,cab,exe,msi,dll,dl_,wim,ost,o,log,m4v" files. They also don't want to backup your /applications, /library, /etc, and so on locations. They also make it clear that backing up a NAS is not the target case for their service.
I can live with that - because, honestly, it's $4/month, and my goal is to keep my working files backed up. System Image backups, I've been using Super Duper to a $50 external hard drive.
Glacier + a product like http://www.haystacksoftware.com/arq/ means I get the benefit of both worlds - Amazon will be fine with me dropping my entire 256 Gigabyte Drive onto Glacier (total cost - $2.56/month) and I get the benefit of off site backup.
The world is about to get a whole lot simpler (and inexpensive) for backups.
Storage experts: I'd love to know more about what might be backing this service.
What kind of system has Amazon most likely built that takes 3-4 hours to perform retrieval? What are some examples of similar systems, and where are they installed?
What kind of system has Amazon most likely built that takes 3-4 hours to perform retrieval? What are some examples of similar systems, and where are they installed?
Beware that retrieval fee!
The retrieval fee for 3TB could be as high as $22,082 based on my reading of their FAQ [1].
It's not clear to me how they calculate the hourly retrieval rate. Is it based on how fast you download the data once it's available, how much data you request divided by how long it takes them to retrieve it (3.5-4.5 hours), or the size of the archives you request for retrieval in a given hour?
This last case seems most plausible to me [6] -- that the retrieval rate is based solely on the rate of your requests.
In that case, the math would work as follows:
After uploading 3TB (3 * 2^40 bytes) as a single archive, your retrieval allowance would be 153.6 GB/mo (3TB * 5%), or 5.12 GB/day (3TB * 5% / 30). Assuming this one retrieval was the only retrieval of the day, and as it's a single archive you can't break it into smaller pieces, your billable peak hourly retrieval would be 3072 GB - 5.12 GB = 3066.88 GB.
Thus your retrieval fee would be 3066.88 * 720 * .01 = $22081.535 (719x your monthly storage fee).
That would be a wake-up call for someone just doing some testing.
--
[1] http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_wh...
[2] After paying that fee, you might be reminded of S4: http://www.supersimplestorageservice.com/
[3] How do you think this interacts with AWS Export? It seems that AWS Export would maximize your financial pain by making retrieval requests at an extraordinarily fast rate.
[(edit) 4] Once you make a retrieval request the data is only available for 24 hours. So even in the best case, that they charge you based on how long it takes you to download it (and you're careful to throttle accurately), the charge would be $920 ($0.2995/GB) -- that's the lower bound here. Which is better, of course, but I wouldn't rely on it until they clarify how they calculate. My calculations above represent an upper bound ("as high as"). Also note that they charge separately for bandwidth out of AWS ($368.52 in this case).
[(edit) 5] Answering an objection below, I looked at the docs and it doesn't appear that you can make a ranged retrieval request. It appears you have to grab an entire archive at once. You can make a ranged GET request, but that only helps if they charge based on the download rate and not based on the request rate.
[(edit) 6] I think charging this way is more plausible because they incur their cost during the retrieval regardless of whether or how fast you download the result during the 24 hour period it's available to you (retrieval is the dominant expense, not internal network bandwidth). As for the other alternative, charging based on how long it takes them to retrieve it would seem odd as you have no control over that.
The retrieval fee for 3TB could be as high as $22,082 based on my reading of their FAQ [1].
It's not clear to me how they calculate the hourly retrieval rate. Is it based on how fast you download the data once it's available, how much data you request divided by how long it takes them to retrieve it (3.5-4.5 hours), or the size of the archives you request for retrieval in a given hour?
This last case seems most plausible to me [6] -- that the retrieval rate is based solely on the rate of your requests.
In that case, the math would work as follows:
After uploading 3TB (3 * 2^40 bytes) as a single archive, your retrieval allowance would be 153.6 GB/mo (3TB * 5%), or 5.12 GB/day (3TB * 5% / 30). Assuming this one retrieval was the only retrieval of the day, and as it's a single archive you can't break it into smaller pieces, your billable peak hourly retrieval would be 3072 GB - 5.12 GB = 3066.88 GB.
Thus your retrieval fee would be 3066.88 * 720 * .01 = $22081.535 (719x your monthly storage fee).
That would be a wake-up call for someone just doing some testing.
--
[1] http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_wh...
[2] After paying that fee, you might be reminded of S4: http://www.supersimplestorageservice.com/
[3] How do you think this interacts with AWS Export? It seems that AWS Export would maximize your financial pain by making retrieval requests at an extraordinarily fast rate.
[(edit) 4] Once you make a retrieval request the data is only available for 24 hours. So even in the best case, that they charge you based on how long it takes you to download it (and you're careful to throttle accurately), the charge would be $920 ($0.2995/GB) -- that's the lower bound here. Which is better, of course, but I wouldn't rely on it until they clarify how they calculate. My calculations above represent an upper bound ("as high as"). Also note that they charge separately for bandwidth out of AWS ($368.52 in this case).
[(edit) 5] Answering an objection below, I looked at the docs and it doesn't appear that you can make a ranged retrieval request. It appears you have to grab an entire archive at once. You can make a ranged GET request, but that only helps if they charge based on the download rate and not based on the request rate.
[(edit) 6] I think charging this way is more plausible because they incur their cost during the retrieval regardless of whether or how fast you download the result during the 24 hour period it's available to you (retrieval is the dominant expense, not internal network bandwidth). As for the other alternative, charging based on how long it takes them to retrieve it would seem odd as you have no control over that.
I think this is more or less the formula for calculating monthly costs (corrections welcome):
0.01S+max(0,7.20*(R-0.0017S)/4)
S is number of GB stored
R is biggest retrieval in the month
4 is the average number of hours a retrieval
For an example with 10TB storage (replace 10000 to change): http://fooplot.com/plot/4pu7u2gpox
x is biggest retrieval in GB, y is $/month
0.01S+max(0,7.20*(R-0.0017S)/4)
S is number of GB stored
R is biggest retrieval in the month
4 is the average number of hours a retrieval
For an example with 10TB storage (replace 10000 to change): http://fooplot.com/plot/4pu7u2gpox
x is biggest retrieval in GB, y is $/month
Does anyone know if there is a CLI tool to interface with this yet? I see SDK's mentioned on the product homepage but I dont see any simple CLI tools for this yet to upload/download data and query etc.
http://aws.amazon.com/developertools/
http://aws.amazon.com/developertools/
> Amazon Glacier is designed to provide average annual durability of 99.999999999% for an archive.
That's pretty impressive. I wonder how many bytes they lost to lose that .0000000001% of data.
That's pretty impressive. I wonder how many bytes they lost to lose that .0000000001% of data.
I just saw this and thought wow finally cheap mass storage. After reading the comments (and the Amazon Glacier web page) it's clear it's cheap archiving but not cheap retrieval.
I wonder if homeowner's insurance would cover the retrieval fee if your computer/hard drives were protected assets on the policy.
Worst. name. ever.
I get a 404, and when I go to http://aws.amazon.com/ I see nothing about glacier in the news feed.
Great. Now if someone would simply create a badass client for this for Mac (like Backblaze or Mozy) … we'd be in business. :)
This is fantastic. I've long searched for a solution like that. This is really suitable for a remote backup that only needs to be accessed if something really bad happens (i.e. a fire breaking out, etc). I'm a lone entrepreneur, so I do have backup hard disks here, but being able to additionally save this data in the cloud is great.
I'm often creating pretty big media assets, so Dropbox doesn't necessarily offer enough space or is - for me - too expensive in the 500gb version (i.e. $50 a month).
Glacier would be $10 a month for 1 terabyte. Fantastic.
I'm often creating pretty big media assets, so Dropbox doesn't necessarily offer enough space or is - for me - too expensive in the 500gb version (i.e. $50 a month).
Glacier would be $10 a month for 1 terabyte. Fantastic.
I had a quick skim through the marketing stuff and the FAQs and didn't see anywhere that actually details what the backend of this is. I'd be curious if they're actually using tape, older machines, Backblaze pods, etc. I guess if it's the latter, the time to recover could be an artificial barrier to prevent people from getting cute.
Since this is built on top of S3, I'd love to be able to store EBS Snapshots in this.
Wow, order of magnitude cheaper than S3 (1 cent per month vs. 12-14 cents for S3). Data transfer priced the same.
Drawback is "jobs typically complete in 3.5 to 4.5 hours."
Seeing as how people tend to be pack rats, I can see this being huge.
Drawback is "jobs typically complete in 3.5 to 4.5 hours."
Seeing as how people tend to be pack rats, I can see this being huge.
I think one of the most interesting parts of this is how they plan to ensure that people do not use it for transient backup: https://aws.amazon.com/glacier/faqs/#How_am_I_charged_for_de...
Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months.
Deleting data from Amazon Glacier is free if the archive being deleted has been stored for three months or longer. If an archive is deleted within three months of being uploaded, you will be charged an early deletion fee. In the US East (Northern Virginia) Region, you would be charged a prorated early deletion fee of $0.03 per gigabyte deleted within three months.
It will be interesting to know if they will upgrade AWS Storage Gateway to use this kind of backend instead of S3
http://aws.amazon.com/storagegateway/
What does "99.999999999% durability" mean? Does it mean they allow themselves to lose one byte per terabyte on average?
This is amazing! Just ~3 weeks ago I finally broke down and started using S3 to store my digital photos and other such crap.
Sure hope transferring 10 or 20 gigabytes of data from S3 to Glacier is easy.
Sure hope transferring 10 or 20 gigabytes of data from S3 to Glacier is easy.
11 9's. Impressive.
What's particularly awesome, is that this likely represents an upper bound on cost. It will only go down as time goes on.
Amazon Glacier is an extremely low-cost, pay-as-you-go storage service that can cost as little as $0.01 per gigabyte per month.
What would be absolutely fascinating is a pay-before-you-go storage service — data cryonics.
Paying $12 to store a gigabyte of data for 100 years seems like a pretty intriguing deal as we emerge from an era of bit rot.
What would be absolutely fascinating is a pay-before-you-go storage service — data cryonics.
Paying $12 to store a gigabyte of data for 100 years seems like a pretty intriguing deal as we emerge from an era of bit rot.
I hope they can get the access time down from 3-5 hours to about 1 hour - that's the difference for me between it being a viable alternative for storing backups of my client's web sites or not.
I might create a script that uploads everything to Glacier and just keeps a couple of the latest backups on S3 though.
I might create a script that uploads everything to Glacier and just keeps a couple of the latest backups on S3 though.
that's certainly interesting. as there will be migration from s3 to glacier, it would be nice if tarsnap had an option to store only the (say) last week in s3 (with .3$/gb/month) and the rest in glacier (with, say, .03$/gb/month).
that would certainly be very nice. cperciva, what do you think?
that would certainly be very nice. cperciva, what do you think?
I'm curious about data security. It says it is encrypted with AES. But is it encrypted locally and the encrypted files are transferred? I.e. does Amazon ever see the encryption keys?
Or is the only way to encrypt it yourself, and then transfer it?
Or is the only way to encrypt it yourself, and then transfer it?
What I find fascinating with Amazon's infrastructure push is the the successful 'homonymization' of their brand name Amazon.
Amazon simultaneously stands for ecommerce and web infrastructure depending on the context. e.g "Hey I want to host my server".. "Why don't you try Amazon". "Do you know where I can get a fair priced laptop?" "Check Amazon".
Is there any other brand that has done this successfully?
Edit: I should have specified internet brand.
Amazon simultaneously stands for ecommerce and web infrastructure depending on the context. e.g "Hey I want to host my server".. "Why don't you try Amazon". "Do you know where I can get a fair priced laptop?" "Check Amazon".
Is there any other brand that has done this successfully?
Edit: I should have specified internet brand.
Looks like the perfect solution to backup all my photos. Considering 100 GB of photos, that's still just 100 * 0.01 * 12 = $12 a year! I'm sold on this!
Wow, the cadence of releases from aws recently has been amazing. Can't wait to see what else they have in store.
This is a really good offering for media that you typically will keep locally for instant access, yet you want to have an off-site backup in a way that lives for a very long time.
Dropbox should work here, but it's simply too expensive. My photo library is 175GB. That isn't excessive when considering I store the digital negatives and this represents over a decade.
I don't mind not being able to access it for a few hours, I'm thinking disaster recovery of highly sentimental digital memories here.
If my flat burns down destroying my local copy, and my personal off-site backup (HDD at an old friends' house) is also destroyed... then nothing would be lost if Amazon have a copy.
In fact I very much doubt anyone I know that isn't a tech actually keeps all of their data backed-up even in that manner.
I find myself already wondering: My 12TB NAS, how much is used (4TB)... could I backup all of it remotely? It's crazy that this approaches being feasible. It's under 30GBP per month for Ireland storage for my all of the data on my NAS.
To be able to say, "All of my photos are safe for years and it's easy to append to the backup.". That would be something.
A service offering a simple consumer interface for this could really do well.
Dropbox should work here, but it's simply too expensive. My photo library is 175GB. That isn't excessive when considering I store the digital negatives and this represents over a decade.
I don't mind not being able to access it for a few hours, I'm thinking disaster recovery of highly sentimental digital memories here.
If my flat burns down destroying my local copy, and my personal off-site backup (HDD at an old friends' house) is also destroyed... then nothing would be lost if Amazon have a copy.
In fact I very much doubt anyone I know that isn't a tech actually keeps all of their data backed-up even in that manner.
I find myself already wondering: My 12TB NAS, how much is used (4TB)... could I backup all of it remotely? It's crazy that this approaches being feasible. It's under 30GBP per month for Ireland storage for my all of the data on my NAS.
To be able to say, "All of my photos are safe for years and it's easy to append to the backup.". That would be something.
A service offering a simple consumer interface for this could really do well.