Farming hard drives: 2 years and $1M later(blog.backblaze.com)
blog.backblaze.com
Farming hard drives: 2 years and $1M later
http://blog.backblaze.com/2013/11/26/farming-hard-drives-2-years-and-1m-later/
14 comments
The second rule of backups is, always test your backups. Never trust a cloud service, backup software or tape drive status. Test a restore periodically, or you can't be sure you're backed up at all.
Very true, but victim blaming gets us nowhere.
Absolutely, Backblaze are at fault here. Having a dashboard lie about files being there is a really, really serious bug.
But computers always lie. Your disk says the file is there, you double click it, and it starts making noises and the file doesn't open. Backup systems are the same. It says the files are safe, but unless you test it, they may not be.
In the end, all I want to say is, I wish ZFS was free software :(
But computers always lie. Your disk says the file is there, you double click it, and it starts making noises and the file doesn't open. Backup systems are the same. It says the files are safe, but unless you test it, they may not be.
In the end, all I want to say is, I wish ZFS was free software :(
Hey Tedivm! Yev from Backblaze here -> It's true that the data was removed from your account. After your trial lapsed and was not paid for, we removed the data after 7 days as is our standard practice. Unfortunately in your case the overview page did not update to show that you had no data associated with your computer, so when you purchased the license, there was no data on the machine that it attached too, so when you erased data from the machine while performing maintenance, you assumed the data was on Backblaze, when it was in fact purged from our system a few weeks prior.
We've since fixed that visual glitch (if you have a trial that expires and is not paid for, once the data has been deleted, that will be reflected on the overview page) AND we're currently working on some other notification systems to help people avoid losing data. As always, I apologize that you had this problem, but I'm happy to inform you that we've since fixed that visual bug on our end.
We've since fixed that visual glitch (if you have a trial that expires and is not paid for, once the data has been deleted, that will be reflected on the overview page) AND we're currently working on some other notification systems to help people avoid losing data. As always, I apologize that you had this problem, but I'm happy to inform you that we've since fixed that visual bug on our end.
So everyone else is clear, we did pay for the account after the trial ended. So we were paying customers, this wasn't an incident where a trial expired and we tried to get our data back.
Also, to be clear, the customer service experience was horrible. The support team repeatedly ignored what we were saying and send us form responses back which were just not relevant. The twitter team claimed we misread the retention policy. The whole thing was kind of ridiculous.
Even more to the point, and the main reason why I tell this story, is that people should know how this company operates. People should know that they don't have backups of backups, that there is no margin for error should your account lapse, and that their systems in general are not robustly programmed in the sense that there are ways that your data could be lost and you wouldn't know it.
From my perspective Backblaze only has one thing going for it over competing companies, and that's it's marketing department.
Also, to be clear, the customer service experience was horrible. The support team repeatedly ignored what we were saying and send us form responses back which were just not relevant. The twitter team claimed we misread the retention policy. The whole thing was kind of ridiculous.
Even more to the point, and the main reason why I tell this story, is that people should know how this company operates. People should know that they don't have backups of backups, that there is no margin for error should your account lapse, and that their systems in general are not robustly programmed in the sense that there are ways that your data could be lost and you wouldn't know it.
From my perspective Backblaze only has one thing going for it over competing companies, and that's it's marketing department.
If I understand it right, you purchased a license after your trial expired and the backup was purged. And you didn't backup anything new upon getting the new license. So the GUI status bug was a real and unfortunate bug on Backblaze's side, but it was only a GUI bug and in fact they didn't lose your data in any unexpected way. This situation seems to be about 2/3rds BackBlaze's fault and 1/3rd your fault.
You also have bonus fault on your side for not testing the BackBlaze backup. Never ever trust a backup, even if it's right in front of you on your own disk, unless you can successfully restore it.
You also have bonus fault on your side for not testing the BackBlaze backup. Never ever trust a backup, even if it's right in front of you on your own disk, unless you can successfully restore it.
Yup, there were serious flaws in our backup plans here. In my defense, we hadn't been dating long enough at the time for me to really get things set and we were on our way to burning man ;-)
I am not claiming to be without fault, but I do think that Backblaze is on the lower end of the reliability spectrum when it comes companies that do this, which is why I bring up these issues. I've switched over to Crashplan and have external harddrives doing local backups.
I should also point out that this isn't the only issue that's come up with this company. Apparently their client side GUI is also not completely truthful about when it's backed things up, and will claim to have backed up files when it's really only backed up the indexes. It then spreads the real backups out in order to increase performance. Although this is a cool feature, it's one people should be aware of and not hidden. I have less knowledge of this though, as I only saw it in a few blog posts (google around instead of taking my word for it).
I am not claiming to be without fault, but I do think that Backblaze is on the lower end of the reliability spectrum when it comes companies that do this, which is why I bring up these issues. I've switched over to Crashplan and have external harddrives doing local backups.
I should also point out that this isn't the only issue that's come up with this company. Apparently their client side GUI is also not completely truthful about when it's backed things up, and will claim to have backed up files when it's really only backed up the indexes. It then spreads the real backups out in order to increase performance. Although this is a cool feature, it's one people should be aware of and not hidden. I have less knowledge of this though, as I only saw it in a few blog posts (google around instead of taking my word for it).
"Lower end of spectrum" - of precisely two companies that I know of, CrashPlan and BackBlaze, that do "Whole Disk Backups at a fixed rate".
I'm a big fan of using a combination of Arq (Love It), Dropbox, BackBlaze, and SuperDuper, but - crashplan/backblaze are both pretty similar, and both have their pros/cons, and, likewise, hundreds of anecdotes from the user community of each.
For what it's worth, I'm in my fifth year of Backblaze, and I've always been able to recover all my files - but I've also kept my subscription paid up.
And, I don't believe I've ever seen your "real backups happening spread over" claim - I am frequently in the field, so I am very aware of exactly when backblaze is sending data (frequently over my iPhone using Roaming Data) - and when backblaze says it's done backing up - that's pretty much exactly when data is no longer sent out.
I'm a big fan of using a combination of Arq (Love It), Dropbox, BackBlaze, and SuperDuper, but - crashplan/backblaze are both pretty similar, and both have their pros/cons, and, likewise, hundreds of anecdotes from the user community of each.
For what it's worth, I'm in my fifth year of Backblaze, and I've always been able to recover all my files - but I've also kept my subscription paid up.
And, I don't believe I've ever seen your "real backups happening spread over" claim - I am frequently in the field, so I am very aware of exactly when backblaze is sending data (frequently over my iPhone using Roaming Data) - and when backblaze says it's done backing up - that's pretty much exactly when data is no longer sent out.
Yev, the same dev who is commenting here, has admitted to the claims about back up times. You should read through this reddit thread, it shows that Backblaze does not do things as you'd think-
http://www.reddit.com/r/YouShouldKnow/comments/1q14ca/ysk_th...
> If you boot up real quickly and want to do a scan one thing you can do is open up the control panel and hit "Alt" + "Restore Options", we'll do a small-file rescan immediately and schedule a large-file scan.
>Small is files under 30MB or so, it's a quick index, the larger files take a bit longer and we try to spread those out over the course of a few hours so as not to be too heavy on your system.
http://www.reddit.com/r/YouShouldKnow/comments/1q14ca/ysk_th...
> If you boot up real quickly and want to do a scan one thing you can do is open up the control panel and hit "Alt" + "Restore Options", we'll do a small-file rescan immediately and schedule a large-file scan.
>Small is files under 30MB or so, it's a quick index, the larger files take a bit longer and we try to spread those out over the course of a few hours so as not to be too heavy on your system.
Yes, that's accurate. The program was designed to be light on systems so we are not constantly scanning and indexing your hard drives. In our best practices we tell people to leave their machines on for a few hours. We cannot run when the machine is off, so if the machine is constantly being turned on and off it'll disrupt our scans. The "Alt + Restore Options" method forces a scan, but you'll still need the machine to be turned on/connected to the internet in order for us to transmit the data. Just pressing "backup now" will start pushing the data that has already been queued for upload during a previous scan (if there was anything left over), but will not necessarily initiate a scan (unless there was nothing left over from the previous uploads).
So, to be perfectly clear, it is more than possible for you to sync the photos from your computer, hit the "backup now" button, have Backblaze respond back that a backup is complete, and then find out that all of those photos you thought you had backed up were not there.
All of this to make it look like your application is performing better than it actually is.
On top of that you keep doing that infuriating thing where you ignore what people are saying and respond as if they're idiots or by trying to deflect the issue. No one expects you to run when the computer is off, and pretending like that's what's being said is downright insulting. What they do expect is that hitting the "backup now" button, and having your application respond that their files have been backed up, should actually mean that the files are backed up.
The more you talk the more you try to twist this around into something it's not- a visual glitch, misreading your policies, or people turning their computers off when they shouldn't- rather than trying to understand why your customers are upset and actually deal with it. That's why this issue has grown, and why we keep having these conversations- you guys have zero concept of responsibility and would rather insult your customers than actually make a viable product.
All of this to make it look like your application is performing better than it actually is.
On top of that you keep doing that infuriating thing where you ignore what people are saying and respond as if they're idiots or by trying to deflect the issue. No one expects you to run when the computer is off, and pretending like that's what's being said is downright insulting. What they do expect is that hitting the "backup now" button, and having your application respond that their files have been backed up, should actually mean that the files are backed up.
The more you talk the more you try to twist this around into something it's not- a visual glitch, misreading your policies, or people turning their computers off when they shouldn't- rather than trying to understand why your customers are upset and actually deal with it. That's why this issue has grown, and why we keep having these conversations- you guys have zero concept of responsibility and would rather insult your customers than actually make a viable product.
No, we should display the last time you were fully backed up in the console. If you turn on your computer, move photos to it, and turn it off, even if you press "backup now" likely they won't be included in the scan. In our best practices and introductory email we say that if you want at least a "daily" backup, you have to let us run for at least 3 hours so that we can fully index your new/changed files and back them up. If you added a lot of files it'll take longer, if you only add a few, it can be pretty quick.
If your complaint is that the Backup Now button does not do a full-system scan, that is totally valid and after the reddit thread our engineering folks are looking in to changing the "backup now" button's behavior. One of the reasons it doesn't do that now is because a full-scan will hang your system, where uploading the remaining files is very light and unnoticeable. Like with any functionality decision it's tough to say what is the best answer, hanging someone's system each time they press a button, or kicking off a remaining files upload and gradually scanning the drive over the next hour.
I sincerely apologize if I am not communicating well though, I am not trying to talk down or assume you or anyone else is an idiot in any way. I've tried to address everything that you bring up on here and on reddit. As far as expecting Backblaze to run when the computer is turned off, you'd be surprised at how many of our support tickets ask, "If my computer is off, are you still working?" so we do see that quite a bit. I don't bring it up to try and dig at anybody. I also think we've taken responsibility for the bug that mislead you in to believing you had data on our system when it had already been removed. Once we realized what had happened in your case we offered a refund and have since fixed it so that it does not happen to anyone else.
We're in the business of backing up data. When customers lose data, whether it's something they did or something that occurred on our end, we feel badly about it and try to make it right. We do have a viable product. We've restored over 4 billion files for the customers that have accounts with us. We take it very seriously.
If your complaint is that the Backup Now button does not do a full-system scan, that is totally valid and after the reddit thread our engineering folks are looking in to changing the "backup now" button's behavior. One of the reasons it doesn't do that now is because a full-scan will hang your system, where uploading the remaining files is very light and unnoticeable. Like with any functionality decision it's tough to say what is the best answer, hanging someone's system each time they press a button, or kicking off a remaining files upload and gradually scanning the drive over the next hour.
I sincerely apologize if I am not communicating well though, I am not trying to talk down or assume you or anyone else is an idiot in any way. I've tried to address everything that you bring up on here and on reddit. As far as expecting Backblaze to run when the computer is turned off, you'd be surprised at how many of our support tickets ask, "If my computer is off, are you still working?" so we do see that quite a bit. I don't bring it up to try and dig at anybody. I also think we've taken responsibility for the bug that mislead you in to believing you had data on our system when it had already been removed. Once we realized what had happened in your case we offered a refund and have since fixed it so that it does not happen to anyone else.
We're in the business of backing up data. When customers lose data, whether it's something they did or something that occurred on our end, we feel badly about it and try to make it right. We do have a viable product. We've restored over 4 billion files for the customers that have accounts with us. We take it very seriously.
We do recommend having a 3-2-1 backup (3 total copies, 2 local (1 original, 1 local external), and 1 offsite (where Backblaze comes in). In Tedivm's defense though, he did find an actual glitch in our UI that occurred in edge cases, and we've since fixed it. The unfortunate part is that he lost data when we no longer had a copy, and we're actively working on additional ways to avoid that in the future.
Can anyone else comment on their experience with Backblaze's systems, and how competitors compare?
well, at least you should be comforted by the fact that your problem (or at least, re-reporting it here on HN) will cause them to fix the bug that got you into this mess.
How did you compensate tedivm for his issues, and how do you plan on fixing the support-related issues tedivm brought up, like your CSRs lying about the problems?
Icelancer, typically when users have any issues we offer a refund if they wish to leave the service, I can say that one was offered in this case as well. I do take issue with our "CSRs lying" though. Reading through the tickets, it is absolutely accurate that our dashboard was misrepresenting the data. It was showing Tedivm's data as still being there (or available), when in reality it was deleted weeks before the license was purchased for the account (after a trial expires we maintain data for 7 days, then remove it, though we do send alert emails stating that the trial is ending and the data will be removed). In Tedivm's case the license was purchased over 20 days after the trial expired.
Our CSR didn't understand the issue that was occurring, because he had not experienced this particular problem before. The CSR was not deliberately trying to mislead anyone, and in truth once the issue was escalated and we realized was going on, we learned that Tedivm had a legitimate complaint, that we had a visual bug that was showing the data as still being there when it was not. Once we realized what was going on, we acknowledged the issue and we've since fixed that glitch as well, so now expired trials with their data deleted show that they have 0 files available online.
I can go in to further into detail, but we try to keep all customer information/support queries private. We did do a bit more training with the CSRs about identifying atypical issues like this one, so hopefully they can be found out earlier in the support process instead of after a bit of back and forth as was the case this time.
Our CSR didn't understand the issue that was occurring, because he had not experienced this particular problem before. The CSR was not deliberately trying to mislead anyone, and in truth once the issue was escalated and we realized was going on, we learned that Tedivm had a legitimate complaint, that we had a visual bug that was showing the data as still being there when it was not. Once we realized what was going on, we acknowledged the issue and we've since fixed that glitch as well, so now expired trials with their data deleted show that they have 0 files available online.
I can go in to further into detail, but we try to keep all customer information/support queries private. We did do a bit more training with the CSRs about identifying atypical issues like this one, so hopefully they can be found out earlier in the support process instead of after a bit of back and forth as was the case this time.
I think Yev put it well, but I just want to add that customer support has two distinct jobs (Tedivm's issue fell under #2):
1) Answering basic general questions like "how much does it cost"?
2) Debugging live customer issues. For example, if crappy anti-virus quarantines one of the Backblaze executables, what the heck is happening and why? Of if a customer's client cannot contact Backblaze's datacenter, which firewall is blocking it (maybe it is the software firewall on the computer, or maybe it is the router not allowing HTTPS through)? Etc....
In the process of debugging problems, the customer service reps are dealing with a truly insane array of possible issues and symptoms and they go back and forth with the customer asking for log files, ask if specific symptoms are happening, and in general COLLABORATE with the customer to get to the true bottom of the issue. Nobody lied. These are good, honest people that are really looking out for our customers and take their responsibilities seriously.
1) Answering basic general questions like "how much does it cost"?
2) Debugging live customer issues. For example, if crappy anti-virus quarantines one of the Backblaze executables, what the heck is happening and why? Of if a customer's client cannot contact Backblaze's datacenter, which firewall is blocking it (maybe it is the software firewall on the computer, or maybe it is the router not allowing HTTPS through)? Etc....
In the process of debugging problems, the customer service reps are dealing with a truly insane array of possible issues and symptoms and they go back and forth with the customer asking for log files, ask if specific symptoms are happening, and in general COLLABORATE with the customer to get to the true bottom of the issue. Nobody lied. These are good, honest people that are really looking out for our customers and take their responsibilities seriously.
No no, I want to be clear, people lied. I never claimed the CSR lied, I just claimed he sent useless form letters and didn't understand the problem. There was no dishonesty there, just a frustrating customer experience. However, the Twitter team made numerous false statements trying to downplay the issue to other people and blame the issue on a policy misunderstanding, rather than a bug. This is what prompted me to post pictures of the dashboard online so they would have no choice but to acknowledge the issue publicly.
This is the kind of thing that happens when dealing with Backblaze. It's constant misrepresentation on the public side of things. Their twitter account people twist things around and misrepresent things and their employees come into discussions like this and defend the company by claiming I said things I didn't say.
This is the kind of thing that happens when dealing with Backblaze. It's constant misrepresentation on the public side of things. Their twitter account people twist things around and misrepresent things and their employees come into discussions like this and defend the company by claiming I said things I didn't say.
Yea, we addressed our twitter folks about this as well. They aren't in support so typically have no context for tweets they are seeing. They just respond in the best way they are able to. In this case I believe that the social team saw that you were having issues with a restore and with data not being there, so they assumed that it had something to do with our 30 day retention window, which is most commonly the case when users are experiencing issues with the dashboard not showing what they think is accurate (typically this is with an unplugged hard drive or something similar).
Once we realized what was going on we told the social folks to try and defer to the support folks. It's hard to have a long conversation on twitter as well, it's not a great medium for full-on support, but it's great for sending links to FAQs and stuff like that.
I do honestly think that the conversations you had with our team on multiple fronts were misinterpreted on both sides which led to a lot of confusion. We've also gotten better at making sure that the social team asks our support folks for guidance if they see a tweet or post that they are not familiar with.
Once we realized what was going on we told the social folks to try and defer to the support folks. It's hard to have a long conversation on twitter as well, it's not a great medium for full-on support, but it's great for sending links to FAQs and stuff like that.
I do honestly think that the conversations you had with our team on multiple fronts were misinterpreted on both sides which led to a lot of confusion. We've also gotten better at making sure that the social team asks our support folks for guidance if they see a tweet or post that they are not familiar with.
I've been trying to reduce my data hoarding. I had a larger back-up hard drive go down and take a lot of data with it. I felt sick knowing I potentially lost years of funny images, videos, music, writings, etc.
What I found is I had back ups of back ups of the things I actually cared about and the things I lost I hadn't truly looked at in a long time. A lot of the music I hadn't listened to in years.
I have found that I can fit on small SSD drives and as a result of the things I do wish I hadn't lost, I'm just keeping up with the practice of keeping multiple back ups.
TL:DR: I have made an effort to keep a smaller data footprint instead of hoarding.
What I found is I had back ups of back ups of the things I actually cared about and the things I lost I hadn't truly looked at in a long time. A lot of the music I hadn't listened to in years.
I have found that I can fit on small SSD drives and as a result of the things I do wish I hadn't lost, I'm just keeping up with the practice of keeping multiple back ups.
TL:DR: I have made an effort to keep a smaller data footprint instead of hoarding.
If the only instance of a thing is on your "back-up" drive then it isn't a back-up.
Data that isn't backed up (as in a second copy) doesn't really exist.
Likewise: data that isn't backed up off site isn't really backed up.
Likewise: data that isn't backed up off site isn't really backed up.
> Data that isn't backed up (as in a second copy) doesn't really exist.
An IMO more realistic statement: Data that is not backed up is implicitly classified as "nice to have" - in many cases a perfectly rational choice, if done consciously.
> Likewise: data that isn't backed up off site isn't really backed up.
You may as well argue that no data is ever "really" backed up because there is always the possibility of simultaneous failure. It's always a choice of how much risk you're willing to accept vs. how much money and effort it would cost to reduce it further.
An IMO more realistic statement: Data that is not backed up is implicitly classified as "nice to have" - in many cases a perfectly rational choice, if done consciously.
> Likewise: data that isn't backed up off site isn't really backed up.
You may as well argue that no data is ever "really" backed up because there is always the possibility of simultaneous failure. It's always a choice of how much risk you're willing to accept vs. how much money and effort it would cost to reduce it further.
Yep. 1 copy of data is not enough, because it can be lost. Now, assuming that N copies of data is not enough, it's easy to see that N+1 copies is also not enough, because that extra copy can be lost, thus reducing to N copies, which is not enough.
This proves by induction that no matter how many backups you have, it's not enough.
This proves by induction that no matter how many backups you have, it's not enough.
Untested backups don't exist either.
There are several horror stories of people going to their offsite tape backups to restore some lost data only to find that their backups didn't work due to a malfunctioning tape drive.
Other stories about encrypted backups where the decryption key is lost (backup data is encrypted with a randomly chosen "session" key, session key is encrypted using a public key, no-one could find the appropriate private key to decrypt the session key.
etc...
There are several horror stories of people going to their offsite tape backups to restore some lost data only to find that their backups didn't work due to a malfunctioning tape drive.
Other stories about encrypted backups where the decryption key is lost (backup data is encrypted with a randomly chosen "session" key, session key is encrypted using a public key, no-one could find the appropriate private key to decrypt the session key.
etc...
I heard about a private school that backed up its fee database (MS Access) to a USB stick every night - in fact what the accountant was doing was backing up the desktop shortcut to the database.
In the event of the failure that revealed the problem, they lost about 7 months worth of data (which was the time since the computer had last been connected to the network, and had synced via Active Directory, or whatever they call it.)
In the event of the failure that revealed the problem, they lost about 7 months worth of data (which was the time since the computer had last been connected to the network, and had synced via Active Directory, or whatever they call it.)
Agreed.
I currently have two external drives: an archive drive and a backup drive: both 2TB. (Most things on the backup drive are compressed and deduped, which is why I can fit everything.)
I currently have two external drives: an archive drive and a backup drive: both 2TB. (Most things on the backup drive are compressed and deduped, which is why I can fit everything.)
The original "Drive Farming" link is at least as interesting as the new article:
http://blog.backblaze.com/2012/10/09/backblaze_drive_farming...
http://blog.backblaze.com/2012/10/09/backblaze_drive_farming...
And here's the discussion of that story:
https://news.ycombinator.com/item?id=4631027
I browsed the comments with two things in mind: was anyone uncomfortable with shucking and farming on this scale, and were the drives of equivalent quality?
Regarding the latter, user rsync of rsync.net claimed that "the drives inside these enclosures are the worst spec'd, highest failure rate drives." User budmang of Backblaze, however, said, "The drives were literally the same inside the external enclosures." User atYevP of Backblaze posted a similar comment.
https://news.ycombinator.com/item?id=4631027
I browsed the comments with two things in mind: was anyone uncomfortable with shucking and farming on this scale, and were the drives of equivalent quality?
Regarding the latter, user rsync of rsync.net claimed that "the drives inside these enclosures are the worst spec'd, highest failure rate drives." User budmang of Backblaze, however, said, "The drives were literally the same inside the external enclosures." User atYevP of Backblaze posted a similar comment.
Data recovery guy here...
Failure of external drives has mostly to do with their handling.
They get bumped around while runnning.
People trip over the cords and knock them over.
They stack them like pancakes and they overheat. They block the fans and cause overheating.
Many enclosures are designed to look stylish - but don't cool well.
Heads fly at nanometers over the surface - they dont tolerate bumps well.
The drives we recover from external enclosures appear to be the same as bare drives that we recover.
Failure of external drives has mostly to do with their handling.
They get bumped around while runnning.
People trip over the cords and knock them over.
They stack them like pancakes and they overheat. They block the fans and cause overheating.
Many enclosures are designed to look stylish - but don't cool well.
Heads fly at nanometers over the surface - they dont tolerate bumps well.
The drives we recover from external enclosures appear to be the same as bare drives that we recover.
Yev here ;-) -> http://blog.backblaze.com/2013/11/12/how-long-do-disk-drives... We posted this earlier in the month. The drives inside the external drives that were shucked performed pretty well. We were guessing that they would fail more frequently but that just didn't turn out to be the case. We're currently collecting even more data and hope to release it soonishly :)
High failure rates of external drives can also be attributed to the USB controllers - they're better than their IDE predecessors (which failed a lot, sometimes taking the drives with them - only low level utilities could revive them).
Otherwise, I don't see how or why the hard drive manufacturers would bother to send the worst batches (which aren't that much worse) to external hard drive assemblers.
I repair computers for friends and families and the majority of the time I get a broken external to look at, it's the enclosure that's at fault and the drive is absolutely fine.
I encourage people to buy their enclosures and drives separately - it's a little more expensive, but this way you can RMA just the enclosure if it fails. With an all-in-one, you either void your warranty to get the data back, or ship the whole unit back for a replacement.
I encourage people to buy their enclosures and drives separately - it's a little more expensive, but this way you can RMA just the enclosure if it fails. With an all-in-one, you either void your warranty to get the data back, or ship the whole unit back for a replacement.
That's an interesting point, since I have a Samsung hard drive I took OUT of an enclosure and out of curiosity I checked the warranty on it - it's a 3 year warranty that is still active and the web page said I could initiate a return.
So, technically, I could RMA the drive myself if it fails... or would they notice it and not process the return?
So, technically, I could RMA the drive myself if it fails... or would they notice it and not process the return?
It could also be that they shut down more frequently. More stop-start cycles mean more wear on the moving parts of the disk.
Isn't that a feature of the USB enclosure? (i.e., not an issue once a drive has been "shucked")
Interesting. I've never once found a standard drive model inside an external.
I mean, of course they are standard form factors, standard interfaces, etc- they are standard 3.5" hard drives- but they are always some odd model number I've never heard of with a 32kB cache, 1 month warranty, and 3600RPM. (slight exaggeration).
I mean, of course they are standard form factors, standard interfaces, etc- they are standard 3.5" hard drives- but they are always some odd model number I've never heard of with a 32kB cache, 1 month warranty, and 3600RPM. (slight exaggeration).
In my experience, most of the 3.5" enclosure drives have been off the shelf. My last batch of HP ones were all WD Greens. I have seen a few WD 2.5" externals with proprietary connections :/
Sadly I read this as a precursor to shortages in water, fish and fruits/vegetables that are coming. The populations of monarch butterflies, bees and other insects are dropping like a stone year over year. The fish near asia's costs are disappearing. The culprits include pesticides, overfarming, overfishing, and pollution. I am not even as worried about the garbage anymore -- scandinavian countries have figured out ways to burn the garbage and "contain" the waste gases. However, the main problem of capitalism is the numerous externalities that are exploited until it's too late to fix. It's not just global warming. It's raping the ecosystems for profit. No futures are going to save us from that. We need sustainable practices but I really don't see a way that'll happen at this point.
Too fatalistic. In 19th century they predicted that Earth can't sustain population over 1 billion. In comes oil and agricultural revolution. I agree what we are doing with ecosystem in general is a crime, but it won't kill us. At least not anytime soon.
Just because advances were able to stave off collapse and allow billions of people to consume resources at an ever greater rate does not mean that the same can be done with the effects of overfarming, overfishing, and pollution. The fallacy is thinking "since we did it once we can always continue doing it." The probability exponentially rises that you can't, since this is basically the same as the probability of always succeeding in time to avert a disaster.
Actually garbage / messing up ecosystems is a different type of problem than obtaining resources. It is somewhat related to the second law of thermodynamics. Once you introduce so much entropy into a system the genie is out of the bottle.of course this isn't exactly thermodynamics but I can say it this way -- putting humpty dumpty together again is different than figuring out how to make humpty give you more food for decades.
Actually garbage / messing up ecosystems is a different type of problem than obtaining resources. It is somewhat related to the second law of thermodynamics. Once you introduce so much entropy into a system the genie is out of the bottle.of course this isn't exactly thermodynamics but I can say it this way -- putting humpty dumpty together again is different than figuring out how to make humpty give you more food for decades.
I really agree with you on the point that we are doing all those things. I am a fan of the http://physics.ucsd.edu/do-the-math/post-index/ blog.
What am I arguing for - that while we do consume more resources, progress is exponential as well. For example if we seriously begin to run out of fossil fuels, amount of effort that is put into other energy sources would increase by order of magnitude and solution will be found. It might not be right away, it might not be silver bullet, but when push comes to shove and there are no alternatives, human ingenuity is frightening.
In the end progress stacks multiplicatively - you add great battery technology to great energy generation technology and you got more than sum of the parts. So more people there are doing research - faster it goes.
This point is only true from certain point of view. Consider genius starting from nill, how far could he take technology within his lifetime? Electricity, probably. Microprocessors and genetics, probably not. Now consider million of people working, each discovery builds on an earlier one, each tool makes other people researching more efficient or enables them to make new discoveries. Now think about multiple generations of people building discoveries on top of each other while sharing information.
What am I arguing for - that while we do consume more resources, progress is exponential as well. For example if we seriously begin to run out of fossil fuels, amount of effort that is put into other energy sources would increase by order of magnitude and solution will be found. It might not be right away, it might not be silver bullet, but when push comes to shove and there are no alternatives, human ingenuity is frightening.
In the end progress stacks multiplicatively - you add great battery technology to great energy generation technology and you got more than sum of the parts. So more people there are doing research - faster it goes.
This point is only true from certain point of view. Consider genius starting from nill, how far could he take technology within his lifetime? Electricity, probably. Microprocessors and genetics, probably not. Now consider million of people working, each discovery builds on an earlier one, each tool makes other people researching more efficient or enables them to make new discoveries. Now think about multiple generations of people building discoveries on top of each other while sharing information.
I disagree -- we leave an impact behind, but I've also seen terribly polluted areas being successfully remediated. So where there's a will, there's a way.
Try to imagine that this is the stock market: http://upload.wikimedia.org/wikipedia/commons/b/b7/Populatio...
Why didn't they touch on why the Hitachi drives weren't the same quality as the other brands with this statement:
"Switching hard drive manufacturers from Hitachi to Seagate and Western Digital. The assumption here is that all drives are the same and as we’ve learned that’s not really the case."
Where is the footnote. Why aren't they the same?
"Switching hard drive manufacturers from Hitachi to Seagate and Western Digital. The assumption here is that all drives are the same and as we’ve learned that’s not really the case."
Where is the footnote. Why aren't they the same?
It doesnt implicate that Hitachi were not the same quality as others. I undestand that the main reason to switch was the availability to price ratio, nothing else. What I think they meant when talking about drives to be different is the variety of sizes you get when purchasing 'a 3TB' drive.
In the parent article they said they switched because of cost: "When Hitachi drives got too expensive we switched to Seagate and Western Digital. When internal drives got too expensive we turned to external drives. When hard drive availability became an issue we turned to Drive Farming."
From their article on v2 they said: "The Western Digital and Seagate equivalents we tested saw much higher rates of popping out of RAID arrays and drive failure. Even the Western Digital Enterprise Hard Drives had the same high failure rates. The Hitachi drives, on the other hand, perform wonderfully." http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...
There are other articles where they mention that they do testing and some brands perform better than others.
From their article on v2 they said: "The Western Digital and Seagate equivalents we tested saw much higher rates of popping out of RAID arrays and drive failure. Even the Western Digital Enterprise Hard Drives had the same high failure rates. The Hitachi drives, on the other hand, perform wonderfully." http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v...
There are other articles where they mention that they do testing and some brands perform better than others.
We used to make PODs like this at work for netbackup storage.. started with WD drives ~128 WD drives, at about two years we were at ~8 WD drives still working. At least once a week we had people in there swapping drives. I have yet to see a WD not fail.
On the other hand, I've got a WD Caviar Black that's been running strong for more than five years. None of my WD drives have ever failed (knock on wood!). I don't run 100+ drives though, so I'm probably just hiding in the statistical margin.
Your Caviar Black is trumped by my two Scorpio Blacks I had to RMA in one year because of strange read/write delays and clicking noise. No actual failure, but it started from one problem/day when I bought them to 3-4/hour when I sent them back.
Also your thermal environment is going to be quite different, which could be a significant factor. How often do you power-cycle that drive BTW? I've had more drives go bad on power-up (just failing to spin up and move the heads correctly) than through surface errors and such.
I try not to restart that machine more than once a month.
That si better than running the drives 24/7 - if one is going to go then that way it is more likely to go alone. If every in an array (anything I even half way care about get s the RAID1 or RAID5 treatment) has been spinning for years then a power cycle can be quite scary. Restarting once month or so increases the chance that if they fail they'll not all fail at the same time.
Having said that, my main little server at home hasn't had a power cycle for somewhat longer than a month as there have been no kernel updates or that like that have required it...
Having said that, my main little server at home hasn't had a power cycle for somewhat longer than a month as there have been no kernel updates or that like that have required it...
The extrapolations in the article were pretty optimistic to begin with.
I mean, when you look at the dotted trendline in this graph: http://blog.backblaze.com/wp-content/uploads/2013/11/blog-co... you'd think by 2016 drives would cost $0 per gigabyte and by 2020, drive companies would be paying you to take drives off them.
I mean, when you look at the dotted trendline in this graph: http://blog.backblaze.com/wp-content/uploads/2013/11/blog-co... you'd think by 2016 drives would cost $0 per gigabyte and by 2020, drive companies would be paying you to take drives off them.
Just like the old razor blades graph: http://imgur.com/IBjL9tT
It scares me that we now have 6 blade razors: http://www.dorcousa.com/sxa1000-6-blade-system-for-men/
I'll stick with a single blade.
I'll stick with a single blade.
That reminds me of Gordon Moore in a commencement address (at Caltech in 2001?) quoting the number of transistors the industry was producing annually per resident of the US (and how that number was growing), and telling the graduating engineers their job was to figure out what to do with all those transistors. The answers turned out to include things like multi-gigabyte smartphones.
So who's going to figure out what to do with our future profusion of razor blades?
So who's going to figure out what to do with our future profusion of razor blades?
I'll just leave this right here:
http://www.theonion.com/articles/fuck-everything-were-doing-...
http://www.theonion.com/articles/fuck-everything-were-doing-...
[deleted]
I believe my Dollar Shave Club Executive razors have 6 blades. So smooth.
Well, at that point, if the current graph charts cost per gigabyte and approaches zero, then you would change the scale of the graph to show costs in terms of terabytes and eventually petabytes.
Ultimately, after traversing enough benchmarks, the scale would go logarithmic, and measurements would have to take orders of magnitude into account.
Ultimately, after traversing enough benchmarks, the scale would go logarithmic, and measurements would have to take orders of magnitude into account.
Indeed, if you click through to the article the historical data is drawn from, you will see it plotted on a semilog plot. The trend is well described by an exponential function.
you are going to get a more intuitively realistic trend-line mapping bytes per dollar rather than dollars per byte.
Aren't they big enough at this point to cut deals directly with the vendors? 10,000 drives at a discount, that sort of thing.
In the previous story, user brianwski of Backblaze said, "We have been told that until we buy blocks of 10,000 drives, we cannot deal directly with the drive manufacturers. We're not there yet."
https://news.ycombinator.com/item?id=4633213
https://news.ycombinator.com/item?id=4633213
brianwski from Backblaze here again - this reminds me to check once again on this info. We heard a rumor that the number of drives might be lower because we only buy the largest and most expensive drive. The 10,000 number might be a rule of thumb for a retailer like Radio Shack that stocks a blend of different drive sizes.
With all that said, EVERY MONTH we bid 20 different suppliers against each other and choose the lowest price. For goodness sake, anybody on earth who wants free money just has to undercut the other suppliers by 1 penny and you'll win our business. Always open to new suppliers, please, take our money!!
With all that said, EVERY MONTH we bid 20 different suppliers against each other and choose the lowest price. For goodness sake, anybody on earth who wants free money just has to undercut the other suppliers by 1 penny and you'll win our business. Always open to new suppliers, please, take our money!!
Slightly off topic but similar, a friend and I tend to have a frustrating talk about some company (startups and old stable businesses alike) who fails to successfully take our money once every week or two. Seriously people, just ask people for their money and then freaking make it easy to do. That's all. It isn't that hard. I want to give you money. Please, let me give you money.
Cool.
I know a few Backblaze people are reading/posting in these comments, so I'll ask here:
Do you guys know the approximate price of when you can purchase them direct in 10,000 blocks?
I know a few Backblaze people are reading/posting in these comments, so I'll ask here:
Do you guys know the approximate price of when you can purchase them direct in 10,000 blocks?
I'm not at Backblaze, but when I was at Google I did know what Google was paying for drives direct from the manufacturer and it was not as much lower than the distributor price as you might expect. Margins are pretty thin in the disk drive business.
Yev from Backblaze here -> It really depends on the type of drive. One of the reasons we don't currently buy in chunks of 10,000 is that we like to forecast how many drives we'll need and not keep a bunch of inventory (if you look at the graphs, you'll see that prices used to drop a little bit each month, so keeping excess inventory was like leaving money on the table).
But they claim to have added 50 Petabytes of storage in 2 years, which surely runs to over 10K drives per year.
It's a cool story but something just doesn't add up here.
It's a cool story but something just doesn't add up here.
(I work at Backblaze.) We buy our drives once per month right now. This is an improvement over the early days where we were so strapped for cash we bought twice a month. :-)
We're building and deploying 17 pods this month, each pod has 45 drives, so we purchased 765 drives this month (plus replacements for failed drives that are out of warranty, etc). So if we would buy 13 months of drives all in one purchase, we might be able to go direct.
THE TRADEOFF (Gamble?) is that drive prices drop every month, so stockpiling a year worth of drives (even at a 10 percent discount) would lose us money. Before the Thailand drive crisis we had a rule of thumb that drive prices dropped 4 percent each month. But this is just a judgement call, your guess is as good as mine whether this makes sense. We're figuring this out like everybody else.
We're building and deploying 17 pods this month, each pod has 45 drives, so we purchased 765 drives this month (plus replacements for failed drives that are out of warranty, etc). So if we would buy 13 months of drives all in one purchase, we might be able to go direct.
THE TRADEOFF (Gamble?) is that drive prices drop every month, so stockpiling a year worth of drives (even at a 10 percent discount) would lose us money. Before the Thailand drive crisis we had a rule of thumb that drive prices dropped 4 percent each month. But this is just a judgement call, your guess is as good as mine whether this makes sense. We're figuring this out like everybody else.
Sounds like you need a hard drive futures market. :)
I'd be willing to bet you wish you'd have purchased 10k drives in August 2011.
Good lord yes. Before the drive crisis we were smug and confident and everything was operating smoothly and according to plan, we just couldn't even imagine what could possibly go wrong. Yep, and we bought our drives JUST in time, the longer you delay the purchase the lower the cost the higher the profit.
What did Andy Grove say? "Only the paranoid survive." Now we run a larger buffer of drives.
What did Andy Grove say? "Only the paranoid survive." Now we run a larger buffer of drives.
If we had the extra cash reserves...yup :)
First of all, 50 PB is less than 20K 3TB drives. Second, they don't buy a year's supply at once.
I have a colleague who works in the industry and purchases a few petabytes of spinning disk a quarter. I love reading the backblaze blogs, and constantly forward them to him. He thinks the backblaze guys are pretty crazy with their "shucking strategy", finds it hard to believe seagate/WD guys will honor warranties on those "shucked" drives, but if they will, all the more power to backblaze.
He does note that his costs per drive didn't go up dramatically after the Thailand floods, and they were pretty close to what backblaze was playing for their "shucked drives" - though, they didn't go down signficantly either.
I'm guessing Backblaze is still at the volume levels where they can't negotiate directly with Drive Vendors - I'm not sure where that level is - 10 Petabbytes/Quarter? 100 Petabytes/quarter?
Regardless, I continue to be very impressed by how backblaze manages to circumvent what others might consider insurmountable hurdles. Truly hacking the system in a good way.
He does note that his costs per drive didn't go up dramatically after the Thailand floods, and they were pretty close to what backblaze was playing for their "shucked drives" - though, they didn't go down signficantly either.
I'm guessing Backblaze is still at the volume levels where they can't negotiate directly with Drive Vendors - I'm not sure where that level is - 10 Petabbytes/Quarter? 100 Petabytes/quarter?
Regardless, I continue to be very impressed by how backblaze manages to circumvent what others might consider insurmountable hurdles. Truly hacking the system in a good way.
Yev w/ Backblaze here -> Yea unfortunately when we shuck the drives the warranties are voided, but we add that factor in to the costs that we are willing to pay for the hard drives. Truth was, it was cheaper for us to buy the externals and shuck them, even with the voided warranty, than pay the larger fees for internal drives.
Part of the is price sensitivity. External drives are geared towards consumers, who simply will not buy the drives if they go up too much in cost. The internal drives we were buying are targeted at business' who have a choice to make, either buy at the higher price, go out of business, or get creative. We try to go the go creative route because we really don't want to go out of business, and paying the higher prices for drives, especially in this case, would have gotten us there.
As for the negotiation, BrianWski mentioned it in another reply on here, but we keep getting weird answers from vendors and manufacturers. Typically they want a minimum order of about 10,000 hard drives in order to work directly with them and we're simply not willing to buy that many at a time, especially since normally the price goes down monthly, meaning carrying inventory is leaving money on the table.
Love that you enjoy your blog posts, stay tuned ;-)
Part of the is price sensitivity. External drives are geared towards consumers, who simply will not buy the drives if they go up too much in cost. The internal drives we were buying are targeted at business' who have a choice to make, either buy at the higher price, go out of business, or get creative. We try to go the go creative route because we really don't want to go out of business, and paying the higher prices for drives, especially in this case, would have gotten us there.
As for the negotiation, BrianWski mentioned it in another reply on here, but we keep getting weird answers from vendors and manufacturers. Typically they want a minimum order of about 10,000 hard drives in order to work directly with them and we're simply not willing to buy that many at a time, especially since normally the price goes down monthly, meaning carrying inventory is leaving money on the table.
Love that you enjoy your blog posts, stay tuned ;-)
I personally (for home use) held off buying drives for two years, found out I did not really need so much space and discovered SSDs would do the trick.
Same here. I bought my last 2 TB for $80 shortly before the crisis, and was able to scrape by at 90% capacity until 4 TB drives became cheap enough.
> When the Drive Crisis started, industry pundits estimated that the hard drive market would take anywhere from 3 months to 1 year to recover. No one guessed two years. Was the delay simply an issue in rebuilding and/or relocating the manufacturing and assembly facilities?
This is a typical response in all markets that are commodities (or approaching commodity status). Once makers have had an excuse to move prices up, they want to hold onto the extra profit for as long as they can, so prices are fast to rise, but slow to fall.
The most direct effect can be witnessed in gasoline prices. Notice how a news report that a camel farted in the desert of Saudi Arabia results in the price of gasoline at your local station rising by 25 cents a gallon (or more) nearly instantly. Even though the gas in the underground tanks was the same gas (at the same cost to the station) that it was 15 minutes ago.
But then, it takes weeks before that 25 cent increase goes away when no more camel farts occur over those same weeks. Same effect here, just with hard drives rather than gasoline.
This is a typical response in all markets that are commodities (or approaching commodity status). Once makers have had an excuse to move prices up, they want to hold onto the extra profit for as long as they can, so prices are fast to rise, but slow to fall.
The most direct effect can be witnessed in gasoline prices. Notice how a news report that a camel farted in the desert of Saudi Arabia results in the price of gasoline at your local station rising by 25 cents a gallon (or more) nearly instantly. Even though the gas in the underground tanks was the same gas (at the same cost to the station) that it was 15 minutes ago.
But then, it takes weeks before that 25 cent increase goes away when no more camel farts occur over those same weeks. Same effect here, just with hard drives rather than gasoline.
Six months after the Thailand drive crisis, drive prices "peaked" at about double their earlier prices and began falling. Just about then, an article came out featuring a bunch of panic inducing quotes from a Seagate VP loudly claiming it might be a year before they could get enough drives made, shortages would be terrible. It felt so disgusting to me, none of it was true, the VP was just trying to encourage pre-buying and drive hoarding to artificially pump sales.
I wonder why external drives are cheaper than internal drives. One would think that they would be more expensive, since the cost to the manufacturer of an external drive includes not only the drive itself but the adapter contained in the enclosure, the enclosure itself, cables, possibly a power supply, more elaborate packaging and the additional labor required to assemble the external drive. I assume that this consumer cost differential arises from differences in the markets for internal and external drives, but naively it seems like the internal drive market would be more price-sensitive and thus lower margin.
Does anyone know why external drives are typically cheaper than internal drives?
Does anyone know why external drives are typically cheaper than internal drives?
Yev from Backblaze here -> Personally, I think that the biggest part of the is price sensitivity. External drives are geared towards consumers, who simply will not buy the drives if they go up too much in cost. The internal drives we were buying are targeted at business' who have a choice to make, either buy at the higher price, go out of business, or get creative. We try to go the go creative route because we really don't want to go out of business, and paying the higher prices for drives, especially in this case, would have gotten us there.
Backblaze tries to avoid the "enterprise tax" as often as we can, and internal vs. external drives are just another example of how "enterprise-grade" items get marked up because the target market will pay for it.
Backblaze tries to avoid the "enterprise tax" as often as we can, and internal vs. external drives are just another example of how "enterprise-grade" items get marked up because the target market will pay for it.
Maybe, with the advent of laptops becoming more popular than desktops, there is just a bigger market for them. Cost is usually due to demand constraints.
I doubt it. Server farms consume an unreal quantity of drives. Desktops consume 1 drive per person.
Server farms store data for actual people. So sit back and think about how much unique data you have on servers somewhere. Add it up and I dont't think the total data storage at all data centers everywhere in the US add up to more that 20GB per person in the US as we are just not that interesting even at 5x redundancy your still at 1/30th a drive per person. Don't forget at 300 million people that would still be 10 Million drives which is a lot of servers.
PS: The NSA could be recording every phone call ever made and it's still not all that much actual data.
PS: The NSA could be recording every phone call ever made and it's still not all that much actual data.
The dropbox model is not the only place for server farms.
How about render farms & their associated data stores. How about Google Maps & the associated scads of satellite imagery.
You could be right, of course, but I'm not sure we can say with certainty.
How about render farms & their associated data stores. How about Google Maps & the associated scads of satellite imagery.
You could be right, of course, but I'm not sure we can say with certainty.
Google Maps stores a 'tiny' amount of data per person. The surface area of the earth is a little less than 200million square miles or a little less than 2/3 of a square mile a person. at 1 foot accuracy that would only be two 10 megapixel pictures per person not that we actually have maps anywhere near that good of most small towns let alone the oceans or arctic. The only thing that really spit's out data is raw scientific data, but even Nasa just dumps well over 99% of what they collect so it's more of a short term problem.
They only have to QC them with the controller they ship with, instead of a hundred different PC's or motherboards. Not sure how much of a difference in cost that is.
I will throw in that, from my experience, external drives usually contain lower quality disks. This is fitting; external drives are primarily used for sequential access on a sporadic basis. E.g., backups, movies, archives
I bet it has to do with the target market.
4tb external drives are something an end-user consumer would shop for based on price.
While a 4tb bare internal is shopped for by techies building systems.
4tb external drives are something an end-user consumer would shop for based on price.
While a 4tb bare internal is shopped for by techies building systems.
I don't know a whole lot about backblaze; as they are a backup service, is there a good reason why they couldn't ramp up usage of tape drives? Perhaps keeping recently uploaded backups on-disk, and shuttling older ones to tape?
Their backups are progressive, so a single disk image might be spread across 100 tapes. A restore would be a nightmare. Tapes really only work if you are doing a full image each time.
or a program of partial backups interleaved with full backups, say 3-to-1. You balance a reasonable restore time with a reasonable storage requirement.
Yes. Tapes and tape drives are expensive and awkward to use.
You also have to store them, test them and then find them when a customer wants their data. With drives in a pod they're always online and in a RAID so when a drive dies they can be alerted quickly and customers can access data essentially 24/7.
This seems like something that should be part of their business instead of waiting for catastrophes.
Since the price of storage is consistently declining, buying any more than necessary is wasteful over the long term.
yeah, but it seems like you'd give yourself more than a few weeks of runway. Don't have to buy 5 years worth, but perhaps 3-6 months would be reasonable. The peace of mind has got to be worth the ever so slight cost increase.
Still... no linux.
Seriously, you can find reports of this all over the internet. It's on reddit, twitter and elsewhere- these guys do not back files up properly, and their systems are so shoddy that they'll report they have files even when they don't.
My girlfriend brought her computer in for repair and they changed the hard drive. She specifically checked before she left that the backups were set, and the dashboard said they were. When we went to restore, however, none of the files could be.
We contacted support and they kept insisting we did something wrong. Finally they admitted that they had erased the files (yes, actually erased them). Due to a "bug" their dashboard reported we still had the files, even though they were gone.
They did everything they could to hide their mistake. They lied to people on twitter about what I was saying (since the only way to get real support was to hound their twitter account). They lied about all sorts of things. I had to take screen shots of everything and post it online before they'd even admit to screwing up. Even at that point their dashboard was still saying it had all of these files, when it in fact had nothing.
Backblaze is a completely untrustworthy company. If they spent half as much effort on backing up files as they do on marketing that may be a different case. Anyone who uses them for backups might as well not have backups at all.