Paranoia and deletion: the wipe man page(boingboing.net)
boingboing.net
Paranoia and deletion: the wipe man page
http://www.boingboing.net/2011/01/14/paranoia-and-deletio.html
18 comments
But how do you know if the data actually gets erased by the drive?
You don't know, you hardly ever know anything. You evaluate the evidence (as laid out by the GP) and conclude that it's most likely the case.
This is really the approach that most conspiracy theories take. The fact that we don't know something to be true (but it's very likely to be) is used as an implication that it's probably false.
This is really the approach that most conspiracy theories take. The fact that we don't know something to be true (but it's very likely to be) is used as an implication that it's probably false.
If that possibility seriously concerns you, follow it up by physical destruction of the drive. Magnets strong enough to actually damage the data aren't a good cost/benefit from what I gather (too strong, too dangerous, too much hassle), so either shred or melt the drive platters.
I'm generally satisfied with df if=/dev/zero of=/dev/sda because I'm not actually dealing with anything that sensitive, but if you are, just take appropriate steps. Defense in depth.
I'm generally satisfied with df if=/dev/zero of=/dev/sda because I'm not actually dealing with anything that sensitive, but if you are, just take appropriate steps. Defense in depth.
It doesn't concern me that much, but I'd rather fill the drive with random data myself first. I can do the ATA-erase thing afterwards.
On a related note, I also have zero trust in manufacturer-supplied encryption that sits in the drives. There is no way for me to verify whether it actually does anything.
On a related note, I also have zero trust in manufacturer-supplied encryption that sits in the drives. There is no way for me to verify whether it actually does anything.
I think that incinerating the disk should erase the data. Magnetic material become normal materials after some temperature http://en.wikipedia.org/wiki/Curie_temperature . Usually it is no more than 1000 C/ 1800 F, so a good fire should be enough.
Warning: I don't know if the fumes are toxic.
Warning: I don't know if the fumes are toxic.
If you're that worried about it, there's always thermite.
Well, you don't know. It all depends on what kind of data was on the target drive.
If it's schoolwork or unimportant business data, a format is good enough. If we're talking CC and SSN numbers, the ATA command with writing 0's is good. If it's sensitive, a sledgehammer and an anvil is suitable.
I inherited a junk 1G hard drive, and just 'filed it away' in my junk drawer. After years, I actually checked the contents. Lo and behold: a flat database full of medical data, including SSN's and other revealing data. It was only 1G so it got the sledge (I use a small section of railroad tie instead of a blacksmiths anvil).
If it's schoolwork or unimportant business data, a format is good enough. If we're talking CC and SSN numbers, the ATA command with writing 0's is good. If it's sensitive, a sledgehammer and an anvil is suitable.
I inherited a junk 1G hard drive, and just 'filed it away' in my junk drawer. After years, I actually checked the contents. Lo and behold: a flat database full of medical data, including SSN's and other revealing data. It was only 1G so it got the sledge (I use a small section of railroad tie instead of a blacksmiths anvil).
Silly conspiracy theory.
1) The scope is too large. Too many engineers at hard drive manufacturing companies would have to know about it.
2) Too America-centric. Engineers in foreign countries, that actually build most of the stuff, would know about it. It would be an enormous security gap that they could use as well.
The sad thing is that such paranoid fantasies, that are easy to debunk, blind you to the actual conspiracies, that are less comprehensive, more subtle and therefore much more threatening. I bet the TLA government agencies love these stories.
1) The scope is too large. Too many engineers at hard drive manufacturing companies would have to know about it.
2) Too America-centric. Engineers in foreign countries, that actually build most of the stuff, would know about it. It would be an enormous security gap that they could use as well.
The sad thing is that such paranoid fantasies, that are easy to debunk, blind you to the actual conspiracies, that are less comprehensive, more subtle and therefore much more threatening. I bet the TLA government agencies love these stories.
The invisible yellow dots identifying printers went a long time before becoming common knowledge. I don't believe TFA for an instance, but it doesn't seem an impossibility.
Ugh, sometimes I wish I could edit later. Make that "I don't believe for an instant". I hate responding on my phone.
Maybe it's the gov who posits these theories?... nah.
I always dissemble my old hard drives, it's therapeutic somehow.
I am not sure how they broke that disc into so many pieces though, in my experience they are incredibly strong and rigid.
The discs make great wind chimes too.
Harbor Freight has a $3 security bit set if you have drives with special screws.
I am not sure how they broke that disc into so many pieces though, in my experience they are incredibly strong and rigid.
The discs make great wind chimes too.
Harbor Freight has a $3 security bit set if you have drives with special screws.
Sorry to be a pedant, but I think you mean "disassemble" - I always get the two mixed up too.
Currently your comment, which adds nothing to the discussion except a simple spelling correction, has more upvotes than the parent. Sign of the times for HN?
Not to say spelling corrections are never useful, I think they often are, but I don't understand the mindset that upvotes them over comments that actually participate in the discussion.
Not to say spelling corrections are never useful, I think they often are, but I don't understand the mindset that upvotes them over comments that actually participate in the discussion.
I learned a new word today: dissemble. I come to HN to learn; therefore, it got my upvote.
Yeah, I'm quite surprised by that. I just thought I would mention it in passing.
upvoted this
Some drives have glass platters AFAIK. This would explain how this one was broken.
My method for dealing with old drives:
- remove top plate of drive - a number of heavy blows with a Sledgehammer. - incineration.
This usually does the trick.
- remove top plate of drive - a number of heavy blows with a Sledgehammer. - incineration.
This usually does the trick.
Liquid nitrogen?
I looked it up on flickr http://www.flickr.com/photos/jon_a_ross/1482849745/
and from the caption it sounds like it actually did that to itself which seems impossible unless maybe the platter was made defectively in the first place ... ? (note it was an "old" 40gb drive - also looks like 2.5 inch)
and from the caption it sounds like it actually did that to itself which seems impossible unless maybe the platter was made defectively in the first place ... ? (note it was an "old" 40gb drive - also looks like 2.5 inch)
Glass platters?
The man page is correct about quite a few things, but it's also a bit sarcastic:
The best way to sanitize a storage medium is to subject it to tempera-
tures exceeding 1500K. As a cheap alternative, you might use wipe at
your own risk. Be aware that it is very difficult to assess whether
running wipe on a given file will actually wipe it -- it depends on an
awful lot of factors, such as : the type of file system the file
resides on (in particular, whether the file system is a journaling one
or not), the type of storage medium used, and the least significant bit
of the phase of the moon.
I sleep at night knowing that the teams of highly effective government data recovery personnel that Hollywood portrays will never exist.
Of course this shifts the trust to the computing system,
the CPU, and so on. I guess there are also "traps" in the
CPU and, in fact, in every sufficiently advanced mass-
marketed chip. Wealthy nations can find those. Therefore
these are mainly used for criminal investigation and
"control of public dissent".
Well, there was/is that trusted computing thing, i.e. putting DRM on the PC in hardware. The benefit would be denying encryption from terrorists/pirates/dissenters, like he just said. Interesting data can be stored just about anywhere, from phoning home to hidden partitions to having a small flash on the BIOS, depending on what's deemed interesting (logs, sector addresses, cryptographic keys, painted files).
Funny, I was just having a conversation today with a friend who works for the company selling http://www.whitecanyon.com/wipedrive-erase-hard-drive.php and remarking that it may keep your data safe from identity thieves, it wouldn't protect you from the FBI.
Did he explain more specifically why he thinks it would "protect you from the FBI"?
The idea that they need actual evidence to lock you up is so 20th century...
*I can't edit my previous message anymore, but the "would" should read as "wouldn't".
To everybody spending thought on this: Relax, nobody is interested in your porn collection.
I've used such tools to wipe my hard drives before selling or donating used computers (although I would use them before disposing of a hard drive in any way, really). I have heard of people buying up used computers and hard drives and extracting saved passwords from them.
A lot of people are smart enough to format it quickly, but not everyone will let (or know to) a computer sit for dozens of hours so that things might be wiped more securely.
A lot of people are smart enough to format it quickly, but not everyone will let (or know to) a computer sit for dozens of hours so that things might be wiped more securely.
I've had the experience where people agree to buy my broken computers for parts and the deal is done, until I mention that the "drives are wiped and ready to go". Then I never hear from them again.
Granted, somehow visiting the craigslist site turns ordinary people into flakes, but the "wiped" statement correlates a bit too well with the vanishing interest.
Granted, somehow visiting the craigslist site turns ordinary people into flakes, but the "wiped" statement correlates a bit too well with the vanishing interest.
+1, but I'd stop mentioning it after the first time (unless of course you're just doing this as an experiment and aren't really interested in selling the stuff)
I generally recommend DBAN[1] for people who are disposing of old boxes.
[1]http://www.dban.org/
[1]http://www.dban.org/
Does anyone know if any such FOSS works with SSDs and their wear-levelling of their writes?
wipe is rather unlikely to overwrite the bits on an SSD. Even with TRIM (which wipe won't use, btw), the disk will prefer using a blank sector to one it has to erase. Reading the currently unmapped spare sectors is probably something anyone with a solder gun can do.
An alternative could be to overwrite the disk with null bytes.
dd if=/dev/zero of=/dev/hda bs=1M
....multiple times.
Overwriting it once prevents software reconstruction of the data, but magnetic analysis of the underlying disk itself can reveal (depending on the voltage returned by the resulting 0 or 1) whether the previous value was (within a degree of certainty) a 0 or 1.
Overwriting it once prevents software reconstruction of the data, but magnetic analysis of the underlying disk itself can reveal (depending on the voltage returned by the resulting 0 or 1) whether the previous value was (within a degree of certainty) a 0 or 1.
You may find this interesting: http://www.nber.org/sys-admin/overwritten-data-guttman.html
It's a counter argument to the urban legend that says data should be wiped multiple times to be truly deleted.
It's worth noting that with today's disks' PRML channels, the signal is barely there already. It's merely "guessed" at (ML in PRML stands for maximum likelyhood). It's seems crazy that anyone could recover something after it's been overwritten. Maybe in the past, but not any more.
It's a counter argument to the urban legend that says data should be wiped multiple times to be truly deleted.
It's worth noting that with today's disks' PRML channels, the signal is barely there already. It's merely "guessed" at (ML in PRML stands for maximum likelyhood). It's seems crazy that anyone could recover something after it's been overwritten. Maybe in the past, but not any more.
It's a legitimate concern given the potential, but perhaps not a realistic concern.
Consider though, the data is digital, and error corrected, but it is written onto a fundamentally analog medium. By reading the medium you can easily determine the last written data which then lets you determine to a rather high precision the signal that was used to write that data (because it's all digital). That then allows you to subtract that signal from your analysis without leaving a ton of residual noise. Now perhaps there isn't much left after that, but what is left will be designed to be read even in the midst of noise, because it employs error correction. Who knows what the theoretical limit of such detection is with state of the art technology.
The evidence does tend to argue against any similar techniques of data recovery being used in practice anywhere today. Does that mean you should feel safe?
Much of Feenberg's argument here rests on technology. STM, MFM scanning. Image storage and processing. Tens of terabytes of data would need to be captured and processed, etc. Technology is not static though. What is the likelihood that there will be significant advances in STM/MFM scanning in the near future? In image storage and processing? In storage capacities in the ten terabyte range? For all of these it's a near certainty that we will continue to see exponential advances for the foreseeable future.
So perhaps abandoning your hard-drive that has been "wiped" once to the vagaries of the world is a safe bet today. But what happens in 10, 20, 30 years when all of those technologies have advanced remarkably and it is not only possible but perhaps even trivial to recover data on such drives? That is the conundrum.
Generally speaking, if you think your drives have contained material which you do very much wish to remain confidential in perpetuity, it probably makes sense to destroy old hard-drives rather than re-sell them. Though the cost/benefit trade-off may be a bit different if you are a business with a lot of data.
Consider though, the data is digital, and error corrected, but it is written onto a fundamentally analog medium. By reading the medium you can easily determine the last written data which then lets you determine to a rather high precision the signal that was used to write that data (because it's all digital). That then allows you to subtract that signal from your analysis without leaving a ton of residual noise. Now perhaps there isn't much left after that, but what is left will be designed to be read even in the midst of noise, because it employs error correction. Who knows what the theoretical limit of such detection is with state of the art technology.
The evidence does tend to argue against any similar techniques of data recovery being used in practice anywhere today. Does that mean you should feel safe?
Much of Feenberg's argument here rests on technology. STM, MFM scanning. Image storage and processing. Tens of terabytes of data would need to be captured and processed, etc. Technology is not static though. What is the likelihood that there will be significant advances in STM/MFM scanning in the near future? In image storage and processing? In storage capacities in the ten terabyte range? For all of these it's a near certainty that we will continue to see exponential advances for the foreseeable future.
So perhaps abandoning your hard-drive that has been "wiped" once to the vagaries of the world is a safe bet today. But what happens in 10, 20, 30 years when all of those technologies have advanced remarkably and it is not only possible but perhaps even trivial to recover data on such drives? That is the conundrum.
Generally speaking, if you think your drives have contained material which you do very much wish to remain confidential in perpetuity, it probably makes sense to destroy old hard-drives rather than re-sell them. Though the cost/benefit trade-off may be a bit different if you are a business with a lot of data.
You should just switch to an SSD and not worry about it anymore
SSDs store data by tunneling charges onto and off of floating gates. I strongly suspect that a TLA entity can recover data from the residual charges just as readily as they can from residual magnetism on a spinning media drive.
Having said that, like many others in this discussion, I'm skeptical of how practical data recovery really is vs. a theoretical issue. I'm guessing that the value of the data has to be extremely high before it would be worth while going to the necessary lengths.
It is usually much easier to use a $5 wrench, ref: http://xkcd.com/538/.
Having said that, like many others in this discussion, I'm skeptical of how practical data recovery really is vs. a theoretical issue. I'm guessing that the value of the data has to be extremely high before it would be worth while going to the necessary lengths.
It is usually much easier to use a $5 wrench, ref: http://xkcd.com/538/.
Wait - how automated is magnetic analysis? How much would it cost to recover a gigabyte disk, for example? What about non-spinning disks, are they cheaper or more expensive?
I recently watched a talk from 27C3 [1] about data recovery by the CEO of just such a data recovery firm [2]. (Very interesting with a rough outline of what they do and many cool anecdotes from the field.)
He more or less said that recovering all the data from a hard drive that cannot be read with its read/write head (either the original or a spare) would not be feasible or economical for his company. He said it would take several years. (And that’s without even considering overwritten data.)
When asked what’s the best way to destroy data forever he said that overwriting the data once is sufficient in any case. I take from this that at least his company and presumably other data recovery companies cannot read data that was overwritten once, even in principle (i.e. it’s not just a question of throwing enough resources at it). I have my doubts that the government has capabilities beyond that.
[1] Link to mp4 (h.264): http://mirror.fem-net.de/CCC/27C3/mp4-h264-HQ/27c3-4231-en-d...; link to slides: http://events.ccc.de/congress/2010/Fahrplan/events/4231.en.h...
[2] http://www.attingo.com/en/
He more or less said that recovering all the data from a hard drive that cannot be read with its read/write head (either the original or a spare) would not be feasible or economical for his company. He said it would take several years. (And that’s without even considering overwritten data.)
When asked what’s the best way to destroy data forever he said that overwriting the data once is sufficient in any case. I take from this that at least his company and presumably other data recovery companies cannot read data that was overwritten once, even in principle (i.e. it’s not just a question of throwing enough resources at it). I have my doubts that the government has capabilities beyond that.
[1] Link to mp4 (h.264): http://mirror.fem-net.de/CCC/27C3/mp4-h264-HQ/27c3-4231-en-d...; link to slides: http://events.ccc.de/congress/2010/Fahrplan/events/4231.en.h...
[2] http://www.attingo.com/en/
It's expensive as hell :)
The companies that do it charge several thousand an hour, IIRC. (this was back in the 90s?)
Depending on the non-spinning disk type in question, it can be either more or less secure than the usual magnetic HDD. You've got "flash" but that's just a nice word for any number of highly-differing technologies such as MLC and SLC on the inside. And you have NAND vs NOR techs to consider as well.
Perhaps the security in non-magnetic-HDDs comes from the fact that they're so new to the table, not many specialize in restoring data from them.
The companies that do it charge several thousand an hour, IIRC. (this was back in the 90s?)
Depending on the non-spinning disk type in question, it can be either more or less secure than the usual magnetic HDD. You've got "flash" but that's just a nice word for any number of highly-differing technologies such as MLC and SLC on the inside. And you have NAND vs NOR techs to consider as well.
Perhaps the security in non-magnetic-HDDs comes from the fact that they're so new to the table, not many specialize in restoring data from them.
Could you give the name of one of these companies?
Here's one: http://www.krollontrack.com/
I have a script that mounts a tiny ramdisk, fetches a statically compiled shred binary I've compiled, and proceeds to shred each disk attached to the system up to 25 times. I generally use this when decommissioning servers not under my direct control.
Couldn't you just use DBAN? http://www.dban.org/
I can't remotely boot servers using a custom disk image unless the provider in question offers KVM-over-IP or something similar.
What happens if you only overwrite a bit if the bit that was there before writing is different? This way your not overwriting every bit, so guessing at the previous bit can lead you astray. Or is that impossible?
It's impossible, because of the way data is encoded before hitting the platters : http://en.wikipedia.org/wiki/Run_Length_Limited
Yeah, I don't think hard drives work like that. You can modally read or write at each head, not simultaneously, and the granularity has to be larger than a bit (maybe a byte?)
Ah good point +1. Did not think about that. Maybe a single pass from
/dev/urandom
[deleted]
Do you have any sources for this? I've heard about this before but I've never seen any details about how it works.
It's explained by Niels Ferguson and Bruce Schneier in Practical Cryptography in section 22.10.2, "Magnetic Storage".
Overwriting does not completely destroy old data. You can think of it as repainting a wall with a single coat of paint. You can still vaguely see the old coat of paint under it. The magnetic domains can also migrate away from the read/write head either to the side of the track or deeper down into the magnetic material, where they can linger for a longer time. Overwritten data is typically not recoverable with the normal read/write head, but an attacker who takes apart a disk drive and uses specialized equipment might be able to retrieve some or all of the old data.
They also advocate multiple, random overwrites using fresh data as a best practice at this time.
Overwriting does not completely destroy old data. You can think of it as repainting a wall with a single coat of paint. You can still vaguely see the old coat of paint under it. The magnetic domains can also migrate away from the read/write head either to the side of the track or deeper down into the magnetic material, where they can linger for a longer time. Overwritten data is typically not recoverable with the normal read/write head, but an attacker who takes apart a disk drive and uses specialized equipment might be able to retrieve some or all of the old data.
They also advocate multiple, random overwrites using fresh data as a best practice at this time.
It seems like that should get harder and harder as the density of disks increases. I wonder how different the ease of recovery is for drives made now compared to when that book was written.
What about overwriting it with random data?
That is nsufficient to be 100% sure that all our data gets overwritten. The disk's firmware could discover that a block is unreliable and never write to it again. Such a block could contain recoverable data.
Also, I am not sure that this is guaranteed to overwrite the last part of a disk whose size is not a multiple of 1M. I guess that will depend on how eagerly the device detects ENOSPC conditions.
Also, I am not sure that this is guaranteed to overwrite the last part of a disk whose size is not a multiple of 1M. I guess that will depend on how eagerly the device detects ENOSPC conditions.
[deleted]
For those on Mac OS X (and perhaps *BSD?), we have srm.
The Secure Empty Trash function is a frontend for this IIRC.
The Secure Empty Trash function is a frontend for this IIRC.
I wish there was a version that only overwrite once with 1's. That would be best for my SSD. (Which has wear leveling and is formatted with a journalling filesystem anyhow, so perhaps it's moot.)
But what to do against those brain-wave readers? I better start thinking encrypted!
I love this section:
"I strongly recommend to call wipe directly on the corresponding block device with the appropriate options. However THIS IS AN EXTREMELY DANGEROUS THING TO DO. Be sure to be sober."
"I strongly recommend to call wipe directly on the corresponding block device with the appropriate options. However THIS IS AN EXTREMELY DANGEROUS THING TO DO. Be sure to be sober."
How would a hard disk controller detect encrypted data? It is essentially a (psudo-) random stream of bytes. Even if the controller is programmed to recognize such streams, it would be easy to first wipe disk by writing small random files all over and then wiping with 0/1 pattern. So any cached data is essentially worthless.
When I was younger, I erased a harddrive by covering it with burning thermite. It was pretty impressive, melting the drive, turning the sand underneath to glass, and burning through the asbestos pad into the cement.
I use DBAN to wipe hard drives before disposal. If you are replacing a failed drive, the only reasonable thing you can do is physically destroy the platters.
Secondly, new drives reserve a percentage of the room (invisible to the user), in case some of the sectors go bad the controller will re-map them transparently to new sectors. This might leave old data in the old sectors, where you can't normally see it but an investigator armed with the proper ATA commands can. (This isn't a conspiracy of the government and drive manufacturers, it's all there in the ATA spec.) The correct way to securely erase a drive is to send the drive the SECURITY-ERASE command. The drive controller will securely erase every part of the drive. https://ata.wiki.kernel.org/index.php/ATA_Secure_Erase The NSA actually recommends this to other government agencies, so it's probably OK.