What It Takes to Truly Delete Data(fivethirtyeight.com)
fivethirtyeight.com
What It Takes to Truly Delete Data
https://fivethirtyeight.com/features/what-it-takes-to-truly-delete-data/
4 comments
The society part is tough and frustrating. I get so tired of arguing with clerks that want my email, zip code, phone #, etc. I don't want to give everyone that, and it really slows down the checkout process in general. Just take my damn money.
A few weeks ago, I walked away from a purchase because the store wanted to take my photo at POS. This was my first encounter with the store, I probably would not be a repeat customer, I just wanted to do a single purchase. Just take my damn money.
The stupid part is, I typically use the same credit card for every purchase. I know they could use that to query info product and know everything about me - and, I don't really care about that (I've given in to it). It's the constant asking for info by companies that I have no ongoing relationship with that I really hate.
Edit: Also, it makes me feel like some weirdo freak when I don't want to participate. It starts to cause a scene. Other customers are staring. My wife is embarrassed. Everyone is thinking, "just let them take the photo" or "the clerk is just doing their job".
A few weeks ago, I walked away from a purchase because the store wanted to take my photo at POS. This was my first encounter with the store, I probably would not be a repeat customer, I just wanted to do a single purchase. Just take my damn money.
The stupid part is, I typically use the same credit card for every purchase. I know they could use that to query info product and know everything about me - and, I don't really care about that (I've given in to it). It's the constant asking for info by companies that I have no ongoing relationship with that I really hate.
Edit: Also, it makes me feel like some weirdo freak when I don't want to participate. It starts to cause a scene. Other customers are staring. My wife is embarrassed. Everyone is thinking, "just let them take the photo" or "the clerk is just doing their job".
Wow, this is the first I've heard of photos being taken at a register. Do you mind sharing what store this was, or if it was triggered by the purchase of a specific item?
I realize this request to reveal where you shopped, or what item you bought, is in response to your objection to being tracked, so I understand if you'd rather not say :)
I realize this request to reveal where you shopped, or what item you bought, is in response to your objection to being tracked, so I understand if you'd rather not say :)
WTF store wants to take your photo just to sell you something? I can't possibly imagine submitting to that unless it was for something amazing or something where the photography was obviously necessary to use the product.
It was just a local business. Clothing boutique. Seemed like the thought was build me a profile, they would send curated picks that fit my style. And of course people with profile pics are more engaged!
The in store onboarding was actually a smart idea. Things like that are applauded by HN folks like us. Increasing LTV,... However, the clerk told me a pic was mandatory. I even said something like "take a picture of the wall then" and he said no like he was offended I asked him to cheat the system. I just turned around and walked out of the shop.
I was just trying to buy a tshirt. I didn't want to establish and ongoing relationship with this place photo or not.
The in store onboarding was actually a smart idea. Things like that are applauded by HN folks like us. Increasing LTV,... However, the clerk told me a pic was mandatory. I even said something like "take a picture of the wall then" and he said no like he was offended I asked him to cheat the system. I just turned around and walked out of the shop.
I was just trying to buy a tshirt. I didn't want to establish and ongoing relationship with this place photo or not.
>Everyone is thinking, "just let them take the photo" or "the clerk is just doing their job".
Maybe, or maybe you made them second guess their own personal policy of what they are comfortable sharing. So long as you are confident I don't think a reasonable person would look down on you for protecting your privacy. I usually just say something like "I don't do that" when asked to share email/phone/zip with stores, never had a clerk get weird about it.
Maybe, or maybe you made them second guess their own personal policy of what they are comfortable sharing. So long as you are confident I don't think a reasonable person would look down on you for protecting your privacy. I usually just say something like "I don't do that" when asked to share email/phone/zip with stores, never had a clerk get weird about it.
I usually do that too. 99% not an issue. But as it gets more pervasive and they ask for more and more. I have encountered at least a dozen places that the clerks workflow can not proceed until I comply. So usually it's a "I don't do that" followed by "it's required" to which I reply "no it's not" and it's down hill from there. Sometimes they're smart enough to feed the machine dummy data and not argue with me. Usually it makes me mad so I don't exactly handle it with grace either (stubbornness I don't get rude or abusive)
Vendors are prohibited by contract from using purchase information tied to your credit card. That's why they have store-specific loyalty cards.
There are also some other things I find people have trouble understanding.
First thing that one needs to understand: you always leak data. It's a feature of reality itself. At the very least, every second of your life, the record of where you are and what you do radiates away from you at the speed of light. Of course interacting with matter, you leave stronger traces everywhere. Mechanical, thermal, accoustic. Most of the data isn't easily accessible in any meaningful way, but it doesn't change the fact it's always there.
Two: progress of technology increases our ability to access this data. There's no way to ensure that data is denied to all people other than stopping the development of new technologies. There's no meaningful way of preventing anyone from accessing the data that's available today other than rolling back the Industrial Revolution and going back to living in caves. Science and technology lets us see the physical world in more details, faster and cheaper, and it also lets us correlate information faster and cheaper. There's no way around that.
So that's about "metadata". As for "personally identifiable information", I may avoid giving some of it to random strangers, but I do assume that PII is public information. Because, again, there's no meaningful way to contain their spread. So pretending to yourself that any PII is private would be stupid.
First thing that one needs to understand: you always leak data. It's a feature of reality itself. At the very least, every second of your life, the record of where you are and what you do radiates away from you at the speed of light. Of course interacting with matter, you leave stronger traces everywhere. Mechanical, thermal, accoustic. Most of the data isn't easily accessible in any meaningful way, but it doesn't change the fact it's always there.
Two: progress of technology increases our ability to access this data. There's no way to ensure that data is denied to all people other than stopping the development of new technologies. There's no meaningful way of preventing anyone from accessing the data that's available today other than rolling back the Industrial Revolution and going back to living in caves. Science and technology lets us see the physical world in more details, faster and cheaper, and it also lets us correlate information faster and cheaper. There's no way around that.
So that's about "metadata". As for "personally identifiable information", I may avoid giving some of it to random strangers, but I do assume that PII is public information. Because, again, there's no meaningful way to contain their spread. So pretending to yourself that any PII is private would be stupid.
There is a contradiction in your line of thinking between:
> data isn't easily accessible in any meaningful way
and
> no meaningful way to contain their spread
If A is not readable then there is no need to contain it.
> data isn't easily accessible in any meaningful way
and
> no meaningful way to contain their spread
If A is not readable then there is no need to contain it.
The first statement you quoted refers to the data you constantly generate, its accessibility being predicated on the level of technology available to humans. The second one refers to the PII - stuff like name, address, SSN, credit card number, etc. - all of which you constantly give away to random parties.
I really don't give out that info on a regular basis. Unlike the first example you can for example pay in cash, which is not necessarily anonymous, but generally acts that way. Store loyalty cards are another example of something used to track people you really don't need to use.
Now, I do carry a cellphone around so that exists. But, the vast majority of organisations have no access to that information.
Now, I do carry a cellphone around so that exists. But, the vast majority of organisations have no access to that information.
If you buy online, you give most of thay information. Maybe US is different, but in Poland getting any kind of service from a bank (e.g. credit) or telco (e.g. new phone contract) involves filling in paperwork that asks for your official ID number and (equivalent of) SSN, etc. I don't believe a random branch of a random company in a random city is secure and trustworthy enough to not have PII leak out. Hell, even libraries have that stuff - at least one used my SSN as a default password in the past. And in fact, in Poland we had cases of PII being available to people who shouldn't have access to them due to bugs in official systems.
It's impossible to be "relevant" without giving away your privacy these days. Doesn't help that there are dozens, if not hundreds of parties with supposedly private information like your SS number, and the odds that all of them (or even a significant proportion of them) have strong data security measures are next to nil.
I think that the issue is not that we leak data, but that in the last 10 years, it's exponentially more accessible. There was a time when finding out who I interacted with would have required many man hours and probably some court orders. Now a carelessly adding a friend on Facebook can do so. Even if I choose not to do business with large companies, so many other people can leak my data (pictures on Facebook, emails on Google servers, etc). And, of course, the ability to algorithmicly correlate it. The only answer here is strong legal protections across borders, and I'm afraid I don't see that on the horizon.
If @realDonaldTrump deletes a tweet, it's against the Twitter ToS to republish it using a bot.
When it does happen I've seen people post screenshots of the old tweet. This doesn't prove anything more than trivial knowledge of the Chrome Web Inspector.
I've thought about distributing a program so people can keep their own local archives of important tweets. There are plenty of ways to achieve this already, but that's besides my point.
How can we make it obvious when an important figure deletes a tweet without breaking Twitter's ToS?
When it does happen I've seen people post screenshots of the old tweet. This doesn't prove anything more than trivial knowledge of the Chrome Web Inspector.
I've thought about distributing a program so people can keep their own local archives of important tweets. There are plenty of ways to achieve this already, but that's besides my point.
How can we make it obvious when an important figure deletes a tweet without breaking Twitter's ToS?
I've found the answer to my own question:
https://projects.propublica.org/politwoops/user/realDonaldTr...
Back story:
https://techcrunch.com/2016/02/09/u-s-version-of-politwoops-...
https://projects.propublica.org/politwoops/user/realDonaldTr...
Back story:
https://techcrunch.com/2016/02/09/u-s-version-of-politwoops-...
Twitter can falsify tweets too. At the bottom of it all, all anyone has is their own memories and records (both can be tampered!) and whatever trust they place in other people's records. That's the history of "escrow".
> doesn't prove anything more than trivial knowledge of the Chrome Web Inspector
That might be generous. They could have used Photoshop as well and had no technical knowledge at all.
That might be generous. They could have used Photoshop as well and had no technical knowledge at all.
Inspector makes it practically impossible to prove wrong. You literally cannot get the font, positioning, kerning, or spacing wrong with the replaced text.
Don't use Twitter to publish that information?
The issue is with consuming the tweets. Publishing the data somewhere else has the same problems as publishing on Twitter.
1. The Twitter ToS prevents scraping the content in section 4.iii.
https://twitter.com/tos?lang=en
2. If you use the API instead, they have this rule:
> Only surface Twitter activity as it surfaced on Twitter. For example, your Service should execute the unlike and delete actions by removing all relevant Content, not by publicly displaying to other users that the Tweet is no longer liked or has been deleted.
https://dev.twitter.com/overview/terms/agreement-and-policy#...
1. The Twitter ToS prevents scraping the content in section 4.iii.
https://twitter.com/tos?lang=en
2. If you use the API instead, they have this rule:
> Only surface Twitter activity as it surfaced on Twitter. For example, your Service should execute the unlike and delete actions by removing all relevant Content, not by publicly displaying to other users that the Tweet is no longer liked or has been deleted.
https://dev.twitter.com/overview/terms/agreement-and-policy#...
While its still no silver bullet, encryption can make the problem more manageable. For example for hard drives with full-disk encryption you should be able to just throw away the key and the data should become inaccessible. Same applies to for cloud backups; you don't know what the cloud provider might do with the data you upload, but they should not be able to access the content if it is properly encrypted, and as such the issues raised in the article become lesser.
Of course this does not apply to data that is actually being used/processed in cloud instead of just being passively stored.
Of course this does not apply to data that is actually being used/processed in cloud instead of just being passively stored.
>That computer, armed with the company’s proprietary wiping software, uses a special algorithm to programmatically access each bit on the drive, stored as either a ‘0’ or ‘1,’ and write a new digit to it. The process must be repeated at least three times and with different numbers to be up to DoD standards.
Maybe a stupid question, but why don't they just write every bit to either 1 or 0 regardless of its current content? Are they trying to preserve some data and not others? Why do you need to do it multiple times if the bit being written is independent of the actual contents?
Maybe a stupid question, but why don't they just write every bit to either 1 or 0 regardless of its current content? Are they trying to preserve some data and not others? Why do you need to do it multiple times if the bit being written is independent of the actual contents?
Drives store analog information and only present it in digital form to you. So a 1 might be a 0.8 and a 0 might be a 0.2. Writing all ones might turn bits that already were "1" to 0.9, while leaving bits that used to be "0" at 0.8.
It is generally accepted that it is impossible to recover data from a single wipe on modern hard drives. There is some mention of it here https://en.wikipedia.org/wiki/Data_erasure#Number_of_overwri... and also I believe I read somewhere that there's unclaimed prizes available for anyone who can prove otherwise (but I do not have a source available for that now).
What can bite you is the drive firmware silently remapping sectors, causing your write request to be mapped to a different sector altogether.
What can bite you is the drive firmware silently remapping sectors, causing your write request to be mapped to a different sector altogether.
There was a (possibly mistaken) belief that some data could be recovered with an electron microscope. AFAIK it is definitively not true with modern spinning drives and its the "AFAIK" portion of that statement that keeps people doing multiple wipes.
I think you might mean something like a Magnetic Force Microscopy.
Edit: It looks like there is something called lorentz microscopy that uses an electron beam.
Edit: It looks like there is something called lorentz microscopy that uses an electron beam.
>Why do you need to do it multiple times if the bit being written is independent of the actual contents?
There is a concern that the write process is not perfect, so the old value of the bit might still be visible under a microscope.
There is a concern that the write process is not perfect, so the old value of the bit might still be visible under a microscope.
It's not because "the bit might still be visible under a microscope" - it's because of something called (IIRC) "intra-track data".
Basically when a hard drive stores the data on the physical platter, it writes it as a series of concentric tracks. These tracks are spaced very close together. When data is written to the drive, the magnetic regions that are written on the track can "bleed over" to the intra-track regions, leaving a faint trace that with the proper equipment ($$$$) can be read, even if the tracks are all zeroed. Think of the magnetic regions as not being (say) "spikes", but rather "hills", where the greatest magnetic intensity is at the peak, tapering off as it reaches the edge/base of the hill. That energy might be small, but it is there.
By overwriting each track multiple times with varying data patterns, these leftover tale-tell magnetic leftovers are essentially pushed into the noise of the rest of the data, and are effectively "wiped". But you have to write the pattern multiple times, over all the tracks.
This effect has "recently" been used to an advantage (depending on how you look at it) for something called an "SMR Hard Drive" - or "Shingled Magnetic Recording":
https://en.wikipedia.org/wiki/Shingled_magnetic_recording
In this kind of hard drive, the tracks are purposefully placed very close together so that the magnetic regions purposefully overlap, allowing for a higher density on fewer platters (so, larger drives in a smaller space for a lower cost). However, they tend to be slower - and not recommended for purposes like RAID NAS or anything where you want speed and proper data retention (my understanding, at least).
Basically when a hard drive stores the data on the physical platter, it writes it as a series of concentric tracks. These tracks are spaced very close together. When data is written to the drive, the magnetic regions that are written on the track can "bleed over" to the intra-track regions, leaving a faint trace that with the proper equipment ($$$$) can be read, even if the tracks are all zeroed. Think of the magnetic regions as not being (say) "spikes", but rather "hills", where the greatest magnetic intensity is at the peak, tapering off as it reaches the edge/base of the hill. That energy might be small, but it is there.
By overwriting each track multiple times with varying data patterns, these leftover tale-tell magnetic leftovers are essentially pushed into the noise of the rest of the data, and are effectively "wiped". But you have to write the pattern multiple times, over all the tracks.
This effect has "recently" been used to an advantage (depending on how you look at it) for something called an "SMR Hard Drive" - or "Shingled Magnetic Recording":
https://en.wikipedia.org/wiki/Shingled_magnetic_recording
In this kind of hard drive, the tracks are purposefully placed very close together so that the magnetic regions purposefully overlap, allowing for a higher density on fewer platters (so, larger drives in a smaller space for a lower cost). However, they tend to be slower - and not recommended for purposes like RAID NAS or anything where you want speed and proper data retention (my understanding, at least).
It was difficult for me to learn this lesson personally. When I understood it, I deleted my Facebook account (best decision, btw), but I assume that even though my account was deleted there are still references to it all over. IDs in log lines, references not cleaned up -- enough to paint something of a picture of who I was and what I did with the site.
Now I'm more judicious in what I share, how I share it, which applications I give permissions. But I'm still leaking metadata everywhere. It's frustrating and terrifying that the cost of participating in our society is either a HUGE cognitive load in keeping an eye on all the data you emit or accepting that basically everyone will have full access to your (meta)data over time.