AI doesn’t understand context either — it can’t tell the difference between an innocent photo of a baby in a bathtub with a parent, a telehealth photo, or something malicious. Google is using AI in addition to hashing, and both systems can get it wrong. With AI you’re always dealing with confidence levels, not certainty. No model in history has ever had 100% confidence on anything.
A scanning system will never be perfect. But there is a better approach: what the FTC now requires Pornhub to do. Before an image is uploaded, the platform scans it; if it’s flagged as CSAM, it simply never enters the system. Platforms can set a low confidence threshold and block the upload entirely. If that creates too many false positives, you add an appeals process.
The key difference here is that upload-scanning stops distribution before it starts.
What Google is doing is scanning private cloud storage after upload and then destroying accounts when their AI misfires. That doesn’t prevent distribution — it just creates collateral damage.
It also floods NCMEC with automated false reports. Millions of photos get flagged, but only a tiny fraction lead to actual prosecutions. The system as it exists today isn’t working for platforms, law enforcement, or innocent users caught in the blast radius.
I’m not a CSAM-detection expert, but after my suspension I ended up doing a lot of research into how these systems work and where they fail. And one important point: Google isn’t just using PhotoDNA-style perceptual hashing.
They’re also running AI-based classifiers on Drive content, and that second layer is far more opaque and far more prone to false positives.
That’s how you get situations like mine: ~700 problematic images in a ~700k-image dataset triggered Google to delete 130,000+ completely unrelated files and shut down my entire developer ecosystem.
Hash-matching is predictable.
AI classification is not.
And Google’s hybrid pipeline:
isn’t independently vetted
isn’t externally audited
isn’t reproducible
has no recourse when it’s wrong
In practice, it’s a black box that can erase an innocent researcher or indie dev overnight. I wrote about this after experiencing it firsthand — how poisoned datasets + opaque AI detection create “weaponized false positives”:
https://medium.com/@russoatlarge_93541/weaponized-false-posi...
I agree with the point above: if open, developer-accessible perceptual hashing tools existed — even via bloom filters or ZK membership proofs — this entire class of collateral damage wouldn’t happen.
Instead, Big Tech keeps the detection tools proprietary while outsourcing the liability to everyone else. If their systems
are wrong, we pay the cost — not them.
I’m the person who got banned. And just to be clear: the only reason I have my account back is because 404 Media covered it. Nobody else would touch the story because it happened to a nobody. There are probably a lot of “nobodies” in this thread who might someday need a reporter like Emanuel Maiberg to actually step in. I’m grateful he did.
The dataset had been online for six years. In my appeal I told Google exactly where the data came from — they ignored it. I was the one who reported it to C3P, and that’s why it finally came down. Even after Google flagged my Drive, the dataset stayed up for another two months.
So this idea that Google “did a good thing” and 404 somehow did something wrong is just absurd.
Im the developers who actually got banned because of this dataset. I used NudeNet offline to benchmark my on-device NSFW app Punge — nothing uploaded, nothing shared.
Your dataset wasn’t the problem. The real problem is that independent developers have zero access to the tools needed to detect CSAM, while Big Tech keeps those capabilities to itself.
Meanwhile, Google and other giants openly use massive datasets like LAION-5B — which also contained CSAM — without facing any consequences at all. Google even used early LAION data to train one of its own models. Nobody bans Google.
But when I touched NudeNet for legitimate testing, Google deleted 130,000+ files from my account, even though only ~700 images out of ~700,000 were actually problematic. That’s not safety — that’s a detection system wildly over firing with no independent oversight and no accountability.
Big Tech designed a world where they alone have the scanning tools and the immunity when those tools fail. Everyone else gets punished for their mistakes.
So yes — your dataset has done good. ANY data set is subject to this. There needs to be tools and process for all.
But let’s be honest about where the harm came from: a system rigged so only Big Tech can safely build or host datasets, while indie developers get wiped out by the exact same automated systems Big Tech exempts itself from.
Couldn't agree more. People thought that may happen when Google lost its case against the DOJ, but they got a tap on the wrist. Hopefully independent developers will file FTC complaints and prompt action.
Thank you so much -- it's tough for people to grasp that why i wrote the Medium post. To show the trend against amongst and the historical context of Big Tech icing out competition
You hit the nail on the head. Google now has buts on a hair trigger banning developers for life--I filed a complaint with the FTC, I hope all developers do
On July 31, Google permanently disabled my account after alleging CSAM was in my Drive. The trigger: I had downloaded and unzipped NudeNet, a dataset hosted on Academic Torrents, which has been used in multiple academic studies.
When Google briefly restored my account weeks later, I discovered that ~136k files had been removed from the “nude” folder — but I was never told which ones. Using Drive Activity logs, I identified at least two specific file IDs that existed in July but were gone after restoration. Then my account was suspended again before I could complete full verification.
This creates a rare opportunity: * If these images are truly CSAM, the dataset owner needs to be notified immediately so researchers stop using it. * If they are false positives, that casts serious doubt on the reliability of Google’s detection methods — and highlights the risks for any researcher working with sensitive datasets.
I’m asking other researchers who have downloaded NudeNet to help: 1. Cross-check whether the missing file IDs I identified exist in your local copies. 2. If found, review whether they are actual violations or misclassifications. 3. If violations exist, notify the dataset owner. If not, this may be a critical case study of a false positive at scale.
It’s important because, in an era where science and research are under increasing scrutiny, researchers need places where they can safely download and share datasets without fear of retribution. If platforms or automated systems treat legitimate research datasets as contraband, it threatens not just one account, but the broader ecosystem of open science.
I built an on-device NSFW detector and used NudeNet, a public dataset hosted on Academic Torrents and cited in multiple research papers, to benchmark it. After uploading the dataset to Google Drive, my account was permanently suspended for “inappropriate material.”
When Google briefly reinstated my account, I discovered 130,000+ files had been silently deleted compared to the original ZIP. While verifying with the Drive Activity API, I was suspended again.
Before losing access, I confirmed actual filenames Google flagged — something they never disclose to users. Example (filename only — not an image):
nude_sexy_safe_v1_x320/training/nude/prefix_reddit_sub_latinasgw_2017! Can't believe it. Feliz año nuevo!-.jpg
This file exists in the public NudeNet dataset, which is widely cited in academic content moderation research.
Because in research circles, it is a good idea — or at least standard practice. Academic Torrents is a public data-sharing platform run for researchers, and the dataset is cited in peer-reviewed AI papers and used in university projects.
I wasn’t browsing porn — I was benchmarking an on-device NSFW detection model. I never reviewed every file; my workflow was automated preprocessing. The only “mistake” was unzipping it in Google Drive, not realizing Google scans those files automatically and can flag them without context.
If Google truly cares about stopping harmful content, why not give researchers the exact file names so they can verify and have dataset maintainers remove it? That’s how you improve the data and prevent future issues — assuming it was a violation at all and not a false positive.
The real issue is that whether it’s for research, unintentional, or a false positive, there’s no fair process outside of Google to resolve it. Instead of an immediate, irreversible account ban, they could quarantine the suspected files while preserving access to the rest of the account. That would protect both children and innocent users.
You’re making a serious accusation without evidence. I don’t know if the file in question was CSAM—because Google won’t tell me. We just have Google’s word for it.
What I do know is that I used publicly available datasets—like NudeNet and COCO—for benchmarking an NSFW detection model. These datasets have been used by researchers for years. If there was anything questionable in them, that should be addressed openly—not used as a reason to erase someone’s digital life without transparency or due process.
I’m not claiming academic immunity. I’m saying it was unintentional, and I had no idea anything I processed could have been flagged. And if there was something problematic in a dataset, I would absolutely want to identify and report it—but I can’t do that if Google refuses to disclose what was flagged.
This isn’t just about me. When a private company has the unchecked power to destroy someone’s livelihood overnight, it stops being a simple platform—it becomes a de facto utility. There needs to be a path for review, correction, and accountability.
Google won’t tell me the name of the file—or files—they believe violated their policy. So IT IS baseless speculation. I only know what I was doing prior to the account deletion—and I’m guessing that’s what triggered the issue.
Google should at minimum publish the filenames of the content they consider violations. I have Colab notebooks for everything I’ve downloaded from the internet. With a filename, I could trace the origin and have the image independently reviewed.
I suspect it wasn’t CSAM at all. We know nothing about how their detection system or hashing works. I’d also like to point out that I benchmarked my model against Google’s own commercially available NSFW detector—and mine actually performed better:
https://medium.com/@russoatlarge/benchmark-study-punge-yolov...
As for datasets: COCO is the most widely used dataset in computer vision, with hundreds of thousands of labeled images. One image in the COCO Train2017 set—000000001790.jpg—shows a child brushing his teeth while looking in the mirror in his “birthday suit.” But if Google's system flagged that as CSAM, then every researcher using COCO is at risk.
If Google genuinely cared about addressing CSAM responsibly, they would share the filenames so they could be independently verified and—if needed—removed from circulation. Instead, they’re silent. That silence creates fear, not safety.
Let me be clear: I deeply appreciate Google’s efforts to detect and report CSAM. These systems are vital for protecting children and stopping abuse, and I fully support that mission.
But good intentions are not a substitute for fairness. Right now, individuals who are falsely flagged have no way to defend themselves, no way to clear their name, and no meaningful path to restore their livelihood.
If a system is powerful enough to destroy someone’s life, it should also be strong enough to offer transparency, review, and correction. That’s all I — and others like me — are asking for: a fair process to protect the innocent while continuing the fight against real harm."
I'm a real person — this actually happened to me last Thursday.
My name is Mark Russo. I’m not going to be shamed or erased. I created an app called Punge that’s live on both iOS and Android. The article wasn't written by ChatGPT — it's my story. I used ChatGPT to help tighten the writing, but every word reflects what I lived through.
Since this happened, I’ve reached out to lawyers, media, lawmakers, and professional contacts — even people I know at Google. I’ve lost access to everything tied to my account: Gmail, Firebase, AdMob, and the infrastructure that powers my apps. Google won’t even tell me which file triggered the CSAM flag.
Even if — by some miracle — I get my account back, I’m committed to making sure this kind of thing doesn’t keep happening. No one should lose their digital life overnight, with no transparency, no due process, and no way to defend themselves.
According to this NYT article: https://www.nytimes.com/2022/12/30/technology/google-appeals... — people who manage to claw their way out of this nightmare usually do so only after getting media attention. A CBS reporter has shown some interest. If anyone here has been through something similar, message me on LinkedIn — I’ll do whatever I can to help share your story too.
A scanning system will never be perfect. But there is a better approach: what the FTC now requires Pornhub to do. Before an image is uploaded, the platform scans it; if it’s flagged as CSAM, it simply never enters the system. Platforms can set a low confidence threshold and block the upload entirely. If that creates too many false positives, you add an appeals process.
The key difference here is that upload-scanning stops distribution before it starts.
What Google is doing is scanning private cloud storage after upload and then destroying accounts when their AI misfires. That doesn’t prevent distribution — it just creates collateral damage.
It also floods NCMEC with automated false reports. Millions of photos get flagged, but only a tiny fraction lead to actual prosecutions. The system as it exists today isn’t working for platforms, law enforcement, or innocent users caught in the blast radius.