One Bad Apple(hackerfactor.com)
hackerfactor.com
One Bad Apple
https://www.hackerfactor.com/blog/index.php?/archives/929-One-Bad-Apple.html
524 comments
This reads like a failure of the NCMEC, and the legal system surrounding it.
It is insane that using perceptual hashes is likely illegal. As the hashes are actually somewhat reversible and so possession of the hash is a criminal offence. It just shows how twisted up in itself the law is in this area.
One independent image analysis service should not be beating reporting rates of major service providers. And NCMEC should not be acting like detection is a trade secret. Wider detection and reporting is the goal.
And the law as setup prevents developing detection methods. You cannot legally check the results of your detection (which Apple are doing), as that involves transmitting the content to someone other than the NCMEC!
It is insane that using perceptual hashes is likely illegal. As the hashes are actually somewhat reversible and so possession of the hash is a criminal offence. It just shows how twisted up in itself the law is in this area.
One independent image analysis service should not be beating reporting rates of major service providers. And NCMEC should not be acting like detection is a trade secret. Wider detection and reporting is the goal.
And the law as setup prevents developing detection methods. You cannot legally check the results of your detection (which Apple are doing), as that involves transmitting the content to someone other than the NCMEC!
> To reiterate: scanning your device is not a privacy risk, but copying files from your device without any notice is definitely a privacy issue.
Not a lawyer, but I believe this part about legality is inaccurate, because they aren’t copying your photos without notice. The feature is not harvesting suspect photos from a device, it is attaching data to all photos before they are uploaded to Apple’s servers. If you’re not using iCloud Photos, the feature will not be activated. Furthermore, they’re not knowingly transferring CSAM, because the system is designed only to notify them when a certain “threshold” of suspect images has been crossed.
In this way it’s identical in practice to what Google and Facebook are already doing with photos that end up on their servers, they just run the check before the upload instead of after. I certainly have reservations about their technique here, but this argument doesn’t add up to me.
Not a lawyer, but I believe this part about legality is inaccurate, because they aren’t copying your photos without notice. The feature is not harvesting suspect photos from a device, it is attaching data to all photos before they are uploaded to Apple’s servers. If you’re not using iCloud Photos, the feature will not be activated. Furthermore, they’re not knowingly transferring CSAM, because the system is designed only to notify them when a certain “threshold” of suspect images has been crossed.
In this way it’s identical in practice to what Google and Facebook are already doing with photos that end up on their servers, they just run the check before the upload instead of after. I certainly have reservations about their technique here, but this argument doesn’t add up to me.
Really nice explanation from someone who knows a thing or two about images/photos (Dr. Neal Krawetz is the creator of https://fotoforensics.com and specializes in computer forensics).
NCMEC has essentially shows that they have zero regard for privacy and called all privacy activists "screeching voices of the minority". At the same time, they're at the center point of a highly opaque, entrenched (often legally mandated) censorhip infrastructure that can and will get accounts shut down irrecoverably and possibly people's homes raided, on questionable data:
In one of the previous discussions, I've seen claims about the NCMEC database containing a lot of harmless pictures misclassified as CSAM. This post confirms this again (ctrl-f "macaque")
It also seems like the PhotoDNA hash algorithm is problematic (to the point where it may be possible to trigger false matches).
Now NCMEC seem to be pushing for the development of a technology that would implant an informant in every single of our devices (mandating the inclusion of this technology is the logical next step that seems inevitable if Apple launches this).
I'm surprised, and honestly disappointed, that the author seems to still play nice, instead of releasing the whitepaper. The NCMEC seems to have decided to position itself directly alongside other Enemies of the Internet, and while I can imagine that they're also doing a lot of important and good work, at this point, I don't think they're salvageable would like to see them disbanded.
Really curious how this will play out. I expect attacks either sabotaging these scanning systems by flooding them with false positives, or exploiting them to get the accounts of your enemies shut down permanently by sending them a picture of a macaque.
In one of the previous discussions, I've seen claims about the NCMEC database containing a lot of harmless pictures misclassified as CSAM. This post confirms this again (ctrl-f "macaque")
It also seems like the PhotoDNA hash algorithm is problematic (to the point where it may be possible to trigger false matches).
Now NCMEC seem to be pushing for the development of a technology that would implant an informant in every single of our devices (mandating the inclusion of this technology is the logical next step that seems inevitable if Apple launches this).
I'm surprised, and honestly disappointed, that the author seems to still play nice, instead of releasing the whitepaper. The NCMEC seems to have decided to position itself directly alongside other Enemies of the Internet, and while I can imagine that they're also doing a lot of important and good work, at this point, I don't think they're salvageable would like to see them disbanded.
Really curious how this will play out. I expect attacks either sabotaging these scanning systems by flooding them with false positives, or exploiting them to get the accounts of your enemies shut down permanently by sending them a picture of a macaque.
Good article, however-
"Due to how Apple handles cryptography (for your privacy), it is very hard (if not impossible) for them to access content in your iCloud account. Your content is encrypted in their cloud, and they don't have access. If Apple wants to crack down on CSAM, then they have to do it on your Apple device"
I do not believe this is true. Maybe one day it will be true and Apple is planning for it, but right now iCloud service data is encrypted in the sense that they are stored encrypted at rest and in transit, however Apple holds the keys. We know this given that iCloud backups have been surrendered to authorities, and of course you can log into the web variants to view your photos, calendar, etc. Not to mention that Apple has purportedly been doing the same hash checking on their side for a couple of years.
Thus far there has been no compelling answer as to why Apple needs to do this on device.
"Due to how Apple handles cryptography (for your privacy), it is very hard (if not impossible) for them to access content in your iCloud account. Your content is encrypted in their cloud, and they don't have access. If Apple wants to crack down on CSAM, then they have to do it on your Apple device"
I do not believe this is true. Maybe one day it will be true and Apple is planning for it, but right now iCloud service data is encrypted in the sense that they are stored encrypted at rest and in transit, however Apple holds the keys. We know this given that iCloud backups have been surrendered to authorities, and of course you can log into the web variants to view your photos, calendar, etc. Not to mention that Apple has purportedly been doing the same hash checking on their side for a couple of years.
Thus far there has been no compelling answer as to why Apple needs to do this on device.
>18 U.S.C. § 2258A is specific: the data can only be sent to NCMEC. (With 2258A, it is illegal for a service provider to turn over CP photos to the police or the FBI; you can only send it to NCMEC. Then NCMEC will contact the police or FBI.) What Apple has detailed is the intentional distribution (to Apple), collection (at Apple), and access (viewing at Apple) of material that they strongly have reason to believe is CSAM. As it was explained to me by my attorney, that is a felony.
I'm not sure, after reading the article, who is/has the most insane system of Apple or NCMEC.
I'm not sure, after reading the article, who is/has the most insane system of Apple or NCMEC.
To help fight back against false positives, why not just repeatedly trigger the code that sends the data to NCMEC (per the article's claimed legal requirements) and create a DoS attack?
There are a lot of articles about Apples hadh algorithm and for me they are mostly irrelevant to the main problem.
The main problem is that Apple has backdoored my device.
More types of bad images or other files will be scanned since now apple does not have plausible deniablity to defend any of ghe government’x requests.
In the future a false? positive that happened? to be of a political file that crept in the list can pin point people to the future dictator wannabe.
It’s always about the children or terrorism.
The main problem is that Apple has backdoored my device.
More types of bad images or other files will be scanned since now apple does not have plausible deniablity to defend any of ghe government’x requests.
In the future a false? positive that happened? to be of a political file that crept in the list can pin point people to the future dictator wannabe.
It’s always about the children or terrorism.
The author of this article purports to have done a ton of research into this system, but appears to have missed basic information that I’ve acquired from a few podcasts.
Namely the “1-in-a-trillion” false positives per account per year is based on the likelihood of multiple photos matching the database (Apple doesn’t say how many are required to trip their manual screening threshold).
Namely the “1-in-a-trillion” false positives per account per year is based on the likelihood of multiple photos matching the database (Apple doesn’t say how many are required to trip their manual screening threshold).
Question: who is or will be making money on this deal? Answer that ("follow the money") and then I think we'll have a handle on what's really going on.
> However, nothing in the iCloud terms of service grants Apple access to your pictures for use in research projects, such as developing a CSAM scanner. (Apple can deploy new beta features, but Apple cannot arbitrarily use your data.) In effect, they don't have access to your content for testing their CSAM system.
> If Apple wants to crack down on CSAM, then they have to do it on your Apple device.
I don’t understand… Apple can’t change their TOS but they can install this scanning service on your device?
> If Apple wants to crack down on CSAM, then they have to do it on your Apple device.
I don’t understand… Apple can’t change their TOS but they can install this scanning service on your device?
The engineers that worked on this should honestly be ashamed of themselves. We need some sort of oath of ethics in computer science.
This feels like missing the forest from the trees — Steve Jobs said many times to the effect ‘it doesn’t matter how any of this stuff happens, GigaHertz, Ram, Speeds, it only matters that the user gets what they want.’
Right now Apple’s biggest unhappy user is the DOJ. As it stands with the legislation coming down the pipe and both previous administrations building on a keenness to ‘get something done’ about big tech, Apple will do as they’ve done in China and ‘obey the laws in each jurisdiction.’
Right now there are a lot of unwritten laws that say Apple better play right or lose quite a bit more —
So, how it’s getting done is a side show.
That said, it wasn’t long ago that they stood toe to toe with the FBI —- but there also weren’t wonderfully strong ‘sanctions’ on the horizon.
Right now Apple’s biggest unhappy user is the DOJ. As it stands with the legislation coming down the pipe and both previous administrations building on a keenness to ‘get something done’ about big tech, Apple will do as they’ve done in China and ‘obey the laws in each jurisdiction.’
Right now there are a lot of unwritten laws that say Apple better play right or lose quite a bit more —
So, how it’s getting done is a side show.
That said, it wasn’t long ago that they stood toe to toe with the FBI —- but there also weren’t wonderfully strong ‘sanctions’ on the horizon.
> Apple then manually reviews each report to confirm there is a match,
This is always the terrifying part for me. They will access your personal photos or data without telling you. I’m surprised how is that even legal given all the law that are already available. Are they immune to those laws stated in thd blog?
Also what happens when they launch this in EU, AU, etc with different privacy laws?
This is always the terrifying part for me. They will access your personal photos or data without telling you. I’m surprised how is that even legal given all the law that are already available. Are they immune to those laws stated in thd blog?
Also what happens when they launch this in EU, AU, etc with different privacy laws?
Somewhat tangential, but people like this author amaze me in their deep knowledge AND ability to communicate it well.
Also, none of this topic is something I would want to deal with.
Also, none of this topic is something I would want to deal with.
That PhotoDNA is reversible is ridiculously shocking to me...
I appreciate just about everything about this post, but this part keeps getting lost in everything I see written about it:
>As noted, Apple says that they will scan your Apple device for CSAM material. If they find something that they think matches, then they will send it to Apple. The problem is that you don't know which pictures will be sent to Apple.
It's iCloud Photos. Apple has explicitly said it's iCloud photos. If it's being synced to iCloud Photos, you know it's getting scanned one way or another (server side, currently, or client side, going forward).
It notes privacy issues, but... iCloud syncs by default. You wouldn't do the kind of work they're talking about (e.g, investigation) and store that kind of material where it could be synced to a server to begin with.
Everyone keeps proclaiming that Apple is scanning your entire device, but that's not what's happening with this change. It's not even comparable to A/V in this respect - it would be a very different story if that was the case. The wording and explanation matters.
>As noted, Apple says that they will scan your Apple device for CSAM material. If they find something that they think matches, then they will send it to Apple. The problem is that you don't know which pictures will be sent to Apple.
It's iCloud Photos. Apple has explicitly said it's iCloud photos. If it's being synced to iCloud Photos, you know it's getting scanned one way or another (server side, currently, or client side, going forward).
It notes privacy issues, but... iCloud syncs by default. You wouldn't do the kind of work they're talking about (e.g, investigation) and store that kind of material where it could be synced to a server to begin with.
Everyone keeps proclaiming that Apple is scanning your entire device, but that's not what's happening with this change. It's not even comparable to A/V in this respect - it would be a very different story if that was the case. The wording and explanation matters.
There was parents arrested over bath time and playing in the yard sprinklers, photos being processed at photo mats, will the same thing happen by apple mistakenly reporting parents?
When I worked telecom, we had md5sum database to check for this type of content. If you emailed/sms/uploaded a file with the same md5sum, your account was flagged and sent to legal to confirm it.
Also if a police was involved, the account was burned to dvd in the datacenter, and only a police officer would touch the dvd, no engineer touched or saw the evidence. (Chain of Evidence maintained)
Prob changed since I haven't worked in telecom in 15 years, but one thing I've read for years, is the feds knew who these people are, where they hang out online, even ran the some of the honeypots. The problem is they leave these sites up to catch the ring leaders, the feds are aware, they have busts almost every month of rings of criminals. Twitter has had accounts reported, and they stay up for years.
I dont think finding the criminals are the problem, seems like every time this happens, theres been people of interest for years, just not enough law enforcement dedicated to investigating this.
All the defund the police, I think moving some police from traffic duty to Internet crimes would be more of an impact on actual cases being closed. Those crimes lead to racketeering and other organized crime anyways.
When I worked telecom, we had md5sum database to check for this type of content. If you emailed/sms/uploaded a file with the same md5sum, your account was flagged and sent to legal to confirm it.
Also if a police was involved, the account was burned to dvd in the datacenter, and only a police officer would touch the dvd, no engineer touched or saw the evidence. (Chain of Evidence maintained)
Prob changed since I haven't worked in telecom in 15 years, but one thing I've read for years, is the feds knew who these people are, where they hang out online, even ran the some of the honeypots. The problem is they leave these sites up to catch the ring leaders, the feds are aware, they have busts almost every month of rings of criminals. Twitter has had accounts reported, and they stay up for years.
I dont think finding the criminals are the problem, seems like every time this happens, theres been people of interest for years, just not enough law enforcement dedicated to investigating this.
All the defund the police, I think moving some police from traffic duty to Internet crimes would be more of an impact on actual cases being closed. Those crimes lead to racketeering and other organized crime anyways.
I think Apple may have really screwed the pooch with this move. They're going to catch hell for being able to take images from your device without warning or consent, and, they'll catch hell if they remove it.
Worse, they've also opened the door to government censorship of images and content and propped that door wide open.
Worse, they've also opened the door to government censorship of images and content and propped that door wide open.
Great title.
>Think of it this way: Your landlord owns your property, but in the United States, he cannot enter any time he wants. In order to enter, the landlord must have permission, give prior notice, or have cause. Any other reason is trespassing. Moreover, if the landlord takes anything, then it's theft. Apple's license agreement says that they own the operating system, but that doesn't give them permission to search whenever they want or to take content.
Yes. That is an analogy only with Apple's software but not hardware. In Apple's view they are selling you the experience. So they are more like Hotels, You dont own the hotel room, the bed, the TV or anything inside that room. And in a Hotel, they can do Room Cleaning anytime they want.
Yes. That is an analogy only with Apple's software but not hardware. In Apple's view they are selling you the experience. So they are more like Hotels, You dont own the hotel room, the bed, the TV or anything inside that room. And in a Hotel, they can do Room Cleaning anytime they want.
https://web.archive.org/web/20210808233609/https://www.hacke...
(I can't seem to access archive.is, so archive.org)
I've been a FOSS dev for 25 years and I remember when everyone else I worked with were avid linux/freeBSD users because 'we didn't trust the big end of town'.. over the years I've watched the vast majority of devs move to apple devices for all sorts of 'just works', 'shinier' reasons that just boil down to 'convenience is more important than privacy'.
Perhaps this is just the benefit of longevity but from my POV it was engineer early adoption and advocacy that made Apple, Google Search etc what they are, and it will be engineer early adoption and advocacy that dethrones these problematic companies from controlling the ecosystem..
Back 20 years ago, before the community filled with $_$ dollars-struck startup founders, software was built by people who wanted to use it.. rather than sell it. There are still some people doing this now, Look at Matrix network for instance.
What will it take for a grass-roots software industry to start building privacy-first apps and systems that don't suck, based on decentralised, distributed principles? We have the skills to build highly polished alternatives to these things, but it takes a determination to step away from convenience for a period of time for the sake of privacy.
How bad does it have to get before the dev community realise this? or are we in a frog boiling slowly scenario and it's hopeless?
Perhaps this is just the benefit of longevity but from my POV it was engineer early adoption and advocacy that made Apple, Google Search etc what they are, and it will be engineer early adoption and advocacy that dethrones these problematic companies from controlling the ecosystem..
Back 20 years ago, before the community filled with $_$ dollars-struck startup founders, software was built by people who wanted to use it.. rather than sell it. There are still some people doing this now, Look at Matrix network for instance.
What will it take for a grass-roots software industry to start building privacy-first apps and systems that don't suck, based on decentralised, distributed principles? We have the skills to build highly polished alternatives to these things, but it takes a determination to step away from convenience for a period of time for the sake of privacy.
How bad does it have to get before the dev community realise this? or are we in a frog boiling slowly scenario and it's hopeless?
I don't see many people pushing back on the child pornography laws themselves that are the cause of this. I'm stepping into a hornets nest by even bringing this up, because any criticism of the laws on the books makes one look they're a pedo, so I'll preface by saying, child pornography (filmed with actual kids) is vile and disgusting, but it is the production of it that is evil to be fought and suppressed, not the possession of it. Remember in the 90s, we had to deal with the Communications Decency Act I & II, because every time they want to crack down on the internet, the excuse is always "it's for the children!" And pedophilia is the go-to excuse in a lot of circumstances.
The CPPA had bans on virtual child porn (e.g. using look-alike adult actresses or CGI), that was overturned by SCOTUS, and then Congress responded with the PROTECT act which tightened up those provisions. These laws on possession are practically unenforceable with modern technology, peer to peer file sharing, onion routing, and encrypted hard drives.
Thus, in order to make them enforcement, the government has to put surveillance at all egress and ingress points of our private/secure enclaves, whether it's at the point of storing it locally, or the point of uploading it to the cloud.
While I agree with the goal of eliminating child porn, should it come at the cost of an omnipresent government surveillance system everywhere? One that could be used for future laws that restrict other forms of content? How about anti-vax imagery? Anti-Semitic imagery? And with other governments of the world watching, especially authoritarian governments, how long until China, which had a similar system with Jingwang Weishi (https://en.wikipedia.org/wiki/Jingwang_Weishi) starts asking: hey, can you extend this to Falun Gong, Islamic, Hong Kong resistance, and Tiananmen square imagery? What if Thailand passes a law that requires Apple to scan for images insulting to the Thai Monarch, does Apple comply?
This is a very bad precedent. I liked the Apple that said no to the FBI instead of installing backdoors. I'd prefer if Apple get fined, and battle all the way to the Supreme Court to resist this.
The CPPA had bans on virtual child porn (e.g. using look-alike adult actresses or CGI), that was overturned by SCOTUS, and then Congress responded with the PROTECT act which tightened up those provisions. These laws on possession are practically unenforceable with modern technology, peer to peer file sharing, onion routing, and encrypted hard drives.
Thus, in order to make them enforcement, the government has to put surveillance at all egress and ingress points of our private/secure enclaves, whether it's at the point of storing it locally, or the point of uploading it to the cloud.
While I agree with the goal of eliminating child porn, should it come at the cost of an omnipresent government surveillance system everywhere? One that could be used for future laws that restrict other forms of content? How about anti-vax imagery? Anti-Semitic imagery? And with other governments of the world watching, especially authoritarian governments, how long until China, which had a similar system with Jingwang Weishi (https://en.wikipedia.org/wiki/Jingwang_Weishi) starts asking: hey, can you extend this to Falun Gong, Islamic, Hong Kong resistance, and Tiananmen square imagery? What if Thailand passes a law that requires Apple to scan for images insulting to the Thai Monarch, does Apple comply?
This is a very bad precedent. I liked the Apple that said no to the FBI instead of installing backdoors. I'd prefer if Apple get fined, and battle all the way to the Supreme Court to resist this.
I haven’t read all the details, articles, and comments. My personal thoughts on the whole situation are the following.
If you are a parent and lose a child then you would want every possible avenue taken to find your child. You would be going mad wanting to find them. If there is a way to match photos to known missing children then I say it should be at least tried.
I equate this to Ring cameras. They are everywhere. You cannot go for a walk without showing up on dozens of cameras, which we know Amazon (god mode) and law enforcement abuse their access privileges. However, if a crime happened to you and a Ring camera captured it, then I know almost everyone would certainly want that footage reviewed. Would you ignore the Ring footage possibility just because you despise Ring cameras? Probably not.
It’s all an invasion of privacy until you’re sitting on the other side of the table where you have a vested interest in getting access to the information.
If you are a parent and lose a child then you would want every possible avenue taken to find your child. You would be going mad wanting to find them. If there is a way to match photos to known missing children then I say it should be at least tried.
I equate this to Ring cameras. They are everywhere. You cannot go for a walk without showing up on dozens of cameras, which we know Amazon (god mode) and law enforcement abuse their access privileges. However, if a crime happened to you and a Ring camera captured it, then I know almost everyone would certainly want that footage reviewed. Would you ignore the Ring footage possibility just because you despise Ring cameras? Probably not.
It’s all an invasion of privacy until you’re sitting on the other side of the table where you have a vested interest in getting access to the information.
As a fan of this blog for longer than I can remember, it's refreshing to hear this particular author's take on this issue, especially considering their background.
I'm glad these issues were addressed in a much more elegant way than I would have put them:
> Apple's technical whitepaper is overly technical -- and yet doesn't give enough information for someone to confirm the implementation. (I cover this type of paper in my blog entry, "Oh Baby, Talk Technical To Me" under "Over-Talk".) In effect, it is a proof by cumbersome notation. This plays to a common fallacy: if it looks really technical, then it must be really good. Similarly, one of Apple's reviewers wrote an entire paper full of mathematical symbols and complex variables. (But the paper looks impressive. Remember kids: a mathematical proof is not the same as a code review.)
> Apple claims that there is a "one in one trillion chance per year of incorrectly flagging a given account". I'm calling bullshit on this.
I'm glad these issues were addressed in a much more elegant way than I would have put them:
> Apple's technical whitepaper is overly technical -- and yet doesn't give enough information for someone to confirm the implementation. (I cover this type of paper in my blog entry, "Oh Baby, Talk Technical To Me" under "Over-Talk".) In effect, it is a proof by cumbersome notation. This plays to a common fallacy: if it looks really technical, then it must be really good. Similarly, one of Apple's reviewers wrote an entire paper full of mathematical symbols and complex variables. (But the paper looks impressive. Remember kids: a mathematical proof is not the same as a code review.)
> Apple claims that there is a "one in one trillion chance per year of incorrectly flagging a given account". I'm calling bullshit on this.
I'm just wondering how long it will take before average people, jokingly or otherwise, imply that buying an android phone makes you a criminal.
> So where else could they get 1 trillion pictures?
That's a real kicker in my opinion. Unless they get training data from NCMEC I struggle to understand how they're training their model? Unless it's entirely algorithmic and not based on ML?
That's a real kicker in my opinion. Unless they get training data from NCMEC I struggle to understand how they're training their model? Unless it's entirely algorithmic and not based on ML?
The "legal" section talks about local scanning, and possible transmission of CSAM from devices to Apple, in pursuit of verification, however Apple have made clear that the scanning happens only for files that have been uploaded to iCloud Photo Library -- in which case they are not deliberately transmitting the CSAM but rather flagging something which the user already sent them.
Likewise the copyright issue; The user has already sent these files to Apple themselves by enabling iCloud photo library, and Apple are not making any additional copies that I am aware of.
It also says "The problem is that you don't know which pictures will be sent to Apple." - but we do know exactly which pictures will and will not be sent to apple; the ones that are already sent by iCloud Photo library.
[To be clear, I don't like the precedent/slippery slope that this kind of technique might lead towards in the future, but it doesn't seem like all the criticisms of it today are valid]
Likewise the copyright issue; The user has already sent these files to Apple themselves by enabling iCloud photo library, and Apple are not making any additional copies that I am aware of.
It also says "The problem is that you don't know which pictures will be sent to Apple." - but we do know exactly which pictures will and will not be sent to apple; the ones that are already sent by iCloud Photo library.
[To be clear, I don't like the precedent/slippery slope that this kind of technique might lead towards in the future, but it doesn't seem like all the criticisms of it today are valid]
I think the article is wrong about this. Or, right-but-situationally-irrelevant. As far as I can tell from Apple's statements, they're doing this only to photos which are being uploaded to iCloud Photos. So, any photo this is happening to is one that you've already asked Apple to copy to their servers.
> In this case, Apple has a very strong reason to believe they are transferring CSAM material, and they are sending it to Apple -- not NCMEC.
I also suspect this is a fuzzy area, and anything legal would depend on when they can actually be said to be certain there's illegal material involved.
Apple's process seems to be: someone has uploaded photos to iCloud and enough of their photos have tripped this system that they get a human review; if the human agrees it's CSAM, they forward it on to law enforcement. There is a chance of false positives, so the human review step seems necessary...
After all, "Apple has hooked up machine learning to automatically report you to the police for child pornograpy with no human review" would have been a much worse news week for Apple. :D