OpenAI releases a free child-abuse image classifier built with NCMEC and adds ChatGPT warnings and suspensions

AI & Compute

OpenAI Releases Free CSAM Classifier Built With NCMEC

OpenAI's free NCMEC-built classifier flagged 48% fewer images while finding the same number of abusive ones. ChatGPT adds 24-hour suspensions for child-sexualization attempts.

By
Grace Kim
Filed
Channel
AI & Compute
Read
3 min read

OpenAI announced on Monday a free image classifier that detects child sexual abuse material, including images generated by AI, alongside new warnings, account suspensions and tools for law enforcement.

The classifier, developed with the National Center for Missing and Exploited Children (NCMEC), scores images it has never seen before for visual signs of abuse. It assigns a score intended to guide human reviewers rather than replace them. OpenAI is offering the tool at no cost to selected companies through its API, and is working with the Tech Coalition and ROOST, a nonprofit that builds open safety tools, to distribute access more broadly.

How accurate is the classifier?

OpenAI reported concrete results from early testing with partners. The classifier flagged 48% fewer images while finding the same number of abusive ones, the company said. Images it did flag were twice as likely to be abusive. For review teams, the practical effect is a smaller queue with a higher hit rate. Human reviewers still make the final call on every flagged image.

The tool works on unseen images rather than matching against a known database. That design matters because generative models can produce novel abuse imagery that never appeared in any existing collection. OpenAI framed the classifier as a response to that gap.

What changes inside ChatGPT?

ChatGPT already blocks CSAM and blocks requests for it. The new measures add an enforcement ladder. When a user attempts a prompt that sexualizes children, ChatGPT will now:

  • show an alert to the user
  • direct the user to specialized support services
  • suspend the account for 24 hours
  • permanently remove the account on repeated offenses

OpenAI said it shaped the deterrent messages with three partners: Protect Children, the Lucy Faithfull Foundation and Joel Scanlan, a researcher at the University of Tasmania. Scanlan independently evaluated a trial of the warning messages before rollout.

What do developers and investigators get?

OpenAI is also publishing guidelines for developers building on its tools. The guidelines cover how to prevent, detect and report CSAM. The company said the guidance has already helped some developers register to file reports with NCMEC.

For law enforcement, OpenAI is offering a separate package. It includes AI plugins, investigative and data-analysis tools managed by program administrators, plus training and a field guide. The company did not disclose how many agencies have joined the program or when the tools will be broadly available.

The release does not come in a legal vacuum. OpenAI faces several lawsuits alleging its technology harmed users, including children. The new safety measures arrive as the company defends its products in court.

Why free access matters

The decision to give the classifier away for free, rather than sell it, signals where OpenAI sees its responsibility landing. Detection of abuse imagery has historically depended on hash-matching databases shared among platforms. A scoring classifier that handles novel and AI-generated images extends that toolkit. Distribution through the Tech Coalition and ROOST, rather than commercial licensing, widens the pool of potential users beyond OpenAI's own customer base.

The 48% reduction in flagged volume, if it holds outside partner testing, could shift reviewer time toward confirmed cases. OpenAI has not yet published full benchmark data or named the partner companies involved in the trials.

The company said it will continue sharing access with selected companies through its API while the NCMEC, Tech Coalition and ROOST partnerships expand the tool's reach.

Source: mlq.ai

Share this article:

More from Grace Kim

Grace Kim

Show full bio

Market editor covering people, launches and day-to-day practice in Semiconductors, chip manufacturing, hardware, AI compute at Chip Dispatch.

1,035 articles

Related articles

« Previous article