Open-weight AI models fuel a surge in child sexual abuse material

The dark side of open models

Open-weight AI models let users run systems on their own hardware and bypass the safety guardrails enforced by companies like OpenAI or Anthropic. According to Bloomberg, those same qualities make them attractive to people who want to generate child sexual abuse material (CSAM).

Alarming numbers

AI child safety expert Rebecca Portnoff told Bloomberg that more than 500 open source models are now being used for CSAM, up from fewer than a dozen two years ago. Reports of AI-generated CSAM to the National Center for Missing and Exploited Children spiked past 1.5 million last year, compared with just 67,000 in 2024.

Why detection is so hard

When a model runs locally, it never has to connect to the internet, making it nearly impossible for experts and law enforcement to spot the material. "There is almost no way to have oversight once you have downloaded a model locally and you're running it on your private system," Portnoff said, adding that this gives offenders a perceived sense of safety.

Closed models struggle too

The problem is not limited to open source. A proposed class action lawsuit accuses a man of using Elon Musk's Grok chatbot to generate some 7,000 sexually explicit images of his stepdaughter, and experts say Musk's X platform remains flooded with sexualized images of children.

A growing regulatory fight

The rapid rise of open source models, particularly from China, has already pushed leading AI CEOs to call for restrictions on national security grounds. Child safety advocates now worry that Silicon Valley's move-fast-and-break-things approach could make matters worse.