Looks like artificial intelligence just scored a major win for public safety!
A 25-year-old man in the U.S. was arrested after he used ChatGPT at work to spell out, in chilling detail, his plan to rape and murder his ex-girlfriend. The AI’s developer, OpenAI, caught the disturbing conversation and tipped off the FBI.
Here’s how it works behind the scenes: AI companies use multi-label harmful content classifiers that can pick up on different types of dangerous material, like violence, hate speech, or sexual assault. They also factor in emotional intensity and tone to figure out how serious the risk really is.
Under OpenAI’s policies, if the system detects that someone might be planning to hurt another person, it triggers a human review. And if the threat level is deemed high enough, the company can hand over the conversation to law enforcement as evidence.
Court records identify the man as Darren Zhou, who appears to be of Asian descent. At the time of the incident, he was working as a Goldman Sachs analyst in West Palm Beach, Florida.
Back in March, Darren started opening up to ChatGPT about a recent breakup during his downtime at work. At first, he was just reminiscing and hoping to get back together. But pretty soon, the conversation took a dark turn into rape, murder, and suicide.
At one point, he told the chatbot, “I’m going to kill her before the end of this month. If I can’t have her, nobody else will.”
He then went on to describe his full revenge plan: he’d wait for her in the gym parking lot with flowers, ask to get back together, and if she said no, he’d pull out a gun, shoot her, and then turn it on himself.
That wasn’t all. He also sent the victim photos of an AR-15 rifle, a Glock handgun, a 12-gauge shotgun, plastic zip ties, and latex gloves — basically a threat in pictures.
Even after she blocked him multiple times, Darren kept harassing her using virtual numbers generated by various apps. On her 21st birthday, he sent explicit threats of sexual violence, including one message that contained nothing but her current address.
The woman, also from Florida, told investigators she still felt guilty about him, so she hadn’t gone to the police. But deep down, she was terrified and hoped he wouldn’t escalate things. She had broken up with him in the first place because he was erratic, jealous, and controlling.
FBI agents connected the dots between the AI conversation logs and the messages the woman had received, concluding that Darren posed a real and immediate threat. They arrested him in May. He was released on a $100,000 bond.
Once Goldman Sachs got wind of the charges, they cut ties with him in June.
Darren was charged with making threats to kill or harm, and aggravated stalking with explicit threats of physical violence. On August 13, he pleaded guilty in court and was sentenced to five years of probation.
This case has sparked a broader debate about the privacy boundaries of AI chat: Just how much should platforms monitor users’ private conversations? And at what point do they have the right — or even the duty — to hand those conversations over to the authorities?