A specialized company that provides crisis intervention services for major artificial intelligence platforms is working on groundbreaking technology to identify and help users displaying signs of violent extremism.
ThroughLine, a New Zealand-based startup that currently assists ChatGPT’s parent company OpenAI, along with competitors Anthropic and Google, plans to expand beyond its existing mental health crisis support to tackle radicalization concerns.
The company’s founder, Elliot Taylor, a former youth worker who operates from rural New Zealand, revealed the initiative comes as AI firms face mounting pressure over safety issues and legal challenges for allegedly failing to prevent violence.
“It’s something that we’d like to move toward and to do a better job of covering and then to be able to better support platforms,” Taylor explained during a recent interview, though he noted no timeline has been established for the project.
The development follows a February incident where OpenAI faced potential government intervention from Canada after disclosing that someone who committed a fatal school shooting had been banned from their platform without notifying authorities.
ThroughLine currently manages an extensive network of 1,600 crisis helplines across 180 countries, which are continuously monitored and updated. When AI systems detect indicators of potential mental health emergencies, self-harm risks, domestic violence situations, or eating disorders, users are automatically connected to ThroughLine’s services and matched with nearby human-operated support resources.
The proposed anti-extremism solution is being developed in partnership with The Christchurch Call, an organization established following New Zealand’s deadliest terrorist incident in 2019 to eliminate online hatred. This collaboration would involve the anti-extremism organization providing expert guidance while ThroughLine creates the intervention technology.
According to Taylor, the new system would likely combine a specially trained chatbot designed to respond to individuals showing extremist tendencies with connections to real-world mental health professionals.
“We’re not using the training data of a base LLM,” Taylor clarified, referring to the standard datasets that large language model platforms use to generate text. “We’re working with the correct experts.”
Galen Lamphere-Englund, who serves as a counterterrorism advisor for The Christchurch Call, expressed optimism about expanding the tool’s use to gaming forum moderators and parents seeking to identify online extremism.
Henry Fraser, an artificial intelligence researcher at Queensland University of Technology, praised the concept as both beneficial and essential, noting it addresses relationship dynamics rather than just problematic content.
However, Fraser cautioned that success would depend on “how good are follow-up mechanisms and how good are the structures and relationships that they direct people into at addressing the problem.”
Taylor acknowledged that follow-up procedures, including potential notifications to law enforcement about dangerous users, remain undetermined but would consider risks of escalating harmful behavior.
He emphasized that individuals in crisis often share information online that they would be too embarrassed to discuss with another person, warning that government pressure on platforms to disconnect users engaging in sensitive conversations could worsen situations.
Research from New York University’s Stern Center for Business and Human Rights indicates that increased platform moderation under law enforcement pressure has driven extremist sympathizers to less regulated alternatives like Telegram.
“If you talk to an AI and disclose the crisis and it shuts down the conversation, no one knows that happened, and that person might still be without support,” Taylor warned.
OpenAI confirmed their partnership with ThroughLine but declined additional comment. Anthropic and Google have not yet responded to requests for information about the initiative.
