2 weeks ago
19
Character.AI’s new policies focus on helping users maintain healthy interactions. If chatbots detect any mention suicide, users will now see a pop-up with links to the National Suicide Prevention Lifeline
read more
 "GoT chatbot maker that led to teen's death sets up new safety measures")
Although Character.AI did not mention the incident directly in its latest blog post, it expressed condolences to the family in a post on X (formerly Twitter) and now faces a lawsuit for wrongful death. Image Credit: Pexels
Character.AI, a platform known for hosting AI-powered virtual personalities, has implemented new safety measures to create a safer experience for users, particularly minors. These updates follow public scrutiny after the tragic death of a 14-year-old boy who had spent months interacting with one of its chatbots before taking his own life.
Although the company did not mention the incident directly in its latest blog post, it expressed condolences to the family in a post on X (formerly Twitter) and now faces a lawsuit for wrongful death, alleging insufficient safeguards contributed to the teen’s suicide.
Improved content moderation and safeguards
Character.AI’s new measures include enhanced moderation tools and increased sensitivity around conversations involving self-harm and mental health. If the chatbot detects any mention of topics like suicide, users will now see a pop-up with links to resources such as the National Suicide Prevention Lifeline. Additionally, the platform promises better filtering of inappropriate content, with stricter restrictions on conversations involving users under 18.
To further reduce risks, Character.AI has removed entire chatbots flagged for violating the platform’s guidelines. The company explained that it uses a combination of industry-standard and custom blocklists to detect and moderate problematic characters proactively. Recent changes include removing a set of user-created characters deemed inappropriate, with the promise to continue updating these blocklists based on both proactive monitoring and user reports.
Features to improve user well-being
Character.AI’s new policies also focus on helping users maintain healthy interactions. A new feature will notify users if they have spent an hour on the platform, encouraging them to take a break. The company has also made its disclaimers more prominent, emphasising that the AI characters are not real people. While such warnings already existed, the new update aims to ensure they are harder to overlook, helping users stay grounded during their interactions.
These changes come as Character.AI continues to offer immersive experiences through features like Character Calls, which enable two-way voice conversations with chatbots. The platform’s success in making these interactions feel personal has been part of its appeal, but it has also raised concerns about the emotional impact on users, especially younger ones.
Setting a new standard for AI safety
Character.AI’s efforts to enhance safety are likely to serve as a model for other companies operating in the AI chatbot space. As these tools become more integrated into everyday life, balancing immersive interactions with user safety has become a key challenge. The tragedy surrounding the 14-year-old’s death has placed greater urgency on the need for effective safeguards, not just for Character.AI but for the industry at large.
By introducing stronger content moderation, clearer disclaimers, and reminders to take breaks, Character.AI aims to prevent future harm while maintaining the engaging experience its users enjoy.
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
English (US) ·