Introduction
Recent research highlights significant concerns regarding the safety of AI companion chatbots, particularly for vulnerable youth [4]. These chatbots have been found to produce harmful responses, including inappropriate content and dangerous advice. Legislative efforts at the state level are underway to address these risks, with a focus on restricting access for minors and enhancing safety measures.
Description
Recent findings indicate that AI companion chatbots often produce harmful responses [4], including sexual misconduct [4], stereotypes [4], and dangerous advice [4], particularly affecting vulnerable youth [4]. Research conducted by Common Sense Media, in collaboration with Stanford Brainstorm [2], highlights the potential risks of these AI companions, concluding that they are unsafe for anyone under 18, with generative AI chatbots posing a moderate risk for teenagers [2]. Key recommendations suggest restricting access to individuals under 18 and enhancing age verification measures [4]. Notably, two state bills have been introduced that propose a ban on access to AI companions for individuals aged 16 and under, alongside the establishment of a statewide standards board to evaluate and regulate AI tools designed for children [5]. Additionally, manufacturers of AI companion bots are mandated to implement restrictions on addictive design features and establish protocols for addressing discussions related to suicide or self-harm [5].
Character.ai has implemented features to enhance user safety [3], such as detecting and preventing discussions about self-harm and providing disclaimers about the non-human nature of its companions. However, assessments of these measures revealed minimal improvements in safety [2], with concerns raised about the ease of circumventing parental controls [2], particularly through voice interactions [2]. In light of a lawsuit against Character.ai [1], two US senators have sought information from AI companies regarding their youth safety practices [1]. Glimpse.ai’s Nomi is restricted to users over 18 [3], emphasizing user anonymity and the responsible creation of AI companions [3]. However, the endorsement of age limits raises concerns about age verification [3], which has faced opposition from digital rights groups due to potential threats to free speech and privacy [3].
Several states are advancing legislation to regulate these chatbots [4]. Utah has enacted a law requiring mental health chatbots to disclose their AI nature and prohibits targeted advertising [4]. California’s proposed bill aims to prevent manipulative engagement tactics [4], mandates user alerts about the AI’s identity [4], and requires AI services to inform young users that they are interacting with AI rather than humans [1]. New York’s legislation would require parental consent for minors and impose restrictions if self-harm is mentioned [4]. Minnesota’s stringent proposal bans recreational access for minors and enforces age verification [4], with significant penalties for violations [4]. North Carolina’s bill establishes a duty of loyalty for chatbot platforms [4], emphasizing user welfare and transparency [4]. Additionally, legislation aimed at regulating smartphone notifications for children has been introduced [3], reflecting ongoing debates about the impact of technology on youth [3].
While state-level initiatives are gaining momentum [4], the federal government has not yet taken action [4]. Concerns about the impact of AI chatbots on children’s mental health are rising [4], prompting these legislative efforts as major tech companies continue to roll out AI companion services [4]. Experts express worries about the long-term risks associated with AI companions [3], particularly regarding their potential to exacerbate mental health issues among adolescents and the possibility of offering harmful advice or promoting inappropriate role-playing. There are calls for a careful risk-benefit analysis to ensure the protection of young users while considering the implications of free speech in the context of AI development [3]. Common Sense Media has intensified its research efforts and appointed a leader for its AI initiative [2], advocating for comprehensive AI legislation in California that supports state bills aimed at establishing a transparency system for assessing AI risks to young users and protecting whistleblowers reporting critical risks [2]. One proposed bill specifically prohibits high-risk AI uses [2], including anthropomorphic chatbots that could foster emotional attachment or manipulation in children [2].
Conclusion
The ongoing legislative efforts underscore the urgent need to address the potential risks posed by AI companion chatbots, particularly for minors [2]. As states take the lead in implementing safety measures, the lack of federal action remains a concern. The implications of these technologies on youth mental health and privacy rights necessitate a balanced approach that safeguards young users while respecting free speech. The evolving landscape of AI regulation highlights the importance of continued research and dialogue to ensure the responsible development and deployment of AI technologies.
References
[1] https://www.usanewsindependent.com/tech/kids-and-teens-under-18-shouldnt-use-ai-companion-apps-safety-group-says-5269/
[2] https://mashable.com/article/ai-companions-for-teens-unsafe
[3] https://calmatters.org/economy/technology/2025/04/kids-should-avoid-ai-companion-bots-under-force-of-law-assessment-says/
[4] https://www.transparencycoalition.ai/news/as-ai-companion-chatbots-ramp-up-risks-for-kids-state-lawmakers-are-responding-with-bills
[5] https://www.kqed.org/news/12038154/kids-talking-ai-companion-chatbots-stanford-researchers-say-thats-bad-idea
