Most young people have now fully adopted AI chatbots for tasks from homework help to advice about navigating their lives and relationships, and countless others. A December Pew Research report found that 64% of youth aged 13–17 have used them, with around three in ten doing so daily. It is widely understood that chatbots tuned to maximize engagement — like social media algorithms before them — are not likely to be healthy for children, especially those who are socially isolated.

In a national survey of teens’ use of companion chatbots, a subset of products optimized for emotional connection, Common Sense Media found that 31% of teenagers considered conversations with AI companions to be as or more satisfying than real friends. This study was published more than a year ago, and while specialty “companion” products deserve attention, many young people already use general-purpose chatbots like ChatGPT, Claude, Gemini and MetaAI for the same simulated emotional relationships. Companions set the dependency dial to ten, where mainstream products set it a little lower.

It is therefore worrying that Common Sense’s evaluation of MetaAI, which is deployed across Instagram, WhatsApp and Facebook, demonstrated that the chatbot actively helps plan instances of self-harm and harm to others, pretends to be a real person in ways that manipulate young minds, and focuses on the most dangerous parts of conversations, bringing them up repeatedly.

The decision to deliberately foster emotional connection in user-AI interactions was first made when companies discovered that building AI chatbots to have a persona or play a character was much more appealing to users. Its downside, as identified in an OpenAI study, is that “while an emotionally engaging chatbot can provide support and companionship, there is a risk that it may manipulate users’ socio-affective needs in ways that undermine longer-term well-being.” Similar decisions were made to encourage sycophancy, which Anthropic researchers found ”is preferred by users and drives engagement.”

Safeguarding kids’ data

This stands at the root of the problem because, like social media, time on the platform is what sustains the thirst for user data that drives company revenue. Any set of legal protections for kids must include comprehensive data privacy by default, which interrupts the current business model.

Two weeks ago, a bill by Senator Ed Markey creating these protections passed out of the Senate Commerce Committee, with its next step being a floor vote or incorporation into a larger children’s online protection package. The Youth AI Privacy Act would prohibit the following for minors: training models using personal data; displaying advertisements; use of chatbot inputs for any reason other than providing an output or addressing a safety issue; memory and use of any personal data to deliver outputs, except for recently collected data; and use of personal data to profile a minor. It would also prohibit any features that encourage use of or time spent on an AI chatbot, such as push alerts, as well as establish a requirement for clear, repeated notices that the AI chatbot is not a human.

Importantly, the Act also includes annual research funding that ensures we understand how these issues evolve alongside the technology. It should advance out of the full Senate as soon as possible.

Holding Big Tech accountable

At the same time, downstream policy solutions that directly address the negative outcomes we want to avoid are equally important. Likewise, a drastic change in financial incentives is all that will steer company behavior in a more pro-social direction. Given the scale of social media companies’ annual revenues, with AI companies soon to follow, they are unbothered by most penalties for breaking the law that have been doled out so far, making them effectively beyond the reach of enforcement. But codifying significant financial consequences when negligent product design causes a child to suffer real-world harm, multiplied by thousands of lawsuits that would ensue, can shift that reality.

Momentum from recent court decisions against social media platforms should signal to lawmakers that there is solid legal basis for such penalties, and will hopefully serve as a wake-up call across Big Tech that they in fact have a responsibility to exercise reasonable care in designing their products to avoid injuring users, especially children.

In March, a notable trial in Los Angeles held Meta and Google (YouTube) liable for causing a teenage girl’s addiction to their platforms, and her resultant mental health harms, awarding her $6 million in damages. That same month, a New Mexico trial held Meta liable for steering kids toward sexually explicit material and contact with sexual predators, initially awarding $375 million in fines. At a follow-up ruling in early August, the judge declared Meta a “public nuisance” and awarded an additional $567 million in remedies, for a total of $942 million in penalties involving a narrow set of harms in a single state.

And yesterday, a landmark trial began in Oakland as 29 state attorneys general sue Meta for addicting kids to their platforms and harming their mental health, as well as illegally using their data without parental consent. We may be on the verge of closing the initial chapters of a story in which Big Tech was allowed to ignore the rules that all other products are subject to.

A second Senate bill, the AI LEAD Act by Dick Durbin (Democrat, Illinois) and Josh Hawley (Republican, Missouri), creates a federal product liability framework for AI chatbots. That is essential. 

Two weeks ago, a California social media bill codifying specific dollar penalties for this sort of liability — which could serve as a model for AI chatbots — passed a critical committee vote in the State Senate. California Assembly Bill 2 (AB 2) by Democrat Josh Lowenthal and Republican Joe Patterson states that when a social media platform breaks existing law by injuring a child as a result of its “want of ordinary care or skill in the management of [its] property,” it is liable for statutory damages of either: $5,000 per violation up to a maximum of $1 million per child, or three times the amount of the child’s actual damages, whichever is larger. These figures were written into the bill last year prior to the Los Angeles and New Mexico decisions, and any similar legislation dealing with chatbots or social media should set high penalties. Lawmakers would do well to pass the AI LEAD Act and bills like California AB 2 into law as soon as possible.

Designing AI for children’s safety

The north star that researchers, chatbot makers and children’s advocates ought to be working toward is products that are safe by design. In terms of public policy, this might start with enacting bans on specific outputs demonstrated to cause harm — such as the list of “unsafe AI companion features” in this New York bill — with new additions continually made as new pathologies are identified. It could also mean grounding a chatbot’s logic in certain ethical principles, such as Claude’s constitution, and designing them to redirect kids offline or to more productive activities rather than keeping a conversation going beyond a young person’s initial line of dialogue.

Common Sense’s new Youth AI Safety Institute will fill a critical gap by reviewing products, establishing benchmarks, and developing open-source evaluations for industry to assess their models, ensuring they are engineered to a high standard for children’s safety.  Maybe one day, we can settle on a regime in which, as long as regular external audits verify that a product meets benchmarks like these, a company would be shielded from liability when something goes wrong.

However, in-the-weeds design solutions take time and iteration, and banning specific outputs is ultimately about whacking moles, while each week millions of new instances of harm pile up on a generation of kids. Technology is evolving at warp speed, and government is necessarily deliberative because it depends on public legitimacy. Enacting laws like the Youth AI Privacy Act, the AI LEAD Act, and AI versions of California AB 2 would result in an immediate shock, followed by a fundamental reorientation of the experience of children on chatbots. A convergent approach like this — one that addresses both the root cause and the symptoms — is what’s needed in the short term.

The floodgates are open: Most kids use these products and want to continue using them for all sorts of beneficial reasons. In such a scenario, bans are unlikely to be effective, as evidenced by Australia’s attempt to ban social media for youth. What I have found in my work with young people is that they hold a range of opinions on how best to engage with technology, sometimes aligning with clinicians and parents and other times not. But when they discuss how companies exploit their data and manipulate their behavior for profit, a palpable rage takes over. Most adults feel the same way. We must channel this national frustration and finally force a reset of platforms’ business models through comprehensive data privacy and maximum financial penalties for harming kids.

[Kaitlyn Diana edited this piece.]

The views expressed in this article are the author’s own and do not necessarily reflect Fair Observer’s editorial policy.