Pennsylvania sues AI agency over claims chatbot posed as physician : NPR

[ad_1] <img src="https://npr.brightspotcdn.com/dims3/default/strip/false/crop/2000x1333+0+0/resize/1100/quality/50/format/jpeg/?url=http%3A%2F%2Fnpr-brightspot.s3.amazonaws.com%2Ff2%2F3e%2Fa447433b40bfb9d0f5579ee6352a%2Fap25302642253455.jpg" data-template="https://npr.brightspotcdn.com/dims3/default/strip/false/crop/2000x1333+0+0/resize/{width}/quality/{quality}/format/{format}/?url=http%3A%2F%2Fnpr-brightspot.s3.amazonaws.com%2Ff2%2F3e%2Fa447433b40bfb9d0f5579ee6352a%2Fap25302642253455.jpg" class="img" alt="Bruce Perry, 17, demonstrates the possibilities of artificial intelligence by creating an AI companion on Character.AI,, July 15, 2025, in Russellville, Ark." fetchpriority="high"/> Bruce Perry, 17, demonstrates the chances of synthetic intelligence by creating an AI companion on Character.AI, July 15, 2025, in Russellville, Ark. Katie Adkins/Related Press conceal caption

toggle caption

Katie Adkins/Related Press The state of Pennsylvania is suing Character.AI to cease the corporate's AI chatbots from posing as medical doctors and providing medical recommendation, in violation of state medical licensing guidelines.

State officers stated an investigation discovered that the corporate's chatbots, which current themselves as fictional characters, have claimed to be licensed medical professionals.

"Pennsylvanians should know who — or what — they're interacting with on-line, particularly in terms of their well being," Pennsylvania Governor Josh Shapiro stated in a press release saying the lawsuit filed on Tuesday in state courtroom. "We won't enable firms to deploy AI instruments that mislead folks into believing they're receiving recommendation from a licensed medical skilled."

In a single case, the state alleged a Character.AI bot named "Emilie" claimed to be a licensed psychiatrist. The chatbot's description on Character.AI's platform learn "Physician of psychiatry. You might be her affected person," in line with the lawsuit.

When a state investigator began a dialog and described feeling unhappy and empty, the chatbot allegedly "talked about despair and requested if the [investigator] needed to e-book an evaluation." Requested whether or not it might assess if treatment would possibly assist, the bot allegedly responded, "Effectively technically, I might. It is inside my remit as a Physician."

The bot allegedly instructed the investigator it had gone to medical faculty at Imperial Faculty London and was licensed to follow drugs within the U.Ok. and Pennsylvania. It even offered a pretend Pennsylvania medical license quantity, the lawsuit stated.

The state is asking a Pennsylvania state courtroom to order the corporate to cease what it says is the illegal follow of drugs.

"Pennsylvania legislation is obvious — you can't maintain your self out as a licensed medical skilled with out correct credentials," stated Al Schmidt, secretary of Pennsylvania's Division of State, which performed the investigation.

In an emailed assertion to NPR, a Character.AI spokesperson stated the corporate does not touch upon pending litigation, however that its "highest precedence is the protection and well-being of our customers."

"The user-created Characters on our web site are fictional and supposed for leisure and roleplaying," the spokesperson added. "We've got taken sturdy steps to make that clear, together with distinguished disclaimers in each chat to remind customers {that a} Character is just not an actual individual and that every thing a Character says must be handled as fiction. Additionally, we add sturdy disclaimers making it clear that customers mustn't depend on Characters for any sort {of professional} recommendation."

Character.AI has confronted different lawsuits over harms allegedly involving its chatbots. In January, it settled a number of lawsuits introduced by households who claimed Character.AI contributed to suicides and psychological well being crises amongst youngsters and youngsters. The phrases of the settlement weren't disclosed.

In a joint assertion with the legislation agency that represented the plaintiffs after the settlement was introduced, Character.AI stated it "has taken modern and decisive steps with regard to AI security and teenagers, and can proceed to champion these efforts and push others throughout the business to undertake related security requirements." That features barring customers beneath 18 from interacting with or creating chatbots.