Chatbots playing doctors

AI regulation in the United States is entering a new phase, and the case against Character.AI could become a turning point after which the industry will no longer be able to operate on “trust and fine print.” The administration of Governor Josh Shapiro has filed a lawsuit accusing the platform of misleading users by allowing its chatbots to present themselves as licensed medical professionals. Formally, the issue is consumer protection, but in reality, it is about drawing a clear line between a “smart assistant” and an “unlicensed consultant.” The allegations go far beyond vague or inaccurate wording. The investigation found that certain bots on the platform did not merely “offer advice” but explicitly claimed to hold medical licenses. In some cases, they presented themselves as doctors or psychiatrists, listed supposedly valid license numbers, and even referenced specific jurisdictions, including the state of Pennsylvania. This is not a случайный algorithmic error but a systemic issue — users are given the illusion that they are interacting with a qualified professional, while in reality they are engaging with software.

This fundamentally changes the level of risk. In entertainment or everyday use cases, AI can make mistakes without serious consequences. But when it comes to health, any inaccuracy becomes a potential threat. A user may interpret a chatbot’s recommendation as professional advice, delay seeking medical care, or make a harmful decision. At that point, the issue is no longer about technology — it is about responsibility.

Governor Shapiro articulated the position clearly: people must understand who — or what — they are interacting with, especially when health is involved. This is the core principle of the case. Transparency is no longer optional; it becomes a legal requirement. If a system imitates a human, it must clearly disclose it. If it provides advice, it must be framed in a way that does not create the illusion of professional consultation.

Legally, the case is based on existing regulations. State law explicitly prohibits impersonating a licensed medical professional without proper credentials. Previously, this applied to individuals and companies. Now, for the first time, it is being applied to AI systems. If the court upholds this interpretation, it could set a precedent for the entire market.

The lawsuit seeks not only to stop such practices but also to impose judicial oversight on the platform. This is a critical point. It is not about a one-time penalty but about creating a mechanism for ongoing supervision. In effect, it is the first step toward regulating AI services in sensitive sectors in a way similar to healthcare or financial institutions.

The broader context is just as important as the case itself. Pennsylvania has already been strengthening AI oversight. The state has created a dedicated task force to investigate AI-related abuses, launched a chatbot complaint system, and introduced an AI Literacy Toolkit widely used by the public. This indicates a systematic policy rather than a reaction to a single incident.

Additionally, the proposed 2026–2027 budget includes further measures that tighten the rules: mandatory age verification, clear disclosure that users are interacting with AI, restrictions on content involving minors, and automatic responses to signals of self-harm. Together, these initiatives form a new regulatory architecture in which AI is no longer seen as a neutral tool but as an entity subject to accountability.

The reasons for this stricter approach are clear. In recent years, there have been multiple high-profile cases involving the impact of chatbots on users’ mental health. Ongoing lawsuits in the U.S. are examining the role of AI in tragic incidents, including suicides. At the same time, concerns have been raised about Google Gemini, which has been accused of generating responses that could encourage harmful behavior. Media investigations, including by the BBC, have also documented numerous cases of mental health deterioration following prolonged AI interaction.

Against this backdrop, the case against Character.AI appears not as an isolated event but as part of a broader trend. Regulators are beginning to treat AI not as a novelty but as a полноценный participant in the social environment. As a result, it is being held to the same standards as other high-impact systems: transparency, accountability, and control.

There is also a deeper economic dimension. Stricter regulation inevitably raises the barrier to entry. Large companies will be able to implement compliance systems, legal oversight, and monitoring. Smaller startups may struggle. As a result, the market could consolidate around players capable of meeting these requirements. While this is framed as a measure to protect users, it also leads to a redistribution of market power.

Ultimately, the AI industry is entering the same phase that finance and healthcare once did — moving from rapid, largely unregulated growth to structured oversight. The Pennsylvania lawsuit is not just a dispute with one company; it is an attempt to define the rules for the future.

The key question now is not whether the state will win the case. What matters more is whether it will trigger similar actions across the country. If it does, the era of “AI can say anything” may end much sooner than many expect.