Introduction
Artificial Intelligence (AI) is rapidly advancing, necessitating robust legal frameworks to govern its development and application. In California [1] [2] [3] [5] [6] [7] [8] [9], existing laws regulate AI systems and automated decision systems (ADS) across various sectors, including healthcare [1] [2] [4] [5] [6] [7] [8]. New regulations [3] [4] [6] [8], effective January 1, 2025 [3] [8] [9], will further clarify these laws, as highlighted by advisories from California Attorney General Rob Bonta. These advisories focus on consumer protection, civil rights [1] [3] [5] [6] [7] [8], and healthcare regulations [5] [8], particularly emphasizing transparency and accountability in the healthcare sector.
Description
AI is rapidly evolving [7], and existing California laws govern both the development and use of AI systems and automated decision systems (ADS) across various sectors, including healthcare [1] [2] [4] [5] [6] [7] [8]. Companies are required to comply with these laws and maintain accountability for their actions [7], decisions [1] [2] [5] [7] [9], and products [7]. On January 1, 2025 [1] [2] [3] [5] [6] [8], new AI regulations will take effect [3] [8], as emphasized in legal advisories issued by California Attorney General Rob Bonta. These advisories clarify the application of California’s consumer protection, civil rights [1] [3] [5] [6] [7] [8], and healthcare regulations to AI technologies, particularly in the healthcare sector [6] [9].
The first advisory outlines the rights and obligations of consumers and entities involved in the development [1], sale [1] [2] [3] [8], and use of AI under California law [1], while the second specifically addresses healthcare providers [1] [3], insurers [1] [8], vendors [1], investors [1], and other healthcare entities [1] [2] [7] [8]. Healthcare providers must inform patients when AI is utilized in diagnostic or treatment decisions [5], promoting transparency and enabling informed patient choices [5]. The advisories detail consumer rights and business obligations under California’s consumer protection [3] [8], civil rights [1] [3] [5] [6] [7] [8], competition [1] [2] [3] [6] [7] [8], and data privacy laws [1] [2] [3] [6] [7] [8], including the California Consumer Privacy Act (CCPA) of 2018 [6], the California Invasion of Privacy Act (CIPA) [3] [6], the Student Online Personal Information Protection Act (SOPIPA) [3] [6], and the Confidentiality of Medical Information Act (CMIA) [2] [3] [5] [6]. AI systems in healthcare are also required to adhere to the Health Insurance Portability and Accountability Act (HIPAA), which mandates strict protections for patient data [5].
The advisories reference the California Fair Employment and Housing Act (FEHA), which provides protections against harassment or discrimination based on various characteristics [3], extending these protections to AI systems used in the workplace. The California Civil Rights Counsel is currently considering new AI regulations under the FEHA [3]. The AG has raised concerns regarding the potential for AI and algorithmic tools to violate state law [7], particularly in relation to racial and ethnic disparities in decision-making processes [1]. In August 2022 [1] [7], the AG reached out to 30 hospitals in California to inquire about their strategies for addressing these disparities [7]. While acknowledging the potential advantages of AI in healthcare—such as enhancing patient health [1] [7], reducing administrative burdens [1] [7], promoting health equity [1] [7], and improving information sharing—the advisories also warn of significant risks, including denial of necessary care [1] [7], resource misallocation [1] [7], discrimination [1] [2] [3] [7] [9], and threats to patient autonomy and privacy [1] [7].
Compliance with California’s consumer protection [1] [5] [6] [7], unfair competition [1] [2] [7], civil rights [1] [3] [5] [6] [7] [8], and patient privacy laws is crucial [7], as the broad scope of the California Unfair Competition Law (UCL) protects consumers from false advertising and anticompetitive practices [7]. Healthcare entities must not overlook data indicating inequities based on race [7], gender [7], or other protected classifications [1] [2] [7], despite AI tools appearing neutral [7]. Investigations into discriminatory practices in healthcare algorithms are already underway [2], and entities receiving state support must not discriminate based on protected classifications [2]. California’s medical privacy laws [1] [7], including the CMIA [7], may offer greater protection than federal regulations [7], mandating that healthcare entities safeguard medical information confidentiality [7], ensure patient access to their data [7], and obtain informed consent [2] [7].
The AG encourages healthcare providers to incorporate AI usage into informed consent processes [7], warning against using coercive tactics to secure consent [2], as this undermines patient autonomy [2]. Many Californians express discomfort with AI’s role in healthcare [2], suggesting that seeking consent for AI use may be beneficial [2]. Access to sensitive data [2], particularly reproductive and sexual health information [2], must be carefully managed [2]. Recent amendments to the CMIA classify certain reproductive and sexual health data as sensitive [2], and healthcare providers must maintain the confidentiality of patient medical information [2].
Developers [2] [8], researchers [8], providers [1] [2] [3] [5] [7] [8], and insurers are urged to rigorously test [8], validate [8], and audit AI systems to ensure their safety [8], ethical use [8], and compliance with legal standards [8]. Ongoing monitoring of AI systems is necessary to mitigate emerging risks that could impact patient outcomes or legal compliance [5]. Transparency with patients regarding the use of their information in AI training and decision-making processes is also emphasized [8]. Additionally, compliance with new California laws [4] [6] [7] [8], such as SB-1120 [7], is essential; this law prohibits healthcare service plans from using AI for final medical necessity determinations [7], requiring licensed healthcare professionals to make these assessments [7]. Potential violations include denying health insurance claims that contradict a physician’s judgment [1] [7], making patient access decisions based on past health claims data with discriminatory effects [7], and generating patient notes or medical orders that contain inaccuracies [7].
Conclusion
The impending AI regulations in California underscore the importance of transparency, accountability [5] [7], and compliance in the use of AI technologies, particularly in healthcare [2] [5] [6] [9]. These regulations aim to protect consumer rights, ensure equitable treatment, and safeguard patient privacy. As AI continues to evolve, ongoing vigilance and adaptation to legal standards will be crucial to harness its benefits while mitigating potential risks.
References
[1] https://www.mintz.com/insights-center/viewpoints/2146/2025-01-22-california-attorney-general-issues-warning-artificial
[2] https://www.jdsupra.com/legalnews/top-ten-takeaways-from-california-ag-s-9384663/
[3] https://www.lexology.com/library/detail.aspx?g=48b499e4-ca10-40a9-bcbd-aeeb525d9ac6
[4] https://natlawreview.com/article/california-ag-issues-legal-advisories-application-california-law-use-ai
[5] https://www.lexology.com/library/detail.aspx?g=50fe29d9-9efb-4701-8354-f07c5028025f
[6] https://www.hunton.com/privacy-and-information-security-law/california-ag-issues-legal-advisories-on-the-application-of-california-law-to-the-use-of-ai
[7] https://www.jdsupra.com/legalnews/california-attorney-general-issues-7789480/
[8] https://oag.ca.gov/news/press-releases/attorney-general-bonta-issues-legal-advisories-application-california-law-ai
[9] https://news.bloomberglaw.com/artificial-intelligence/california-attorney-general-warns-businesses-on-ai-compliance
