AI adoption in healthcare in 2026 is no longer experimental.
According to Doximity’s 2026 State of AI in Medicine Report, 63% of physicians surveyed between November 2025 and January 2026 use AI in clinical practice.
The number was 47% just nine months earlier.
However, 56% of medical group leaders say their organization does not have a formal policy on AI use.
For healthcare organizations deploying AI, the harder question is what needs to be verified before an AI tool can access PHI, interact with an EHR, generate clinical documentation, or store patient data during its workflow.
There is no universal HIPAA-compliant AI label you can rely on. Compliance depends on how your organization deploys the tool, what data it handles, where that data goes, who can access it, whether a BAA is in place, and how the system is monitored.
In this blog, we will explore how you can ensure HIPAA compliant use of AI in healthcare in 2026.
What Exactly is HIPAA Compliant AI?
AI can be used in a HIPAA-compliant manner, but no AI model is automatically HIPAA compliant on its own. Compliance depends on how your healthcare organization deploys the system, what PHI it can access, where that data moves, which vendors handle it, and which safeguards you put around it. That means you need to look beyond the model itself.
A HIPAA compliant AI deployment includes several connected layers:
- AI model: the model generating, analyzing, or retrieving information.
- AI application: the tool clinicians or staff actually use.
- Infrastructure: where the application processes and stores PHI.
- Integrations: APIs, EHR connections, retrieval systems, and other services that move data.
- Data: what PHI enters the system, where copies remain, and how long they are retained.
- Users: who can access the AI and what patient information each user is allowed to see.
For example, an enterprise AI product may support HIPAA-regulated workloads, but the deployment can still create risk if its EHR integration has excessive access, PHI appears in logs, or another provider receives patient data outside the approved data flow.
That leads to the next question: What exactly has to be true for an AI system to be considered HIPAA compliant in practice?
What Makes Your AI System HIPAA Compliant?
No single feature, certification, or vendor claim makes an AI system HIPAA compliant.
Compliance depends on how your healthcare organization deploys it. In practice, that means verifying the legal relationship, PHI flow, access boundaries, integrations, and the controls actually enforced in production.
Is the AI Vendor Legally Allowed to Handle PHI?
If an AI vendor creates, receives, maintains, or transmits PHI on behalf of your healthcare organization, you need a signed Business Associate Agreement (BAA) before you send PHI to the service.
But signing a BAA does not finish the review. For an AI deployment, it must cover how the AI system actually handles your data.
Consider an ambient clinical scribe. It may capture patient audio, send it to a transcription provider, pass the transcript to another model, and then write the generated note into the EHR. The company you contracted with may therefore be only one party handling the PHI.
Before approval, verify:
- whether PHI can be used for model training or fine-tuning;
- which prompts, audio, transcripts, outputs, and logs are retained;
- retention and deletion periods;
- every model provider, cloud service, and downstream vendors that receives PHI;
- whether the same HIPAA restrictions flow down to those external providers handling PHI; and
- how quickly the vendor must report a security incident or breach.
Your organization does not necessarily need to sign a separate BAA with every downstream provider. The AI vendor, as your business associate, must ensure that any subcontractor handling PHI agrees to the same applicable HIPAA restrictions and safeguards.
Pay particular attention to model training. HIPAA does not create a separate rule that simply prohibits AI training on PHI. The question is whether the use is permitted under HIPAA and your agreement. If training is unnecessary for the service, prohibit it contractually.
Does the BAA Cover the AI Product You’re Actually Using?
A vendor’s BAA may not cover every product, subscription tier, API, or feature it offers. So do not approve the vendor for PHI. Approve the specific deployment.
A consumer chatbot, enterprise workspace, healthcare product, and API may use the same underlying model but operate under different contracts and security controls. Coverage can even vary between features within the same enterprise product.
For example, OpenAI currently lists ChatGPT for Healthcare, ChatGPT Enterprise with a Regulated Workspace, ChatGPT for Clinicians, and certain API configurations among its HIPAA-eligible products. But it excludes some functionality even inside eligible environments. Improved memory, for example, is not covered by the BAA for PHI use.
Your approval should therefore identify:
- the exact product and subscription tier;
- the organization account or workspace;
- whether the active BAA explicitly covers that product;
- included and excluded features;
- any HIPAA-specific settings that administrators must enable;
- whether PHI moves through an API, connector, plugin, or other service; and
- whether those components introduce additional processors.
Cloud-hosted models need the same scrutiny. Using a model through Azure or AWS, for example, may create a different contractual and processing relationship than calling the model provider directly.
Treat configuration as part of the compliance boundary. If an administrator can enable an uncovered feature without security review, the deployment can move outside the conditions you originally approved.
So, document approval at the product, account, feature, and configuration level first. Then require review when any of those conditions change.
Can You Trace PHI Through the Entire AI Workflow?
Before you approve an AI system for PHI, map every place patient data enters, moves, gets copied, gets stored, and leaves the system.
But do not stop at the AI model. The vendor you contract with may not operate every part of that chain. One provider may transcribe the audio, another may process the prompt, and another may host temporary files or logs.
PHI may therefore exist in the original audio, transcripts, prompts, retrieved EHR data, generated notes, application logs, caches, backups, and systems operated by third-party providers.
The same issue appears in RAG-based clinical assistants. The model may receive only a small amount of patient context, while the retrieval system indexes entire clinical records. Logs may also capture patient identifiers or clinician queries even when the model itself does not retain prompts.
You therefore need two views of the architecture. The first is logical data flow. The second is the actual technical flow. The implication is simple. You cannot assess whether an AI deployment can handle PHI compliantly until you can trace that PHI from entry to the last place a copy can remain.
Is the AI Limited to the PHI It Actually Needs?
Give the AI access only to the PHI required for the task it performs.
Where HIPAA’s minimum necessary standard applies, your organization must make reasonable efforts to limit the PHI used, disclosed, or requested to what is necessary for the intended purpose.
There is an important exception: the minimum necessary standard generally does not apply to disclosures to, or requests by, healthcare providers for treatment.
That does not mean a treatment-related AI tool should automatically receive unrestricted access to the patient record. From a security perspective, you should still apply least-privilege access and give the system only the permissions its clinical function requires.
Suppose you deploy an AI assistant to help staff schedule follow-up appointments. It may need the patient’s name, contact details, appointment history, and care-team information.
It probably does not need psychotherapy notes, laboratory results, medication history, imaging reports, or the patient’s entire longitudinal record.
The same issue appears in RAG systems. A clinical assistant may need three relevant documents to answer a question, while the retrieval layer can search thousands of patient records. The model may receive only selected content, but the application behind it still has broader access.
Review both sides of the workflow:
- Input: What PHI can the system read or retrieve?
- Output: What PHI can it return, summarize, export, or write back?
- User access: Does the AI respect the permissions of the clinician or staff member making the request?
- Task scope: Can one use case reach data that belongs to another clinical or administrative purpose?
Do not rely on the application’s intended function to define access. Verify the permissions enforced by the EHR, API, retrieval layer, and application itself.
This will ensure that you are separating what HIPAA legally requires from what a mature healthcare security team should still enforce technically.
Can the Data AI Uses Be Treated as De-Identified Under HIPAA?
De-identification determines whether the data your AI uses still falls under HIPAA.
If your organization plans to train, test, retrieve, or analyze patient data without treating it as PHI, you first need to verify that the data meets HIPAA’s de-identification requirements.
HHS recognizes two methods for de-identification:
- Safe Harbor, which requires removal of 18 specified identifiers.
- Expert Determination, where a qualified expert determines that the risk of re-identification is very small.
Until one of those methods is satisfied, treat the data as PHI.
Also, do not assume embedding is de-identified simply because it is numerical. If derived data remains identifiable, continue treating it as PHI. If you intend to treat it as de-identified, the dataset still needs to satisfy HIPAA’s de-identification standard.
The same issue applies to training and testing datasets.
A dataset may have direct identifiers removed but still contain enough information, such as dates, rare diagnoses, locations, or combinations of attributes, to make re-identification possible when combined with other data.
Whether de-identification was done properly determines if the resulting data still counts as PHI, and therefore whether it falls under HIPAA.
Before treating any AI dataset, embedding store, test set, or training corpus as outside HIPAA, document which de-identification method you used and verify that the resulting data cannot reasonably be linked back to an individual.
What PHI Can the AI Access Through Your EHR?
Connecting an AI tool to your EHR can expose far more PHI than the tool actually needs. The key control is the integration layer.
Many modern healthcare AI applications connect to EHR systems through FHIR APIs. The application receives an access token that defines which resources it can read or modify. Suppose an ambient scribe only needs access to encounter details and permission to write a clinical note. If its token can also read medications, lab results, diagnoses, imaging reports, and other records, the integration is over-scoped.
The same applies to write access. A summarization tool that only needs to read patient data should not receive permission to create, update, or delete EHR records.
Review:
- which FHIR resources the application can access;
- whether access is read-only or includes write permissions;
- whether scopes apply at the record, resource, or broader application level;
- how long access tokens remain valid;
- how credentials are stored and rotated; and
- whether every AI-driven access or change appears in your audit logs.
Do not rely only on the vendor’s architecture diagram or stated integration requirements. Test the permissions directly. A token issued for one narrow workflow can become a much larger exposure if its actual scope allows access to the full patient record.
So, before deployment, inspect the integration credentials, verify the exact resources they expose, remove unnecessary permissions, and confirm that every PHI access and write-back action is logged.
Are the HIPAA Controls Actually Enforced in the Deployment?
A signed BAA, SOC 2 report, and clean architecture diagram do not prove that the deployed AI system enforces the controls you approved. You need to test the implementation as well.
For an AI-EHR integration, that means checking the actual API permissions, access tokens, credentials, logging, and downstream connections.
A review may find that:
- a token created for appointment data can also read labs, medications, and clinical notes;
- credentials issued during testing still work months later;
- a tool that only needs read access can also write to unrelated EHR resources;
- AI activity is logged under one generic service account instead of the clinician, patient encounter, or session that triggered it; or
- middleware handling PHI sits between the AI tool and EHR without appearing in the original vendor review.
None of these problems require the AI model itself to fail. They come from how the system was integrated and configured. A security gap analysis can identify where required controls are missing or incomplete. Whereas manual penetration testing shows whether the deployed controls can actually be bypassed.
For systems handling PHI, review the effective permissions instead of the requested permissions. Test token scope, credential expiry, write access, audit events, and every service that receives data after it leaves the EHR.
Before production use, validate the deployed controls against the approved design. Then repeat that testing after major integration, permission, or architecture changes.
Not sure how far one compromised account could reach in your healthcare environment? KLEAP can assess access paths, exposed systems, third-party connections and recovery gaps through a healthcare security risk assessment and manual penetration testing.
Book a slot with us:
Does Self-Hosting Make HIPAA Compliant AI?
Self-hosting an AI system can reduce third-party exposure, but it does not make the deployment HIPAA compliant by default. It simply moves more responsibility back to your organization.
If the model runs inside your own environment, PHI may no longer pass through a public AI provider. That can simplify the vendor chain and give you more control over storage, retention, and access.
But the same HIPAA safeguards still apply. You still need to verify:
- encryption at rest and in transit;
- role-based access to the model, retrieval layer, and underlying data;
- complete audit logging for PHI access;
- secure credential and key management;
- network segmentation between the AI environment and other clinical systems;
- retention and deletion controls;
- backup and recovery protections; and
- BAAs for any third party that can still access the environment or PHI.
Self-hosted RAG systems need particular attention. Because a model may run locally while the retrieval database, monitoring platform, backup service, or management console still sends data to an external provider. In that case, PHI has not truly stayed inside your boundary.
The operational burden also increases. Your team now owns patching, hardening, access reviews, logging, incident response, and infrastructure security that a managed provider may otherwise handle. So, treat self-hosting as an architectural choice, not a compliance shortcut.
Before approving an on-premises or self-hosted AI deployment, map every remaining PHI dependency and confirm that your own controls meet the same HIPAA requirements you would expect from an external vendor.
Whether you host the model yourself or use a managed provider, the same deployment-level checks apply. They become especially important with LLMs, where the same model can be offered through several products with different BAA and data-handling terms.
Are LLMs HIPAA Compliant?
An LLM is not HIPAA compliant by itself. A healthcare organization can use an LLM with PHI only when the specific product, deployment, BAA, configuration, and data flow meet HIPAA requirements.
This follows directly from the checks above. The difference with LLMs is that the same underlying model can reach your organization through several different routes.
You might use it through:
- an enterprise AI workspace;
- a direct LLM API;
- a cloud platform such as Azure or AWS;
- an application built by another healthcare vendor; or
- your own self-hosted environment.
Those deployments do not automatically have the same HIPAA status. So, if you are looking for a HIPAA compliant LLM, start with four checks:
1. Which exact product are you using?
Do not approve a model name such as GPT, Claude, or Gemini. Identify the exact product, account, API, workspace, or cloud service that will process PHI.
2. Is that service covered by your BAA?
A vendor may offer a BAA for one product without covering every feature or service it sells. Before signing the healthcare BAA, you must ensure that it applies to the deployment that will actually handle PHI.
3. What happens to PHI after you send it?
Verify whether prompts, responses, uploaded files, embeddings, logs, cached data, and fine-tuning data are retained. Also check whether any of that data can be used for model training.
4. Which other providers can receive the PHI?
Trace the full processing path. Your application may involve the LLM provider, cloud infrastructure, retrieval databases, monitoring systems, connectors, and other third-party providers.
HIPAA Coverage Across Major LLM Platforms
The table below is a verification reference, not a list of certified HIPAA compliant LLMs. HHS does not provide a HIPAA certification for AI products, and each organization remains responsible for how it deploys the service.
Here’s how verification looks across some major LLM platforms:
OpenAI / ChatGPT
OpenAI currently offers HIPAA-eligible options including ChatGPT for Healthcare, ChatGPT Enterprise with a Regulated Workspace, ChatGPT for Clinicians, and qualifying API accounts.
Confirm the exact ChatGPT workspace or API account covered by your BAA. Also check whether every feature clinicians plan to use is covered.
Some functionality can sit outside BAA coverage even inside an eligible workspace.
Anthropic Claude
Anthropic provides BAA coverage for HIPAA-ready Claude Enterprise and its first-party API. Enterprise coverage requires the organization’s Primary Owner to activate HIPAA compliance and accept the BAA.
Verify that HIPAA mode is actually active. For API use, check the exact API feature: Anthropic covers some functions under its BAA while excluding others such as Batch API, Files API beta, Code Execution, and Computer Use.
Microsoft Azure OpenAI
Microsoft provides HIPAA BAA coverage through its Product Terms and Data Protection Addendum for in-scope Azure services. Microsoft explicitly states that using Azure does not by itself make your application HIPAA compliant.
Confirm that the Azure service used by the workload is in scope, then review your own identity controls, network configuration, logging, storage, and surrounding services. Do not assume every model available elsewhere in the Azure AI ecosystem has identical coverage.
AWS Bedrock
Amazon Bedrock is currently included in AWS’s list of HIPAA-eligible services. AWS states that customers should process PHI only through services covered by their Business Associate Addendum.
Confirm that your AWS account is covered by the BAA and that every AWS service storing or processing PHI is also HIPAA eligible.
A Bedrock application may also use S3, databases, API Gateway, logging, retrieval systems, or other services that process PHI.
Google Cloud / Gemini Enterprise
Google currently lists Gemini Enterprise, Gemini Enterprise Agent Platform, and Generative AI on Gemini Enterprise Agent Platform among products covered by its Cloud BAA.
Execute the Google Cloud BAA and confirm that every service handling PHI appears on Google’s covered-products list.
Google also tells customers not to use products outside BAA coverage or pre-GA services with PHI unless specifically permitted.
The provider may be the same, but the HIPAA implications are very different because the product, agreement, and configuration are different.
Therefore, “HIPAA compliant LLM” is a deployment question, not a model-selection question. Identify the exact service, confirm BAA coverage, verify how PHI is retained and used, and trace every system that receives it before allowing the LLM into a clinical workflow.
How to Choose a HIPAA Compliant AI Tool?
Whether you are deploying an LLM, ambient scribe, clinical assistant, RAG system, or another AI tool, the approval process should answer one question: Can this specific deployment handle PHI without creating access, dataflow, or security risks that fall outside your HIPAA controls?
Use this checklist before allowing the tool into a production healthcare workflow.
Check
What your security team should verify
Why it matters
BAA coverage
Confirm whether the vendor will handle PHI and whether the exact product, tier, API, and features are covered by your BAA.
A vendor may support HIPAA use without covering every product it offers.
PHI data flow
Trace PHI from input through processing, storage, logs, retrieval systems, third-party providers, and final output.
You need to know every place where PHI can exist before you can secure it.
Minimum necessary access
Confirm that the AI can access only the patient data required for its approved task.
A narrow use case should not automatically receive access to the complete patient record.
Data retention
Check how long prompts, outputs, audio, uploaded files, logs, embeddings, and backups remain stored.
PHI may survive in supporting systems even after the main application deletes it.
Model training
Verify whether your PHI can be used for model training, fine-tuning, or service improvement.
Training rules may differ between products, accounts, and configurations.
Third-party providers
Identify every external provider that can process, store, or access PHI as part of the AI workflow.
Your HIPAA boundary extends beyond the AI vendor if other providers also handle PHI.
EHR and API permissions
Check which records the AI can read, whether it can write to the EHR, how broad its API permissions are, and how long its credentials remain valid.
Over-scoped permissions can expose far more PHI than the workflow requires.
Authentication and access control
Verify who can use the AI, what each role can access, and whether the AI respects the permissions of the user making the request.
The AI should not become a shortcut around existing clinical access controls.
Audit logging
Confirm that PHI access, retrieval, changes, exports, and AI-generated write-backs can be traced to the relevant user, session, or workflow.
You need enough evidence to investigate unauthorized access or misuse.
De-identification
Verify that data treated as de-identified actually meets a recognized HIPAA de-identification method.
Removing obvious identifiers does not automatically take data outside HIPAA.
Technical validation
Test the live application, APIs, permissions, authentication, data flows, and integrations against the approved design.
Vendor questionnaires tell you how the system should work. Testing shows you what it can actually do.
Ongoing change control
Reassess new AI features, integrations, permissions, vendors, and configuration changes after deployment.
The system you approved today may handle PHI differently after the next product update.
A useful final test is whether your team can answer five questions without going back to the vendor:
What PHI can the AI access? Where does it go? Who else receives it? What can the AI do with it? And can you prove those controls work in the deployed environment?
The first four questions can often be answered through architecture reviews, contracts, vendor documentation, and configuration checks.
The fifth usually cannot.
A signed BAA will not tell you whether an API token can access more EHR data than intended. An architecture diagram will not show whether authorization can be bypassed. A security questionnaire will not tell you whether PHI leaks through an API response, application log, retrieval layer, or misconfigured integration.
Those require hands-on security testing.
Where KLEAP Fits into Security Validation for HIPAA Compliant AI
KLEAP focuses on the part of the HIPAA review that needs technical evidence rather than another questionnaire.
Its healthcare security testing capabilities include LLM VAPT, AI penetration testing, API VAPT, web and cloud testing, security architecture review, and source code review. That allows the assessment to follow the same surfaces where an AI deployment can expose PHI: the model-facing application, APIs, authentication, permissions, cloud infrastructure, and the integration layer around them.
For a healthcare AI deployment, that means testing questions such as:
- Can the AI or its API retrieve PHI outside the approved scope?
- Can one user access another patient’s data?
- Do API tokens or service accounts have excessive permissions?
- Can authentication or authorization controls be bypassed?
- Does PHI appear in responses, logs, or integrations where it should not?
- Can the AI workflow reach systems or data that the architecture says should be inaccessible?
This is the difference between documenting that an AI deployment should meet your HIPAA controls and verifying that those controls actually hold when the system is used.
Planning to deploy HIPAA compliant AI in a healthcare environment? Book a security assessment with KLEAP to test the application, API, LLM, and integration layers before AI touches your PHI.
Frequently Asked Question
Can I use AI with PHI and remain HIPAA compliant?
Yes. You can use AI with PHI when the specific deployment has appropriate safeguards, access controls, data handling practices, and contractual protections in place. If an AI vendor handles PHI on your behalf, you also need to address the applicable Business Associate Agreement and security requirements.
Which AI tools or LLMs are HIPAA compliant?
There is no universal list of HIPAA-compliant AI tools or LLMs. Whether you can use a given AI tool with PHI depends on factors such as its service tier, BAA, data retention, model-training practices, access controls, logging, configuration, and deployment architecture. Assess the specific product and implementation you’re using rather than relying on a vendor’s HIPAA claim alone.
Can I use public cloud AI with PHI?
Public cloud infrastructure can support HIPAA-regulated workloads, but you cannot assume that any public AI service is appropriate for PHI. Evaluate the service, contractual arrangement, configuration, access controls, data handling, and applicable BAA before you use it.
Does using an open-source model remove the need for BAAs?
No. The model license and the HIPAA obligations around your deployment are separate issues. Running an open-source model locally may remove the model provider from the PHI flow, but other vendors may still handle PHI through hosting, storage, monitoring, support, or integrations, you still need to review those relationships.
What does AI HIPAA compliance actually require?
A HIPAA compliant AI platform needs more than a HIPAA statement or BAA. You should evaluate how it handles PHI, limits access, protects data in transit and at rest, manages retention and deletion, records activity, connects to other systems, and prevents unauthorized disclosure. The security of your actual deployment and integrations matters just as much.
Does HIPAA-compliant AI software require a BAA?
If the AI vendor creates, receives, maintains, or transmits PHI on your behalf, the vendor generally acts as a business associate and you need a BAA. Make sure the BAA covers the specific AI service you use, not simply the vendor as a company.
Are on-premises generative AI platforms HIPAA compliant?
Not automatically. Keeping the model inside your own environment can reduce third-party exposure, but you still need to protect ePHI with appropriate administrative, physical, and technical safeguards. Verify access controls, encryption, audit logging, credential management, retention, backups, network segmentation, and incident response, and if any external vendor can create, receive, maintain, or transmit ePHI on your behalf, you may also need a BAA with that vendor.
How do I securely integrate AI with an EMR?
Start by mapping exactly what PHI your AI tool can access and where that information travels. Review the API layer, authentication, OAuth scopes, access permissions, the data the system returns, and audit logging before you connect it to production.
What HIPAA controls do I need when using AI tools?
It depends on your AI workflow and the PHI involved, but you should address risk analysis, minimum-necessary access, encryption, access control, audit logging, data-flow management, vendor risk, de-identification where applicable, incident response, and workforce training.
Do I need a BAA to use an AI tool with PHI?
If the AI vendor is acting as a business associate and handles PHI on your behalf, you generally need a BAA. But signing one doesn’t by itself prove the implementation is secure — you should also examine the vendor’s data handling, security controls, retention practices, subprocessors, and actual deployment configuration.
