AI medical scribes are now core clinical tool. Adoption is moving fast. The AMA’s 2026 survey found 81% of US physicians use AI at work, up from 38% in 2023.
But that growth puts your product under a microscope. Your scribe listens to visits, reads outside records, and writes into your customers’ EHRs. Health system security teams now want to know how you’ve tested all of that for a breach precaution.
Compliance badges no longer settle the question. In September 2026, HP shared a case where an attacker fed an AI scribe malicious data. That data exploited a flaw in the tool the scribe used to create PDFs and gave the attacker full EHR access. The vendor had passed compliance reviews and claimed SOC 2 and HIPAA compliance
So, our advice to you is blunt. If hospitals stop accepting a SOC 2 report alone, you need to bring more proof to the deal. That’s exactly why an LLM pentest report matters.
Our Analysis: Where AI Scribe Security Broke in 2026
We looked at four public AI scribe cases from the last year. None involved a clever hacker. Each broke at an ordinary point in the workflow, and that is something a pentest checks.
Here are the four cases we reviewed:
Case
What happened
Heidi Health jailbreak (disclosed to vendor February 2026)
Researchers at Mindgard bypassed the scribe’s guardrails with prompts. It renamed itself “Nexus” and started making clinical decisions. Heidi said it had already fixed the issue and no patient data was exposed.
Model: The limits that keep the scribe to note taking
Ontario hospital AI scribe breach (regulator findings, October 2025)
A former physician’s unapproved AI scribe joined a virtual rounds meeting through an old calendar invite. It recorded talk about 7 patients and emailed a summary to 65 people, including former staff.
Capture: Who and what can record a clinical conversation
Sutter Health and MemorialCare lawsuit (filed April 2026)
Patients allege two health systems used an AI scribe to record their visits without informed consent, then sent the audio to the vendor’s servers.
Capture and Vendor Storage: Consent before recording, and who sees the audio after
Ontario Auditor General audit (May 2026)
Auditors tested 20 approved AI scribes. 12 recorded the wrong drug and 9 invented treatment details nobody mentioned.
Outputs: What the scribe writes into the record

We found four patterns that stand out:
- If any user can talk the scribe past its limits, those limits aren’t a security control. A pentest checks what happens when someone tries.
- In the case of Ontario Hospital, nobody actually hacked anything. The hospital’s systems worked as designed. Its offboarding didn’t. Mapping who and what can join a clinical conversation finds this gap fast.
- If a scribe invents a drug by accident, a poisoned template or hidden instruction can make it do so on purpose. That’s why you need to check what lands in the record.
- For the case of Sutter Health and MemorialCare, the lawsuit does not claim breach. It claims the recording itself was unlawful, because patients weren’t asked first. California requires consent from everyone in the room. If your scribe can start recording before consent is captured, that’s a flaw you can test, not just a legal risk.
Our take? Compliance reviews check the vendor on paper. These incidents happened in the daily workflow, so that’s where your testing belongs. And testing the workflow takes a different kind of pentest.
How Is LLM Pentesting Different from Web App Pentesting?
A web app pentest asks whether flaws in code let an attacker in. An LLM pentest asks whether an attacker can talk the model into working against you.

Four differences shape how testers approach your AI scribe.
1. Attack Surface: Code vs Plain Language
Web apps take input in fixed shapes, like forms and URLs. Your scribe treats every word as input, from patient speech to last month’s notes. So testers write attacks in everyday language and hide them where staff won’t look.
2. Test Results: Repeatable vs Unpredictable
A model might resist an attack nine times and give in on the tenth. Testers repeat each attack with new wording. One success counts, because real attackers can retry for free.
3. Permissions: User Access vs Scribe Access
Your scribe has its own permissions, including write access to patient charts. Testers check what an attacker can make it do with that access.
4. Fixes: Code Patches vs Design Changes
You can’t patch a model’s habit of following instructions. OWASP warns against relying on the system prompt for security. Real fixes come from design:
- Tighter scribe permissions
- Treating outside documents as untrusted
- Checking output before other systems use it
- A doctor’s review before notes post
But, you still need both tests. A manual pentest for your SOC 2 Type II program covers your infrastructure. Wheras an LLM pentest covers the AI layer on top of it.
Not sure your vendor’s last pentest covered the AI layer? We will review it for you and show you what’s missing.
LLM Pentest Scope: 8 Attack Types to Test in Healthcare AI Scribes
A full LLM pentest scope covers 8 attack types across one patient visit, from the microphone to the vendor’s storage. Each maps to the OWASP Top 10 for LLM Applications (2025).

Here’s what each one means, how testers check it, and how to protect your clinic.
1. Prompt Injection Through Clinical Inputs
Prompt injection is hidden text that tricks your scribe into following it like a command. It’s the top risk for scribes, because one outside document is enough.
Testers hide instructions where the scribe will find them:
- White text in a referral letter
- File metadata or scanned images
- Spoken lines during a mock visit
- A prior note the scribe reads for context
If the scribe fails, one bad document can change a chart. It might add a fake allergy, drop a real finding, or open a path into the vendor’s systems.
To protect your clinic:
- Ask your vendor, in writing, how the scribe stops documents from acting as instructions.
- Turn off file and old note imports you don’t need.
- Have doctors review every note and flag details nobody said.
2. Data Leakage Across Patients and Clinics
Data leakage is when one patient’s details reach another patient’s note or another clinic’s account. Two walls must hold: between patients, and between clinics on your vendor’s platform.
Testers try to break both walls:
- Asking the scribe’s chat or edit tools about other patients
- Reusing one tablet session across visits
- Checking whether old transcripts carry into new notes
- Trying to reach one test clinic from another, with vendor approval
- Checking shared templates, support tools, and logs for patient data
If the scribe fails, one patient’s diagnosis lands in another’s note, or another clinic sees your visits. Either way, you likely face a reportable breach you didn’t cause.
To protect your clinic:
- Give every clinician a personal login. No shared tablet sessions.
- Set the scribe to clear its session after each visit.
- Ask your vendor for proof that customer accounts stay separate.
3. Guardrail Bypass and System Prompt Leakage
Guardrail bypass pushes the scribe past its limits, like writing diagnoses instead of notes. System prompt leakage exposes its hidden setup. Both show attackers how your scribe works.
Testers try to:
- Pull out the scribe’s hidden instructions
- Use role play or rewording to get diagnoses or drug changes
- Talk it into dropping its limits
If the scribe fails, attackers learn what they need for their next move. The scribe may also write a diagnosis no doctor made, and a rushed signature puts it in the legal record.
To protect your clinic:
- Turn off chat or suggestion features you don’t use.
- Ask your vendor to block diagnoses and treatment plans.
- Train doctors to read notes before signing, not skim them.
4. EHR Privileges and Integration
EHR privilege testing checks how much damage someone could do with your scribe’s access to patient charts. That access decides how far every other attack can go.
Testers check:
- What the scribe’s account can read and write
- Whether it can post to another visit or edit a signed note
- Whether its tokens can be stolen from the app and reused
- Whether notes post as drafts that need a doctor’s approval
If this fails, the scribe becomes a back door into your EHR. An attacker could read full histories or change records at scale, and it would look like normal scribe activity.
To protect your clinic:
- Limit the scribe to draft notes in the open visit.
- Require a doctor to approve every note.
- Rotate the scribe’s tokens and review its logs every month.
5. Output Handling in PDFs, Portals, and Exports
Output handling testing checks whether harmful text from the scribe can damage the systems that show it. Your portal, PDF tools, and billing exports trust whatever the scribe writes.
Testers try:
- Pushing code through the scribe into the patient portal
- Feeding crafted content to the PDF tool to reach internal files
- Slipping formulas into billing exports
If this fails, a bad note can run code in a patient’s browser or expose internal files. That’s exactly how HiddenLayer’s researchers reached the vendor’s systems.
To protect your clinic:
- Ask your portal and PDF vendors to strip code from scribe text.
- Confirm your scribe vendor’s PDF service can’t reach internal files.
- Have billing staff open exports as plain text.
This is where AI bugs become classic web bugs. A regular web app pentest won’t catch them unless testers attack through the model.
6. Capture Devices and the Audio Pipeline
Capture device testing checks whether visit audio stays safe from the microphone to the cloud. That audio is the most sensitive data your scribe handles, and it starts on a tablet that moves between rooms.
Testers check:
- Device locks, settings, and stored audio
- Encryption in transit and separation from clinical networks
- How long devices keep recordings
- Whether recording can start before patient consent
If this fails, one lost tablet exposes hours of recorded visits. Or recording starts before the patient agrees, which is now a legal risk too.
To protect your clinic:
- Manage scribe devices centrally, with auto lock and remote wipe.
- Keep them on their own wireless network.
- Use a short consent script, and record only after the patient agrees.
7. Data Poisoning and Fabricated Clinical Details
Data poisoning is when someone corrupts the sources your scribe reuses, like templates, prior notes, and phrase libraries. False details then spread into new notes.
Testers check:
- Whether they can plant false details in a template or phrase library
- Who can edit templates, and whether edits get logged
- Whether the scribe repeats details nobody said
If this fails, a fake medication quietly spreads across many patients’ notes. With no record of its source, your team may not catch it until it affects care.
To protect your clinic:
- Limit who can edit templates, and log every change.
- Review shared templates every quarter.
- Ask doctors to flag details nobody discussed.
8. Data Retention and Model Training
Retention testing checks whether your vendor really deletes audio and keeps your data out of model training. Most vendors promise both. Testers check whether the promises hold:
- Deleting test visits, then looking for leftover audio or transcripts in the app, API, or exports
- Checking what stays on devices and in browser caches
- Watching app traffic for data sent to outside AI providers
- Comparing results with your BAA
If this fails, “deleted” recordings stay retrievable, or patient conversations reach an AI model your BAA never covered. Either way, the exposure lands on you.
To protect your clinic:
- Put deletion timelines and a training ban in your BAA.
- Ask for proof of deletion, not just a promise.
- Get the names of every outside AI provider that sees your data.
Once testing is done, the report is what you actually use.
Securing AI Tools in Healthcare: What You Get and How Often to Test
A good pentest report shows what’s broken, how to fix it, and proof the fix worked. Expect:
- Every finding ranked by severity, with proof
- A map to the OWASP Top 10 for LLM Applications
- Fixes for your IT team and asks for your vendor
- Evidence for your HIPAA risk analysis, SOC 2 report, or ISO 27001 program
- A retest that confirms your fixes
Test at least once a year, and again after any big change:
- A new scribe or vendor
- A new EHR integration
- A model update
- New data sources, like templates
HIPAA doesn’t name pentesting today, but it does require regular technical checks of your safeguards. The proposed Security Rule update would require a pentest every 12 months. HHS now expects final action in July 2027.
Don’t wait for the final rule. Your scribe handles patient data today. If you haven’t mapped your AI tools yet, start with a security gap analysis.
How KLEAP Runs LLM Pentests for AI Medical Scribes
KLEAP runs the whole engagement for clinics that use AI scribes but don’t have an AI security team. Here’s how:
- We scope around your clinic. We map how visit data moves, so you pay only for what matters.
- People do the testing, not scanners. We craft poisoned referral letters, templates, and records by hand.
- We work with your vendor. We handle approvals and check their pentest report for gaps.
- You get usable evidence. Every finding maps to OWASP and fits your HIPAA, SOC 2, or ISO 27001 records.
- Fixes stick. We give your team clear steps, then confirm each fix holds.
Your scribe already hears every patient visit. Find out what an attacker could make it do before someone else does.
Frequently Asked Question
What is an LLM pentest for an AI medical scribe?
It’s a manual security test. Ethical hackers try to trick your scribe the way a real attacker would. They feed it poisoned documents and crafted prompts. Then they check if it leaks patient data, writes to the wrong chart, or opens a path into your EHR.
Does my clinic need one if the vendor is HIPAA compliant?
Yes. Compliance shows that controls exist on paper. An LLM pentest shows whether they hold up in your clinic, with your EHR, staff, and devices.
Who owns the pentest: the clinic or the vendor?
Both share it. Your clinic owns tests of the EHR link, devices, and staff roles. Your vendor owns tests of its model and cloud, and should share that report. Once you know what to buy, use this checklist to vet any proposal you get.
Will an LLM pentest disrupt patient care?
It shouldn’t. Testers work in a test account or EHR sandbox with fake patient data. They never touch live visits.
Does an LLM pentest check whether notes are clinically accurate?
No. Your clinical team owns note accuracy. The pentest checks whether attackers can manipulate the scribe.
How does an indirect prompt injection attack work against an AI scribe?
An attacker hides instructions in something the scribe reads later, like a referral letter. The scribe treats that hidden text as a command. It may change the note, leak data, or trigger a backend action. The attacker never touches your network.
How do you prevent prompt injection in an AI medical scribe?
You can’t fully prevent it, so you limit the damage. Keep the scribe’s EHR access narrow and treat outside documents as untrusted. Check output before other systems use it, and require a doctor’s review before notes post.
What's the difference between LLM security testing and an LLM pentest?
LLM security testing is the broad category. It includes automated scans, model evaluations, and red teaming. An LLM pentest is a scoped, manual test where people attack your setup and prove what they can exploit.
Can automated tools replace a manual LLM pentest?
No. Open source tools like Garak and PyRIT run big libraries of known attacks fast. They don’t know your clinic’s workflow, so they miss attacks built around referral letters, EHR access, and staff roles.
How is an LLM pentest different from a vulnerability scan?
What should we ask our AI scribe vendor about penetration testing?
Ask for their latest LLM pentest report, plus its date and scope. Check that it covered indirect prompt injection and separation between clinics. Then ask how they fixed critical findings and whether they retest after model updates.
