AI for incident reporting: what it can and cannot safely do

The short answer: AI drafts, people decide

AI is useful where the task is language: turning a messy account into a structured record, spotting what is missing, and summarizing long text. It is unsafe where the task is judgment or accountability: severity, cause, fault, deadlines and closing an action.

The rule is simple. The AI drafts. A named person reviews, edits and signs. The record shows both steps. AAAHC's AI governance guide for ambulatory care (August 2026) calls this meaningful human oversight. See how AI intake changes incident reporting.

AI can ask, fill and flag at intake

Keep the reporter's own words as the source of truth. Everything the AI adds sits beside them, never replaces them.

A worked example (illustrative)

1. The reporter writes"Found a resident on the floor beside the bed around 2 a.m. He said he was going to the bathroom. I didn't see an injury."
2. The assistant asksWhat time exactly? Was the bed alarm on? Who found the resident and checked for injury? Who was told, and when?
3. The draft fills inFields take only what was said, such as "about 2 a.m." and "no injury seen," each marked draft. Unanswered fields stay blank, not guessed.
4. The reviewer editsAfter the nurse's assessment, the reviewer changes the injury field to a skin tear on the left forearm and records the notifications.
5. The draft stops short of a causeIf nobody said the alarm failed, the draft does not say so. Causes come later, from the investigation.

AI can draft and sort in investigations

AI should not decide, close or contact

• Set final severity or harm class. It drives escalation and outside reporting.
• Decide if a deadline applies. A model can remind. A person decides.
• State a root cause or assign fault. Causes are findings, not predictions.
• Recommend discipline. That belongs in HR, outside root cause analysis.
• Close or verify a corrective action. Verification needs real-world evidence.
• Contact patients, families or regulators. A named person must do it.
• Change the record silently. Every AI edit must be visible and reversible.

See incident reporting in healthcare for deadlines.

Hallucination is a real risk

NIST's Generative AI Profile calls false output confabulation: confidently stated but erroneous content. It follows from how models work: they predict likely text. NIST adds that people tend to over-trust automated output, which it calls automation bias.

A 2025 npj Digital Medicine study tested clinical note generation across 18 configurations and 12,999 clinician-annotated sentences. It found a 1.47 percent hallucination rate and a 3.45 percent omission rate. Refining prompts and workflows brought major errors below previously reported human note-taking rates.

That study covers clinical notes, not incident intake, so it is not a benchmark for any product. Three lessons carry over. Error rates are not zero. Omissions can outnumber inventions. Design and testing change the result.

A missing fact, such as a witnessed fall, can matter as much as an invented one.

Test AI on your own incidents

Do not rely on a vendor demo or figures from other tasks. Test on your own text, and set the pass mark before you see results.

1. Collect real casesPull 50 to 100 past narratives, including a few hard ones. Use a BAA or proper de-identification.
2. Set the pass mark firstDecide what rate of inventions, omissions and wrong categories you accept, and which errors are never acceptable.
3. Compare drafts with signed recordsHave clinicians mark each difference: invention, omission, misattribution or harmless rewording.
4. Test the review step tooDo reviewers catch planted errors? A rubber-stamp review makes any draft unsafe.
5. Repeat after every changeRe-run when the vendor changes the model, prompts or form.

Human-in-the-loop design that holds up

1. Preserve the reporter's accountStore what was written or dictated, unchanged.
2. Label AI output as draftAI-filled fields stay marked and unapproved until a person acts.
3. Review each fieldA named reviewer approves, edits or rejects each one.
4. Sign by a named personA person signs the report or investigation.
5. Log the whole chainThe audit trail records the AI suggestion, the human edit, who approved it and when.
6. Audit samplesCompare drafts with final records on a schedule to measure edit rates and error types.

The last step counters automation bias. Put the most friction where a wrong draft hurts most: severity, cause and reportability.

IncidentKit follows this design. Lauren asks follow-up questions, fills the form and drafts the investigation. Every AI-drafted field shows "Lauren · draft" until approved, and a person always reviews, edits and signs. Staff report by text now. Voice and human-authored RCA templates are rolling out.

HIPAA and BAA: the model provider counts too

If narratives contain patient information, the vendor is a business associate. HIPAA names patient safety activities (42 CFR 3.20) as covered functions. The agreement must include the terms in 45 CFR 164.504(e). See business associate agreement and HIPAA.

The AI model provider is the next link. A subcontractor that handles protected health information for the vendor is itself a business associate, and the vendor's contract must bind it to the same limits.

Ask which provider processes your data, whether a BAA covers that exact service, and whether your data trains models. Return-or-destroy terms cover logs and backups.

• Add the AI feature to your HIPAA risk analysis.
• Send the model only what the task needs.
• Ask for a breach notice window far shorter than 60 days.
• With a patient safety organization, ask counsel about your evaluation system.

Log enough to rebuild who decided what

HIPAA requires mechanisms that record and examine activity in systems holding electronic protected health information, and controls against improper alteration. For AI features, keep enough to reconstruct who decided what.

Store each draft as it was shown, because models change. Retention runs for years: OSHA records five years, process safety reports five years, HIPAA policy documents six years. See the audit trail.

Accreditors now write AI expectations

In September 2025 the Joint Commission and the Coalition for Health AI released guidance with seven elements: AI policies and governance, patient privacy and transparency, data security, ongoing quality monitoring, voluntary blinded reporting of AI safety events, risk and bias assessment, and training.

The fifth element points back to your incident system. AI errors are reportable events. Make sure staff can file one, and review them like any other.

AAAHC's v45 standards (August 2026) add an AI governance framework: leadership accountability, risk assessment, cybersecurity, human oversight and transparency. They apply to surveys on or after December 15, 2026. AAAHC's guide includes an AI inventory and vendor checklist.

Add your incident software's AI to the inventory. See AAAHC and Joint Commission.

Questions to ask vendors

Design

• Which fields can the AI draft, and which can it never change?
• Is every AI-drafted field marked until a person approves it?
• Can a reviewer see the original text beside each draft?
• Can we turn each AI feature off?

Data

• Which model providers process our data, and where?
• Does a BAA cover that exact service?
• Is our data used to train models, by you or your provider?
• What do you retain, and how is data deleted at exit?

Evidence

• What do you log for each AI action?
• How do you measure fabrication and omission on incident text?
• What happens to drafts and tests when the model changes?

Pair these with the questions in the buyer's guide.

Frequently asked questions

Sources

• NIST, Artificial Intelligence Risk Management Framework: Generative AI Profile (NIST AI 600-1)
• npj Digital Medicine, A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation (2025)
• Joint Commission Perspectives, November 2025 (Joint Commission and CHAI responsible AI guidance)
• AAAHC, v45 Standards press release (August 18, 2026)
• AAAHC, 1095 Advance AI Governance Guide press release (August 17, 2026)
• 45 CFR 160.103, Definitions, including business associate (eCFR)
• 45 CFR 164.504, Business associate contracts (eCFR)
• 45 CFR 164.410, Notification by a business associate (eCFR)
• 45 CFR 164.308, Administrative safeguards, including risk analysis (eCFR)
• 45 CFR 164.312, Technical safeguards (eCFR)
• 45 CFR 164.316, Policies, procedures and documentation requirements (eCFR)
• 42 CFR 3.20, Patient safety work product definitions (eCFR)
• IHI and NPSF, RCA2: Improving Root Cause Analyses and Actions to Prevent Harm

Reviewed against the sources above on Oct 5, 2026. Rules change: confirm current requirements with the issuing body or your counsel before relying on any summary.