Data
AI
The Stack Underneath
Hello, we are BRILLIAN’S. Trying to make an effort to put the right people for you to get the best results. Just insight !
Most coverage of agentic AI in healthcare stops at the slide-deck use cases, drug discovery, hospital-of-the-future, the dream demo. The actual problem is that almost none of those use cases survive a hospital security review.
The pilots that ship in 2026 are narrower than the headlines suggest. They run inside a tight set of guardrails: BAA-covered model providers, structured audit logs, a human signature on anything clinical. The pilots that fail tend to fail on the same three issues — hallucination consequences, audit trail, and the wrong human-in-the-loop boundary.
This post covers the four use cases that are actually shipping, the named radiology workflow already inside production worklists, the three risk classes that kill most deployments, and what a HIPAA-aware rollout looks like.
Agentic AI is autonomy plus multi-step reasoning plus tool use, scoped by guardrails. An agent reads a goal, plans steps, calls tools (an EHR query, a scheduler, a payer-portal API), checks the result, and either advances or escalates. It is not a single model call.
That is a meaningful step away from the AI most hospitals already have. Predictive ML scores a patient for sepsis risk and stops there. Generative AI summarizes a note and stops there. Agentic AI takes action across systems — it queries the EHR, schedules the follow-up, opens the prior-auth case, then circles back to confirm.
For a clearer split between the two terms most readers conflate, see Agentic AI vs AI Agents. For the engineering pattern underneath, see Agentic AI Software Development.
The action-taking is exactly what makes healthcare deployment harder than software-engineering deployment. A wrong tool call in a coding agent costs an hour of debug. A wrong tool call in a clinical agent can land in someone's chart.
The four patterns below are in production at health systems and at trial sponsors today. They share two traits: the agent operates inside a tightly scoped tool set, and a human owns the consequential decision.
Patient-screening agents read inclusion and exclusion criteria against EHR data, surface candidates, schedule consent calls, and flag suspected adverse events for the principal investigator. The pattern replaces research-coordinator chart review — work that was previously two coordinators reading the same patient's notes for an hour to confirm trial eligibility.
The build is straightforward: the agent has a tool definition for the EHR query layer, a tool for the scheduler, and a structured-output schema for the screening verdict. Where it breaks: trial-protocol amendments. When the sponsor amends inclusion criteria mid-study, the agent's tool definitions have to change with them, and most teams forget to wire that into the change-control process. The fix is to treat the protocol-to-tool mapping as a versioned artifact, not a prompt.
Multi-step agents pull patient context across visits, draft the after-visit summary, and queue the order set for the physician to sign. The pattern replaces around two hours per day of physician charting — a major driver of clinician burnout in most US health systems.
Where it breaks: silent hallucination of medications the patient is not on. A summarization model that invents one extra med is annoying. A documentation agent that drops that med into a draft order set is dangerous. Mitigation is retrieval-grounded generation, structured-output schemas tied to the medication list, and a hard rule that the physician signs every order.
Agents read the payer's policy PDF, assemble the authorization packet from the patient chart, submit through the portal, and fight the denial with a structured appeal. The pattern replaces a $20-per-hour staff team plus a roughly 30-day turnaround.
Where it breaks: payer policy changes mid-flow. Major payers update medical-necessity policies on a quarterly cadence, sometimes mid-month. An agent caching last quarter's policy will assemble a packet that is correct in form and wrong in substance. The fix is a freshness check on policy retrieval and a confidence threshold that escalates ambiguous cases to a human reviewer.
Pre-visit symptom-collection agents capture history, surface red-flag presentations to a nurse, and draft the visit summary for the clinician before the appointment starts. The pattern replaces nurse-line triage queues for routine intake.
Where it breaks: under-triage of cardiac and stroke symptoms. The agent's prompt is usually tuned to avoid over-escalation — patients flagged unnecessarily clog the ED and erode trust in the tool. Tune too far that direction and the agent under-flags chest pain in a 62-year-old. Mitigation is a small, hard-coded set of pattern-match rules that always escalate, layered under the LLM-driven triage. Belt and suspenders, deliberately.
Imaging AI is mostly not agentic — most production radiology AI is a CNN reading a single study. The agentic workflow is everything around that read.
A radiology agent routes the study to the right worklist based on modality, body part, and stat status. It pulls priors from the PACS for comparison and stages them for the radiologist before the read begins. After the dictation, it populates a structured report from the draft, fills in the recommended follow-up interval per the relevant Lung-RADS or BI-RADS template, and queues the patient-facing letter for sign-off.
Where it breaks is the same as the documentation case: the structured report has to be grounded in what the radiologist actually said, not in what the model thinks they meant. The mitigation pattern is identical — retrieval-grounded fields, structured-output schemas, a hard sign-off step. The radiologist owns the impression. The agent owns the routing, the priors, and the paperwork.
Three risk classes kill most deployments. Each has a known mitigation pattern; the mistake is treating any of them as something the model vendor will solve later.
The same hallucination that costs a software team a wasted hour can cost a patient an unnecessary procedure, a missed allergy, or a delayed diagnosis. The asymmetry is the point — healthcare does not have the cost-of-being-wrong tolerance that lets generic copilots get away with optimistic generation.
Mitigation is layered. Structured-output schemas constrain the shape of the agent's response. Retrieval-grounded responses tie every clinical assertion to a source document. Real-EHR-data evals — not synthetic benchmarks — catch the failure modes that show up only on real, messy data. And human sign-off is non-negotiable on any clinical assertion that reaches a chart.
HIPAA covers PHI handling and breach response. FDA Software-as-a-Medical-Device guidance covers clinical decision support that meets the device threshold. State privacy law — Texas, Colorado, Washington, California — adds residency and biometric requirements that apply even when HIPAA does not.
The audit-trail requirement is the part most teams underestimate. The audit log has to capture every tool call, every retrieved document, every model version — not just the final answer. That log has to be immutable and retained on the same schedule as the medical record. Building this in from day one is straightforward. Retrofitting it after a security review is brutal.
A fully autonomous clinical action is not deployable today, and pretending otherwise is how programs get killed. The real question is which checkpoint pattern makes the agent useful instead of useless.
The boundary that matters is between "agent drafts, human signs" and "agent acts, human reviews after-the-fact." The first is defensible across clinical care. The second is rarely defensible in clinical care and often defensible in revenue-cycle and admin work, where the consequence of a wrong action is a reversible billing entry, not a wrong dose.
This is the section the buyer is actually scrolling for.
The deployment shape is consistent across health systems with mature security teams: BAA-covered model providers only, no PHI in logs, no PHI in prompts that route to non-BAA tools, regional data residency that matches the system's compliance footprint, encryption at rest and in transit, and deterministic redaction of PHI before any external API call. The redaction step is the one teams skip and regret — token-level redaction at the boundary, before the request leaves your VPC.
Three checkpoint patterns cover the realistic surface. Pre-action approval — the human signs before the agent acts — applies to anything clinical. Post-action review — the human audits a sample after the fact — applies to revenue-cycle and other reversible domains. Exception-only review — the human inspects only flagged cases — applies to low-risk admin work and only after the eval is mature enough to trust the flag rate.
Pick the right one per use case. The mistake is using one pattern across the board.
Real EHR-data evals beat synthetic benchmarks every time. Add adversarial prompts that specifically probe the failure modes flagged earlier — hallucinated meds, wrong-payer policy, under-triaged red flags. Canary the deployment with a small physician group for two to four weeks. Run a weekly error review with that group. Only scale when the error rate is below the unit-economics threshold and stable across two consecutive review cycles.
For the deeper engineering pattern, see Agentic AI Software Development.
We deploy agentic AI under regulatory constraint for clients in privacy-sensitive industries. The patterns above — BAA-covered providers, structured audit logs, redaction at the boundary, the right human-in-the-loop checkpoint per use case — are the same patterns we apply when the constraint is SOC 2 or PCI rather than HIPAA. The framing changes; the engineering does not.
We do not yet have a healthcare-specific case study. We have track record in agentic engineering and in regulated-data deployment, and we are explicit about the difference. If you are evaluating a healthcare agentic AI pilot and the vendor's pitch deck claims a stack of named hospital deployments, ask which ones you can call.
If you want to talk through the implementation pattern for your specific use case, our AI development services page is the entry point.
HIPAA compliance is a property of the deployment, not the model. An agentic AI deployment is HIPAA-compliant when every PHI-touching component is covered by a Business Associate Agreement, PHI does not appear in logs or in calls to non-BAA tools, and the audit trail meets the system's medical-records retention policy. The model provider is one variable in that picture, not the whole picture.
The terms are used interchangeably in marketing. We use "AI agents" for the singular component — one LLM, one tool set, one task — and "agentic AI" for the system pattern that orchestrates multiple agents, retrieval, structured outputs, and human checkpoints. The distinction matters when the buyer is comparing vendor pitches; see Agentic AI vs AI Agents for the longer version.
No, and the framing is wrong. The deployments that ship are the ones that take charting, prior-auth, scheduling, and triage prep off the clinician's plate. The clinical decision stays with the clinician. The deployments that fail are the ones that try to invert that.
Patient-screening agents read inclusion and exclusion criteria against EHR data, schedule consent calls with eligible candidates, and flag suspected adverse events for the principal investigator. The hard part is keeping the agent's tool definitions in sync with mid-study protocol amendments — treat that mapping as a versioned artifact.
Three: hallucination consequences (mitigated by retrieval grounding, structured outputs, and human sign-off on clinical assertions), audit and regulatory exposure (mitigated by immutable per-tool-call audit logs and BAA-covered providers), and the wrong human-in-the-loop boundary (mitigated by picking the checkpoint pattern that fits the consequence class — pre-action for clinical, post-action for revenue-cycle, exception-only for low-risk admin).
You might also like
