I Built PA-Trace: An On-Device MedGemma Workflow for Prior Authorization

I believe AI can improve humanity.

But in healthcare, that future may start with something less glamorous: removing administrative friction.

That was the idea behind PA-Trace, my submission to Google Research's MedGemma Impact Challenge on Kaggle, which ran from January 13, 2026 to February 24, 2026.

An estimated $35 billion of annual US healthcare administrative spending is tied to prior-authorization administration, according to a 2024 Health Affairs Scholar paper. In the public PA-Trace Kaggle writeup, I framed one narrow slice of that problem: imaging orders that require staff to dig through unstructured clinic notes, cross-check payer criteria, and assemble a packet that can survive first-pass review.

Earlier this year I built Checkstand, a local Gemma 3n submission focused on offline receipt intelligence. PA-Trace came from the same instinct: keep the model close to the data, keep privacy boundaries simple, and solve one workflow well instead of promising magic.

If you read my earlier post Hands-on Agentic AI: LangChain 1.0, the simple mental model still applies: model + tools = agentic AI.

For healthcare admin workflows, I would extend it like this:

model + tools + validation + deterministic rules = something a human can actually trust

What Is Prior Authorization, And Why Does It Hurt So Much?

In plain English, prior authorization is the process of proving to a payer that a requested treatment, scan, or medication meets the insurer's criteria before it gets approved.

For imaging, that often means showing things like symptom duration, conservative care history, and whether red-flag indications are present. None of that is conceptually exotic. The pain is operational.

The clinic note has the facts. The payer policy has the rules. The staff member has to bridge the two.

That is why this kind of work matters. Delays here do not look like dramatic AI demos. They look like back-and-forth paperwork, rejected submissions, and more waiting before the next step in care can happen.

What I Built

PA-Trace is an on-device, agentic workflow that turns:

Clinic note
Imaging order
Payer criteria

into:

Submission-ready prior-authorization draft
Criteria checklist
Missing-evidence signal
Exact evidence tracing

The public repo describes it as a documentation-assembly prototype, and that is exactly the right framing.

It is not trying to diagnose the patient.

It is not trying to recommend treatment.

It is not submitting anything to a payer portal.

It is trying to make one ugly administrative handoff more structured, faster, and easier to review.

The output bundle is concrete:

packet.json
checklist.json
provenance.json
packet.md
highlights.html

That last artifact is especially useful. It renders the clinic note with highlighted evidence spans so you can see exactly where the system found the facts it used.

Clinical note with highlighted evidence spans for symptoms, treatments, and conservative care. — Evidence tracing in the clinical note, with extracted symptoms, treatments, and conservative-care spans highlighted for review.

Why I Kept The Scope Narrow

This is the part that made the project more credible.

A lot of healthcare AI discussion jumps quickly to diagnosis, triage, treatment planning, or clinical decision support. Those are important areas, but they also invite bigger claims, bigger safety burdens, and more room for overreach.

PA-Trace stays deliberately smaller.

It focuses on documentation assembly for prior authorization, which is a real bottleneck, but also a better fit for a system that should be abstention-first and human-in-the-loop.

That design choice shows up everywhere in the repo:

Synthetic notes only
No PHI
Refusal guardrails for diagnosis or treatment questions
Deterministic checklist logic instead of free-form final reasoning

That is also why I think this project says something useful about healthcare AI more broadly.

If AI is going to improve humanity, some of that progress will not come only from grand medical breakthroughs.

Some of it will come from smaller systems that remove invisible friction from care.

How The Pipeline Actually Works

The local architecture is simple enough to understand, which is part of the point.

The public GitHub repo uses a compact Python pipeline:

A policy store loads chunked payer criteria
Lightweight retrieval scores chunks by token overlap
MedGemma 4B-IT GGUF via llama-cpp-python extracts structured facts from the note
A validator rejects evidence that is not an exact substring of the source text
Baseline regex logic boosts missed treatments, conservative-care duration, and red flags
A deterministic checklist engine emits MET, NOT_MET, or UNKNOWN

Here is the workflow in one view:

flowchart TD
  A[Clinic note] --> D[Policy retrieval]
  B[Imaging order] --> D
  C[Payer policy] --> D
  D --> E[MedGemma extraction]
  E --> F[Evidence validation + baseline boost]
  F --> G[Deterministic checklist]
  G --> H[Packet bundle + highlights]

The trust boundary is not the model alone.

It is the harness around the model: retrieval, evidence validation, fallback logic, and deterministic output rules. That is why the system is more interesting to me as an engineering pattern than as a raw model demo.

Structured extraction and overall packet status in PA-Trace. — Structured extraction and deterministic checklist status, designed to make missing evidence visible instead of hiding uncertainty.

What Still Held Up When I Revisited It

The original Kaggle writeup is still the public source of truth for this project, and I still trust it.

Coming back to the repo later, the main picture still held. The documentation-first, abstention-first design is still the most important thing about PA-Trace, and the headline numbers from the Kaggle writeup are still the right ones to reference for the LLM workflow:

Metric	Kaggle writeup score
`decision_accuracy`	1.00
`provenance_valid_rate`	1.00
`abstention_precision_on_unknown`	1.00
`conservative_care_weeks`	1.00
`red_flags_present`	0.91
`symptoms_duration_weeks`	0.82

One implementation detail is worth calling out for readers who inspect the repo directly: the checked-in runs/eval/metrics.json is a baseline-mode artifact, while the headline numbers above correspond to the LLM evaluation path.

Another small nuance is that the current LLM evaluation covers 11 synthetic cases. That does not change the public story, but it helps explain the repo as it exists today.

What The Misses Actually Look Like

This is the most important nuance in the whole project.

Field extraction is not perfect.

The reason the system still looks strong is that the final checklist is conservative and deterministic.

When I revisited the LLM path, three concrete misses stood out:

case_01: the gold label says red_flags_present = false, but the output marked cauda_equina from the phrase positive straight leg raise
case_08: the note says three months, but the extracted symptom duration stayed at 3 instead of being normalized to 12 weeks
case_10: the note says 1 week, but the extracted symptom duration came out as 7

Those are real misses, and they should be described honestly.

But they also explain why the project architecture matters.

The system is designed so that missing or uncertain evidence pushes the result toward review rather than confident hallucination. That is why the final decision behavior remained stronger than the raw extraction fidelity.

In other words, the project is better because it does not ask the model to do everything.

Why On-Device Mattered Here

The Kaggle writeup and the local repo are consistent on the core deployment idea: keep the workflow local.

That had a few benefits:

No cloud API dependency in the demo path
Simpler privacy boundary
No PHI exposure in the project data because the notes are synthetic
Easier alignment between evidence extraction and the source text on the same machine

The README recommends roughly 6 GB of VRAM for the CUDA path, with a slower CPU fallback if needed. That is not “runs anywhere instantly,” but it is still a very different posture from shipping notes to a remote model and hoping the rest of the control stack behaves.

This is also where PA-Trace feels like a natural follow-up to Checkstand for me.

Both projects came from the same local-first instinct. Keep the model close to the data. Keep the system bounded. Make privacy and observability easier, not harder.

Reality Check

PA-Trace is still a hackathon prototype, and its limits matter:

The repo ships a paraphrased demo policy snippet, not a production payer-policy ingestion system
Retrieval is intentionally lightweight, not a full retrieval stack
The eval set is small and synthetic
There is no payer portal integration
The system is explicitly not a medical device
It does not provide diagnosis, treatment advice, or autonomous care decisions

So no, this is not a finished healthcare product.

But I do think it is a meaningful prototype.

It demonstrates a shape of healthcare AI that I trust more than grand claims: narrow scope, visible evidence, deterministic final logic, and a willingness to say UNKNOWN when the documentation is not strong enough.

The Takeaway

I still believe AI can improve humanity.

What PA-Trace reminded me is that the version people will trust does not begin with pretending the model can do everything.

It begins with narrow systems that solve real problems without pretending to do more than they can.

In healthcare, that can mean better diagnostics one day.

But it can also mean something smaller and more immediate: less paperwork friction, fewer resubmission loops, and more human time for the parts of care that actually need humans.

That is not the loudest version of AI progress.

It might be one of the most useful.

Search This Blog

Eido Askayo