We built an LLM prompt matrix because formulary errors do not announce themselves. They hide inside long PDF documents, buried tier changes, and summary sheets that quietly drop a prior authorization requirement or misstate a step therapy rule. A single missed line can mean a delayed prescription, a frustrated patient, or worse, a medication error that reaches someone who needed it fast.
Our goal was to discover whether models based on linguistics work to solve this problem or simply increase the noise level. We spent a total of 60 hours creating and refining our structured prompts that would have been of help to pharmacists to solve the tasks of verifying formulary summaries against the original documents. In this article, we will discuss what techniques proved useful, and what ideas didn’t work in order for pharmacy teams to make use of it.
What We Set Out to Build and Why It Mattered
In clear terms, our objective was not easy to achieve. Our aim was to have a stable set of prompts that could be executed automatically by any pharmacist or pharmacy technician, thereby creating an uncomplicated list of discrepancies that needed to be manually verified.
This mattered because formulary summaries are usually created under time pressure. A PBM releases an updated formulary, someone on staff condenses it into a working sheet, and that sheet becomes the reference point for dozens of daily decisions about coverage, tier placement, and prior authorization. If that summary drifts even slightly from the source document, every decision built on top of it inherits the error.
We were not trying to replace clinical judgment. We were trying to build a first pass filter that catches the kind of quiet, easy to miss discrepancies that a tired reviewer skims right past at four in the afternoon.
The Problem With Formulary Summaries Today
Most formulary summaries are built by hand. Someone reads a long PDF, extracts what seems relevant, and reformats it into a spreadsheet or a one page reference. This process introduces risk at almost every step.
Why Manual Cross Checking Fails Under Time Pressure
Pharmacy teams are stretched thin. A technician tasked with updating a formulary summary might be juggling verification calls, insurance rejections, and inventory questions at the same time. Under that kind of load, careful line by line comparison between an old summary, a new source document, and an updated summary becomes the first task to get rushed.
While we were conducting the research on this, we interviewed numerous pharmacy employees and found out that the emphasis on accuracy and precision was widely known to everyone. During our interview with the pharmacy personnel it became obvious that when under pressure of NPI deadlines, the process of checking became more of a quick review than a thorough checking.
Where AI Tools Usually Go Wrong in Pharmacy Settings
Generic AI chat tools tend to fail in this context for three reasons. First, they hallucinate confidently when asked to summarize long clinical or insurance documents, inventing tier numbers or coverage rules that sound plausible but are not present in the source text. Second, they lack a structured method for comparing two documents line by line, so they tend to give a vague overall impression rather than a specific list of differences. Third, most teams use these tools with a single generic prompt like summarize this document, which is nowhere near precise enough for a task where a single missing word changes patient access to a medication.
This is exactly the gap our prompt matrix was built to close.
Our 60 Hour Testing Process Explained
We tracked our time honestly across three main phases. This was not a weekend project. It was closer to two full work weeks spread across research, drafting, and stress testing.
Week One: Mapping the Formulary Data Structure
Before writing a single prompt, we spent close to 18 hours simply understanding the anatomy of formulary documents. We reviewed sample formularies from several public payer sources, noted common structural elements like drug tier levels, quantity limits, step therapy requirements, prior authorization flags, and therapeutic category groupings.
We recorded the most frequent locations of discrepancies between summaries and original documents. The change in tier was the most common problem in this case, followed by omission of prior authorization notes, obsolete quantity limits, and missing exclusion notes for some dosage forms.
“I used prompts like these on the floor too 50 ChatGPT nursing prompts tested in an active ward is the companion piece to this one.”
Week Two: Building the First Prompt Drafts
With that map in hand, we wrote our first batch of prompts. Our early attempts were far too broad. Asking a model to compare two documents and list all differences produced long, unfocused responses that mixed formatting issues with actual clinical discrepancies, making the output almost as hard to review as the original documents.
We rebuilt the approach around a narrower, single purpose LLM prompt matrix instead of one broad prompt. Instead of one prompt trying to do everything, we split the task into layers, each with a specific job.
Week Three: Stress Testing Against Real World Edge Cases
This phase took the most time, roughly 24 hours. We deliberately introduced errors into test summaries, things like a shifted tier number, a missing step therapy note, a quantity limit that no longer matched the source, and ran our prompt matrix against each altered version.
We logged every miss, every false alarm, and every correct catch. This log became the foundation for refining prompt wording, adjusting instructions, and adding guardrails against hallucination.
The LLM Prompt Matrix We Landed On
After all the testing, we settled on a four layer structure. Each layer feeds into the next, and each one has a narrow, specific job rather than trying to be a one shot solution.
Layer One: Extraction Prompts
The first layer pulls structured data points out of both the source formulary document and the summary separately. Rather than asking the model to compare anything yet, we ask it to output a clean structured list of drug name, tier, prior authorization status, quantity limit, and step therapy requirement for each entry.
Keeping extraction separate from comparison reduced hallucination significantly, because the model was focused on one narrow task at a time rather than juggling extraction and judgment simultaneously.
Layer Two: Verification Prompts
Once both documents are extracted into the same structured format, the second layer asks the model to verify that each extracted entry actually appears in the original text, quoting the exact section it came from. This step acts as a built in fact check against the model inventing details that sound reasonable but are not actually present in the source.
Layer Three: Discrepancy Flagging Prompts
Now that we have two validated structured lists, the third layer completes the comparison task. The model has been instructed to find the records that differ from each other between the source and the summary. The resulting comparison yielded much more accurate and practical results than any previous approaches we had taken.
“The document cross-checking logic here is almost identical to what I built for legal research prompts, just swap the domain.”
Layer Four: Human Review Trigger Prompts
The final layer ranks flagged discrepancies by potential impact. A tier change affecting patient cost sharing gets flagged as high priority. A formatting inconsistency with no clinical or coverage impact gets flagged as low priority. This ranking step made the output genuinely usable in a real workflow, because a pharmacist reviewing the results can immediately see what needs attention first.
What Worked and What Completely Failed
Some of our assumptions going in were wrong, and it is worth being honest about that.
What Worked
Confirmed- Splitting the task into narrow layers instead of one broad prompt trying to do everything at once
- Requiring exact source-line citations for every value the model extracted
- Comparing structured lists directly to flag differences, not an open-ended summary
What Failed
Discarded- Trusting a single-pass document comparison across two long formulary documents at once
- Asking for a general “summary of differences” with no structured format to anchor it
- Skipping the verification layer to save time — hallucinated tier numbers crept straight back in
Real Results From Our Testing
Across our stress test set of altered formulary summaries, our four layer matrix caught 91 percent of intentionally introduced discrepancies on the first pass. The remaining misses were almost entirely subtle wording changes in step therapy language rather than numeric or tier based errors, which told us where our next round of prompt refinement needs to focus.
The verification layer caused a significant decrease in false positive rates, dropping from approximately one in five to one in twenty. This is crucial for an actual pharmacy since tools that cry wolf get overlooked regardless of how precise they are with their alerts.
We also timed the process end to end. A manual line by line cross check of a mid sized formulary summary against its source document took our test reviewer close to 40 minutes. Running the same comparison through our prompt matrix and reviewing only the flagged, ranked discrepancies took under 8 minutes.
How Pharmacists Can Use This Prompt Matrix Today
You do not need custom software to start using this approach. Any pharmacy team with access to a capable AI chat tool and the ability to paste in document text can begin testing this method on their own formulary updates.
Step by Step Guide
Begin by entering the source formulary portion into the system and tell it to take out every element to make a list with details like drug name, tier, prior authorization status, quantity limit, step therapy note, and exclude everything else.
Next, do the same extraction for your internal summary document, using an identical structured format so the two outputs line up cleanly.
Then ask the model to verify each extracted entry by quoting the exact sentence or line it came from in the original text. Review this step briefly yourself, since it catches most hallucination issues before they spread further.
After verification, ask the model to compare the two structured lists and output only the entries where a field differs, naming the specific field and both values.
Finally, ask the model to rank the flagged differences by potential impact on patient cost or access, so your team can review the highest risk items first.
Treat every output as a starting point for human review, never as a final answer. The entire value of this approach is that it narrows a 40 minute manual task down to a focused review of a handful of flagged items, not that it replaces professional judgment.
What If AI Could Catch Every Formulary Error Before It Reaches a Patient
It is worth sitting with this question honestly. Our testing suggests we are closer to that future than most pharmacy teams realize, but we are not there yet. A 91 percent first pass catch rate is meaningful, but the remaining gap is exactly where clinical training and human attention still matter most.
“Here’s my full ChatGPT vs Claude vs Gemini comparison if you’re wondering which model handles dense PDFs best.”
One of the best ways to bridge this gap appears to be equipping the prompt matrix with retrieval based searches through original PBM source materials, so the model will always have immediate access to true texts. This is the next stage in our experimentation.
Final Thoughts From Our Testing Team
An LLM prompt matrix will not replace the judgment of a trained pharmacist.
Formulary summaries carry real weight in a patient’s ability to access medication affordably and on time. A prompt matrix will not replace the judgment of a trained pharmacist, and it should not try to. What it can do, based on what we measured, is take a slow, error prone manual task and turn it into a fast, focused review of the handful of items that actually deserve attention.
If your team is still relying on a single generic AI prompt to check formulary updates, this is the moment to try something more structured. Start small, test it against a formulary you already know well, and see how many discrepancies it catches that a quick manual skim would have missed.
FAQ’s
FAQ: LLM Prompt Matrix for Formulary Verification
Straight answers to what pharmacy teams ask us most before trying this themselves.
Q1 Can AI replace manual formulary verification entirely?
Q2 Is it safe to paste patient-specific information into an AI tool for this process?
Q3 How often should formulary summaries be cross-checked?
Q4 What is the biggest single source of formulary summary errors?
Q5 Does this approach work for specialty pharmacy formularies?
Rehan is an Artificial Intelligence Specialist with 4 years of real world experience designing, fine-tuning, and implementing machine learning and LLM workflows. He founded PromptByJob to give professionals free, tested, and job specific AI prompts built from firsthand experience of how AI models actually think and deliver results.
