Evaluating AI reps for regulated life-sciences promotion.
Most AI sales tools are judged on pipeline. An AI rep in regulated medicine is judged on something that comes first: what happens to each sentence before an HCP reads it — because every one of those sentences is a regulated statement, attributable to your company. This guide is the evaluation frame for that judgment: ten questions to put to any vendor, what a good answer looks like, the red flags that generalize, and how to run a pilot that actually settles the question. It is vendor-neutral by construction — the questions apply to any system that puts promotional words in front of clinicians, ours included.
Judge the sentence, not the demo
A demo shows you the system on its best day. Regulated promotion is judged on the other days — the off-label question asked politely, the adverse event mentioned in passing, the check that fails mid-conversation, the record pulled two years later. So the right evaluation runs compliance-first, sales-second: establish what the system is permitted to say, what it does when it should not speak, and what evidence survives — then evaluate whether it sells.
The questions below follow that order. Each comes with what a good answer looks like at the outcome level — no vendor should need to disclose proprietary internals to answer any of them — and the red flag that tells you the design was never meant for this job.
Ten questions to put to any vendor
Each with what a good answer looks like — and the red flag that should end the meeting early.
What is the system able to say — and who controls that set?
A closed answer: the system draws only on material your MLR process approved, loading that material requires no vendor rewriting, and when a document is withdrawn or superseded, what the system may say changes with it. The approved corpus should be the ceiling — not a starting point the model can improvise beyond.
Any version of "the model falls back on its general knowledge when the approved content runs out." In regulated promotion, that fallback is the incident.
Is each outgoing claim verified before the HCP sees it — or is generation merely constrained, with review after the fact?
A per-claim, pre-send answer: each statement in each reply is checked against approved material before delivery, and every check leaves a recorded verdict. Constrained generation and post-hoc review both have value, but neither is verification — a sentence reviewed after sending has already reached the clinician.
The words "constrained," "tuned," or "instructed" doing the work the word "verified" should be doing — with no per-claim record to show for it.
What happens when a check cannot complete, or a claim cannot be supported?
The system declines — visibly, gracefully, and on the record. Refusal is a designed behavior with its own wording, its own routing to a human, and its own audit entry, and the vendor talks about correct refusals as successes.
Fail-open defaults — "if the check times out, the reply goes through" — or a vendor who treats refusals as an embarrassment to be minimized in the demo.
How does required safety information travel with claims — enforced by the system, or by authoring convention?
Pairing enforced per reply: benefit statements carry the balancing risk information drawn from the approved source, in the conversation the HCP actually reads — not a static footer authored once and hoped onto every message.
Fair balance treated as a template problem — a disclaimer block appended to everything — rather than a property checked on each communication.
Which jurisdictions' promotional rules are encoded — and which are honestly not yet?
Named frameworks — PAAB for Canada, OPDP for the United States, their counterparts elsewhere — encoded as distinct rulesets the system enforces, with a straight answer about which jurisdictions are live and which are still in development. The honest tense is itself the signal: a vendor precise about what is not yet live is a vendor whose live claims you can trust.
"You can configure the rules in plain language" as the whole answer. Customer-authored prose guardrails put the regulatory judgment back on you — with the vendor collecting the fee.
Can anyone — including the vendor — edit or delete the record of what was said?
Append-only enforced at the storage layer, not by policy: no role edits or deletes a turn after the fact, refusals and escalations land on the same record as answers, and the whole trail exports per-turn in a format your MLR or inspection team can work with.
Logs that administrators can amend, aggregate dashboards standing in for records, or exports available "on request" with no committed per-turn shape.
Pick one exchange from the pilot at random — can the vendor reconstruct it completely?
A full reconstruction on demand: what was asked, what was answered, what the system decided, and which approved passage the answer was anchored to. If a regulator asked about one conversation two years from now, this is the drill — run it now, while it is cheap.
Reconstruction that requires the vendor's engineers, arrives partial, or cannot tie the answer back to the approved source it came from.
If an HCP mentions a patient reaction mid-conversation, what happens — step by step?
Recognition that goes beyond keyword matching (real reports arrive as hedges, shorthand, and second-hand accounts), a structured intake record in the reporter's own words with patient identifiers stripped, an alert to your pharmacovigilance team, and a human who grades and files. Equally important is what the vendor does not claim: the tool routes the report; it does not discharge the reporting obligation.
"Our agent avoids medical topics" offered as the answer — adverse events do not wait for permission — or, at the other extreme, a claim that the system files regulatory reports itself.
Where does conversation data live, who can access it, and what happens to patient identifiers?
A named residency posture per jurisdiction, a stated approach to patient-identifying information, defined access boundaries between customers, and security documentation available under NDA. Vagueness here is a finding: this data describes interactions with your HCPs about your products.
No residency answer, silence on identifier handling, or your conversation data feeding the vendor's models by default.
Was the compliance layer designed for regulated promotion — or adapted from general-purpose safety guardrails?
The vocabulary of the trade present in the product's own design targets: approved claims, fair balance, ISI, off-label boundaries, pharmacovigilance routing. A system built for regulated promotion can tell you where each of those lives; a general-purpose system can only tell you which filter it mapped them onto.
Toxicity filters, topic blocklists, and brand-safety controls presented as promotional compliance. They are necessary, useful — and not the same discipline.
Red flags that generalize
Beyond any single question, these patterns tell you how a vendor thinks about the job.
- The refusal-free demo
- A system that never declines is not demonstrating compliance; it is demonstrating that nobody asked it a hard question. Bring the off-label question yourself and watch what happens.
- Present-tense roadmaps
- Channels, jurisdictions, and certifications described in the present tense that turn out to be forthcoming under questioning. Make the vendor sort every claim into live, in development, or planned — in writing.
- Metrics without provenance
- Accuracy figures with no date, method, or test conditions attached. Ask for the figures under NDA, dated, with how they were measured — a serious vendor keeps them in exactly that form.
- Compliance as a mode
- A "compliance setting" bolted onto a general-purpose sales agent. If the enforcement can be toggled, it is a preference, not a control.
- The policy-binder defense
- Governance that lives entirely in policy documents rather than in system behavior. Policies matter — but a regulator reads the record of what was actually said, so evaluate the system that writes that record.
How to run a pilot evaluation
A pilot settles in two weeks what a slide deck argues for two quarters. Run it like this:
Bring your own corpus.
Run the pilot on a real, MLR-approved subset of your material — not vendor demo content. Half of what you are evaluating is how your material behaves: how it loads, what the system does when your corpus does not contain the answer, and how updates propagate.
Script the hard questions.
On-label questions your reps hear weekly, off-label questions phrased politely, adjacent-indication traps, dosing edge cases, competitor comparisons — and a handful of messages carrying adverse events phrased the way clinicians actually talk: hedged, abbreviated, second-hand.
Grade the refusals as results.
Count correct declines as passes, wrong answers as failures, and needless refusals of fair questions as failures too. A compliant system earns its keep at the boundary — that is where the pilot should spend its time.
Read the record, not the transcript.
After the sessions, pull the audit trail and try to reconstruct each exchange from it alone: question, answer, decision, source. If the record cannot tell the story without the demo, the record is the finding.
Put your reviewers in the room.
MLR, pharmacovigilance, and regulatory colleagues should judge the outputs directly — they are the buyers of record for this class of tool, and they will see failure modes a commercial evaluation misses.
Ask what failed.
End the pilot by asking the vendor what went wrong during it — what was refused incorrectly, what was escalated, what they changed. A vendor with nothing to show for that question was not watching closely enough.
Put these questions to us.
This guide is the standard Zenreps expects to be measured against. Bring all ten questions to a demo, ask a live gated rep the hard ones, and read the record it writes while it answers.