What Specializing a Model for Medicine Measurably Bought

Barakaeli Lawuo, Jun 28, 2026

In December 2022 a Google team asked a panel of clinicians to rate answers to 140 consumer health questions, most of them drawn from what people type into search engines, without being told who wrote each answer. For answers from Flan-PaLM, a 540 billion parameter general model tuned to follow instructions, the clinicians judged only 61.9% to be aligned with scientific consensus. For answers written by clinicians the figure was 92.9%. The same raters judged 29.7% of Flan-PaLM's answers as potentially leading to harm, against 5.7% of the clinicians' answers.¹

The same model had just set a record on MedQA, a set of multiple-choice questions in the style of the US Medical Licensing Examination, with 67.6% accuracy.¹ So one model, in one paper, looked strong by one yardstick and unsafe by another. A version of Flan-PaLM adapted with 40 clinician-written examples, called Med-PaLM, reached 92.6% consensus alignment on the same questions.¹ This post follows that thread. It asks what adapting a model to a specialized field buys, and it keeps asking which yardstick each result was measured with.

A licensing-exam record that clinicians did not trust

Before large general models, the standard answer to "medicine needs its own model" was domain pretraining. Pretraining is the first, largest stage of training, where a model learns to predict text from a big corpus. Domain pretraining means that corpus comes from the field itself. Gu and colleagues at Microsoft showed that for biomedicine, where unlabeled text is abundant, pretraining a BERT model from scratch on biomedical text gave substantial gains over continuing to pretrain a general-domain model. It also gave the model a vocabulary built from biomedical words.⁴ The resulting model, PubMedBERT, set new records on their biomedical benchmark.⁴

Scale changed that picture on multiple-choice exams. The Med-PaLM paper lists prior MedQA results next to its own: PubMedBERT with 100 million parameters scored 38.1%, a 2.7 billion parameter model called PubMedGPT, one of the biomedical models pretrained on domain text, scored 50.3%, and the general Flan-PaLM scored 67.6%, beating the previous best by more than 17%.^1,3 The Medprompt authors put it plainly: first-generation models got a clear advantage from domain-specific pretraining, but it was unclear whether that still held for modern models pretrained at much larger scale.³

Flan-PaLM reached its score through prompting, not medical training. It combined three prompting strategies. Few-shot prompting places a handful of solved examples in front of the question. Chain-of-thought makes those examples include step-by-step reasoning, so the model writes its own reasoning before answering. Self-consistency samples several reasoning paths and takes the majority answer.¹ For scale, the USMLE website says examinees must answer roughly 60% of questions correctly to pass.⁶

The Med-PaLM authors did not treat the exam score as the finish line. Their argument was that multiple-choice accuracy is a robust measure but omits important details. A model answering a real person writes paragraphs, not a letter from A to E. Existing benchmarks, they wrote, were limited to classification accuracy or automated text-overlap metrics like BLEU, which do not allow the detailed analysis needed for clinical use.¹ The reference answers bundled with some datasets came from inconsistent sources, sometimes automated tools or librarians, so scoring model text against them would not measure clinical quality either.¹

Twelve questions a clinician asks of an answer

The replacement was a rubric, built with focus groups and interviews with clinicians in the UK, US and India. It has twelve items. Raters say whether an answer agrees with scientific and clinical consensus, how severe any possible harm would be and how likely it is, whether there is evidence of correct or incorrect reading comprehension, recall of knowledge and reasoning, whether the answer includes content it should not or omits content it should, and whether it contains information that is inapplicable or inaccurate for a particular demographic.¹ Two more axes went to lay raters without medical training: does the answer address the intent of the question, and is it helpful.¹

The harm items borrowed their severity options from the AHRQ Common Formats, a scale that runs from death through severe, moderate and mild harm to none. The authors note that even in hospital settings, where the context of a harm is known in detail, physicians vary a lot in how they rate severity, so their own ratings should be read as subjective estimates.¹ The evaluation set was 100 questions from HealthSearchQA, a new dataset of 3,375 commonly searched consumer questions, plus 20 each from LiveQA and MedicationQA. A panel of nine clinicians rated the answers, one clinician per answer, blind to the source.¹

MultiMedQA

the benchmark the Med-PaLM paper assembled: six existing medical question sets (exam, research and consumer questions) plus the new HealthSearchQA.

Instruction tuning

further training a pretrained model on many tasks phrased as instructions, so it follows instructions better. Flan-PaLM is PaLM after this step.

Soft prompt

a short sequence of learned vectors placed before the input. They sit where word embeddings would go but do not correspond to any words, and they are trained by backpropagation while the model stays frozen.

Instruction prompt tuning

the Med-PaLM method: a soft prompt learned from clinician-written examples, placed in front of an ordinary text prompt of instructions and examples.

Forty examples and 1.84 million trainable parameters

Retraining a 540 billion parameter model on medical text was not what the team did. They chose prompt tuning because of compute cost and the cost of getting clinicians to write data.¹ Prompt tuning, from Lester and colleagues, freezes the whole model and learns only a soft prompt. Lester's experiments found that as models grow past billions of parameters, this matches the quality of tuning every weight.⁵ Med-PaLM's variant keeps the ordinary text prompt too: the learned vectors come first, then the hand-written instructions and examples, then the question.¹

The training data was tiny. Five clinicians in the US and UK wrote exemplar answers to sampled consumer questions, then threw out pairs where they could not write an ideal answer, for instance when the needed information was not known. That left 40 examples.¹ The soft prompt was 100 vectors long. With PaLM's embedding size of 18,432, that is 1.84 million trainable parameters, trained for 200 steps. The team did not pick the final checkpoint by an automatic metric; a clinician ranked answers on held-out questions, because automatic scores on long answers may not track human judgment.¹

Most axes moved toward the clinicians. Evidence of correct knowledge recall rose from 76.3% of Flan-PaLM answers to 95.4% for Med-PaLM, against 97.8% for clinicians. Answers missing important information fell from 47.2% to 15.1%.¹ One axis went the wrong way. Answers with inappropriate or incorrect content rose from 16.1% to 18.7%, while clinicians were at 1.4%. The authors suggest a reason: the tuned model writes more detailed answers, which cuts omissions but gives more room for wrong content.¹ Even so, the headline in the abstract is careful. Med-PaLM "performs encouragingly, but remains inferior to clinicians."¹

Med-PaLM 2 changed the recipe and the test

The follow-up, in May 2023, changed three things at once, which matters when reading its gains. It started from a stronger base model, PaLM 2. It replaced the soft prompt with full instruction finetuning on the training splits of MultiMedQA: about 10,000 MedQA and 183,000 MedMCQA questions plus 64 long-form consumer answers, mixed in empirically chosen ratios. And it added a prompting method called ensemble refinement: the model samples several reasoning paths, then reads all of them and writes a refined answer, and the final answer is a vote over many such refinements.² On multiple choice that used 11 samples in the first stage and 33 in the second.²

The unified Med-PaLM 2 scored 85.4% on MedQA with ensemble refinement. The best figure, 86.5%, came from a separate variant finetuned only on MedQA and not aligned for consumer answers.² Med-PaLM had scored 67.2% on the same benchmark.²

The human evaluation grew too, because the old one had stopped telling models apart. On the same 140 questions, physicians rated Med-PaLM 2 as generally comparable to both physicians and Med-PaLM, and the authors say that analysis was underpowered for the differences they saw.² So they added pairwise ranking: a rater sees two anonymous answers to the same question, in random order, and picks the better one, or a tie, on each of nine axes. Physicians from a pool of fifteen in the US, UK and India ranked answers to 1,066 consumer questions.²

Physicians picked Med-PaLM 2's answer as better reflecting consensus 72.9% of the time, physicians' answers 11.8% of the time, and called 15.3% a tie. The model was preferred on eight of nine axes.² The ninth is the same axis that got worse for Med-PaLM: raters flagged Med-PaLM 2's answer as containing more inaccurate or irrelevant information 26.6% of the time, against 14.1% for the physician's answer.² Med-PaLM 2's answers were also much longer, a median of 794 characters against 337.5 for physicians, and the authors list length as something that may have helped its ratings.² On 240 adversarial questions written to draw out harmful or biased answers, 90.6% of Med-PaLM 2's answers were rated low risk of harm, against 79.4% for Med-PaLM.²

Medprompt: GPT-4, no medical training, a higher exam score

In November 2023 a Microsoft team asked whether any of this medical tuning was needed. Their earlier study had shown GPT-4 passing the USMLE by over 20 points with simple prompting and no special training.⁶ The new paper searched systematically for better prompts and settled on three general-purpose parts, which together they called Medprompt.³

• Dynamic few-shot: for each test question, embed it, find the most similar questions in the training set by k-nearest neighbors, and use those as the examples.³
• Self-generated chain-of-thought: ask GPT-4 to write the reasoning for each training example itself, and discard any example where its reasoning ends at the wrong answer.³
• Choice-shuffle ensemble: run the question several times with the answer options in different orders and take the most common answer, which also counters the model's preference for certain option positions.³

Nothing in the pipeline is medical, and no clinician wrote any of its reasoning examples. With 5 retrieved examples and 5 ensemble calls, GPT-4 reached 90.2% on MedQA, against Med-PaLM 2's best of 86.5%. With 20 examples and 11 calls it reached 90.6%.³ The authors call this a 27% cut in error rate over the best specialist result, and note that Med-PaLM 2's scheme used 44 model calls per question.³ Medprompt set the top score on all nine MultiMedQA multiple-choice datasets. On MMLU professional medicine it only tied Med-PaLM 2, at 95.2%.³

The paper tests its own result for overfitting. Choosing a prompt strategy is like tuning a hyperparameter, so before starting they hid a random 20% of each dataset, which they call eyes-off, and only scored it at the end. Average accuracy was 90.6% on the data they had worked with and 91.3% on the hidden data.³ Adding the parts one at a time on MedQA moved accuracy from 81.7% with a zero-shot prompt to 83.9%, 87.3%, 88.4% and finally 90.2%. Self-generated reasoning was the largest single step.³ In a direct comparison with fixed examples, GPT-4's own reasoning chains scored 86.9%, while the expert-written chains from the Med-PaLM 2 paper scored 83.8%.³

Where the two results actually meet

Put side by side, the papers make a narrower claim than the headline race suggests. Medprompt shows that a strong general model, prompted well, beats a finetuned specialist on the multiple-choice yardstick. The Med-PaLM paper built its whole evaluation on the argument that this yardstick was not enough for medicine. Its own evidence supports that: Flan-PaLM set the MedQA record and still had 29.7% of its answers flagged as potentially harmful.¹ Nothing in the Medprompt paper measures consensus, harm or omissions in long answers. So the two results never meet on the same axis.

One finding does cross over. In both lines of work, the example-writing that most people picture as expert work turned out to be replaceable or small. Med-PaLM needed 40 clinician answers and 1.84 million parameters to move its consensus score by about 30 points.¹ Medprompt's model-written reasoning beat clinician-crafted reasoning on the exam by 3.1 points.³ The Med-PaLM 2 variant tuned only on MedQA scored 86.5% and was, in the paper's words, "not aligned for consumer medical question answering."² That trade shows up inside a single paper.

The author's reading: when specializing is worth paying for

What follows is my interpretation of these papers, not a finding any of them states.

If your task has a checkable answer, like picking the right option, extracting a code or labeling a document, start with the strongest general model and invest in prompting. Retrieved examples, model-written reasoning and ensembling are cheap to try, and on MedQA they took GPT-4 from 81.7% to 90.2% without touching a weight.³ Domain pretraining from scratch made sense when models were small.⁴ These results give little reason to expect it to pay for a question-answering task at today's scale.

If the output is open-ended text that an expert will judge on safety, completeness and fit, the evidence points the other way, and cheaply. Med-PaLM's largest measured gains came on exactly the axes an exam cannot see, from 40 carefully filtered expert examples and a parameter-efficient method.¹ What seems worth paying for is not a medical model as such. It is expert time: writing a small set of ideal answers, and above all rating outputs on a rubric like the twelve-item one. Without that rating step, you cannot tell which of these approaches worked for your task. The highest exam score in this post belongs to the one system whose answers no physician rated.

Specialization is not free on the rubric either. Med-PaLM and Med-PaLM 2 both wrote longer answers than what they were compared with, both omitted less, and both did worse than physicians on incorrect or irrelevant content.^1,2 Any specialized model should be checked for that trade, not just for its headline score.

The limitation Medprompt states about its own score

The Medprompt authors are explicit that their results cannot be taken to demonstrate real-world efficacy. Competency exams framed as multiple-choice questions, they write, do not capture the range and complexity of tasks that healthcare professionals face in actual practice.³ They also raise a risk that applies to every method in this post. Better prompting may reduce hallucinations and raise accuracy, but it may also make the remaining hallucinations even harder to detect.³ The Med-PaLM 2 authors add their own caveat. Their adversarial questions are limited in scope and should not be read as a complete assessment of safety, bias and equity.²