Auditing a leaderboard number: what MMLU hides and what to measure instead

Barakaeli Lawuo, Jul 4, 2026

In 2024 a team at the University of Edinburgh and partner institutions sat down with 100 randomly chosen questions from the Virology section of MMLU, the widely used multiple-choice benchmark, and checked each one against its original source. They reported that 57% of those questions had some kind of error.¹ One question asked for the best option for preventing future outbreaks of Ebola. The answer key picked sending American and European army teams into West Africa.¹

Across all 57 subjects, their re-annotated set of 5,700 questions, built by 14 human experts, led them to estimate that 6.49% of MMLU questions contain errors.¹ In the subjects where errors cluster, that was enough to reorder the top models.¹ So before trusting a leaderboard number to pick a model, it helps to audit what the number is made of. This post does that for MMLU, line by line, and then asks what a builder should measure instead.

Benchmark

A fixed set of test items with a scoring rule. MMLU is one: multiple-choice questions, scored by the fraction answered correctly.

Ground truth

The answer the benchmark counts as correct. If the key is wrong, a model that answers correctly gets marked down.

Calibration

Whether a model's stated confidence matches how often it is right. A calibrated model that says 80% is right about 80% of the time.

Robustness

How much accuracy holds up when the input is perturbed in ways that should not change the answer, such as typos.

Pairwise preference

A human sees two anonymous answers to the same prompt and picks the better one. Chatbot Arena ranks models from many such votes.

Line item one: the answer key is wrong more often than you would guess

The Edinburgh study, published as "Are We Done with MMLU?" by Gema and colleagues, sorted each bad question into a small tree of error types. A question can be unclear, for example by referring to "question 21" that the model never sees. Its options can be garbled, often from a parsing mistake that split one option into two. Or the key itself can fail: no option is correct, several are, or the marked answer is simply the wrong one.¹

The errors were not spread evenly. In Virology, the figure caption reports 33% of the sampled questions with a wrong ground truth, 14% with unclear questions, and 4% with more than one correct answer.¹ More than 20% of sampled questions were wrong in Logical Fallacies and College Chemistry, and more than 10% in Professional Law, Business Ethics, Formal Logic, Human Aging, Global Facts, Machine Learning, Miscellaneous and Public Relations.¹ The causes ranged from the plain to the strange. In College Chemistry, questions that spanned several lines in the source were parsed so that part of the question became option A and the real option D fell off.¹ In Formal Logic, one question marks (F ∧ L) ∧ ¬C as correct and F ∧ L ∧ ¬C as incorrect, though the two formulas are equivalent.¹

Does this change which model looks best? The authors re-scored ten leading models on the five subjects with the most errors, once on all questions and once on only the questions their annotators judged correct. In Virology, Llama 3.1 Instruct Turbo (405B) ranked 16th on all questions and first on the clean ones.¹ In Human Sexuality, Gemini 1.5 Pro (001) went from 0.37 exact match and 55th place to 0.94 and 6th.¹ The ranks here are positions on the larger HELM MMLU leaderboard the authors compared against, so the ten models below are a slice of a longer list.

There is a second, odder signal in the same data. Scored against the original key, models should do worse on the questions flagged as erroneous, and in most subjects they did. In Professional Law and Formal Logic they did about as well, or better.¹ A model should not be able to agree with a broken key by reasoning. The authors read this as possible evidence that those questions were memorised during pretraining.¹

Line item two: which capability a question actually tests

MMLU was built to measure breadth. Its original paper describes 57 tasks and 15,908 questions, collected by graduate and undergraduate students from freely available sources such as practice questions for the GRE and the US Medical Licensing Examination.² The same paper found that models were "lopsided": strong in some subjects and near chance in others, including socially important ones such as morality and law.² A single average hides that shape by design.

Reading the questions closely shows a second problem, which is that a subject label does not tell you what a question asks for. Gema and colleagues note that Professional Law and Accounting assume US jurisdiction and practice without saying so.¹ Almost every Global Facts question needed an outside source to check, often a specific report from ourworldindata.org, and for several questions the sources disagreed with each other.¹ Some Machine Learning quiz questions rely on older knowledge that may no longer apply.¹ A high score on the Professional Law subset is partly a score for knowing that the unstated context is American. A high Global Facts score partly measures whether a model absorbed particular statistics tables.

If a question can be answered by recall of a leaked answer key, it tests memory. If it depends on unstated national context, it tests an assumption. Neither is the capability named on the subject label, and neither shows up in the leaderboard cell.

Line item three: the metrics that accuracy leaves out

Accuracy is only one column. The Holistic Evaluation of Language Models project (HELM), led by Liang and colleagues at Stanford, measured seven metrics for each of 16 core scenarios: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency.³ Before HELM, the 30 models it studied had on average been evaluated on just 17.9% of its core scenarios, and some prominent models did not share a single scenario. HELM raised that to 96.0% under the same prompting conditions.³ Many of the comparisons people quoted before HELM were between scores on different tests.

Once every model sat the same tests, the metrics did not move together. Calibration was the clearest case. On HellaSwag, higher accuracy went with worse calibration; on OpenBookQA, a similar commonsense question set, higher accuracy went with better calibration.³ The authors called the finding that more robust and fairer models can be less well calibrated "counter-intuitive and surprising."³ The MMLU paper had already found GPT-3 poorly calibrated on its own test, with the gap between confidence and accuracy reaching 24% on some subjects.²

Robustness mostly tracked accuracy, but not always. On NarrativeQA, TNLG v2 (530B), the third most accurate model there, dropped from 72.6% accuracy to 38.9% when HELM applied robustness perturbations such as typos.³ In the overall head-to-head comparisons, BLOOM (176B) did better on robustness and fairness than its accuracy suggested, and OPT (175B) and GLM (130B) roughly swapped places between the accuracy and robustness rankings.³ The harm metrics diverged further. T0++ (11B) was the most toxic model compared with all others but among the three least gender-biased; davinci (175B) was among the four most biased but one of the less toxic.³

The figure makes the point quickly. text-davinci-002 won more than 90% of its accuracy comparisons, the clearest leader in the study.³ In the calibration error panel, where a higher bar means the model more often had the larger error, it sits ninth from the worst end of 24 models.³ Efficiency behaved differently again. HELM saw only weak correlations between efficiency and the other metrics, no strong overall trade-off between accuracy and efficiency, and only a subset of models on the accuracy and efficiency frontier for any given scenario.³ The authors left efficiency out of the head-to-head rankings entirely, because they did not think it meant anything without accuracy beside it.³

HELM's authors draw the consequence themselves. They write that they "cannot simply rank models by accuracy to get a total order," and that they do not believe a universal aggregation exists that "satisfies all preferences, reflects all values, or captures all circumstances appropriately."³ Their own example of a circumstance is an organization deploying on mobile, which should give the efficiency results more weight.³

Line item four: what people prefer is a different ranking

Chatbot Arena, described by Chiang and colleagues at Berkeley, ranks models a different way. A visitor types a prompt, two anonymous models answer, and the visitor votes for the better one; the names appear only after the vote. At the time of the paper it had collected over 240,000 votes from about 90,000 users.⁴ Because the prompts are open ended and fresh, there is no fixed answer key to go wrong or leak into training data.⁴ The authors checked vote quality by having graduate students label a sample blind, with fact-checking. Crowd votes agreed with those experts 72% to 83% of the time, and the two experts agreed with each other 79.4% and 89.8% of the time.⁴

Two things in the Arena paper matter for model choice. First, strength depends on topic. The authors' topic clustering found 600 clusters, the largest covering only 1% of prompts.⁴ In one comparison on prompts sampled from several clusters, scored by a GPT-4 judge rather than by users, GPT-4 beat Llama-2-70b-chat in 96.7% of Python game programming prompts but only 53.3% of movie recommendation prompts.⁴ Second, the ranking has error bars. In the paper's ranking, gpt-4-0613 could sit anywhere from 3rd to 7th, and claude-2.1 from 6th to 18th.⁴ Much of a leaderboard's middle is a statistical tie.

How far does a preference ranking differ from MMLU's ranking? A separate study, MixEval by Ni and colleagues, measured the Spearman rank correlation between many benchmarks and Arena Elo. MMLU came in at 0.83.⁵ That is well above chance, and still leaves room for models to trade places. The same study found that 10 of 13 general-domain benchmarks correlated above 0.5 with Arena, but only 1 of 8 domain-specific ones did.⁵ If your product lives in one domain, a general preference ranking and a domain test can point different ways.

Reading the audit for your own application (the author's reading)

What follows is my interpretation of these papers, not a finding in any of them. The audit shows that a leaderboard number bundles four things: the quality of the answer key, the capabilities the items happen to test, the single metric chosen, and the population of prompts or voters. When you pick a model, you get to choose all four for yourself, and you should.

Start with the metric list, not the model list. HELM's seven metrics are a good menu. For a support bot that routes tickets, calibration may matter more than raw accuracy, since the system has to know when to hand off to a person. For something that reads messy user text, robustness to typos is the capability to test, and the NarrativeQA drop shows that accuracy on clean input can hide a large gap there.³ For a phone app or a high-volume pipeline, HELM's own advice is to weight efficiency.³

Then write items that test the capability you named. The MMLU audit is a warning about what happens when you do not. If your legal assistant serves Kenya, a test that silently assumes US law measures the wrong thing.¹ Keep the set small enough that someone can check every answer key by hand, because errors at MMLU's rate were enough to reorder close models in its worst subjects.¹ Watch for items the models might have seen: questions lifted from public exams or documentation invite recall rather than reasoning.¹

Finally, treat public rankings as a filter, not a verdict. A model near the top of Arena or MMLU is a reasonable candidate. The difference between the third and seventh place models may not be real, and the topic breakdown suggests your task may sit in a cluster where the gap is narrow or reversed.⁴ A shortlist of three models run on a hundred of your own checked prompts, scored on the two or three metrics that fit the job, will usually tell you more than the leaderboard did.

What the Arena authors say their own ranking cannot see

The Chatbot Arena paper is candid about its reach. Its authors expect their users to be mostly LLM hobbyists and researchers keen to try the newest models, which may skew who is voting.⁴ The prompts come from their own chat interface, which, they write, "might not accurately reflect the real-world usage of LLMs in production environments or specialized domains."⁴ And the study assesses helpfulness but overlooks safety, which they say calls for a parallel evaluation mechanism of its own.⁴