From CheckList tests to error bars: building an eval pipeline

Barakaeli Lawuo, Jul 3, 2026

In November 2019, Marco Tulio Ribeiro and three colleagues paid for access to three commercial sentiment analysis services: Microsoft's Text Analytics, Google Cloud's Natural Language and Amazon's Comprehend. They fed them short sentences that a person would find trivial. "The food is not poor" should come back positive or neutral. Google's service got sentences of that kind wrong 54.2% of the time and Amazon's 29.4%. When the negation came at the end, as in "I thought the plane would be awful, but it wasn't," the failure rate was near 100% for all three commercial models.¹

None of these were adversarial tricks. The paper, which introduced a testing method called CheckList, also found that the services changed their answer on 7.0% to 20.8% of airline tweets when only a city name was swapped, for example Cuba for Canada. And two research models fine-tuned on movie reviews, BERT and RoBERTa, did better than all three commercial models on almost every test that did not depend on predicting "neutral". That surprised the authors, since the commercial services listed social media as a use case and were regularly tested and improved using customer feedback.¹

A single accuracy number on a held-out test set would not have shown any of this. That is the problem an evaluation pipeline for an LLM application has to solve, and it has two halves. You need test cases that ask about specific behaviors, so a failure tells you what broke. Then you need enough statistics to know whether a change in the numbers is a change in the system or just noise from which cases you happened to pick. CheckList supplies the first half. Evan Miller's 2024 paper "Adding Error Bars to Evals" supplies the second.^1,2 What follows builds the pipeline in the order you would build it.

Stage 1: write tests that each name a capability

CheckList organizes tests as a grid. The rows are capabilities, the linguistic skills the task depends on: vocabulary, named entities, negation, time order, coreference (knowing who "he" or "her" refers to), and so on. The columns are three test types. The paper asks you to fill as many cells as make sense for your task, and the capability list is meant as a starting point you extend for your own domain.¹

Minimum functionality test (MFT)

A small set of simple, labeled examples that check one behavior, like a unit test. "I didn't love the flight" should be negative. The paper says MFTs are good at catching models that use shortcuts on complex inputs without having learned the skill.¹

Invariance test (INV)

Apply a change that should not affect the answer, and check that the answer stays the same. Swapping one city name for another, or adding a typo, should not change the sentiment of a tweet.¹

Directional expectation test (DIR)

Apply a change whose effect you know in advance, and check the output moves the right way. Adding "You are lame." to a complaint to an airline should not make it read more positive.¹

INV and DIR tests have a practical advantage: they need no ground-truth labels. They check a relationship between two outputs, before and after a perturbation, so you can run them over unlabeled real traffic. The CheckList authors ran their sentiment INV and DIR tests over a dataset of unlabeled airline tweets.¹

Writing test cases one at a time is slow, so CheckList generates them from templates. "I {NEGATION} {POS_VERB} the {THING}." with lists such as {didn't, can't say I} for the negation and {love, like} for the verb expands into every combination. To fill those lists, the tool can ask a masked language model for suggestions: "I really {mask} the flight." returns verbs like enjoyed, liked, loved and regret, which the user sorts into positive, negative and neutral lists.¹ The variety matters because, as the paper notes, a small set of cases can miss a failure when a model handles some forms of negation and not others.

The method held up on a team that already tested heavily. The authors ran a session of about five hours with the Microsoft team responsible for the sentiment service, whose evaluation already included public benchmarks and in-house ones for things like negation and emojis. The team brainstormed about 30 tests, implemented about 20 in the time available, and found many bugs they had not known about. Among the new tests were camel-cased hashtags like "#IHateYou" and implicit negation like "I wish it was good".¹ In a separate study, 18 participants given two hours to test a BERT model on duplicate-question detection wrote 5.8 tests on average without CheckList and 13.5 with the full template tools, and found 2.2 versus 6.2 bugs they rated severity 3 or higher on a five-point scale.¹

For an LLM application the same grid carries over. This is my reading, not the paper's: the rows become the behaviors your product promises, such as refusing out-of-scope requests or quoting a policy correctly, and the columns stay the same. An MFT is a set of plain requests with known right answers. An INV test rephrases a request or changes a customer's name. A DIR test adds a detail that should change the answer in a known direction.

Stage 2: turn each test into a score

CheckList reports each test as a failure rate: the fraction of its generated cases the model got wrong. For MFTs, wrong means the prediction did not match the label. INV and DIR tests need a tolerance, because a model's confidence always moves a little. In the paper, an INV case fails only when the predicted label changes and the probability moves by more than 0.1. A DIR case adding positive phrases fails if the sentiment score goes down by more than 0.1.¹ Choosing that tolerance is a design decision, and writing it down is part of the test.

A score per test case is also the raw material for everything statistical that comes next. Call the score on case $i$ $s_i$. It can be 0 or 1 (pass or fail), or a continuous value such as a probability. Continuous scores tend to be less noisy. Miller shows that for multiple-choice questions answered in a single token, scoring with the probability of the correct token instead of grading one sampled answer removes the sampling noise entirely.² Madaan and colleagues, who trained a set of 7B models that differed only in random seed, found that continuous metrics tracked training progress more steadily than discrete ones on almost every benchmark they studied.³

When a probability is not available, for example with long generated answers, Miller suggests sampling each case $K$ times and scoring the case by the mean of its $K$ results. He advises against lowering the sampling temperature to make scores less noisy. That changes the model you are measuring, and in his worked examples it can raise the variance of per-question scores rather than lower it.²

Stage 3: put an error bar on every number

Miller's starting assumption is that your test cases are a random sample from a much larger, unseen population of cases you could have written. The mean score over your $n$ cases is an estimate of the mean over that population, and it has a standard error: roughly, how far the estimate would typically move if you drew a fresh set of $n$ cases.²

Here $n$ is the number of test cases, $s_i$ is the score on case $i$, and $\bar{s}$ is their average. The sum under the root is the sample variance of the scores, and dividing it by $n$ gives the variance of the mean. When every score is 0 or 1, the formula reduces to $\sqrt{\bar{s}(1-\bar{s})/n}$, and a 95% confidence interval is $\bar{s} \pm 1.96 \times \mathrm{SE}$.² A pass rate of 80% on 400 cases has a standard error of 2 points, so its interval runs from about 76% to 84%. Miller also notes the Central Limit Theorem makes bootstrapping unnecessary for plain averages over many cases.²

The formula assumes the cases were drawn independently. Many evals break that: reading comprehension sets ask several questions about one passage, and a multilingual set can contain the same question in several languages. For grouped questions, Miller gives a clustered standard error that adds the correlations within each group.² On real Anthropic model results he reports, the clustered error was 3.05 times the naive one on DROP, 1.88 times on MGSM and 1.10 times on RACE-H.² Template-generated CheckList cases look like clusters to me, since every case from one template shares its wording. That link is my inference; neither paper tests it.

Stage 4: compare two versions on the same cases

A pipeline is rarely asked "how good is this model?" It is asked whether version B beats version A. The naive way treats the two scores as independent, so the standard error of the difference is $\sqrt{\mathrm{SE}_A^2 + \mathrm{SE}_B^2}$. Both versions usually run on the same cases, though, and Miller's recommendation is to analyze the difference case by case.²

$d_i$ is the score difference on case $i$, $\bar{d}$ is the mean difference, and $z$ says how many standard errors that mean lies from zero. If $|z|$ exceeds 1.96, the difference is significant at the 5% level, which is the same as saying the 95% interval $\bar{d} \pm 1.96 \times \mathrm{SE}_{\text{paired}}$ excludes zero. Pairing helps whenever the two versions agree on which cases are easy and which are hard, because that shared difficulty cancels in $d_i$. Miller shows the paired variance equals the unpaired variance minus twice the covariance of the two versions' true per-case scores, divided by $n$. In his example with a correlation of 0.5, pairing cuts the variance by a third.²

His motivating example uses two fictional models. "Dreadnought" beats "Galleon" on two of three evals, HumanEval and MGSM, and looks like the better model. With paired standard errors, only Galleon's 2.5-point lead on MATH has an interval that excludes zero; Dreadnought's leads of 3.1 and 2.7 points do not. The careful reading reverses the first impression.²

How many test cases a 3-point gain needs

The last formula runs the other way: given the difference you care about, how many cases do you need? The inputs are a significance level $\alpha$ (the false-alarm rate you accept), a power $1-\beta$ (how often a real difference of that size should be detected), and the minimum detectable effect $\delta$.²

$z_p$ is the point of the standard normal distribution with probability $p$ above it, so $z_{0.025} = 1.96$ and $z_{0.20} \approx 0.84$. $\omega^2$ is the variance of the true per-case differences between the versions. $\sigma_A^2$ and $\sigma_B^2$ are the average sampling noise per case for each version, and $K_A$, $K_B$ are how many times each case is sampled. With $K=1$, the three terms together are the variance of the observed $d_i$, which a pilot run can estimate. That last step is my reading of his variance decomposition.² Miller's own example, with $\omega^2 = 1/9$ and $\delta = 0.03$, needs about 969 questions, and he concludes new evals should contain at least 1,000.²

The example shows where the leverage is. Most of a paired comparison's noise comes from the cases where the versions disagree, so two prompts that mostly agree can be compared on fewer cases than two unrelated models. And because $n$ scales with $1/\delta^2$, halving the gain you want to detect quadruples the cases you need.²

Deciding when a change is real

The rule that falls out is short. Before running, decide the smallest difference worth acting on and check that your case count can detect it. After running, ship only if the paired interval excludes zero. A difference smaller than the minimum detectable effect is not evidence of no change. It means the test could not tell.

Skipping the first step has a cost that Dallas Card and colleagues measured across NLP. They estimated, for several GLUE test sets, the smallest improvement over the top model that a test of that size could detect with 80% power, and compared it with the average improvement papers claiming a new state of the art had reported. For the three smallest test sets, WNLI, MRPC and SST-2, the typical claimed gain was well below that threshold.⁴ They also point out that when underpowered experiments do come out significant, the measured difference tends to overstate the true one.⁴

Their machine translation estimate gives a sense of scale: a typical test set of 2,000 sentences has about 75% power to detect a 1 BLEU point difference.⁴ By Miller's formula, a suite of a few dozen hand-written cases can only detect large differences, on the order of the collapses in the chart at the top of this post.

There is a second way a static suite goes stale. The Dynabench authors point out that GLUE saturated within a year of release, and that challenge sets often lack statistical power, which is one reason they argue for collecting test data dynamically against current models.⁵ A pipeline's test cases need the same upkeep: once a model passes a test every time, the test tells you only about regressions.

What a clean run does not prove

CheckList's own authors state the limit plainly. The commercial models passed their simple fairness checks, such as always predicting "I am a black woman." as neutral. "Similar to software engineering," they write, "absence of test failure does not imply that these models are fair," only that they are "not unfair enough to fail these simple tests."¹ They also list what behavioral testing cannot reach at all: data versioning problems, labeling errors, annotator biases, worst-case security issues and lack of interpretability.¹ A pipeline built this way measures, with honest error bars, the behaviors someone thought to write down. Everything else is outside it.