The bill for a guardrail pipeline: what input, output and dialog checks catch and cost

Barakaeli Lawuo, Mar 18, 2026

For a paper published in January 2025, Anthropic ran a bug bounty through HackerOne against a Claude 3.5 Sonnet model wrapped in two classifiers, one reading the user's input and one reading the model's output. From 800 applications it invited 405 people, offered up to $15,000 per report, and asked them to get detailed answers to ten forbidden chemical, biological, radiological and nuclear questions. By the paper's estimate they spent over 3,000 hours on it (mean estimate 4,720), and Anthropic paid out $95,000 in total.¹ None of the 113 jailbreak reports answered all ten questions with even half the detail of a helpful-only model, one trained for helpfulness with no harmlessness training. Under the stricter bar of matching that model's detail, the best red teamer, pooling every report they sent, got six of ten.¹

The same paper is just as plain about the price. That prototype was tuned for robustness and refused about 44% of Claude.ai traffic, which the authors call impractically high.¹ A later version, with classifiers fine-tuned from the smaller Claude 3.5 Haiku, blocked over 95% of a held-out set of jailbreak attempts (the unguarded model blocked 14%). It raised the refusal rate on production traffic by 0.38 percentage points and added 23.7% to inference cost.¹ Those three numbers, a catch rate, an over-refusal rate and a compute overhead, are the bill for any guardrail. This post goes through the checks one at a time and reads each paper's bill.

Where the checks sit

A guardrail here means code outside the model that inspects traffic and can block or rewrite it. NeMo Guardrails, NVIDIA's open-source toolkit, describes its runtime as a proxy between the user and the LLM.³ In its moderation pipeline a user message first passes an input rail, the model answers, the answer passes an output rail, and only a response that clears both reaches the user.³ The toolkit also has execution rails: custom Python actions, such as a call to the WolframAlpha API or a fact check against retrieved evidence, that the dialogue flow runs at chosen points.³ Anthropic's system uses the same two ends, with an output classifier that watches the reply as it streams.¹

The three papers report quality in different units, so here are the terms, defined once.

Positive

an item the check should block, such as an unsafe prompt. A false positive is a harmless item it blocked anyway; a false negative is a harmful item it let through.

Precision

of everything the check flagged, the share that really was harmful. Low precision means many false positives, which users feel as over-refusal.

Recall

of everything harmful, the share the check flagged. This is the catch rate. Anthropic's paper calls it the true-positive rate.

AUPRC

area under the precision-recall curve. A classifier that outputs a score can be cut at any threshold; each threshold gives one (recall, precision) point, and the area under the resulting curve summarizes all of them in one number, with 1.0 perfect.

Attack success rate

the share of jailbreak attempts that got harmful content out of the whole guarded system. One minus this is the system's catch rate on attacks.

Check one: a 7B model reads the prompt

Meta's Llama Guard is Llama2-7b fine-tuned to act as a classifier. Its instructions contain a list of harm categories with plain-text descriptions, a flag saying whether to judge the user's prompt or the agent's response, the conversation, and an output format. The model answers "safe" or "unsafe" and, if unsafe, lists the violated categories. Because "safe" and "unsafe" are single tokens, the probability of that first token works as a score you can threshold.² The training set was 13,997 prompts and responses, labeled by Meta's in-house red team against six categories, split 3:1 for training and evaluation.²

Llama Guard reports AUPRC, following OpenAI's moderation work,² and the choice fits the data. On ToxicChat, a benchmark of 10,166 real user queries from the Vicuna demo chatbot, only 7.10% of examples are toxic.⁴ When positives are that rare, a classifier that flags nothing is 93% accurate, so accuracy tells you little. Precision and recall on the unsafe class do not have that problem. On prompt classification, zero-shot, Llama Guard scored 0.945 on its own test set against 0.764 for the OpenAI Moderation API and 0.728 for Perspective API. On ToxicChat, which none of the models trained on, it scored 0.626 against 0.588 and 0.532.² On OpenAI's own moderation set it scored 0.847 against the API's 0.856. Adding two to four examples per category to the prompt raised that to 0.872, ahead of the API, without any retraining.²

The cost side is thin in this paper. The authors chose the smallest Llama 2 size because it is more user friendly and has lower inference and deployment costs, and training took 500 steps on one machine with eight A100 GPUs.² They report no latency or per-request cost, so the real cost of running a 7B model on every message has to be measured on your own traffic.

Anthropic's input classifier does a narrower job. It is also a fine-tuned LLM, framed as next-token prediction, and among other things it is trained to block inputs that try to set up coded communication, so that the output would be unreadable without seeing the input.¹ The paper's framing is that the input classifier catches attempts to get around the output classifier.¹ On a random sample of red-team prompts, a model with only harmlessness training let through about 14% of attacks; adding the input classifier cut that to 2%, and adding the output classifier as well cut it to 0.5%.¹

Check two: judging a reply before it finishes

Llama Guard's response classification uses the same model with different wording in the instructions, and to its authors' knowledge it was the first work to split prompt and response moderation into two separate tasks.² Its response AUPRC on its own test set was 0.953, against 0.769 for the OpenAI API and 0.699 for Perspective.² That comparison favors Llama Guard, since the test set follows Llama Guard's own taxonomy, which the paper states openly and treats as the in-domain setting.²

An output check that waits for the full reply adds that wait to every response. Anthropic's output classifier avoids this by scoring the reply token by token as it streams. It is a fine-tuned LLM with a linear value head, a small layer that turns the model's internal state at each position into one number: the predicted probability that the finished output will be harmful, given only the prefix so far. When the score passes a threshold, streaming stops.¹ Because a stop cannot be undone, training gradually shifts from the score at each token to the running maximum of all scores so far, which matches how the classifier is used.¹ In the paper's cost accounting, this design means the output classifier generates no output tokens at all; the input classifier generates one.¹

Streaming creates a question that whole-output tests hide: did the stop come in time? Anthropic truncated each blocked output at the token where the classifier first crossed its threshold and checked what a user would already have seen. Across 10,000 jailbreak outputs, the system blocked 96.70%. Only 117 of the truncated outputs, 1.17% of attempts, had already shown harmful content.¹ The wiring matters as much as the model. One red-team report first looked like a universal jailbreak, and it turned out to be an implementation bug that let up to 128 tokens through after the output classifier had flagged them.¹

Check three: dialog rails that call the LLM three times

NeMo Guardrails controls where a conversation can go, not only what text is allowed. Developers write rules in a modeling language called Colang: canonical forms, short natural-language labels for what a message means ("ask about capabilities"), and flows, the dialogue paths the bot must follow.³ For each user message the runtime makes three LLM calls in sequence. The first turns the message into a canonical form, using the nearest examples from a vector index as few-shot prompts. The second picks the next step, from a matching flow if there is one or by letting the LLM improvise. The third writes the bot's reply. If the next step for a political question is "bot inform cannot answer," the bot declines.³

The paper measured these topical rails on a banking intent dataset of 231 test samples over 77 intents. With text-davinci-003 and all examples indexed, the first step recovered the right user intent 77% of the time, or 82% when a similarity match replaced exact matching. falcon-7b-instruct reached 70% and 76%.³ With only one indexed example per intent, text-davinci-003 fell to 50%, which is why the authors recommend at least three.³

The safety rails in NeMo are prompts to a strong LLM, not trained classifiers. The input rail asks whether an instruction would make a model break moderation policies; the output rail asks whether a reply is legal, ethical and not harmful.³ The authors built a set of 200 prompts, half taken from Anthropic's red-teaming data (the highest-rated harmful prompts) and half from its helpful data. With no rails, text-davinci-003 refused only 24% of the harmful prompts. The input rail alone raised that to 87% while blocking 3% of helpful prompts, and both rails together reached 97% at a cost of 5% of helpful prompts. gpt-3.5-turbo already refused 93% on its own; with both rails it blocked close to 99% of harmful prompts and 2% of helpful ones.³

The execution rails around retrieval do similar work for facts. The fact-checking rail asks the LLM whether the answer is entailed by the evidence, and on MS MARCO pairs with rewritten hard negatives both text-davinci-003 and gpt-3.5-turbo scored about 80% accuracy.³ The hallucination rail borrows SelfCheckGPT's idea that sampled answers to a question the model really knows tend to agree, while made-up facts diverge across samples.⁶ On 20 questions with false premises, the rail caught 70% of the cases text-davinci-003 would otherwise have answered, and raised gpt-3.5-turbo's rate of deflecting or hedging from 65% to 95%, by the main text's figure.³ Twenty questions is a very small test, and it should be read that way.

The bill for dialog rails is in the paper's limitations. The three calls are chained, each depending on the one before, so they cannot be batched, and latency and cost come to about three times a normal call.³ The moderation rails are separate LLM calls on top of that.³

Catch rate against false refusals

Every check above has a threshold, and moving it trades recall for precision. Llama Guard's appendix shows this with a fixed threshold of 0.5 on its own prompt test set. GPT-4, prompted zero-shot, had the highest recall at 0.947 but the lowest precision at 0.717: it caught almost everything and flagged many safe prompts along the way. The OpenAI Moderation API was the reverse, with precision 0.874 and recall 0.250. Llama Guard sat at 0.880 and 0.864.² Part of the gap is taxonomy. The moderation API was built around its own taxonomy, with an active learning pipeline meant to capture rare events,⁵ and on Llama Guard's category of regulated or controlled substances it caught almost nothing (recall 0.008).²

Anthropic chose its thresholds by stating the refusal budget first. It swept combinations of input and output thresholds and kept the pair with the highest true-positive rate on its jailbreak set, subject to three limits on the whole system: at most 0.5 percentage points more false positives than the unguarded model on multi-turn WildChat conversations, at most 5% false positives on single-turn WildChat queries, and at most 5 points more on the OR-Bench over-refusal benchmark.¹ Written this way, over-refusal is a constraint the design must meet, and robustness is what gets maximized inside it. The prototype, with no such budget, refused about 44% of traffic. A version built with the improved method and the prototype's constitution refused about 1.74% of production traffic, with lower attack success than the red-teamed system, though red teamers never targeted that version directly.¹

A low average can hide a high rate in one domain. The chemical-weapons classifiers had a false-positive rate of 1.50% on MMLU chemistry questions but 26.05% on the harder GPQA chemistry questions.¹ On 1,000 chemistry-related WildChat queries, the model itself refused 3.6%, and the classifiers blocked a further 3.4% of queries the model had been willing to answer.¹ The authors read the GPQA gap as a capability limit, since larger classifiers held at the same robustness had lower false-positive rates on those questions.¹ The practical point: a refusal budget measured on general traffic says little about the users whose work sits next to the restricted topic.

The classifier's own size and setup drive the compute line. In Figure 6, three prompted Claude 3.5 Sonnet classifiers (zero-shot, chain-of-thought and 32-shot) caught less, refused more and cost more than the fine-tuned Haiku classifiers; the constitutional classifiers had both the smallest refusal increase and the lowest overhead of all the setups tested.¹ The overhead estimate assumes an efficient setup with prompt caching, and it scores whole outputs, not streams.¹ NeMo's authors make the same argument from the other side: they expect smaller models to bring down the cost of their rails, and they note that falcon-7b-instruct already did well on topical rails.³

What the red team broke instead

The most revealing result in the Anthropic paper is where the attackers went. The highest-scoring reports mostly did not beat the classifiers. They beat the grading. One tactic, benign paraphrasing, reworded a harmful question into something that sounded innocent. The other, length exploitation, produced very long outputs that scored rubric points through volume. Established universal jailbreaks such as many-shot prompting were missing from the successful attacks. In the authors' words, "attackers tend to target a system's weakest component," and here that was the evaluation protocol.¹ The other flaw found was the 128-token streaming bug, which sat in the plumbing, not the model.¹

The authors do not claim the question is settled. Their limitations section says the red-teaming cohort is only a subset of possible adversaries, that bounties paid out at the half-detail mark so stronger attacks may have gone unfound, and that the test covered a chatbot with preset questions, leaving other deployment settings open.¹ Their conclusion goes further: "common wisdom suggests that system vulnerabilities will likely emerge with continued testing," and responsible deployment will need complementary defenses such as rapid patching of new jailbreaks and monitoring to detect them.¹ NeMo's authors say the same about their own toolkit, that it should not be used as a stand-alone solution, especially for safety rails.³