Function Calling, MCP and the API Underneath: A Stack Read Bottom Up

Barakaeli Lawuo, Jul 26, 2026

The Berkeley Function Calling Leaderboard (BFCL) paper, presented at ICML 2025, has a category called Parallel: one tool is available and the right answer is to call it several times in a single turn.¹ OpenAI's o1-2024-12-17 scored 91.5 on it when the function definitions were pasted into its system prompt. With the same definitions supplied through the model's native tools field, it scored 0.0. Claude-3.5-Sonnet-20241022 went from 70.5 to 3.5 the same way.¹ The authors' reading is that the parallel call feature "appears to have regressed or even removed" in those flagship versions, and they guess why: parallel calls can hurt accuracy when each call depends on the result of the previous one.¹

The model did not get worse at knowing what to call. On the Multiple category, where it has to pick one function out of several, o1 scored 93.5 in prompt mode and 93.0 in native mode.¹ What changed was the channel the call went through. That is a useful way into a question people ask a lot: should I use MCP, function calling or just call the API? The three are not competing answers to one question. They are layers stacked on top of each other, and each one has its own way of failing.

Plain API

An ordinary endpoint, usually HTTP, that does the actual work: reads a database, converts a currency, runs a search. It has no idea a model exists.

Function calling

The model writes a structured request, a function name plus arguments, instead of prose. The model does not run anything. Some other code does.

JSON Schema

A standard way to describe the shape of a JSON object: which fields exist, their types, which are required. Tool definitions use it to describe arguments.

MCP

The Model Context Protocol. A client asks a server which tools it has and then asks it to run one, using JSON-RPC messages.^5,6

Application loop

The code around the model that runs each call, puts the result back into the conversation and asks the model what to do next, until the task ends.

The HTTP API does the work and knows nothing about models

At the bottom sits code that already existed before anyone attached a language model to it. When BFCL built its executable test category, part of it was Python functions that wrap public REST services such as ExchangeRate API, OMDb API and a geocoding API. They stuck to GET requests because those are the most common kind in real use.¹ Nothing in those endpoints changes when a model is the caller. They get a URL and parameters and return a response.

This layer adds the only thing that actually touches the world. It also has a demand of its own. Toolformer, the 2023 Meta paper that taught a model to insert its own API calls, put exactly two constraints on its tools: their inputs and outputs have to be representable as text, and the authors need a few demonstrations of how each tool is used.² Whatever the API returns has to become a string the model can read. The response for each call, in their words, needs to be "a single text sequence."²

What this layer returns also puts a ceiling on everything above it. Toolformer answered open questions with a Wikipedia search tool and still trailed the much larger GPT-3. The authors blame the simplicity of their search engine, which in many cases returned results that were clearly not a good match for the query.² ReAct gave its models a deliberately weak Wikipedia API with three actions, search, lookup and finish, which the authors describe as "significantly weaker" than modern retrievers.³ They then hand-labeled 200 HotpotQA trajectories. In the ReAct failures it sampled, 23% were search result errors, where the search came back empty or held nothing useful. The authors note this "derails the model reasoning" and makes it hard to recover.³ The API answered correctly by its own standards. It just had nothing useful to say.

The structured call: a name and some arguments, nothing more

One layer up, the model's job is to turn a sentence into a request the API layer can run. Toolformer did this with special tokens: the model learned to write something like [Calculator(400 / 1400) → 0.29] inside ordinary text. During decoding, when the model produced the arrow token, the system paused, called the API, pasted the result in and let the model continue.² The model was a 6.7B GPT-J, fine-tuned on text where calls had been kept only if their results lowered the loss on the tokens that followed.² The ability showed up late. In their scaling experiment with smaller GPT-2 models, useful tool use only emerged at around 775M parameters.²

Current APIs ask for a JSON object instead of inline tokens, but the idea is the same. The model's entire output at this layer is small:

{ "name": "get_exchange_rate",
  "arguments": { "base": "USD", "target": "EUR" } }

That object is not an HTTP request. Some other code has to map it to one. That split is also what makes this layer testable without the layer below. BFCL scores most calls without running them at all. It parses the model's output into an abstract syntax tree (a tree of the call's name and arguments) and checks that the function name matches exactly and that every argument falls within a set of accepted values. Strings are compared case-insensitively after stripping whitespace and some punctuation, so "NYC" and "New York City" can both pass.¹ On a subset of the data, those scores tracked scores from actually running the calls closely.¹

By this measure, single calls are close to solved for strong models. gpt-4o-2024-11-20 in native mode scored 93.5 on Multiple and 83.1 on Irrelevance, the category where tools are offered but the right move is to call none of them.¹ The failures that remain look like the opening chart. When a model supports both modes, native function-calling output is more rigidly structured, which reduces parsing errors. The paper says prompted models had on average three times more decoding problems. Its own figures, 412.93 versus 182.5 out of 4,251 entries, work out to about 2.3 times.¹ But among outputs that did decode, native mode got the number of calls wrong more often on the Multiple category, 77.5 versus 21 on average, and the authors conclude that prompted models are "generally more flexible in complex scenarios."¹ The structure that makes the output easy to parse also limits what the model will produce.

The protocol: listing tools and routing calls

A structured call assumes the model was told which functions exist. In a single app, a developer writes those definitions by hand. MCP moves that job into a protocol. A client sends a tools/list request, and the server answers with each tool's name, description and an inputSchema written in JSON Schema. To run one, the client sends tools/call with the name and an arguments object.⁵ Messages are JSON-RPC, carried over one of two standard transports: stdio for a local subprocess, or Streamable HTTP for a server running as its own process.⁶ A server can also send notifications/tools/list_changed when its set of tools changes.⁵

Note what is missing from that list. MCP does not choose a tool. The model still produces the same structured call as before, and the client passes it on. The protocol adds discovery at run time and a single envelope for calls and results, including two kinds of errors: protocol errors such as an unknown tool, and tool execution errors reported with isError set to true, such as a failed upstream API.⁵ That second kind is the HTTP failure from the bottom layer, packaged so the model can read it.

Discovery brings its own failure: the model now sees tools it has never seen before, and more of them. BFCL measures tool choice under distractors. Its Multiple category adds unrelated function documents to a query to check the model is not misled.¹ Its crowd-sourced set, built from 64,517 real queries sent to the team's hosted endpoint, averaged 3 function choices per entry, with one entry offering 37. Real users asked for choosing between functions much more often than for parallel calls.¹

MCP-Universe, a 2025 Salesforce benchmark built on 11 real MCP servers, measured both problems directly.⁴ The authors call the first the unknown-tools challenge. For example, a Yahoo Finance server needs different start and end dates to fetch a stock price, and models often set them to the same day, which caused execution errors.⁴ Letting the model explore the tools first helped in some domains and not others. GPT-4.1 gained 7.69 points in browser automation, while Claude-4.0-Sonnet got worse in that domain.⁴ Then they connected 7 servers with 94 tools in total, most unrelated to the task. Claude-4.0-Sonnet's success rate in location navigation fell from 22.22% to 11.11%, and GPT-4.1's in browser automation fell from 23.08% to 15.38%.⁴ Nothing in the protocol went wrong. The protocol did its job, and the model had a longer menu to choose from.

The application loop: where single calls turn into tasks

The top layer is the code that runs a call, reads the result and goes back to the model. ReAct gave this loop a common shape: the model alternates a written thought, an action and an observation from the environment.³ On ALFWorld, a text-based household game, the best ReAct run reached a 71% success rate against 45% for the same prompting without thoughts. Without thoughts, the authors saw the model fail to break goals into subgoals or lose track of the environment's state.³ The loop is also where ReAct's most common failure lived. Reasoning errors made up 47% of its sampled HotpotQA failures, including one pattern specific to ReAct: the model repeats its previous thoughts and actions and cannot get out of the loop.³

BFCL's multi-turn categories test this layer, and the scores drop sharply. The same gpt-4o-2024-11-20 that scored 93.5 on picking a single function scored 62.5 on basic multi-turn tasks in native mode. On Missing Parameters, where it should ask the user for a detail instead of guessing, it scored 37.5. On Missing Functions, where it should notice that no available tool can do the job, it scored 6.0.¹ In the memory category even the leader, o1 in native mode, reached only 12%. The paper describes models guessing keys instead of listing them first, and many giving up after one failed lookup.¹

When the BFCL authors had GPT-4o act as a judge and classify the root cause of multi-turn failures, the most common was failing to understand the environment state: assuming or hallucinating a state that was not there.¹ One of the few-shot examples in their judge prompt shows it plainly. The user asks for just enough fuel to reach a town. The model calls fillFuelTank with 50 gallons, and the labeled answer first calls displayCarStatus to check the fuel level, then adds 44.¹ Each individual call there was well formed and would have passed a single-turn check. The mistake was at the loop layer, in the decision not to look before acting.

MCP-Universe ran almost every model through a ReAct loop over real MCP servers, and the best result was GPT-5 at 43.72% overall.⁴ The authors also report that input tokens grow quickly with the number of interaction steps.⁴ That growth is a cost only the loop pays. Neither the API, nor a single structured call, nor the protocol keeps the history. The loop does, and it resends the whole history on every step.

What Toolformer said it could not do

The layers above the structured call exist mostly to handle work that a single call cannot. Toolformer's authors said this about their own method in its limitations section. Toolformer could not use tools in a chain, where one tool's output becomes another tool's input, because every API call in its training data was generated independently and so no chained example ever appeared.² It also could not use a tool interactively, for example browsing through a search engine's results or refining a query, which they say certain applications need.² Their own TempLAMA result showed the cost. The best approach there would have been to ask the calendar for today's date and then ask the question-answering tool with that date. Their limit of one API call per input prohibited that, and it was hard for Toolformer to learn anyway.²

A protocol that lists tools and a loop that feeds results back were built for those two gaps. BFCL's multi-turn scores, measured on the model versions in its table, show the gaps are narrower but still open. And the paper that first stated them added one more limitation that none of these layers has addressed yet: when Toolformer decides whether to make an API call, it "does not take into account the tool-dependent, computational cost incurred from making an API call."²