One tool call, traced through four hops

Barakaeli Lawuo, Mar 15, 2026

In 2023 a team from Alibaba Group, HKUST and Peking University built 73 working APIs, from weather lookups to account management and alarm clocks, and spent 98 person-days of senior engineering time making them run inside one test system. Annotators then wrote 314 dialogues by hand that needed 753 API calls between them.¹ The benchmark, API-Bank, scores three levels of skill. At Call, the model is shown the API descriptions and has to call the right one. At Retrieve+Call, it is not told which APIs exist and must first find one with a search tool. At Plan+Retrieve+Call, it has to find and chain several calls on its own.¹

The gpt-4-0613 checkpoint got 63.66% of calls right at the first level, 37.04% at the second and 70.00% at the third. The gpt-3.5-turbo-0613 checkpoint scored 59.40%, 38.52% and 22.00%. GPT-3 Davinci, a 175 billion parameter model, managed 0.50% at the easiest level; the authors suspect that calling an API needs the instruction following that only instruction tuning unlocks.¹ The hardest level holds only 50 of the 314 test dialogues, so the 70% rests on a small sample.¹ Even so, the pattern says something about how tool use works. A single accuracy number hides a chain of steps, and each step fails in its own way.

This post follows one request through those steps. The API field names come from Anthropic's Messages API, but the shape is the same in any provider's function calling. Function calling, or tool use, means the model can ask for a function to be run and see what it returned. For tools the application defines, the model never runs anything itself. It writes a request, and the application's own code does the rest.⁵

Qin and colleagues, in a 2023 survey of tool learning, gave the moving parts names. The controller is the foundation model that plans. The tool set is the collection of callable tools. The environment is where a tool actually executes, and the perceiver turns what came back, from the environment or from the user, into feedback for the controller.² At each step $t$ the controller picks an action $a_t$ given the summarized feedback $x_t$, the history $H_t$ of earlier feedback and actions, and the user's query $q$. The process repeats for several rounds until the task is done.²

Take an illustrative request: a user asks what Alibaba's stock is trading at, and the application offers the model a single stock price tool.

Hop one: the model reads a description, not a function

Before the model sees the question, the application sends a list of tools with it. In the Messages API each tool has a "name", a "description" and an "input_schema", which is a JSON Schema listing the parameters, their types and which ones are required.⁵ For the stock question, that means a tool called get_stock_price with one string parameter, the ticker. That text is all the model knows about the tool. It cannot inspect the code behind it. API-Bank's prompt worked the same way: each API arrived as a JSON line with a name, a description, and typed input and output parameters.¹

Qin et al. split what the controller has to do at this point into two parts. Their equation 2 factors the choice of action into picking a tool and then deciding what to do with it:²

The first thing that breaks at this hop is simply fitting the tools in. API-Bank's authors note that a user might define two or three APIs, which all fit in the prompt, or hundreds, which do not. Past that point the model has to retrieve candidate tools before it can call one.¹ Their Retrieve+Call level tests exactly that, with an API Search tool that embeds the model's keywords and returns the single closest API description.¹ When they sorted GPT-4's errors, 67.86% were failed retrievals: the model could not use the search step to reach the API it needed.¹ GPT-3.5 lost about 21 points going from Call to Retrieve+Call.¹

The second thing is how the tool is described. Qin et al. tested 18 tools with text-davinci-003 and ChatGPT, once with only a task description and API information (zero-shot) and once with worked usage examples added (few-shot).² On a curated map tool with 11 APIs, ChatGPT went from 29.7% zero-shot to 86.8% few-shot. On a knowledge graph tool, it reached only 14.1% even with examples.² The authors single out tools that need generated code as an API argument, such as that tool's search_by_query API, as much harder to use.²

Hop two: the call is text the model writes

When the model decides to use a tool, the API response ends with "stop_reason" set to "tool_use", and the content holds a "tool_use" block with an "id", the tool's "name" and an "input" object.⁵ The whole mechanism, cut down to the definition and the call:

{ "name": "get_stock_price",
  "description": "Latest price for one stock ticker.",
  "input_schema": {
    "type": "object",
    "properties": { "ticker": { "type": "string" } },
    "required": ["ticker"] } }

{ "type": "tool_use", "id": "toolu_01A",
  "name": "get_stock_price",
  "input": { "ticker": "BABA" } }

The second object is generated token by token like any other text, and every way it can go wrong was counted in API-Bank. The authors defined six error types.¹ No API Call means the model answered without calling anything. API Hallucination means the name in the prediction does not match the ground truth. False API Call Format means the call could not be parsed. Invalid Input Parameters and Miss Input Parameters cover bad or missing arguments, and Has Exception means running the call raised a Python exception.¹

Which error dominates depends on the model. For the original Alpaca-7B, 36.77% of errors were no call at all and 23.65% were unparseable calls.¹ After fine-tuning on API-Bank's training data, only 5.29% of Lynx's errors were skipped calls, but 61.38% were hallucinated API names. Sometimes it called APIs it had seen during training that were not in the test prompt.¹ Argument problems, which the authors say surface as exceptions, invalid parameters or unparseable calls, made up about 32% of Lynx's errors. The examples the authors list are concrete: a placeholder passed as a value, a date not in the required format, a missing parameter, and a company name passed where the API wanted a stock code.¹

GPT-4's second largest error class, 17.86%, was calls the system could not parse. The cause was that GPT-4 sometimes made several API calls at once, which the test prompt did not allow.¹ API-Bank asked for calls in a bracketed text format written into the prompt, such as [ApiName(key1='value1')].¹ In the author's reading, part of that error class is about the prompt's own format rules rather than about the model misunderstanding the tool. The authors ended by calling for decoding methods that force strict adherence to the input parameter definitions.¹

Providers now ship something close to that. Setting "strict: true" on a tool definition constrains sampling so the tool input always matches the JSON Schema.⁷ Reading that against API-Bank's list, it can remove unparseable calls and wrongly typed or missing required arguments. It cannot catch a company name in a field typed as a string, because "Alibaba" is a valid string. That is the author's reading; API-Bank did not test constrained decoding.

Hop three: the runtime runs it and reports what happened

The call block is a request, not an action. Nothing has looked up BABA yet. The application reads the "name", runs its own function with the "input", and replies in a new user message containing a "tool_result" block. That block carries the "tool_use_id" of the call it answers, the output as "content", and optionally "is_error" when the tool failed.⁶ In Qin et al.'s terms this is the environment producing feedback $e_t$, and the perceiver can be as simple as a fixed template that wraps it before the controller sees it.²

API-Bank is built around this hop. The authors required evaluation to use a working system: the model offers a call, the system executes it and returns the result, and scoring looks at the effect of the execution.¹ Database-backed APIs start each run from the same default entries. APIs that reach outside, like a search engine, return answers recorded at one moment and hard-coded so the benchmark repeats.¹ A prediction is judged correct when it performs the same database queries or changes as the annotated call and returns the same result.¹ Under that rule, a call that looks right but writes the wrong row is wrong.

Failures surface here as exceptions. In the Lynx evaluation, 16.40% of errors were calls that raised an exception when executed.¹ Shen et al., describing HuggingGPT, a system in which ChatGPT plans tasks and dispatches them to models hosted on Hugging Face, list instability as one of its limits: the LLM can fail to follow instructions or give wrong answers, which leads to exceptions in the program workflow.⁴ Chameleon, a planner system from UCLA and Microsoft Research, puts a Program Verifier module in front of its Program Executor to check generated programs for syntax and logic errors before they run.³

Hop four: the result goes back and the loop decides

The model is then called again with everything so far: the question, its own "tool_use" block and the "tool_result". It either writes an answer, here a sentence quoting the price, or emits another call.⁵ Qin et al.'s formulation has the same shape. The next feedback goes to the perceiver, and the cycle repeats until the controller finishes the task. After that, the controller may turn the execution results into a response for the user.²

A correct call does not guarantee a better answer. In Qin et al.'s experiments ChatGPT answered 91.7% of ASDiv math word problems without any tool. Given a calculator with zero-shot prompting, it fell to 74.1%. With few-shot examples it reached 92.5%.² On HotpotQA with a Wikipedia tool the drop was steeper: 34.5% with no tool, 8.5% zero-shot and 19.0% few-shot.² The authors' conclusion is that sub-optimal use of a tool can make results worse than not using it.²

HuggingGPT's human evaluation shows how steps compound. Across 130 requests, GPT-3.5 had a passing rate of 91.22% for task planning and 93.89% for model selection, yet the success rate, whether the user's request was finally resolved, was 63.08%.⁴ API-Bank scored the final reply separately with ROUGE-L, a word-overlap measure. GPT-4 made more correct calls than GPT-3.5 at the Call level but scored 0.3691 against GPT-3.5's 0.4598 on the reply that followed.¹

A request that needs several tools goes through these hops more than once. Chameleon asks an LLM planner to write the whole sequence up front as a natural-language-like program, such as a list of module names, and then runs the modules in order.³ With GPT-4 as the planner it reached 86.54% on the ScienceQA benchmark and 98.78% on TabMWP, a benchmark of math problems over tables.³ The planners differed in which tools they reached for. ChatGPT called Knowledge Retrieval on 72% of ScienceQA queries and Bing Search on 3%, which the authors attribute to the in-context examples. GPT-4 called them 81% and 11% of the time.³

The error counts show both sides. Starting from 50 questions plain ChatGPT got wrong, Chameleon with ChatGPT cut knowledge errors from 37 to 6 and image errors from 32 to 10. It also made 13 tool planning mistakes, a category where the tool-free baseline shows zero.³ With GPT-4 as the planner, tool planning mistakes fell to 1.³ The paper's appendix traces other failures to a hop: a search query too vague to bring back the needed facts, a plan that skipped the text detector and knowledge retrieval, and a picture of a food web with arrows that no module in the inventory could read.³

What Chameleon's authors said would not hold

Chameleon's limitations section names two assumptions that touch hops one and four. The system generates its program in one step, with no re-planning as the modules run. It also assumes the list of modules and their descriptions will fit within the LLM's context window, which the authors say may not always be the case as the module inventory grows.³ The second assumption is the one API-Bank built its retrieval levels around, and failed retrieval made up 67.86% of GPT-4's errors there.¹