Generating tool-use training data from executable workflows

Training a language model to use software tools requires examples that connect a request to the operations needed to carry it out. Each example must specify suitable tools, supply valid arguments, and account for the information returned by earlier calls. Producing such examples becomes difficult when a request requires several operations across a large collection of tools.

One approach begins by generating a request, then searching for a sequence of calls that can fulfill it. This creates a source of wasted work: the generated request may be impossible to satisfy with the available tools. A search can continue through many unsuccessful calls without producing a usable training example.

ToolGrad, developed by Zhongyi Zhou and colleagues, reverses this construction process. It builds an executable sequence of tool calls and generates a corresponding request from the resulting workflow. Their study examines both the efficiency of producing these examples and the performance of language models trained on them. Google Research described the work in September; the paper is listed in ACL 2026 Findings. 1 2

Constructing a request from execution

A software tool is exposed to a model through an application programming interface, or API. Its definition describes an operation and the arguments that operation accepts. Selecting a tool therefore involves more than recognizing its name. The model must relate the user's intent to the operation, identify the required inputs, and express them in the expected format.

For a sequence of operations, the relationships between calls also matter. An identifier returned by one call may become an argument to another. A successful response establishes that a particular invocation can run under the tested conditions. Retaining that response gives the data generator evidence about what the sequence accomplished.

ToolGrad organizes construction into four stages. A proposer selects candidate APIs from a sampled group. Executors attempt the candidates and return execution reports. A selector uses those reports to choose an extension to the workflow. An updater revises the synthetic request and response to reflect the expanded sequence. Repeating these stages produces a request paired with tool usage grounded in execution. 2

!Four stages of ToolGrad: propose candidate tools, execute candidate calls, select a successful extension, and update the synthetic request. A return arrow indicates another construction iteration.

Figure 1. ToolGrad extends a workflow through proposal, execution, selection, and annotation. The request changes as the workflow grows. Original schematic based on the authors' method; it represents training-data construction. 1

The term textual gradients refers to feedback expressed through language that guides this construction. It is an analogy to an optimization signal: the feedback helps determine the next change. Ordinary numerical gradients are used separately when the resulting examples train a model.

This ordering changes which uncertainty the generator must resolve. With a request specified first, the search must discover whether the available operations can fulfill it. With an executed workflow in hand, the annotation step has a concrete result to describe. The remaining difficulty is to produce a request that accurately expresses that result and resembles something a person would ask.

Measuring data-generation efficiency

The authors compare ToolGrad with a depth-first search procedure used in ToolBench. Depth-first search explores a candidate sequence and can return to earlier decisions when that sequence fails. The reported comparison measures the success of data generation, the length of the retained workflow, and the work performed while constructing it.

Table 1. Results from Table 2 of the ToolGrad paper. Call and step counts describe generation work; retained tool uses describe the resulting examples. The pass rate concerns data generation under this experimental setup. 1

The table separates two quantities that could otherwise be confused. A generator may execute several candidate calls while retaining only a few in its final example. Here, ToolGrad performs fewer tool-use steps during generation while producing examples with longer retained workflows. The language-model call counts are similar.

These measurements support a claim about the construction procedure. They do not measure the reliability of an agent responding to new users. They also leave the monetary cost unresolved: a call count does not specify token volume, model pricing, execution latency, or the charges associated with external APIs.

Evaluating models on unfamiliar tools

The usefulness of the data depends on what a model learns from it. A model could become better at reproducing the generator's preferred sequences without becoming better at interpreting unfamiliar tool definitions.

The authors use Gemini 2.5 Flash-Lite to construct ToolGrad-500 through 500 generation runs, each with ten iterations. They format the workflows for supervised fine-tuning, in which a model learns to predict the supplied target calls. The released processing pipeline also adds negative examples with unsuitable tools and an empty call list as the target. These teach the model to withhold a call when the available tools cannot satisfy the request. 1 3

The resulting Gemma 3 models are evaluated on ToolBench-I3 and versions 1 and 2 of the Berkeley Function Calling Leaderboard, or BFCL. BFCL includes tests of tool selection, arguments, and multiple calls within a single response. Its structural checks compare predicted calls against accepted function names and arguments; its execution checks examine the outputs of callable functions. 4

In the reported BFCL comparison, the authors' 12-billion-parameter model scores 83.1, close to the 83.2 reported for Gemini 2.5 Pro. The study uses prompted calls for models supporting both prompting and native function-calling modes, to match the Gemma 3 setup. The scores therefore describe performance under that particular comparison. 1 2

The component tasks show uneven changes. In Table 5, the 12-billion-parameter model's score on the BFCL Live parallel-call task falls from 87.50 to 75.00 after training, while its score on the Live multiple-parallel task rises from 45.83 to 62.50. The former asks for several invocations of one tool; the latter also requires choosing among tools. These results indicate that aggregate improvement does not establish improvement in every form of tool use. 1

The evaluated models predict their calls together in one response. This differs from an agent that reads a tool result and then decides its next action. The study leaves performance in those iterative systems, and in multi-turn conversations, to future work. Recovering from service failures or revising a plan after a user changes the request requires evaluations with those interaction sequences. 1

Diversity limits the benefit of more examples

Scaling the generation procedure introduces another problem. If independent runs repeatedly construct similar workflows, the number of examples can grow faster than the range of behavior they cover. The model receives more observations of familiar patterns while uncommon combinations remain poorly represented.

The authors observe this limitation in their scaling experiments: increasing the amount of synthetic data eventually reduces performance. They identify repetitive generation as a possible explanation and propose sharing memory across generation runs to encourage diversity. They also note that synthetic requests may not match the variety of human language. 1

The released repository provides generation code, model and dataset links, and instructions for reproducing evaluation and fine-tuning. These resources make it possible to investigate how generation choices affect the resulting examples. They also allow the method to be adapted to other tool collections. EFFOMA has not independently reproduced the experiments. 3

An executable workflow provides a useful constraint on synthetic data: the annotation can be tied to operations that were actually attempted. Representativeness remains a separate requirement. A dataset intended for real users must cover varied requests, dependencies between tools, and situations in which the available operations cannot satisfy the request.

The next research question is how to direct generation toward missing behaviors while preserving that connection to execution. Testing such a method would require measuring both the diversity of generated workflows and performance on held-out human requests. That would help establish when additional synthetic examples extend a model's competence beyond the patterns the generator finds easiest to produce.

References

1. Zhongyi Zhou, Kohei Uehara, Haoyu Zhang, Jingtao Zhou, Lin Gu, Ruofei Du, Zheng Xu, and Tatsuya Harada. ToolGrad: Efficient Tool-use Dataset Generation with Textual “Gradients”. Findings of ACL, 2026; arXiv:2508.04086, version 3. Method, Table 2, scaling experiments, limitations, and Appendix C.
2. Zhongyi Zhou and Ruofei Du. ToolGrad: Efficient tool-use dataset generation with textual “gradients”. Google Research, September 10, 2026. Authors' account of the construction procedure and model evaluation.
3. Zhongyi Zhou and collaborators. ToolGrad source code and reproduction instructions. Official project repository, accessed September 26, 2026. Generation, fine-tuning, evaluation, and links to released artifacts.
4. Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Ion Stoica, Joseph E. Gonzalez, Tianjun Zhang, and Shishir G. Patil. Berkeley Function-Calling Leaderboard. Berkeley Gorilla project, 2024. Function-calling task definitions, structural checks, and executable evaluations.