Machine Intelligence. Zhongyi Zhou, Research Scientist, and Ruofei Du, Interactive Perception & Graphics Lead, Google XR ToolGrad is a data generation framework that reverses the traditional paradigm by first generating tool-use answers before user queries.

What happened

Zhongyi Zhou, Research Scientist, and Ruofei Du, Interactive Perception & Graphics Lead, Google XR ToolGrad is a data generation framework that reverses the traditional paradigm by first generating tool-use answers before user queries. We show this design enables LLMs to achieve better tool-use performance.

AI agents have shown great potential in automating real-world tasks, such as conducting a Google Search, reading local computer files, or executing generated Python scripts. To achieve such agentic workflows, LLMs need to learn how to use tools correctly and efficiently.

To teach large language models tool uses, we need datasets of tool-use chains and their corresponding user queries.

The important details

In our prior work introduced in InstructPipe, we manually annotated our evaluation data, but it is impractical to scale up the human annotation for advanced LLM fine-tuning workstreams. To streamline the data workstream, prior work, e.g., ToolBench and ToolACE, explored using an agent to automatically search a tool-use path with trial and error.

This representative annotation approach involves two steps: (1) generate a hypothetical user instruction from a sampled API pool, and (2) use a depth-first search (DFS) agent to find its tool-use solution. This approach is inherently inefficient because its core concept is to distill valuable trajectories from a complex agent exploration for training an LLM.

In “ToolGrad: Efficient Tool-use Dataset Generation with Textual ‘Gradients’”, presented at ACL 2026, we introduce an alternative solution paradigm.

Context and impact

ToolGrad first generates a ground-truth tool-use chain and then annotates its corresponding user prompt. Intuitively, an explicit tool-use solution provides more unambiguous information than a prompt, making the annotation, from tool usage to the use query, much easier and requiring only one LLM step.

Our result shows that our answer-first approach can generate more complex (long-horizon) tool-use data with lower cost. LLMs trained on our generated data also outperform those trained on baseline methods, and even match SoTA proprietary LLMs on out-of-distribution (OOD) datasets with unseen tools.

While prior art generates tool-use datasets by searching solutions of user queries with low pass rate, ToolGrad generates successful tool-use chains before generating prompts, yielding high pass rate.

What comes next

Standard machine learning (ML) systems improve by computing numerical loss gradients across mini-batches of training samples, which are then used by an optimization algorithm to update model weights. Recently, TextGrad adapted this paradigm for prompt engineering using an LLM critic to provide rich, descriptive feedback in plain text — feedback called “textual gradients”.

These textual gradients then guide the refinements of a given prompt into a new draft that can better resolve the target task. ToolGrad adapts the concept of textual gradients from prompt optimization to synthetic dataset generation.

Rather than optimizing a static text prompt, ToolGrad uses these gradients to iteratively construct complex, valid API workflows from large tool libraries.

Key signals

• AI agents have shown great potential in automating real-world tasks, such as conducting a Google Search, reading local computer files, or executing generated Python scripts.
• Our result shows that our answer-first approach can generate more complex (long-horizon) tool-use data with lower cost.
• ToolGrad generates successful tool-use chains before generating prompts, yielding a high pass rate.

What to watch

Watch for follow-on benchmarks, developer adoption, pricing changes, and reliability feedback.