Researchers have taken a significant step toward improving artificial intelligence (AI) agents that can use tools to complete complex, multi-step tasks. The new study focuses on how the composition of training data—specifically, synthetically generated environments—affects the performance of large language models (LLMs) designed to interact with tools. Understanding these relationships is crucial as AI systems become more capable and are expected to assist with increasingly sophisticated real-world problems.
Key Takeaways
- The study introduces SynthAPI, a pipeline that generates synthetic training environments by controlling three key aspects: the variety of tools, the complexity of tasks, and the databases those tools operate on.
- By training 849 models across four different sizes on nearly half a million tasks, researchers identified a scaling law that predicts how well AI agents generalize to new tasks based on the scale and makeup of their training data.
- Increasing the diversity of tools and databases in training data is more effective than simply adding more tasks, especially as the total data budget grows.
- Training with more complex data leads to bigger improvements in agent performance, helping AI systems outperform previous models on multiple benchmarks.
In AI research, “tool-use agents” are systems that can leverage external tools—like calculators, search engines, or databases—to extend their capabilities beyond just generating text. However, training these agents to handle complicated sequences of actions in varied environments is challenging. To tackle this, the team behind this study developed SynthAPI, a method that creates synthetic environments with precise control over three dimensions: the kinds of tools available, the tasks to be performed, and the data those tools access.
SynthAPI allows researchers to systematically vary these factors to see how they influence the learning process. For example, they can generate training data with many different tools but fewer tasks, or vice versa. By training hundreds of models on this controlled data, the researchers discovered a predictable scaling law—a kind of mathematical rule—that links the composition and size of training data to how well the agents perform on new, unseen tasks.
This scaling law helps clarify where to invest effort when building training datasets. The findings suggest that as training resources increase, it’s more beneficial to focus on expanding the variety of tools and the underlying databases rather than just increasing the number of tasks. Additionally, introducing more complex data scenarios leads to larger gains in the AI’s ability to generalize and solve problems effectively, compared to simpler training setups.
These insights could guide future efforts in developing AI agents that assist with intricate, multi-step workflows—such as data analysis, software development, or even scientific research—where using multiple tools seamlessly is essential. By better understanding how to structure training data, developers can build more capable and reliable AI helpers without relying solely on massive amounts of indiscriminate data.
Looking ahead, this research opens the door to more efficient and targeted training strategies for tool-using AI systems. The scaling laws derived from SynthAPI’s controlled experiments provide a valuable framework for optimizing data generation processes. While the study focuses on synthetic environments, the principles may eventually help improve AI performance in real-world applications where diverse tools and complex tasks are the norm.
This work was detailed in a newly published research paper by Andrew Szot, Samira Abnar, Omar Attia, and colleagues, advancing our understanding of how to train AI agents that effectively combine language understanding with practical tool use.
Based on research published on arXiv by Andrew Szot, Samira Abnar, Omar Attia et al..