As governments worldwide open up their data through public APIs (Application Programming Interfaces), artificial intelligence (AI) systems face the challenge of effectively using these tools to complete complex, multi-step tasks. A newly published research paper tackles this problem by introducing a benchmark focused on Korean open public APIs and a novel method to improve AI performance in this setting. This work is important because it addresses how AI can better interact with live government services while respecting data sovereignty rules, which require sensitive data to stay within local infrastructure.
Key Takeaways
- The researchers created KOPA-Bench, a benchmark consisting of 145 real-world tasks that require chaining multiple calls to Korean government APIs.
- They developed EDGE, a data synthesis method that builds a dynamic graph of API tool interactions verified through live execution, enabling the generation of multi-step task examples.
- Fine-tuning a 9-billion-parameter language model on EDGE-generated data nearly matched the performance of a much larger 27-billion-parameter model without fine-tuning.
- The approach improved AI performance not only on the new KOPA-Bench but also on an existing multi-step tool-calling benchmark called BFCL.
Public APIs allow software programs to communicate with government services to retrieve data or perform actions like checking eligibility for benefits, verifying documents, or submitting forms. However, many tasks require multiple API calls in sequence, where the output of one call becomes the input for the next. This “multi-step tool-calling” is difficult for current AI models, especially open-source ones that run locally to comply with strict data sovereignty laws demanding that sensitive data not leave national borders.
To better understand and improve AI capabilities in this scenario, the researchers introduced KOPA-Bench. This benchmark reflects realistic tasks involving Korean government APIs, providing a standardized way to measure how well AI systems can chain multiple API calls. To generate training data for this challenging task, they designed EDGE (Execution-grounded Dynamic Graph for tool-calling data synthEsis). EDGE constructs a graph mapping how outputs from one API can serve as valid inputs to another, but crucially, it verifies these connections by actually executing the API calls live. This ensures the synthesized multi-step sequences are executable and realistic rather than hypothetical.
Using EDGE, the team synthesized a large dataset of multi-step API call sequences. They then fine-tuned a 9-billion-parameter open-source language model with a reinforcement learning technique called GRPO (Guided Reinforcement Policy Optimization) on this data. The result was striking: the fine-tuned smaller model performed almost as well as an untuned 27-billion-parameter model from the same family, demonstrating that targeted fine-tuning with realistic multi-step data can significantly boost performance without requiring massive model sizes.
This advance has practical implications for building AI agents that interact with government services in countries with strict data sovereignty laws. By enabling smaller, open-source models to handle complex multi-step API tasks effectively, institutions can deploy AI tools on-premises without relying on large cloud-based models. Looking ahead, this research opens the door to expanding similar benchmarks and data synthesis approaches to other languages and government ecosystems, further supporting the development of responsible, locally compliant AI services.
Based on research published on arXiv by Dain Kim, Eungi Cho, Kyumin Kim et al..