Artificial intelligence has made impressive strides in completing well-defined tasks, but delivering effective AI solutions in real-world customer environments remains a major challenge. A newly published research paper introduces FDE-Gym, a benchmark designed to test AI systems on the complex, ambiguous, and interactive nature of real customer delivery projects. This work sheds light on why AI models that excel in controlled settings often fall short when deployed to solve messy, real-life problems where requirements are unclear and constraints are unstated.
Key Takeaways
- FDE-Gym includes 95 realistic end-to-end delivery tasks drawn from 20 customer scenarios across 18 industries, reflecting true-to-life challenges faced by forward-deployed engineers (FDEs).
- The best AI model tested achieved an average score of 0.56 out of 1, successfully resolving just 14 out of 95 tasks with high-quality, incident-free solutions.
- Discovering unstated constraints—hidden rules or requirements not explicitly communicated—proved to be the biggest obstacle for AI agents.
- Even when AI models obtained all key information from stakeholders, their ability to deliver effective business improvements was limited, highlighting deeper challenges beyond information gathering.
Traditional AI benchmarks typically evaluate models on tasks with clear instructions and well-specified goals, such as solving puzzles or coding problems. However, real-world AI deployment often involves forward-deployed engineers (FDEs) working directly with customers to clarify ambiguous requirements, interact with live systems, and navigate unstated constraints that can dramatically affect outcomes.
To better capture these complexities, the researchers developed FDE-Gym, a simulated environment where AI agents engage in realistic customer delivery scenarios. These simulations include scripted stakeholders—virtual customer representatives who provide feedback and information—and modifiable customer systems that mimic production environments. The AI agents must communicate with stakeholders, perform trial runs, and iteratively improve solutions, much like human engineers do in practice.
Performance in FDE-Gym is measured by how much the AI improves the customer’s success metric, a normalized score between 0 (no improvement) and 1 (matching a reference solution). The scoring also penalizes excessive messaging, trial runs, and waiting time, encouraging efficiency and effectiveness. An online platform accompanying FDE-Gym allows human users to practice these delivery skills under the same evaluation criteria, bridging human and AI performance comparisons.
Testing 13 different AI configurations from seven providers revealed that current models struggle with the “last mile” of AI delivery. The primary bottleneck is the discovery of unstated constraints—requirements or limitations that customers don’t explicitly mention but are critical to success. When AI agents had to uncover these constraints without explicit guidance, their performance dropped sharply. Yet even when all key facts were obtained, AI systems often failed to translate that information into effective business results, indicating that information gathering alone isn’t enough.
This research highlights the substantial gap between solving neat, well-defined tasks and delivering robust AI-driven solutions in unpredictable, real-world settings. By focusing on end-to-end delivery and practical challenges faced by engineers, FDE-Gym offers a new way to benchmark progress and guide future AI development.
Looking ahead, improving AI’s ability to handle ambiguity, communicate effectively with stakeholders, and adapt to unstated requirements will be crucial for bringing AI out of the lab and into everyday customer environments. FDE-Gym provides a valuable testbed for researchers and practitioners to push the boundaries of what AI can achieve in the “last mile” of delivery, ultimately aiming for more reliable and impactful AI solutions across industries.
Based on research published on arXiv by Dongxu Yang, Hongda Yan, Renxin Wang et al..