Robots that can pick up, move, and use objects have long relied on specialist programs trained for specific tasks. But new research reveals that generalist artificial intelligence (AI) models—designed to handle a wide range of tasks—can actually control robot arms more reliably than these traditional specialist systems, even without any prior training on the tasks at hand. This breakthrough could simplify how robots learn and adapt in dynamic environments, opening doors for more versatile and robust automation.
Key Takeaways
- Generalist AI models can command robot manipulators directly, adapting actions on the fly without task-specific training or demonstrations.
- A reference system called AMP1 achieved an 85.6% success rate on a challenging benchmark, far outperforming specialist vision-language-action (VLA) and world-action models (WAMs).
- In real-world trials across multiple tasks and robot types, the generalist approach reduced failure rates by up to 20 times compared to specialist baselines.
- Adding a memory component that lets the AI learn from experience further decreased failure rates, showing potential for continual improvement.
Traditionally, robot manipulation systems have been built as specialists—programs trained with demonstrations to perform particular tasks like grasping or pushing objects. These approaches often require extensive data collection and fine-tuning for each new task or robot setup, limiting flexibility. The new study explores “agentic manipulation policies” (AMPs), where a generalist AI model takes direct control of the robot’s manipulator, making decisions in real time as the task unfolds.
To test this, the researchers developed AMP1, a unified software framework that works across different robot configurations—including single-arm, two-arm, and mobile manipulators—without changing the underlying code. Remarkably, AMP1 does not come pre-programmed with specific skills like grasping or placing objects. Instead, the AI model learns to develop its own strategies, such as searching for hidden objects, using tools, and calibrating its sensors, purely through interaction.
The team evaluated AMP1 on two established benchmarks: RoboLab-120, which includes a variety of manipulation challenges, and CaP-Gym, designed for bimanual robot tasks. AMP1 achieved state-of-the-art results, succeeding in 85.6% of tasks on RoboLab-120 without any prior demonstrations or task-specific training—a substantial improvement over specialist models that scored below 50%. In real-world experiments involving 108 trials across twelve different tasks and three robot instances, AMP1 maintained high success rates and dramatically reduced failure rates compared to leading specialist approaches.
Beyond zero-shot performance, the researchers enhanced AMP1 with a persistent memory system that allows the AI to carry knowledge across multiple task episodes. This memory-augmented version of AMP1 further improved performance, reducing failure rates by nearly fivefold on complex bimanual manipulation tasks. This suggests that generalist AI-driven robots can not only adapt instantly but also learn and improve from experience over time.
While the study focuses on benchmarking and controlled experiments, the findings have exciting implications for real-world robotics. Robots equipped with generalist AI controllers could handle a wider variety of tasks without needing task-specific programming or retraining, making them more flexible in manufacturing, logistics, home assistance, and beyond. The ability to learn from experience also points to more autonomous and resilient robotic systems that improve with use.
As robotics continues to evolve, this research highlights a promising direction: leveraging powerful, adaptable AI models to simplify and enhance robot manipulation. Future work will likely explore integrating these generalist policies into more complex environments and extending their learning capabilities further, bringing us closer to robots that can seamlessly assist with everyday tasks.
Based on research published on arXiv by Chen Chen, Omid Bagherieh, Naveen Nikamanth Appiah Balaji et al..