Researchers have taken a closer look at how the latest multimodal large language models (MLLMs), like GPT-6-Astra, can control robots by interpreting visual and sensory information without needing robot-specific training. This new study explores how well these advanced AI agents perform alone and when paired with specialized control policies designed for robot tasks. Understanding these capabilities is crucial as AI-driven robots become more common in manufacturing, healthcare, and everyday environments.
Key Takeaways
- GPT-6-Astra, a cutting-edge AI language model, can directly control robots and performs competitively on short, simple tasks compared to specialized, post-trained robot control policies.
- For complex, longer tasks or precise manipulations, GPT-6-Astra’s direct control lags behind specialized policies trained specifically for those tasks.
- Combining GPT-6-Astra with post-trained policies through an “agent harness” improves overall success rates and reduces computational costs, leveraging strengths of both approaches.
- Current integration methods have limitations, such as fixed instructions and difficulties resuming control mid-task, indicating room for future improvements in AI-robot collaboration.
The study centers on EBench, a benchmarking environment that evaluates how well AI agents manage robotic manipulation tasks based on visual and proprioceptive inputs—the robot’s “eyes” and “sense of position.” GPT-6-Astra was tested in two main setups. In the first, called direct control, the AI generated waypoints—specific positions or movements—for the robot to follow without any additional guidance. In the second, known as harness with policy, the AI could call upon specialized, post-trained policies that have been fine-tuned for particular tasks to execute actions.
These post-trained policies fall into two categories: vision-language-action (VLA) policies, which integrate visual data and language instructions to guide robot movements, and world action model (WAM) policies, which focus on understanding the physical environment and planning accordingly. By comparing GPT-6-Astra alone and combined with these policies, the researchers assessed strengths, weaknesses, and potential synergies.
Results showed that GPT-6-Astra’s direct control is robust for short-horizon tasks—those requiring fewer steps or simpler manipulations—and it maintains consistent performance across different generalization challenges. However, it struggles with longer, more complex sequences, delicate tabletop manipulations, and tasks requiring precise contact, where the specialized policies excel. When GPT-6-Astra was paired with the post-trained policies, the combined system achieved task success rates comparable to the best standalone policies while using fewer computational resources, thanks to more efficient “token” usage (the units of information processed by language models).
Despite these advances, the integration isn’t perfect. The current system uses fixed instructions without breaking down tasks into smaller subgoals, which can limit flexibility. Additionally, switching control back to the AI mid-task sometimes disrupts robot actions, such as grasping objects. These challenges highlight opportunities for future research to develop smarter ways for AI agents and specialized policies to collaborate seamlessly.
By profiling the capabilities and limitations of GPT-6-Astra in robotic control, this research offers valuable insights for designing next-generation AI agents that better harness the complementary strengths of large language models and dedicated robot policies. As AI-driven robots become more prevalent, improvements in these hybrid systems could lead to more adaptable, efficient, and reliable automation in real-world settings—from factories to homes. The project page with detailed results and code is available for those interested in exploring this promising frontier further.
Based on research published on arXiv by Yifei Yao, Yude Zou, Guanbo Jiao et al..