Researchers have developed a new AI technique that helps computers better understand the 3D shape and depth of objects in images and videos—without needing massive amounts of labeled training data. This advance could improve technologies like autonomous driving, robotics, and augmented reality, where accurately estimating the geometry of the environment is crucial. The study, recently published on arXiv, introduces a method that repurposes advanced video-generating AI models to predict depth and surface details from single images more efficiently and effectively than previous approaches.
Key Takeaways
- The new method, called GeoNeXt, uses pretrained video generative models to estimate depth and surface normals (the orientation of surfaces) from images.
- GeoNeXt formulates geometry estimation as a “next-frame prediction” task, leveraging the natural temporal structure of video models.
- It achieves strong zero-shot performance—meaning it can estimate geometry on new images without additional training—outperforming earlier generative methods.
- The approach requires substantially less labeled data, rivaling state-of-the-art methods that are trained on over 100 times more data.
Traditional AI methods for estimating 3D geometry from images often rely on large amounts of labeled data, such as images paired with depth maps or surface orientation information. Some recent attempts have adapted image-based generative models (which create images from noise) to predict depth or surface normals, but these either treat each geometry type separately or require extensive fine-tuning with labeled data. This can limit how well the models learn the natural relationships between different geometric features and makes scaling difficult.
In this new work, the researchers take a different approach by using video generative models, which are AI systems originally designed to predict future frames in a video sequence. These models inherently understand how objects and scenes evolve over time, capturing rich spatial and temporal information. GeoNeXt leverages this capability by framing geometry estimation as a next-frame prediction problem: instead of generating a future video frame, the model predicts geometric information related to the current image.
By integrating images and their geometric targets into a unified modeling framework, GeoNeXt benefits from the pretrained video model’s structured knowledge and richer priors. The model learns to jointly represent images and geometry, capturing intrinsic correlations between depth and surface normals more naturally. This joint learning framework allows the model to generalize better and be more data efficient—performing well even with limited labeled training examples.
Extensive experiments on diverse datasets demonstrate that GeoNeXt outperforms previous generative approaches tailored specifically for geometry tasks. Remarkably, it also competes with discriminative models that rely heavily on large-scale supervised training, despite using much less data. This shows the promise of video generative models as a powerful backbone for 3D scene understanding tasks.
Looking ahead, this research suggests a new direction for building AI systems that understand the geometry of the world more effectively and with fewer data requirements. Potential applications include improving the perception systems of self-driving cars, enabling more realistic augmented or virtual reality experiences, and aiding robots in navigating complex environments. Future work may explore extending this framework to other geometric attributes or integrating it with real-time systems, further bridging the gap between generative video AI and practical 3D understanding.
Based on research published on arXiv by Haosen Yang, Jifei Song, Zhensong Zhang et al..