Creating 3D worlds—from video game environments to virtual reality scenes—has traditionally required skilled designers and programmers working with complex tools. A new research project called 3DCodeVerse aims to change that by enabling AI agents to write and verify 3D content as executable code, making the process more scalable, editable, and reliable. This advancement could pave the way for faster, more accessible 3D content creation, impacting industries like gaming, simulation, and digital design.
Key Takeaways
- 3DCodeVerse introduces a large, high-quality dataset of 3D executable code, called 3DCodeVerse-2M, containing millions of standalone 3D programs with all necessary assets and verification data.
- The project includes a unique “harness” — a multi-agent system that automatically checks, measures, and visually evaluates 3D code to refine and verify content without human intervention during generation.
- Fine-tuning existing open-source AI models on the 3DCodeVerse dataset significantly boosts their ability to generate accurate and functional 3D content.
- In human evaluations, the 3DCodeVerse harness outperformed a leading AI design system, achieving a 90% win rate in generating 3D objects when paired with advanced language models.
At the heart of 3DCodeVerse is the idea of treating 3D scenes and objects as code that can be executed, tested, and modified—similar to how software developers write and debug programs. To support this, the researchers built 3DCodeVerse-2M, a massive collection of 3D “programs” gathered from open-source projects, specialized formats converted into code, and newly generated examples produced by AI agents themselves. Each program comes with everything needed to run it in its intended environment, including source files, assets (like textures or models), descriptions, and records that verify its correctness.
To make sure the generated 3D content is not only syntactically correct but also spatially meaningful and visually coherent, the team developed the 3DCodeVerse Harness. This system uses multiple AI agents organized in a directed acyclic graph—a structure that manages dependencies and workflows—to automatically generate, check, and improve 3D code. The harness performs “execution checks” to ensure the code runs properly, “spatial measurements” to verify geometric relationships, and uses visual feedback to spot and fix errors. Importantly, this feedback loop happens without human input during the generation process, allowing for scalable and autonomous 3D content creation.
Through extensive experiments, the researchers found that training AI models on the 3DCodeVerse dataset greatly enhances their 3D coding skills. They also benchmarked their system against Claude Design, a known AI tool for 3D object generation, finding that their harness combined with the Claude Opus 5.5 language model won 90% of the time in side-by-side human evaluations. This suggests that 3DCodeVerse not only pushes the technical boundaries but also delivers practical improvements in quality.
Looking ahead, 3DCodeVerse offers a promising framework for automating parts of 3D content creation, which could reduce costs and lower the barrier to entry for designers and developers. By enabling AI to generate editable and verifiable 3D code at scale, this research opens the door to more dynamic and customizable virtual environments. Future work may explore integrating this approach into commercial design tools or expanding it to more complex and interactive 3D worlds, potentially transforming how digital spaces are built and experienced.
Based on research published on arXiv by Yipeng Gao, Ziyao Zeng, Jinfa Huang et al..