Imagine being able to create and tweak videos of moving objects—like billiard balls bouncing or dominoes falling—in real time, with precise control over how each object moves based on real-world physics. That’s the promise of new research from a team including Chuhao Chen and Peter Wonka, who have developed PhysStream, an AI system that generates videos grounded in physical laws and lets users interactively guide the motion of objects as the video is being created. This advance matters because it moves beyond earlier AI video tools that offered only broad or static control, enabling much finer, physics-based manipulation of dynamic scenes.
Key Takeaways
- PhysStream generates videos of multiple moving objects by modeling their physical dynamics, rather than just their positions.
- The system uses “structured scene memory”—maps tracking object positions and movements from previous frames—to maintain consistency over time.
- Users can control the video mid-generation by providing sparse signals that represent changes in object velocity, allowing intuitive, physics-based adjustments.
- On benchmark tests, PhysStream reduced motion errors by up to 33% compared to state-of-the-art methods and was preferred by human judges in over 85% of real-world comparisons.
At its core, PhysStream is an autoregressive model, meaning it generates video frames one after another, with each new frame depending on the ones before it. What sets it apart is its use of structured scene memory, which keeps track of where objects are and how they have moved in earlier frames. This memory includes positional maps (showing object locations) and object tracking maps (monitoring identities and paths), updated continuously as the video progresses. This approach helps the AI maintain physical consistency—objects behave as expected over time, avoiding sudden or unrealistic jumps.
Another innovative feature is PhysStream’s fine-grained motion control. Instead of simply telling the AI where to place objects, users provide sparse “velocity-increment signals” that represent changes in objects’ speeds and directions—physical quantities that describe motion. This lets the AI learn the underlying physics governing the scene, resulting in more natural and controllable movement. The model is trained in two stages: first, a bidirectional version is fine-tuned to understand motion control, and then a causal autoregressive model is trained with the structured memory to improve the physical accuracy of generated videos.
By enabling interactive, mid-generation control over scenes with multiple rigid objects—like those on a tabletop—PhysStream opens the door to a range of applications. For instance, game designers could create realistic animations that respond to player input in real time, educators might generate physics demonstrations tailored to specific lessons, and filmmakers could prototype complex motion sequences more efficiently. While the system currently focuses on synthetic tabletop scenes, future work could extend these principles to more complex environments and object types. As AI-generated video continues to evolve, PhysStream represents a significant step toward more controllable and physically grounded content creation.
Based on research published on arXiv by Chuhao Chen, Peter Wonka, Chaoyang Wang et al..