arXiv:2510.134541 PaperLens breakdown
Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator
VIST3A introduces a novel framework for high-quality text-to-3D generation by intelligently combining powerful, pre-trained text-to-video models with advanced feedforward 3D reconstruction systems. It achieves this through two key innovations: model stitching to seamlessly integrate the networks and direct reward finetuning to align the generative process with 3D consistency and visual quality. This approach significantly outperforms previous methods, producing superior 3D Gaussian splats and pointmaps from text prompts.
VIST3A combines text-to-video generators and 3D reconstruction models for text-to-3D.
Model stitching integrates pre-trained networks by finding compatible latent spaces and adding a linear layer.
Direct reward finetuning aligns the video generator with the 3D decoder for consistent, high-quality outputs.
The method avoids slow per-scene optimization and rebuilding 3D capabilities from scratch.
VIST3A generates superior 3D Gaussian splats and high-quality pointmaps compared to prior art.
Reward signals are based on multi-view image quality, 3D representation quality, and 3D consistency.
VIST3A Framework
VIST3A is a modular, efficient, and effective framework for text-to-3D that reuses and aligns existing powerful AI components.
Model Stitching
Model stitching efficiently integrates pre-trained models by finding optimal linear connections between their latent representations.
Direct Reward Finetuning
Direct reward finetuning directly optimizes a generative model to produce outputs that meet specific quality and consistency criteria by using a comprehensive reward signal.
Latent Text-to-Video Model (Generator)
The latent text-to-video model is the text-conditioned visual content creator, producing compressed video representations.
VIST3A's ability to generate high-quality, geometrically consistent 3D content directly from text has transformative potential. It significantly lowers the barrier to creating complex 3D assets, accelerating development in fields like augmented/virtual reality, video game design, architectural visualization, and robotics simulation. By making 3D content creation more accessible and efficient, it can democratize access to advanced 3D modeling for a wider range of creators and industries.
Rapid prototyping of 3D assets for video games and virtual environments.Generating realistic 3D scenes for AR/VR applications from simple text descriptions.Creating diverse 3D training data for robotics and autonomous driving simulations.Personalized 3D content creation for e-commerce and digital marketing.Architectural and interior design visualization by generating 3D models from textual briefs.