The paper introduces Interactive3D, a framework for user-envisioning 3D object generation that allows for precise user control through interactive operations. The framework consists of two stages with distinct 3D representations: Gaussian Splatting in the first stage and InstantNGP-based refinement in the second stage. In the first stage, users can interact with the 3D model through operations like add/remove parts, geometry transformation, deformable or rigid dragging, and semantic editing. The Gaussian blobs are converted to InstantNGP using NeRF distillation in the second stage, followed by an Interactive Hash Refinement Module for further detailed enhancements.

In the first stage of Interactive3D, Gaussian Splatting is used for flexible user interactions, enabling operations such as deformable dragging, stretching, rigid dragging, and local semantic editing. These interactions allow users to make specific modifications to the 3D model, such as adding wings, changing sizes, repositioning elements, and applying visual effects like flames. The second stage involves converting the Gaussian representation into an InstantNGP structure, followed by an Interactive Hash Refinement process that refines the 3D model based on user interactions and feedback.

The implementation details of Interactive3D involve training the framework on one NVIDIA A100 GPU for 20k steps, utilizing the AdamW optimizer with specific parameters, and incorporating Stable Diffusion 2.1 for guidance during training. The framework is designed to be efficient, with the interactive operations integrated into the optimization process without the need for extra training steps. The framework's architecture allows for real-time user interactions and precise control over 3D object generation, making it a versatile tool for creating customized 3D models based on user input.