Make-it-Real provides a novel approach that leverages Multimodal Large Language Models (MLLMs) to assign realistic materials to 3D objects. The method involves three main steps: creating a detailed material library using GPT-4V, segmenting 3D meshes into distinct components, and matching suitable materials to each part. The material library contains over 1,300 unique materials with detailed descriptions generated by GPT-4V. Multi-view image segmentation is used to accurately identify and segment different material regions on 3D meshes based on diffuse RGB information. MLLM-based material matching is then employed to assign appropriate materials to these segmented regions using visual and hierarchical text prompts.

The pipeline of Make-it-Real includes Multi-View Image Segmentation, MLLM-based Material Matching, and SVBRDF Maps Generation. The material library is meticulously generated and cataloged with comprehensive descriptions for each material, enabling hierarchical querying for material allocation. The method utilizes GPT-4V for material recognition and assignment, enhancing the realism and visual quality of 3D assets. By integrating a fine-grained, annotated PBR material library with detailed textual annotations, Make-it-Real provides rich and precise material descriptions for 3D meshes, improving material assignment processes significantly.

Make-it-Real employs innovative segmentation strategies based on multi-view 2D image rendering and Semantic-SAM model for precise segmentation of rendered images. The material matching process utilizes a material retrieval strategy powered by MLLMs, combining visual cues and hierarchical text prompts with GPT-4V for accurate material assignment. The method focuses on enhancing the realism and depth effects of 3D objects by applying realistic materials automatically, streamlining the 3D asset creation process for developers.