Researchers have unveiled a pioneering method for sketch-guided image inpainting, aimed at seamlessly integrating hand-drawn sketches into the object completion process, yielding realistic and contextually appropriate results.

At the heart of this approach lies a partial discrete diffusion process (PDDP), which enables the incorporation of visual information from sketches into the inpainting model. Leveraging a bidirectional transformer architecture, the model effectively bridges the domain gap between sketches and natural images, enhancing the quality of inpainting outcomes.

The architecture comprises a sketch encoder and a diffusion decoder, with the former extracting features from input sketches and incorporating positional information. The diffusion decoder, functioning as a bidirectional encoder-only transformer, estimates the distribution of the inpainted region based on the input sketch and the latent representation of the image. Through a conditioning mechanism, the model predicts the inpainted region by concatenating sketch embeddings with the latent image representation and passing them through a series of transformer blocks.