Content-Based Search for Deep Generative Models

Abstract

The growing proliferation of customized and pretrained generative models has made it infeasible for a user to be fully cognizant of every model in existence. To address this need, we introduce the task of content-based model search: given a query and a large set of generative models, finding the models that best match the query. As each generative model produces a distribution of images, we formulate the search task as an optimization problem to select the model with the highest probability of generating similar content as the query. We introduce a formulation to approximate this probability given the query from different modalities, e.g., image, sketch, and text. Furthermore, we propose a contrastive learning framework for model retrieval, which learns to adapt features for various query modalities. We demonstrate that our method outperforms several baselines on Generative Model Zoo, a new benchmark we create for the model retrieval task.

arXiv commentOur project page is hosted at https://generative-intelligence-lab.github.io/modelverse/

References in corpus (5)

• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Blended Diffusion for Text-driven Editing of Natural Images
• Barbershop: GAN-based Image Compositing using Segmentation Masks
• Imagic: Text-Based Real Image Editing with Diffusion Models
• Multi-Concept Customization of Text-to-Image Diffusion

Cited by in corpus (1)

• Advances in Artificial Intelligence: A Review for the Creative Industries