Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communities

Abstract

Text-to-Image generation (TTI) technologies are advancing rapidly, especially in the English language communities. However, apart from the user input language barrier problem, English-native TTI models inherently carry biases from their English world centric training data, which creates a dilemma for development of other language-native TTI models. One common choice is to fine-tune the English-native TTI model with translated samples. It falls short of fully addressing the model bias problem. Alternatively, training non-English language native models from scratch can effectively resolve the English world bias, but model trained this way would diverge from the English TTI communities, thus not able to utilize the strides continuously gaining in the English TTI communities any more. To build Chinese TTI model meanwhile keep compatibility with the English TTI communities, we propose a novel model structure referred as "Bridge Diffusion Model" (BDM). The proposed BDM employs a backbone-branch network structure to learn the Chinese semantics while keep the latent space compatible with the English-native TTI backbone, in an end-to-end manner. The unique advantages of the proposed BDM are that it's not only adept at generating images that precisely depict Chinese semantics, but also compatible with various English-native TTI plugins, such as different checkpoints, LoRA, ControlNet, Dreambooth, and Textual Inversion, etc. Moreover, BDM can concurrently generate content seamlessly combining both Chinese-native and English-native semantics within a single image, fostering cultural interaction.

Accepted as Oral at AAAI 2025. 8 pages, 5 figures. Published in Proceedings of the 39th AAAI Conference on Artificial Intelligence. Code: https://github.com/360CVGroup/Bridge_Diffusion_Model

References in corpus (21)

• Learning Transferable Visual Models From Natural Language Supervision
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Score-Based Generative Modeling through Stochastic Differential Equations
• Generative Modeling by Estimating Gradients of the Data Distribution
• LAION-5B: An open large-scale dataset for training next generation image-text models
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
• Adding Conditional Control to Text-to-Image Diffusion Models
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models
• Stable Bias: Analyzing Societal Representations in Diffusion Models
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
• RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths
• ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation
• JPEG Artifact Correction using Denoising Diffusion Restoration Models
• Threat Model-Agnostic Adversarial Defense using Diffusion Models
• AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities
• HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation