Zero-Shot Text-to-Image Generation

Abstract

Text-to-image generation has traditionally focused on finding better modeling assumptions for training on a fixed dataset. These assumptions might involve complex architectures, auxiliary losses, or side information such as object part labels or segmentation masks supplied during training. We describe a simple approach for this task based on a transformer that autoregressively models the text and image tokens as a single stream of data. With sufficient data and scale, our approach is competitive with previous domain-specific models when evaluated in a zero-shot fashion.

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Learning Transferable Visual Models From Natural Language Supervision
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Categorical Reparameterization with Gumbel-Softmax
• Neural Discrete Representation Learning
• Generative Adversarial Text to Image Synthesis
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• YFCC100M: The New Data in Multimedia Research
• Mixed Precision Training
• Generating Long Sequences with Sparse Transformers
• Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Jukebox: A Generative Model for Music
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Generating Images from Captions with Attention
• On the Binding Problem in Artificial Neural Networks
• Understanding the Difficulty of Training Transformers
• X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

Cited by in corpus (173)

• A Survey on Visual Transformer
• On the Opportunities and Risks of Foundation Models
• Diffusion Models Beat GANs on Image Synthesis
• Evaluating Large Language Models Trained on Code
• BEiT: BERT Pre-Training of Image Transformers
• Alias-Free Generative Adversarial Networks
• Blended Diffusion for Text-driven Editing of Natural Images
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• CogView: Mastering Text-to-Image Generation via Transformers
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Unleashing the potential of prompt engineering for large language models
• Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• Adversarial Text-to-Image Synthesis: A Review
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• A Taxonomy of Prompt Modifiers for Text-To-Image Generation
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Artificial-intelligence-based molecular classification of diffuse gliomas using rapid, label-free optical imaging
• Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging
• Video Transformers: A Survey
• Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art
• Multimodal datasets: misogyny, pornography, and malignant stereotypes
• SpaText: Spatio-Textual Representation for Controllable Image Generation
• VLT: Vision-Language Transformer and Query Generation for Referring Segmentation
• A Review of Machine Learning Methods Applied to Structural Dynamics and Vibroacoustic
• VideoGPT: Video Generation using VQ-VAE and Transformers
• Pix2seq: A Language Modeling Framework for Object Detection
• Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• DiffEdit: Diffusion-based semantic image editing with mask guidance
• CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding
• OperA: Attention-Regularized Transformers for Surgical Phase Recognition
• CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders
• Vector-quantized Image Modeling with Improved VQGAN
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
• CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
• AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia
• ViTGAN: Training GANs with Vision Transformers
• Copyright in Generative Deep Learning
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• Benchmarking Detection Transfer Learning with Vision Transformers
• TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Long-Short Transformer: Efficient Transformers for Language and Vision
• Multiscale Vision Transformers
• Txt2Img-MHN: Remote Sensing Image Generation from Text Using Modern Hopfield Networks
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• M6: A Chinese Multimodal Pretrainer
• Transformer-based Map Matching Model with Limited Ground-Truth Data using Transfer-Learning Approach
• AI Art is Theft: Labour, Extraction, and Exploitation, Or, On the Dangers of Stochastic Pollocks
• A Survey of Visual Transformers
• SGDiff: A Style Guided Diffusion Model for Fashion Synthesis
• ClipFace: Text-guided Editing of Textured 3D Morphable Models
• Computer-Aided Design as Language
• Vector Quantized Diffusion Model for Text-to-Image Synthesis
• LAFITE: Towards Language-Free Training for Text-to-Image Generation
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesis
• VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligence
• NormFormer: Improved Transformer Pretraining with Extra Normalization
• Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
• Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models
• ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Design
• Assaying Out-Of-Distribution Generalization in Transfer Learning
• Improving Text-to-Image Synthesis Using Contrastive Learning
• Causal Reinforcement Learning using Observational and Interventional Data
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Towards Open-World Text-Guided Face Image Generation and Manipulation
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• Towards Efficient Post-training Quantization of Pre-trained Language Models
• Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
• OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation
• Continual World: A Robotic Benchmark For Continual Reinforcement Learning
• Bayesian Renormalization
• On the Utility of Gradient Compression in Distributed Training Systems
• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
• Zero-Shot Text-Guided Object Generation with Dream Fields
• Adma-GAN: Attribute-Driven Memory Augmented GANs for Text-to-Image Generation
• SimMIM: A Simple Framework for Masked Image Modeling
• Foundation Models and Transformers for Anomaly Detection: A Survey
• Pre-Trained Models: Past, Present and Future
• Does CLIP Know My Face?
• M6-T: Exploring Sparse Expert Models and Beyond
• Deception for Cyber Defence: Challenges and Opportunities
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• Interactive Interior Design Recommendation via Coarse-to-fine Multimodal Reinforcement Learning
• Differentiable Physics: A Position Piece
• Discrete Representations Strengthen Vision Transformer Robustness
• End-to-End Trainable Multi-Instance Pose Estimation with Transformers
• AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation
• Prompting Video-Language Foundation Models with Domain-specific Fine-grained Heuristics for Video Question Answering
• 3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers
• Translation-Enhanced Multilingual Text-to-Image Generation
• Interactive Neural Painting
• Enhance Image-to-Image Generation with LLaVA-generated Prompts
• Study of positional encoding approaches for Audio Spectrogram Transformers
• Active Divergence with Generative Deep Learning -- A Survey and Taxonomy
• Evaluating the Fairness of Discriminative Foundation Models in Computer Vision
• AI-generated art perceptions with GenFrame -- an image-generating picture frame
• Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• Generative Models as a Data Source for Multiview Representation Learning
• Paint4Poem: A Dataset for Artistic Visualization of Classical Chinese Poems
• An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wild
• GAN Cocktail: mixing GANs without dataset access
• Self-supervised Video Object Segmentation by Motion Grouping
• Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Don't waste SAM
• Automating Generative Deep Learning for Artistic Purposes: Challenges and Opportunities
• Text2Poster: Laying out Stylized Texts on Retrieved Images
• Image2Lego: Customized LEGO Set Generation from Images
• Visual Conceptual Blending with Large-scale Language and Vision Models
• 8-bit Optimizers via Block-wise Quantization
• Cycle-Consistent Inverse GAN for Text-to-Image Synthesis
• Telling Creative Stories Using Generative Visual Aids
• SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches
• Improved Transformer for High-Resolution GANs
• Unfinished Architectures: A Perspective from Artificial Intelligence
• High-Resolution Complex Scene Synthesis with Transformers
• A Mathematical Framework for Learning Probability Distributions
• Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model
• SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentation
• Conditional Generation of Periodic Signals with Fourier-Based Decoder
• Level generation and style enhancement -- deep learning for game development overview
• NWT: Towards natural audio-to-video generation with representation learning
• From Pampas to Pixels: Fine-Tuning Diffusion Models for Gaúcho Heritage
• MuG: A Multimodal Classification Benchmark on Game Data with Tabular, Textual, and Visual Fields
• Intelligent Anomaly Detection for Lane Rendering Using Transformer with Self-Supervised Pre-Training and Customized Fine-Tuning
• Natlog: Embedding Logic Programming into the Python Deep-Learning Ecosystem
• Scaling Law for Recommendation Models: Towards General-purpose User Representations
• FooDI-ML: a large multi-language dataset of food, drinks and groceries images and descriptions
• Universal Performance Gap of Neural Quantum States Applied to the Hofstadter-Bose-Hubbard Model
• Learning Signal-Agnostic Manifolds of Neural Fields
• Disentanglement of Latent Representations via Causal Interventions
• StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis
• A Systematic Review of Open Datasets Used in Text-to-Image (T2I) Gen AI Model Safety
• Addressing Weak Decision Boundaries in Image Classification by Leveraging Web Search and Generative Models
• The Image Local Autoregressive Transformer
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Semantic Text-to-Face GAN -ST^2FG
• Towards Better Adversarial Synthesis of Human Images from Text
• eProduct: A Million-Scale Visual Search Benchmark to Address Product Recognition Challenges
• Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning
• Realistic Image Synthesis with Configurable 3D Scene Layouts
• CARMS: Categorical-Antithetic-REINFORCE Multi-Sample Gradient Estimator
• Training Efficiency and Robustness in Deep Learning
• Ownership and Creativity in Generative Models
• Compositional generalization in semantic parsing with pretrained transformers
• An implementation of the "Guess who?" game using CLIP
• On the Limitations of Multimodal VAEs
• CIGLI: Conditional Image Generation from Language & Image
• Mutltimodal AI Companion for Interactive Fairytale Co-creation
• Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation
• Tensorization of neural networks for improved privacy and interpretability
• MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning
• Variational latent discrete representation for time series modelling
• EdiBERT, a generative model for image editing
• Exploration into Translation-Equivariant Image Quantization
• Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers
• ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation
• Generative Adversarial Learning via Kernel Density Discrimination
• Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization
• PatchGame: Learning to Signal Mid-level Patches in Referential Games