DRAW: A Recurrent Neural Network For Image Generation

Abstract

This paper introduces the Deep Recurrent Attentive Writer (DRAW) neural network architecture for image generation. DRAW networks combine a novel spatial attention mechanism that mimics the foveation of the human eye, with a sequential variational auto-encoding framework that allows for the iterative construction of complex images. The system substantially improves on the state of the art for generative models on MNIST, and, when trained on the Street View House Numbers dataset, it generates images that cannot be distinguished from real data with the naked eye.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Multiple Object Recognition with Visual Attention
• Attention for Fine-Grained Categorization
• Deep AutoRegressive Networks
• A Deep and Tractable Density Estimator
• On Learning Where To Look

Cited by in corpus (337)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Teaching Machines to Read and Comprehend
• Attention Mechanisms in Computer Vision: A Survey
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Unsupervised Learning of Video Representations using LSTMs
• Tutorial on Variational Autoencoders
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• End-To-End Memory Networks
• Adversarially Learned Inference
• Revisiting Distributed Synchronous SGD
• Recent Advances in Recurrent Neural Networks
• Deep Reinforcement Learning: An Overview
• Layer Normalization
• Image and Video Compression with Neural Networks: A Review
• Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Adaptive Computation Time for Recurrent Neural Networks
• Count-Based Exploration with Neural Density Models
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Residual Attention Network for Image Classification
• Inversion using a new low-dimensional representation of complex binary geological media based on a deep neural network
• Ladder Variational Autoencoders
• Image Captioning with Semantic Attention
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Multi-Object Representation Learning with Iterative Variational Inference
• RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism
• Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
• Composing graphical models with neural networks for structured representations and fast inference
• Learning What and Where to Draw
• Adversarial Text-to-Image Synthesis: A Review
• Emergence of grid-like representations by training recurrent neural networks to perform spatial localization
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
• AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
• An Introduction to Image Synthesis with Generative Adversarial Nets
• Semantic Jitter: Dense Supervision for Visual Comparisons via Synthetic Images
• Conditional Computation in Neural Networks for faster models
• Variational inference for Monte Carlo objectives
• Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities
• Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
• Blocks and Fuel: Frameworks for deep learning
• Learning Contextual Dependencies with Convolutional Hierarchical Recurrent Neural Networks
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• Land Cover Classification from Multi-temporal, Multi-spectral Remotely Sensed Imagery using Patch-Based Recurrent Neural Networks
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Variable Rate Image Compression with Recurrent Neural Networks
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• TAC-GAN - Text Conditioned Auxiliary Classifier Generative Adversarial Network
• Learning to Decompose and Disentangle Representations for Video Prediction
• Age Progression/Regression by Conditional Adversarial Autoencoder
• Deep Variational Canonical Correlation Analysis
• Scene-LSTM: A Model for Human Trajectory Prediction
• Differential Recurrent Neural Networks for Action Recognition
• Super Mario as a String: Platformer Level Generation Via LSTMs
• Unsupervised Learning of 3D Structure from Images
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• Attribute2Image: Conditional Image Generation from Visual Attributes
• Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
• GP-GAN: Towards Realistic High-Resolution Image Blending
• Deep Successor Reinforcement Learning
• Super-Resolution with Deep Convolutional Sufficient Statistics
• Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
• DenseCap: Fully Convolutional Localization Networks for Dense Captioning
• Deep Divergence-Based Approach to Clustering
• Convergence guarantees for RMSProp and ADAM in non-convex optimization and an empirical comparison to Nesterov acceleration
• Photographic Image Synthesis with Cascaded Refinement Networks
• ChatPainter: Improving Text to Image Generation using Dialogue
• Hierarchical Attention Network for Action Recognition in Videos
• Attentive Recurrent Comparators
• LFADS - Latent Factor Analysis via Dynamical Systems
• Generating Images from Captions with Attention
• Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs
• Back to the Basics: Bayesian extensions of IRT outperform neural networks for proficiency estimation
• Selective Feature Connection Mechanism: Concatenating Multi-layer CNN Features with a Feature Selector
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Adversarial Images for Variational Autoencoders
• Visual Translation Embedding Network for Visual Relation Detection
• Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents
• Accurate and Lightweight Image Super-Resolution with Model-Guided Deep Unfolding Network
• Topic-Guided Variational Autoencoders for Text Generation
• ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs
• Look, Listen and Learn - A Multimodal LSTM for Speaker Identification
• Attentive Generative Adversarial Network for Raindrop Removal from a Single Image
• Attention Mesh: High-fidelity Face Mesh Prediction in Real-time
• Inverting face embeddings with convolutional neural networks
• Constrained Bayesian Optimization for Automatic Chemical Design
• First Step toward Model-Free, Anonymous Object Tracking with Recurrent Neural Networks
• Memory-Efficient Backpropagation Through Time
• Conditional Neural Processes
• Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition
• Seeing with Humans: Gaze-Assisted Neural Image Captioning
• Non-linear Functional Modeling using Neural Networks
• Convolutional Network for Attribute-driven and Identity-preserving Human Face Generation
• Discriminative Regularization for Generative Models
• Dual Attention Networks for Multimodal Reasoning and Matching
• Res3ATN -- Deep 3D Residual Attention Network for Hand Gesture Recognition in Videos
• Stacked Generative Adversarial Networks
• Visual7W: Grounded Question Answering in Images
• A Survey of Optimization Methods from a Machine Learning Perspective
• Attention-Aware Face Hallucination via Deep Reinforcement Learning
• Conditional Variational Autoencoder for Neural Machine Translation
• Neural Adaptive Sequential Monte Carlo
• Multi-View Image Generation from a Single-View
• Generating Text with Deep Reinforcement Learning
• Discrete Variational Autoencoders
• Generative Temporal Models with Memory
• Language as a Latent Variable: Discrete Generative Models for Sentence Compression
• Nonparametric Variational Auto-encoders for Hierarchical Representation Learning
• Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
• An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
• Recurrent Spatial Transformer Networks
• Saccader: Improving Accuracy of Hard Attention Models for Vision
• Deep Generative Models in the Real-World: An Open Challenge from Medical Imaging
• Simple and Effective VAE Training with Calibrated Decoders
• Temporal Generative Adversarial Nets with Singular Value Clipping
• Variational Neural Machine Translation
• Deep Recurrent Generative Decoder for Abstractive Text Summarization
• Efficient inference in occlusion-aware generative models of images
• Semi-Amortized Variational Autoencoders
• A Deep Learning Approach for Joint Video Frame and Reward Prediction in Atari Games
• Z-Forcing: Training Stochastic Recurrent Networks
• Efficient inverse design and spectrum prediction for nanophotonic devices based on deep recurrent neural networks
• Creativity: Generating Diverse Questions using Variational Autoencoders
• Spherical Latent Spaces for Stable Variational Autoencoders
• A Novel Weight-Shared Multi-Stage CNN for Scale Robustness
• Hierarchical Attentive Recurrent Tracking
• The Variational Homoencoder: Learning to learn high capacity generative models from few examples
• Multi-focus Attention Network for Efficient Deep Reinforcement Learning
• Convolutional Recurrent Neural Networks for Dynamic MR Image Reconstruction
• Semi Conditional Variational Auto-Encoder for Flow Reconstruction and Uncertainty Quantification from Limited Observations
• High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis
• Joint Modeling of Event Sequence and Time Series with Attentional Twin Recurrent Neural Networks
• Scribbler: Controlling Deep Image Synthesis with Sketch and Color
• Modeling documents with Generative Adversarial Networks
• Video Imagination from a Single Image with Transformation Generation
• A Deep Memory-based Architecture for Sequence-to-Sequence Learning
• deepTarget: End-to-end Learning Framework for microRNA Target Prediction using Deep Recurrent Neural Networks
• Facial Landmark Detection with Tweaked Convolutional Neural Networks
• Semi-Implicit Variational Inference
• Learning FRAME Models Using CNN Filters
• Style Mixer: Semantic-aware Multi-Style Transfer Network
• Enabling hyperparameter optimization in sequential autoencoders for spiking neural data
• Iterative Retraining of Quantum Spin Models Using Recurrent Neural Networks
• Discriminative Particle Filter Reinforcement Learning for Complex Partial Observations
• Emergence of foveal image sampling from learning to attend in visual scenes
• Learning to Sketch with Deep Q Networks and Demonstrated Strokes
• Gang of GANs: Generative Adversarial Networks with Maximum Margin Ranking
• Improving Clinical Predictions through Unsupervised Time Series Representation Learning
• A Data-Efficient Deep Learning Approach for Deployable Multimodal Social Robots
• Graph Element Networks: adaptive, structured computation and memory
• Variational Memory Addressing in Generative Models
• Noisin: Unbiased Regularization for Recurrent Neural Networks
• Learning Wake-Sleep Recurrent Attention Models
• On Generalization Bounds of a Family of Recurrent Neural Networks
• Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks
• Recurrent Attention Unit
• Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
• Representation of linguistic form and function in recurrent neural networks
• Every Smile is Unique: Landmark-Guided Diverse Smile Generation
• Emergent Symbols through Binding in External Memory
• Degeneration in VAE: in the Light of Fisher Information Loss
• Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
• Neural Dynamics Discovery via Gaussian Process Recurrent Neural Networks
• Generative One-Shot Learning (GOL): A Semi-Parametric Approach to One-Shot Learning in Autonomous Vision
• Dynamics Transfer GAN: Generating Video by Transferring Arbitrary Temporal Dynamics from a Source Video to a Single Target Image
• Learning abstract structure for drawing by efficient motor program induction
• Associative Compression Networks for Representation Learning
• Deep Feature Consistent Variational Autoencoder
• Dirichlet Variational Autoencoder for Text Modeling
• Variational Inference with Hamiltonian Monte Carlo
• Cross Domain Knowledge Transfer for Person Re-identification
• Learning Latent Sub-events in Activity Videos Using Temporal Attention Filters
• Fast Convergence of DETR with Spatially Modulated Co-Attention
• To Create What You Tell: Generating Videos from Captions
• Unsupervisedly Prompting AlphaFold2 for Few-Shot Learning of Accurate Folding Landscape and Protein Structure Prediction
• World Discovery Models
• Learning Bidirectional LSTM Networks for Synthesizing 3D Mesh Animation Sequences
• End-to-End Localization and Ranking for Relative Attributes
• Variational Laplace Autoencoders
• Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net
• RRA: Recurrent Residual Attention for Sequence Learning
• Variational Recurrent Neural Machine Translation
• Sparsity in Variational Autoencoders
• Reweighted Expectation Maximization
• Learning Residual Images for Face Attribute Manipulation
• Signal retrieval with measurement system knowledge using variational generative model
• Unsupervised Image to Sequence Translation with Canvas-Drawer Networks
• Bounded Information Rate Variational Autoencoders
• Sampled Training and Node Inheritance for Fast Evolutionary Neural Architecture Search
• Learning Visual Storylines with Skipping Recurrent Neural Networks
• Time Perception Machine: Temporal Point Processes for the When, Where and What of Activity Prediction
• DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation
• Variational Neural Discourse Relation Recognizer
• Disentangling neural mechanisms for perceptual grouping
• Autoencoding sensory substitution
• The Neural Painter: Multi-Turn Image Generation
• Design, Benchmarking and Explainability Analysis of a Game-Theoretic Framework towards Energy Efficiency in Smart Infrastructure
• AMTnet: Action-Micro-Tube Regression by End-to-end Trainable Deep Architecture
• Information Maximizing Visual Question Generation
• Conservative Policy Construction Using Variational Autoencoders for Logged Data with Missing Values
• Deep Co-attention based Comparators For Relative Representation Learning in Person Re-identification
• Generative Adversarial Networks with Decoder-Encoder Output Noise
• Collaborative Attention Mechanism for Multi-View Action Recognition
• Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet
• Generative Knowledge Transfer for Neural Language Models
• Face Parsing via Recurrent Propagation
• Learning Diverse Image Colorization
• Dynamic Capacity Networks
• Compositional Visual Generation and Inference with Energy Based Models
• CAR-Net: Clairvoyant Attentive Recurrent Network
• MBA-RainGAN: Multi-branch Attention Generative Adversarial Network for Mixture of Rain Removal from Single Images
• Variational Bi-LSTMs
• Rethinking Recurrent Neural Networks and Other Improvements for Image Classification
• Teaching GANs to Sketch in Vector Format
• Towards Proof Synthesis Guided by Neural Machine Translation for Intuitionistic Propositional Logic
• Variational Autoencoder with Implicit Optimal Priors
• Distributed Sequence Memory of Multidimensional Inputs in Recurrent Networks
• DeepFace: Face Generation using Deep Learning
• Text-to-Image Generation with Attention Based Recurrent Neural Networks
• Robust LSTM-Autoencoders for Face De-Occlusion in the Wild
• Where to Look Next: Unsupervised Active Visual Exploration on 360° Input
• Composing Music with Grammar Argumented Neural Networks and Note-Level Encoding
• PixelCNN Models with Auxiliary Variables for Natural Image Modeling
• Low-rank passthrough neural networks
• Text-guided Attention Model for Image Captioning
• Generating new concepts with hybrid neuro-symbolic models
• LS-Tree: Model Interpretation When the Data Are Linguistic
• Online Signature Verification using Recurrent Neural Network and Length-normalized Path Signature
• Attention-GAN for Object Transfiguration in Wild Images
• Action-Driven Object Detection with Top-Down Visual Attentions
• Unveiling the Dreams of Word Embeddings: Towards Language-Driven Image Generation
• A Deep Learning and Gamification Approach to Energy Conservation at Nanyang Technological University
• Hybrid VAE: Improving Deep Generative Models using Partial Observations
• An Attention-Based Approach for Single Image Super Resolution
• mu-Forcing: Training Variational Recurrent Autoencoders for Text Generation
• Benchmarking Deep Sequential Models on Volatility Predictions for Financial Time Series
• Attention Mechanisms for Object Recognition with Event-Based Cameras
• Data Augmentation for Enhancing EEG-based Emotion Recognition with Deep Generative Models
• Learning Temporal Transformations From Time-Lapse Videos
• An Uncertain Future: Forecasting from Static Images using Variational Autoencoders
• Unsupervised and interpretable scene discovery with Discrete-Attend-Infer-Repeat
• Real-time interactive sequence generation and control with Recurrent Neural Network ensembles
• Active Perception and Representation for Robotic Manipulation
• Double Backpropagation for Training Autoencoders against Adversarial Attack
• A Biologically Inspired Visual Working Memory for Deep Networks
• DNA Steganalysis Using Deep Recurrent Neural Networks
• Differential Recurrent Neural Network and its Application for Human Activity Recognition
• Variational Rejection Sampling
• StampNet: unsupervised multi-class object discovery
• Endowing Deep 3D Models with Rotation Invariance Based on Principal Component Analysis
• CDVAE: Co-embedding Deep Variational Auto Encoder for Conditional Variational Generation
• AutoScaler: Scale-Attention Networks for Visual Correspondence
• Generative Mixture of Networks
• Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks
• Locally Smoothed Neural Networks
• CNN based texture synthesize with Semantic segment
• Adversarial Framework for Unsupervised Learning of Motion Dynamics in Videos
• Sensorimotor Visual Perception on Embodied System Using Free Energy Principle
• From Language Games to Drawing Games
• Modality-specific Cross-modal Similarity Measurement with Recurrent Attention Network
• Joint Training of Variational Auto-Encoder and Latent Energy-Based Model
• Depth Structure Preserving Scene Image Generation
• Generative Image Modeling using Style and Structure Adversarial Networks
• Stochastic Sequential Neural Networks with Structured Inference
• VQ-DRAW: A Sequential Discrete VAE
• Parametrized Accelerated Methods Free of Condition Number
• Learning a Hierarchical Latent-Variable Model of 3D Shapes
• Encoding Spatial Relations from Natural Language
• CRAM: Clued Recurrent Attention Model
• Video Time: Properties, Encoders and Evaluation
• EagleEye: Attack-Agnostic Defense against Adversarial Inputs (Technical Report)
• Time-Aware and View-Aware Video Rendering for Unsupervised Representation Learning
• Multi-Task Spatiotemporal Neural Networks for Structured Surface Reconstruction
• Channel-Recurrent Autoencoding for Image Modeling
• Instance-Level Salient Object Segmentation
• DeepCoder: Semi-parametric Variational Autoencoders for Automatic Facial Action Coding
• Deep Sensory Substitution: Noninvasively Enabling Biological Neural Networks to Receive Input from Artificial Neural Networks
• Interpretable Visual Understanding with Cognitive Attention Network
• Improving Fair Predictions Using Variational Inference In Causal Models
• RNN-based Generative Model for Fine-Grained Sketching
• Bidirectional Inference Networks: A Class of Deep Bayesian Networks for Health Profiling
• SAdam: A Variant of Adam for Strongly Convex Functions
• Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification
• Modeling Latent Attention Within Neural Networks
• Enhanced Neural Machine Translation by Learning from Draft
• Symmetries and control in generative neural nets
• Texture Synthesis with Recurrent Variational Auto-Encoder
• Max-Margin Deep Generative Models for (Semi-)Supervised Learning
• Towards universal neural nets: Gibbs machines and ACE
• Think Visually: Question Answering through Virtual Imagery
• Generative Creativity: Adversarial Learning for Bionic Design
• Deep Markov Random Field for Image Modeling
• A Fully Trainable Network with RNN-based Pooling
• Generative Models for Security: Attacks, Defenses, and Opportunities
• Connectionist Recommendation in the Wild: On the utility and scrutability of neural networks for personalized course guidance
• Pre-training Attention Mechanisms
• Re-examination of the Role of Latent Variables in Sequence Modeling
• Computational Graph Approach for Detection of Composite Human Activities
• Ensemble Kalman Variational Objectives: Nonlinear Latent Trajectory Inference with A Hybrid of Variational Inference and Ensemble Kalman Filter
• Note on Equivalence Between Recurrent Neural Network Time Series Models and Variational Bayesian Models
• Blur-Attention: A boosting mechanism for non-uniform blurred image restoration
• Artificial Intelligence in Quantitative Ultrasound Imaging: A Review
• A Generative Map for Image-based Camera Localization
• Neuro-symbolic EDA-based Optimisation using ILP-enhanced DBNs
• LSTM and GPT-2 Synthetic Speech Transfer Learning for Speaker Recognition to Overcome Data Scarcity
• Data Augmentation for Deep Candlestick Learner
• Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation
• Visual Attention in Imaginative Agents
• No Representation without Transformation
• 3D Conceptual Design Using Deep Learning
• Learning in Gated Neural Networks
• Biological Blueprints for Next Generation AI Systems
• Crossing-Domain Generative Adversarial Networks for Unsupervised Multi-Domain Image-to-Image Translation
• Binarized Convolutional Neural Networks for Efficient Inference on GPUs
• Understanding in Artificial Intelligence
• An Interpretable Generative Model for Handwritten Digit Image Synthesis
• Long Short Term Memory Networks for Bandwidth Forecasting in Mobile Broadband Networks under Mobility
• One-element Batch Training by Moving Window
• Recurrent Existence Determination Through Policy Optimization
• Coarse Grained Exponential Variational Autoencoders
• CanvasGAN: A simple baseline for text to image generation by incrementally patching a canvas
• NP-DRAW: A Non-Parametric Structured Latent Variable Model for Image Generation
• A Reinforcement Learning Approach for Sequential Spatial Transformer Networks
• MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
• Bio-inspired visual attention for silicon retinas based on spiking neural networks applied to pattern classification
• Mutltimodal AI Companion for Interactive Fairytale Co-creation
• Image Disguise based on Generative Model
• Unsupervised Holistic Image Generation from Key Local Patches
• Deep Nonparametric Estimation of Discrete Conditional Distributions via Smoothed Dyadic Partitioning
• Reinforced Decision Trees
• Learning Fixation Point Strategy for Object Detection and Classification
• A Generative Learning Approach for Spatio-temporal Modeling in Connected Vehicular Network
• A backward pass through a CNN using a generative model of its activations
• Variational Autoencoded Regression: High Dimensional Regression of Visual Data on Complex Manifold