Modulating early visual processing by language

Abstract

It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic input are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the \emph{entire visual processing} by linguistic input. Specifically, we condition the batch normalization parameters of a pretrained residual network (ResNet) on a language embedding. This approach, which we call MOdulated RESnet (\MRN), significantly improves strong baselines on two visual question answering tasks. Our ablation study shows that modulating from the early stages of the visual processing is beneficial.

Advances in Neural Information Processing Systems 30 (NIPS 2017)

References in corpus (8)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Multimodal Residual Learning for Visual QA
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering

Cited by in corpus (80)

• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• FiLM: Visual Reasoning with a General Conditioning Layer
• Adversarial Video Generation on Complex Datasets
• Selective Synthetic Augmentation with HistoGAN for Improved Histopathology Image Classification
• Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform
• ContraGAN: Contrastive Learning for Conditional Image Generation
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• Controlling generative models with continuous factors of variations
• On Self Modulation for Generative Adversarial Networks
• U-Sleep's resilience to AASM guidelines
• Dynamic Neural Networks: A Survey
• A Highly Adaptive Acoustic Model for Accurate Multi-Dialect Speech Recognition
• Efficient Video Object Segmentation via Network Modulation
• End-to-End Adversarial Text-to-Speech
• Spatially Controllable Image Synthesis with Internal Representation Collaging
• Interactive Grounded Language Acquisition and Generalization in a 2D World
• Data-driven Crop Growth Simulation on Time-varying Generated Images using Multi-conditional Generative Adversarial Networks
• Continuous conditional generative adversarial networks for data-driven solutions of poroelasticity with heterogeneous material properties
• Question-Guided Hybrid Convolution for Visual Question Answering
• Reinforcing Generated Images via Meta-learning for One-Shot Fine-Grained Visual Recognition
• Controllable Video Captioning with an Exemplar Sentence
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis
• Facial Synthesis from Visual Attributes via Sketch using Multi-Scale Generators
• What You See is Not What the Network Infers: Detecting Adversarial Examples Based on Semantic Contradiction
• Multimodal Grounding for Language Processing
• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• Twin-GAN -- Unpaired Cross-Domain Image Translation with Weight-Sharing GANs
• PcDGAN: A Continuous Conditional Diverse Generative Adversarial Network For Inverse Design
• NeeDrop: Self-supervised Shape Representation from Sparse Point Clouds using Needle Dropping
• Learning Mesh Representations via Binary Space Partitioning Tree Networks
• Modulating and attending the source image during encoding improves Multimodal Translation
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• Generating unrepresented proportions of geological facies using Generative Adversarial Networks
• cGANs with Multi-Hinge Loss
• Spotting Virus from Satellites: Modeling the Circulation of West Nile Virus Through Graph Neural Networks
• COCO-GAN: Generation by Parts via Conditional Coordinating
• A Survey on Deep Domain Adaptation and Tiny Object Detection Challenges, Techniques and Datasets
• Adaptive Denoising via GainTuning
• Generating Electronic Health Records with Multiple Data Types and Constraints
• Uncertainty in Multitask Transfer Learning
• Twin Auxiliary Classifiers GAN
• Learning Visual Question Answering by Bootstrapping Hard Attention
• Segmenting Objects in Day and Night:Edge-Conditioned CNN for Thermal Image Semantic Segmentation
• Continuous Conditional Generative Adversarial Networks: Novel Empirical Losses and Label Input Mechanisms
• Conditional GANs with Auxiliary Discriminative Classifier
• Positional Normalization
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Image Synthesis From Reconfigurable Layout and Style
• JGAN: A Joint Formulation of GAN for Synthesizing Images and Labels
• Image Generation from Layout
• Label-Noise Robust Generative Adversarial Networks
• Self-Supervised GANs via Auxiliary Rotation Loss
• ManiGAN: Text-Guided Image Manipulation
• 3D Teeth Reconstruction from Panoramic Radiographs using Neural Implicit Functions
• cGANs with Conditional Convolution Layer
• Telescoping Density-Ratio Estimation
• Class-Distinct and Class-Mutual Image Generation with GANs
• Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise
• Multilevel Language and Vision Integration for Text-to-Clip Retrieval
• Deep Context-Aware Novelty Detection
• M2FN: Multi-step Modality Fusion for Advertisement Image Assessment
• Deep HMResNet Model for Human Activity-Aware Robotic Systems
• Question Guided Modular Routing Networks for Visual Question Answering
• Deep Video Codec Control for Vision Models
• Continuous Face Aging via Self-estimated Residual Age Embedding
• AdaCrowd: Unlabeled Scene Adaptation for Crowd Counting
• An Empirical Study of Batch Normalization and Group Normalization in Conditional Computation
• Attribute-Modulated Generative Meta Learning for Zero-Shot Classification
• Fine-grained Image-to-Image Transformation towards Visual Recognition
• Conditioning Trick for Training Stable GANs
• Scale Calibrated Training: Improving Generalization of Deep Networks via Scale-Specific Normalization
• Cascaded Mutual Modulation for Visual Reasoning
• Semantic Text-to-Face GAN -ST^2FG
• Point Discriminative Learning for Data-efficient 3D Point Cloud Analysis
• Pixel Sampling for Style Preserving Face Pose Editing
• Understanding Invariance via Feedforward Inversion of Discriminatively Trained Classifiers
• Towards Generalising Neural Implicit Representations
• FoCL: Feature-Oriented Continual Learning for Generative Models
• Efficient Video Understanding via Layered Multi Frame-Rate Analysis