FiLM: Visual Reasoning with a General Conditioning Layer

Abstract

We introduce a general-purpose conditioning method for neural networks called FiLM: Feature-wise Linear Modulation. FiLM layers influence neural network computation via a simple, feature-wise affine transformation based on conditioning information. We show that FiLM layers are highly effective for visual reasoning - answering image-related questions which require a multi-step, high-level process - a task which has proven difficult for standard deep learning methods that do not explicitly model reasoning. Specifically, we show on visual reasoning tasks that FiLM layers 1) halve state-of-the-art error for the CLEVR benchmark, 2) modulate features in a coherent manner, 3) are robust to ablations and architectural modifications, and 4) generalize well to challenging, new data from few examples or even zero-shot.

AAAI 2018. Code available at http://github.com/ethanjperez/film . Extends arXiv:1707.03017

References in corpus (8)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• A simple neural network module for relational reasoning
• Modulating early visual processing by language
• Learning to Compose Neural Networks for Question Answering
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Visual Interaction Networks
• Dynamic Layer Normalization for Adaptive Neural Acoustic Modeling in Speech Recognition

Cited by in corpus (32)

• Robust Motion In-betweening
• Slimmable Neural Networks
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• TADAM: Task dependent adaptive metric for improved few-shot learning
• Differentiable Learning-to-Normalize via Switchable Normalization
• FigureQA: An Annotated Figure Dataset for Visual Reasoning
• Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform
• Object Level Visual Reasoning in Videos
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• "No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy
• A Highly Adaptive Acoustic Model for Accurate Multi-Dialect Speech Recognition
• Efficient Video Object Segmentation via Network Modulation
• HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling
• Learning from Few Samples: A Survey
• CLOSURE: Assessing Systematic Generalization of CLEVR Models
• Learning to Count Isomorphisms with Graph Neural Networks
• Overcoming Language Priors in Visual Question Answering with Adversarial Regularization
• Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
• Image-Question-Answer Synergistic Network for Visual Dialog
• Diffusion-Based Audio Inpainting
• Segmenting Objects in Day and Night:Edge-Conditioned CNN for Thermal Image Semantic Segmentation
• CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation
• Long-Term Cloth-Changing Person Re-identification
• Back-Projection Diffusion: Solving the Wideband Inverse Scattering Problem with Diffusion Models
• SCHA-VAE: Hierarchical Context Aggregation for Few-Shot Generation
• Neural Drum Machine : An Interactive System for Real-time Synthesis of Drum Sounds
• Deep Context-Aware Novelty Detection
• Fine-grained Image-to-Image Transformation towards Visual Recognition
• Challenging Images For Minds and Machines
• Semantic Segmentation with Scarce Data
• Conditional Fetal Brain Atlas Learning for Automatic Tissue Segmentation