CLEVRER: CoLlision Events for Video REpresentation and Reasoning

Abstract

The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human casual judgment, CLEVRER includes four types of questions: descriptive (e.g., "what color"), explanatory ("what is responsible for"), predictive ("what will happen next"), and counterfactual ("what if"). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.

arXiv commentThe first two authors contributed equally to this work. Accepted as Oral Spotlight as ICLR 2020. Project page: http://clevrer.csail.mit.edu/

References in corpus (11)

• Distributed Representations of Words and Phrases and their Compositionality
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• The Kinetics Human Action Video Dataset
• Interaction Networks for Learning about Objects, Relations and Physics
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• MONet: Unsupervised Scene Decomposition and Representation
• A Compositional Object-Based Approach to Learning Physical Dynamics
• Learning Physical Intuition of Block Towers by Example
• DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer
• CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning

Cited by in corpus (16)

• The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
• MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
• Dynamic Inference with Neural Interpreters
• AlignNet: Unsupervised Entity Alignment
• Learning Object-Centric Video Models by Contrasting Sets
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning
• Compositional Video Synthesis with Action Graphs
• CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoning
• HySTER: A Hybrid Spatio-Temporal Event Reasoner
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• Location-aware Graph Convolutional Networks for Video Question Answering
• SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environments
• AGENT: A Benchmark for Core Psychological Reasoning
• Learning Object Permanence from Video
• Understanding in Artificial Intelligence
• Knowledge Fusion Transformers for Video Action Recognition