Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction

Abstract

Future prediction is a fundamental principle of intelligence that helps plan actions and avoid possible dangers. As the future is uncertain to a large extent, modeling the uncertainty and multimodality of the future states is of great relevance. Existing approaches are rather limited in this regard and mostly yield a single hypothesis of the future or, at the best, strongly constrained mixture components that suffer from instabilities in training and mode collapse. In this work, we present an approach that involves the prediction of several samples of the future with a winner-takes-all loss and iterative grouping of samples to multiple modes. Moreover, we discuss how to evaluate predicted multimodal distributions, including the common real scenario, where only a single sample from the ground-truth distribution is available for evaluation. We show on synthetic and real data that the proposed approach triggers good estimates of multimodal distributions and avoids mode collapse. Source code is available at $\text{href}$

arXiv commentIn CVPR 2019

References in corpus (18)

• Unsupervised Learning of Video Representations using LSTMs
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Deep multi-scale video prediction beyond mean square error
• Learning to Generate Long-term Future via Hierarchical Prediction
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Hierarchical Long-term Video Prediction without Supervision
• Deep Learning Driven Visual Path Prediction from a Single Image
• Learning A Physical Long-term Predictor
• Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving
• Predicting Deeper into the Future of Semantic Segmentation
• Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypotheses
• Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks
• Time-Agnostic Prediction: Predicting Predictable Video Frames
• Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods
• Probabilistic Prediction of Vehicle Semantic Intention and Motion
• Learning what you can do before doing anything
• Multimodal Interaction-aware Motion Prediction for Autonomous Street Crossing
• Forecasting Hands and Objects in Future Frames

Cited by in corpus (44)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Human Motion Trajectory Prediction: A Survey
• A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving
• A Review on Deep Learning Techniques for Video Prediction
• Probabilistic neural networks for fluid flow surrogate modeling and data recovery
• Injecting Knowledge in Data-driven Vehicle Trajectory Predictors
• Continuous and Distribution-free Probabilistic Wind Power Forecasting: A Conditional Normalizing Flow Approach
• Semi Conditional Variational Auto-Encoder for Flow Reconstruction and Uncertainty Quantification from Limited Observations
• Diverse Multiple Trajectory Prediction Using a Two-stage Prediction Network Trained with Lane Loss
• What-If Motion Prediction for Autonomous Driving
• PLOP: Probabilistic poLynomial Objects trajectory Planning for autonomous driving
• Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals
• SimAug: Learning Robust Representations from Simulation for Trajectory Prediction
• Vision-based Multi-future Trajectory Prediction: A Survey
• Towards trustworthy multi-modal motion prediction: Holistic evaluation and interpretability of outputs
• Diverse Multimedia Layout Generation with Multi Choice Learning
• Social-VRNN: One-Shot Multi-modal Trajectory Prediction for Interacting Pedestrians
• Probabilistic Multi-modal Trajectory Prediction with Lane Attention for Autonomous Vehicles
• Decoder Fusion RNN: Context and Interaction Aware Decoders for Trajectory Prediction
• The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
• Scene Gated Social Graph: Pedestrian Trajectory Prediction Based on Dynamic Social Graphs and Scene Constraints
• Long-term Human Motion Prediction with Scene Context
• Multiple Object Tracking with Mixture Density Networks for Trajectory Estimation
• Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes
• A novel Deep Learning approach for one-step Conformal Prediction approximation
• Improving Confidence Estimates for Unfamiliar Examples
• Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks
• Exploring Dynamic Context for Multi-path Trajectory Prediction
• AMENet: Attentive Maps Encoder Network for Trajectory Prediction
• Over-crowdedness Alert! Forecasting the Future Crowd Distribution
• Recovering the Imperfect: Cell Segmentation in the Presence of Dynamically Localized Proteins
• Interpretable and Flexible Target-Conditioned Neural Planners For Autonomous Vehicles
• From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video
• 6D Camera Relocalization in Ambiguous Scenes via Continuous Multimodal Inference
• Learning to Predict Diverse Human Motions from a Single Image via Mixture Density Networks
• Distributional Depth-Based Estimation of Object Articulation Models
• Maximum Likelihood Estimation of Flexible Survival Densities with Importance Sampling
• Transformer-based few-shot learning for modeling Electricity Consumption Profiles with minimal data across thousands of domains
• Probability Trajectory: One New Movement Description for Trajectory Prediction
• Jointly Learning Agent and Lane Information for Multimodal Trajectory Prediction
• Deep Neural Network in Cusp Catastrophe Model
• Modeling continuous-time stochastic processes using $\mathcal{N}$-Curve mixtures
• Learning Proposals for Practical Energy-Based Regression
• Handwriting Prediction Considering Inter-Class Bifurcation Structures