Multi-modal Experts Network for Autonomous Driving

Abstract

End-to-end learning from sensory data has shown promising results in autonomous driving. While employing many sensors enhances world perception and should lead to more robust and reliable behavior of autonomous vehicles, it is challenging to train and deploy such network and at least two problems are encountered in the considered setting. The first one is the increase of computational complexity with the number of sensing devices. The other is the phenomena of network overfitting to the simplest and most informative input. We address both challenges with a novel, carefully tailored multi-modal experts network architecture and propose a multi-stage training procedure. The network contains a gating mechanism, which selects the most relevant input at each inference time step using a mixed discrete-continuous policy. We demonstrate the plausibility of the proposed approach on our 1/6 scale truck equipped with three cameras and one LiDAR.

arXiv commentPublished at the International Conference on Robotics and Automation (ICRA), 2020

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• End to End Learning for Self-Driving Cars
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Deep Convolutional Networks using Vector Quantization
• From Perception to Decision: A Data-driven Approach to End-to-end Motion Planning for Autonomous Ground Robots
• An Empirical Evaluation of Deep Learning on Highway Driving
• Conditional Computation in Neural Networks for faster models
• Adaptive Neural Networks for Efficient Inference
• Learning Factored Representations in a Deep Mixture of Experts
• Choosing Smartly: Adaptive Multimodal Fusion for Object Detection in Changing Environments
• Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation