Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net

Abstract

In this paper we propose a novel deep neural network that is able to jointly reason about 3D detection, tracking and motion forecasting given data captured by a 3D sensor. By jointly reasoning about these tasks, our holistic approach is more robust to occlusion as well as sparse data at range. Our approach performs 3D convolutions across space and time over a bird's eye view representation of the 3D world, which is very efficient in terms of both memory and computation. Our experiments on a new very large scale dataset captured in several north american cities, show that we can outperform the state-of-the-art by a large margin. Importantly, by sharing computation we can perform all tasks in as little as 30 ms.

CVPR 2018 Oral

References in corpus (1)

• Very Deep Convolutional Networks for Large-Scale Image Recognition

Cited by in corpus (102)

• Deep Continuous Fusion for Multi-Sensor 3D Object Detection
• Argoverse: 3D Tracking and Forecasting with Rich Maps
• MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction
• LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving
• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• 3D-SiamRPN: An End-to-End Learning Method for Real-Time 3D Single Object Tracking Using Raw Point Cloud
• PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences
• Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data
• Stereo R-CNN based 3D Object Detection for Autonomous Driving
• Map-Adaptive Goal-Based Trajectory Prediction
• Joint Interaction and Trajectory Prediction for Autonomous Driving using Graph Neural Networks
• EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection
• Auto4D: Learning to Label 4D Objects from Sequential Point Clouds
• Multi-View Adaptive Fusion Network for 3D Object Detection
• What-If Motion Prediction for Autonomous Driving
• Self-Supervised Velocity Estimation for Automotive Radar Object Detection Networks
• MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences
• PLOP: Probabilistic poLynomial Objects trajectory Planning for autonomous driving
• PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Sampling
• RetinaTrack: Online Single Stage Joint Detection and Tracking
• Associate-3Ddet: Perceptual-to-Conceptual Association for 3D Point Cloud Object Detection
• V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction
• LiDAR-based Online 3D Video Object Detection with Graph-based Message Passing and Spatiotemporal Transformer Attention
• FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras
• Neural Bridge Sampling for Evaluating Safety-Critical Autonomous Systems
• Discrete Residual Flow for Probabilistic Pedestrian Behavior Prediction
• Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks
• Improving 3D Object Detection through Progressive Population Based Augmentation
• LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar Fusion
• Probabilistic Multi-modal Trajectory Prediction with Lane Attention for Autonomous Vehicles
• An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds
• RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects
• Pit30M: A Benchmark for Global Localization in the Age of Self-Driving Cars
• Spatial-Channel Transformer Network for Trajectory Prediction on the Traffic Scenes
• STINet: Spatio-Temporal-Interactive Network for Pedestrian Detection and Trajectory Prediction
• AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles
• FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection
• MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps
• PolarStream: Streaming Lidar Object Detection and Segmentation with Polar Pillars
• PnPNet: End-to-End Perception and Prediction with Tracking in the Loop
• StrObe: Streaming Object Detection from LiDAR Packets
• Rethinking Trajectory Forecasting Evaluation
• Improving Movement Predictions of Traffic Actors in Bird's-Eye View Models using GANs and Differentiable Trajectory Rasterization
• Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild
• TPCN: Temporal Point Cloud Networks for Motion Forecasting
• MVLidarNet: Real-Time Multi-Class Scene Understanding for Autonomous Driving Using Multiple Views
• SoDA: Multi-Object Tracking with Soft Data Association
• Streaming Object Detection for 3-D Point Clouds
• Motion Prediction using Trajectory Sets and Self-Driving Domain Knowledge
• 3D Object Detection From LiDAR Data Using Distance Dependent Feature Extraction
• P2B: Point-to-Box Network for 3D Object Tracking in Point Clouds
• Exploring Uncertainty in Conditional Multi-Modal Retrieval Systems
• 3D-MAN: 3D Multi-frame Attention Network for Object Detection
• Pseudo-labeling for Scalable 3D Object Detection
• Cross-Modality 3D Object Detection
• Fooling LiDAR Perception via Adversarial Trajectory Perturbation
• End-to-end Interpretable Neural Motion Planner
• SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving
• Safety-Oriented Pedestrian Motion and Scene Occupancy Forecasting
• F-Siamese Tracker: A Frustum-based Double Siamese Network for 3D Single Object Tracking
• When We First Met: Visual-Inertial Person Localization for Co-Robot Rendezvous
• Crowd Density Forecasting by Modeling Patch-based Dynamics
• Monocular 3D Object Detection with Decoupled Structured Polygon Estimation and Height-Guided Depth Estimation
• Investigating the Effect of Sensor Modalities in Multi-Sensor Detection-Prediction Models
• Winning the ICCV 2019 Learning to Drive Challenge
• Segment as Points for Efficient Online Multi-Object Tracking and Segmentation
• PanoNet3D: Combining Semantic and Geometric Understanding for LiDARPoint Cloud Detection
• Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs
• Uncertainty-Aware Vehicle Orientation Estimation for Joint Detection-Prediction Models
• Single Shot Multitask Pedestrian Detection and Behavior Prediction
• Just Label What You Need: Fine-Grained Active Selection for Perception and Prediction through Partially Labeled Scenes
• AutoSelect: Automatic and Dynamic Detection Selection for 3D Multi-Object Tracking
• Long-term Prediction of Vehicle Behavior using Short-term Uncertainty-aware Trajectories and High-definition Maps
• Drosophila-Inspired 3D Moving Object Detection Based on Point Clouds
• Ellipse Loss for Scene-Compliant Motion Prediction
• 3D-FCT: Simultaneous 3D Object Detection and Tracking Using Feature Correlation
• Iterative Imitation Policy Improvement for Interactive Autonomous Driving
• Towards Robust Human Trajectory Prediction in Raw Videos
• Leveraging Temporal Information for 3D Detection and Domain Adaptation
• Learning Scene Dynamics from Point Cloud Sequences
• Deep Structured Reactive Planning
• Sensor Fusion: Gated Recurrent Fusion to Learn Driving Behavior from Temporal Multimodal Data
• Deep Crowd-Flow Prediction in Built Environments
• Learning to Optimally Segment Point Clouds
• Efficient Latent Representations using Multiple Tasks for Autonomous Driving
• Object Tracking by Detection with Visual and Motion Cues
• 3D Siamese Voxel-to-BEV Tracker for Sparse Point Clouds
• Tracking from Patterns: Learning Corresponding Patterns in Point Clouds for 3D Object Tracking
• Human Segmentation with Dynamic LiDAR Data
• Deep Learning-Based Autonomous Driving Systems: A Survey of Attacks and Defenses
• Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds
• Active Perception with Neural Networks
• Space-Time-Separable Graph Convolutional Network for Pose Forecasting
• Can't Fool Me: Adversarially Robust Transformer for Video Understanding
• Stereo Frustums: A Siamese Pipeline for 3D Object Detection
• A two-stage data association approach for 3D Multi-object Tracking
• Extending Multi-Object Tracking systems to better exploit appearance and 3D information
• Uncertainty-aware Self-supervised 3D Data Association
• Motion Prediction Performance Analysis for Autonomous Driving Systems and the Effects of Tracking Noise
• Convolutions for Spatial Interaction Modeling
• 3D Object Detection and Tracking Based on Streaming Data
• Relation3DMOT: Exploiting Deep Affinity for 3D Multi-Object Tracking from View Aggregation