Semantics for Robotic Mapping, Perception and Interaction: A Survey
arXiv:2101.00443 · doi:10.1561/2300000059
Abstract
For robots to navigate and interact more richly with the world around them, they will likely require a deeper understanding of the world in which they operate. In robotics and related research fields, the study of understanding is often referred to as semantics, which dictates what does the world "mean" to a robot, and is strongly tied to the question of how to represent that meaning. With humans and robots increasingly operating in the same world, the prospects of human-robot interaction also bring semantics and ontology of natural language into the picture. Driven by need, as well as by enablers like increasing availability of training data and computational resources, semantics is a rapidly growing research area in robotics. The field has received significant attention in the research literature to date, but most reviews and surveys have focused on particular aspects of the topic: the technical research issues regarding its use in specific robotic topics like mapping or segmentation, or its relevance to one particular application domain like autonomous driving. A new treatment is therefore required, and is also timely because so much relevant research has occurred since many of the key surveys were published. This survey therefore provides an overarching snapshot of where semantics in robotics stands today. We establish a taxonomy for semantics research in or relevant to robotics, split into four broad categories of activity, in which semantics are extracted, used, or both. Within these broad categories we survey dozens of major topics including fundamentals from the computer vision field and key robotics research areas utilizing semantics, including mapping, navigation and interaction with the world. The survey also covers key practical considerations, including enablers like increased data availability and improved computational hardware, and major application areas where...
arXiv comment81 pages, 1 figure, published in Foundations and Trends in Robotics, 2020
References in corpus (65)
- Deep Learning in Neural Networks: An Overview
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- R-FCN: Object Detection via Region-based Fully Convolutional Networks
- A Survey of Deep Learning Techniques for Autonomous Driving
- Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
- MOT16: A Benchmark for Multi-Object Tracking
- Semantic Foggy Scene Understanding with Synthetic Data
- DeepGlobe 2018: A Challenge to Parse the Earth through Satellite Images
- ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
- Joint 2D-3D-Semantic Data for Indoor Scene Understanding
- Video Salient Object Detection via Fully Convolutional Networks
- Associative Embedding: End-to-End Learning for Joint Detection and Grouping
- MOT20: A benchmark for multi object tracking in crowded scenes
- FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review
- Deep Continuous Fusion for Multi-Sensor 3D Object Detection
- A Survey of Neuromorphic Computing and Neural Networks in Hardware
- SfM-Net: Learning of Structure and Motion from Video
- Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
- Vision Meets Drones: A Challenge
- Graduated Non-Convexity for Robust Spatial Perception: From Non-Minimal Solvers to Global Outlier Rejection
- Rank Pooling for Action Recognition
- Deep Drone Racing: From Simulation to Reality with Domain Randomization
- A2D2: Audi Autonomous Driving Dataset
- Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects
- IntentNet: Learning to Predict Intention from Raw Sensor Data
- HDNET: Exploiting HD Maps for 3D Object Detection
- Co-Fusion: Real-time Segmentation, Tracking and Fusion of Multiple Objects
- Spatially-sparse convolutional neural networks
- A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones
- SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection
- The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
- End-to-end Learning of Multi-sensor 3D Tracking by Detection
- IPOD: Intensive Point-based Object Detector for Point Cloud
- 3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents
- Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
- A Survey of Knowledge Representation in Service Robotics
- Pre-training Tasks for Embedding-based Large-scale Retrieval
- Hierarchy-based Image Embeddings for Semantic Image Retrieval
- Finding beans in burgers: Deep semantic-visual embedding with localization
- Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing
- DAC-h3: A Proactive Robot Cognitive Architecture to Acquire and Express Knowledge About the World and the Self
- The History of Mobile Augmented Reality
- A Survey of Semantic Segmentation
- Learning to Localize Using a LiDAR Intensity Map
- Navigating the Landscape for Real-time Localisation and Mapping for Robotics and Virtual and Augmented Reality
- Fast Scene Understanding for Autonomous Driving
- Semi-Dense 3D Semantic Mapping from Monocular SLAM
- Semantically-Guided Representation Learning for Self-Supervised Monocular Depth
- RefinedMPL: Refined Monocular PseudoLiDAR for 3D Object Detection in Autonomous Driving
- Learning Deep NBNN Representations for Robust Place Categorization
- FutureMapping: The Computational Structure of Spatial AI Systems
- DeepSignals: Predicting Intent of Drivers Through Visual Signals
- Noise-resistant Deep Learning for Object Classification in 3D Point Clouds Using a Point Pair Descriptor
- Affordances in Robotic Tasks -- A Survey
- Semantic Localization in the PCL library
- Semantic Image Based Geolocation Given a Map
- SegVoxelNet: Exploring Semantic Context and Depth-aware Features for 3D Vehicle Detection from Point Cloud
- End-to-End Deep Structured Models for Drawing Crosswalks
- Towards a Domain Specific Language for a Scene Graph based Robotic World Model
- Inferring Spatial Uncertainty in Object Detection
- FutureMapping 2: Gaussian Belief Propagation for Spatial AI
- Service-Oriented Software Architecture for Cloud Robotics
- Semantic Image Search for Robotic Applications
Cited by in corpus (11)
- The Revisiting Problem in Simultaneous Localization and Mapping: A Survey on Visual Loop Closure Detection
- VPR-Bench: An Open-Source Visual Place Recognition Evaluation Framework with Quantifiable Viewpoint and Appearance Change
- Multi-Robot Collaborative Perception with Graph Neural Networks
- MultiRes-NetVLAD: Augmenting Place Recognition Training with Low-Resolution Imagery
- Air-Ground Collaboration with SPOMP: Semantic Panoramic Online Mapping and Planning
- A Taxonomy of Semantic Information in Robot-Assisted Disaster Response
- Active Exploration based on Information Gain by Particle Filter for Efficient Spatial Concept Formation
- Hierarchical Path-planning from Speech Instructions with Spatial Concept-based Topometric Semantic Mapping
- Semantic 3D Grid Maps for Autonomous Driving
- Improving Road Segmentation in Challenging Domains Using Similar Place Priors
- CurriculumLoc: Enhancing Cross-Domain Geolocalization through Multi-Stage Refinement