Challenges for Monocular 6D Object Pose Estimation in Robotics
arXiv:2307.12172 · doi:10.1109/TRO.2024.3433870
Abstract
Object pose estimation is a core perception task that enables, for example, object grasping and scene understanding. The widely available, inexpensive and high-resolution RGB sensors and CNNs that allow for fast inference based on this modality make monocular approaches especially well suited for robotics applications. We observe that previous surveys on object pose estimation establish the state of the art for varying modalities, single- and multi-view settings, and datasets and metrics that consider a multitude of applications. We argue, however, that those works' broad scope hinders the identification of open challenges that are specific to monocular approaches and the derivation of promising future challenges for their application in robotics. By providing a unified view on recent publications from both robotics and computer vision, we find that occlusion handling, novel pose representations, and formalizing and improving category-level pose estimation are still fundamental challenges that are highly relevant for robotics. Moreover, to further improve robotic performance, large object sets, novel objects, refractive materials, and uncertainty estimates are central, largely unsolved open challenges. In order to address them, ontological reasoning, deformability handling, scene-level reasoning, realistic datasets, and the ecological footprint of algorithms need to be improved.
Accepted for publication in Transactions on Robotics: https://ieeexplore.ieee.org/document/10609560 arXiv admin note: substantial text overlap with arXiv:2302.11827
References in corpus (42)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- YOLOv4: Optimal Speed and Accuracy of Object Detection
- Denoising Diffusion Probabilistic Models
- YOLOX: Exceeding YOLO Series in 2021
- DINOv2: Learning Robust Visual Features without Supervision
- Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols
- Mistral 7B
- Satellite Pose Estimation Challenge: Dataset, Competition Design and Results
- Probabilistic two-stage detection
- Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation
- TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models
- Rearrangement: A Challenge for Embodied AI
- MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare
- SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
- Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects
- DPODv2: Dense Correspondence-Based 6 DoF Pose Estimation
- Category-Level Metric Scale Object Shape and Pose Estimation
- A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
- Towards Real-World Category-level Articulation Pose Estimation
- DGCM-Net: Dense Geometrical Correspondence Matching Network for Incremental Experience-based Robotic Grasping
- Advancing 6-DoF Instrument Pose Estimation in Variable X-Ray Imaging Geometries
- DISP6D: Disentangled Implicit Shape and Pose Learning for Scalable 6D Pose Estimation
- Foundation Models in Robotics: Applications, Challenges, and the Future
- Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models
- VisDA-2021 Competition Universal Domain Adaptation to Improve Performance on Out-of-Distribution Data
- Unseen Object 6D Pose Estimation: A Benchmark and Baselines
- FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
- Robust RGB-based 6-DoF Pose Estimation without Real Pose Annotations
- Deep Learning on Monocular Object Pose Detection and Tracking: A Comprehensive Overview
- OSSID: Online Self-Supervised Instance Detection by (and for) Pose Estimation
- iCaps: Iterative Category-level Object Pose and Shape Estimation
- StereoPose: Category-Level 6D Transparent Object Pose Estimation from Stereo Images via Back-View NOCS
- POPE: 6-DoF Promptable Pose Estimation of Any Object, in Any Scene, with One Reference
- HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios
- Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image
- CASAPose: Class-Adaptive and Semantic-Aware Multi-Object Pose Estimation
- CheckerPose: Progressive Dense Keypoint Localization for Object Pose Estimation with Graph Neural Network
- 3rd Place Solution for VisDA 2021 Challenge -- Universally Domain Adaptive Image Recognition
- Zero123-6D: Zero-shot Novel View Synthesis for RGB Category-level 6D Pose Estimation
- KDFNet: Learning Keypoint Distance Field for 6D Object Pose Estimation
- Open-vocabulary object 6D pose estimation