Challenges for Monocular 6D Object Pose Estimation in Robotics
arXiv:2307.12172 · doi:10.1109/TRO.2024.3433870
Abstract
Object pose estimation is a core perception task that enables, for example, object grasping and scene understanding. The widely available, inexpensive and high-resolution RGB sensors and CNNs that allow for fast inference based on this modality make monocular approaches especially well suited for robotics applications. We observe that previous surveys on object pose estimation establish the state of the art for varying modalities, single- and multi-view settings, and datasets and metrics that consider a multitude of applications. We argue, however, that those works' broad scope hinders the identification of open challenges that are specific to monocular approaches and the derivation of promising future challenges for their application in robotics. By providing a unified view on recent publications from both robotics and computer vision, we find that occlusion handling, novel pose representations, and formalizing and improving category-level pose estimation are still fundamental challenges that are highly relevant for robotics. Moreover, to further improve robotic performance, large object sets, novel objects, refractive materials, and uncertainty estimates are central, largely unsolved open challenges. In order to address them, ontological reasoning, deformability handling, scene-level reasoning, realistic datasets, and the ecological footprint of algorithms need to be improved.
Accepted for publication in Transactions on Robotics: https://ieeexplore.ieee.org/document/10609560 arXiv admin note: substantial text overlap with arXiv:2302.11827
References in corpus (32)
- YOLOv4: Optimal Speed and Accuracy of Object Detection
- YOLOX: Exceeding YOLO Series in 2021
- DINOv2: Learning Robust Visual Features without Supervision
- Mistral 7B
- Probabilistic two-stage detection
- TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models
- Rearrangement: A Challenge for Embodied AI
- MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare
- SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation
- Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects
- DPODv2: Dense Correspondence-Based 6 DoF Pose Estimation
- Category-Level Metric Scale Object Shape and Pose Estimation
- A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
- Advancing 6-DoF Instrument Pose Estimation in Variable X-Ray Imaging Geometries
- Foundation Models in Robotics: Applications, Challenges, and the Future
- Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models
- VisDA-2021 Competition Universal Domain Adaptation to Improve Performance on Out-of-Distribution Data
- FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
- Unseen Object 6D Pose Estimation: A Benchmark and Baselines
- Robust RGB-based 6-DoF Pose Estimation without Real Pose Annotations
- OSSID: Online Self-Supervised Instance Detection by (and for) Pose Estimation
- StereoPose: Category-Level 6D Transparent Object Pose Estimation from Stereo Images via Back-View NOCS
- iCaps: Iterative Category-level Object Pose and Shape Estimation
- POPE: 6-DoF Promptable Pose Estimation of Any Object, in Any Scene, with One Reference
- HouseCat6D -- A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios
- CASAPose: Class-Adaptive and Semantic-Aware Multi-Object Pose Estimation
- Zero123-6D: Zero-shot Novel View Synthesis for RGB Category-level 6D Pose Estimation
- CheckerPose: Progressive Dense Keypoint Localization for Object Pose Estimation with Graph Neural Network
- KDFNet: Learning Keypoint Distance Field for 6D Object Pose Estimation
- 3rd Place Solution for VisDA 2021 Challenge -- Universally Domain Adaptive Image Recognition
- Open-vocabulary object 6D pose estimation