CoDEPS: Online Continual Learning for Depth Estimation and Panoptic Segmentation

Abstract

Operating a robot in the open world requires a high level of robustness with respect to previously unseen environments. Optimally, the robot is able to adapt by itself to new conditions without human supervision, e.g., automatically adjusting its perception system to changing lighting conditions. In this work, we address the task of continual learning for deep learning-based monocular depth estimation and panoptic segmentation in new environments in an online manner. We introduce CoDEPS to perform continual learning involving multiple real-world domains while mitigating catastrophic forgetting by leveraging experience replay. In particular, we propose a novel domain-mixing strategy to generate pseudo-labels to adapt panoptic segmentation. Furthermore, we explicitly address the limited storage capacity of robotic systems by leveraging sampling strategies for constructing a fixed-size replay buffer based on rare semantic class sampling and image diversity. We perform extensive evaluations of CoDEPS on various real-world datasets demonstrating that it successfully adapts to unseen environments without sacrificing performance on previous domains while achieving state-of-the-art results. The code of our work is publicly available at http://codeps.cs.uni-freiburg.de.

arXiv commentAccepted for "Robotics: Science and Systems (RSS) 2023"

References in corpus (19)

• Gradient Episodic Memory for Continual Learning
• Gradient based sample selection for online continual learning
• Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images
• Unsupervised Monocular Depth Learning in Dynamic Scenes
• Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation
• Continual SLAM: Beyond Lifelong Simultaneous Localization and Mapping Through Continual Learning
• D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry
• KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D
• CoVIO: Online Continual Learning for Visual-Inertial Odometry
• Unsupervised Domain Adaptation for LiDAR Panoptic Segmentation
• Unsupervised Monocular Depth and Ego-motion Learning with Structure and Semantics
• Monocular Depth Estimation with Self-supervised Instance Adaptation
• Cross-View Regularization for Domain Adaptive Panoptic Segmentation
• ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation
• Self-Supervised Deep Visual Odometry with Online Adaptation
• Continual Test-Time Domain Adaptation
• ConfMix: Unsupervised Domain Adaptation for Object Detection via Confidence-based Mixing
• Continual BatchNorm Adaptation (CBNA) for Semantic Segmentation
• Perceiving the Invisible: Proposal-Free Amodal Panoptic Segmentation

Cited by in corpus (5)

• A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
• CoVIO: Online Continual Learning for Visual-Inertial Odometry
• Few-Shot Panoptic Segmentation With Foundation Models
• Syn-Mediverse: A Multimodal Synthetic Dataset for Intelligent Scene Understanding of Healthcare Facilities
• A Good Foundation is Worth Many Labels: Label-Efficient Panoptic Segmentation