Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning

Abstract

Vision-and-language navigation (VLN) asks an agent to follow a given language instruction to navigate through a real 3D environment. Despite significant advances, conventional VLN agents are trained typically under disturbance-free environments and may easily fail in real-world scenarios, since they are unaware of how to deal with various possible disturbances, such as sudden obstacles or human interruptions, which widely exist and may usually cause an unexpected route deviation. In this paper, we present a model-agnostic training paradigm, called Progressive Perturbation-aware Contrastive Learning (PROPER) to enhance the generalization ability of existing VLN agents, by requiring them to learn towards deviation-robust navigation. Specifically, a simple yet effective path perturbation scheme is introduced to implement the route deviation, with which the agent is required to still navigate successfully following the original instruction. Since directly enforcing the agent to learn perturbed trajectories may lead to inefficient training, a progressively perturbed trajectory augmentation strategy is designed, where the agent can self-adaptively learn to navigate under perturbation with the improvement of its navigation performance for each specific trajectory. For encouraging the agent to well capture the difference brought by perturbation, a perturbation-aware contrastive learning mechanism is further developed by contrasting perturbation-free trajectory encodings and perturbation-based counterparts. Extensive experiments on R2R show that PROPER can benefit multiple VLN baselines in perturbation-free scenarios. We further collect the perturbed path data to construct an introspection subset based on the R2R, called Path-Perturbed R2R (PP-R2R). The results on PP-R2R show unsatisfying robustness of popular VLN agents and the capability of PROPER in improving the navigation robustness.

Accepted by TPAMI 2023

References in corpus (22)

• A Simple Framework for Contrastive Learning of Visual Representations
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Graph Contrastive Learning with Augmentations
• GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training
• Learning Representations by Maximizing Mutual Information Across Views
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multi-View Representation Learning on Graphs
• Speaker-Follower Models for Vision-and-Language Navigation
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• Vision-and-Dialog Navigation
• Adversarial Self-Supervised Contrastive Learning
• Embodied Visual Navigation with Automatic Curriculum Learning in Real Environments
• Group Surfing: A Pedestrian-Based Approach to Sidewalk Robot Navigation
• GraphCL: Contrastive Self-Supervised Learning of Graph Representations
• SADA: Semantic Adversarial Diagnostic Attacks for Autonomous Applications
• A Deep Reinforcement Learning Approach to Audio-Based Navigation in a Multi-Speaker Environment
• Vision-based navigation and obstacle avoidance via deep reinforcement learning