Practical Cross-modal Manifold Alignment for Grounded Language

Abstract

We propose a cross-modality manifold alignment procedure that leverages triplet loss to jointly learn consistent, multi-modal embeddings of language-based concepts of real-world items. Our approach learns these embeddings by sampling triples of anchor, positive, and negative data points from RGB-depth images and their natural language descriptions. We show that our approach can benefit from, but does not require, post-processing steps such as Procrustes analysis, in contrast to some of our baselines which require it for reasonable performance. We demonstrate the effectiveness of our approach on two datasets commonly used to develop robotic-based grounded language learning systems, where our approach outperforms four baselines, including a state-of-the-art approach, across five evaluation metrics.

References in corpus (6)

• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• In Defense of the Triplet Loss for Person Re-Identification
• Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
• A Survey of Machine Learning Methods and Challenges for Windows Malware Classification
• Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTA
• Stochastic Optimization for Deep CCA via Nonlinear Orthogonal Iterations