Learning Multi-modal Similarity

Abstract

In many applications involving multi-media data, the definition of similarity between items is integral to several key tasks, e.g., nearest-neighbor retrieval, classification, and recommendation. Data in such regimes typically exhibits multiple modalities, such as acoustic and visual content of video. Integrating such heterogeneous data to form a holistic similarity space is therefore a key challenge to be overcome in many real-world applications. We present a novel multiple kernel learning technique for integrating heterogeneous data into a single, unified similarity space. Our algorithm learns an optimal ensemble of kernel transfor- mations which conform to measurements of human perceptual similarity, as expressed by relative comparisons. To cope with the ubiquitous problems of subjectivity and inconsistency in multi- media similarity, we develop graph-based techniques to filter similarity measurements, resulting in a simplified and robust training procedure.

References in corpus (1)

• Consistency of the group Lasso and multiple kernel learning

Cited by in corpus (27)

• Clustering on Multi-Layer Graphs via Subspace Analysis on Grassmann Manifolds
• Multi-View Matrix Completion for Multi-Label Image Classification
• Learning Features of Music from Scratch
• Large Margin Multi-modal Multi-task Feature Extraction for Image Classification
• A Similarity Measure for Material Appearance
• Drug Similarity Integration Through Attentive Multi-view Graph Auto-Encoders
• Multimodal diffusion geometry by joint diagonalization of Laplacians
• Making Laplacians commute
• Generalization Bounds for Metric and Similarity Learning
• Lens depth function and k-relative neighborhood graph: versatile tools for ordinal data analysis
• Adaptive Regularization for Weight Matrices
• Active Perceptual Similarity Modeling with Auxiliary Information
• Generating Labels for Regression of Subjective Constructs using Triplet Embeddings
• PerceptNet: Learning Perceptual Similarity of Haptic Textures in Presence of Unorderable Triplets
• Cross Euclidean-to-Riemannian Metric Learning with Application to Face Recognition from Video
• Learning content similarity for music recommendation
• Jointly Learning Multiple Measures of Similarities from Triplet Comparisons
• Heat kernel coupling for multiple graph analysis
• Multimodal similarity-preserving hashing
• Geometry of Similarity Comparisons
• Interpolated Discretized Embedding of Single Vectors and Vector Pairs for Classification, Metric Learning and Distance Approximation
• Robust Ordinal Embedding from Contaminated Relative Comparisons
• Positive semidefinite support vector regression metric learning
• MULTIMODAL ANALYSIS: Informed content estimation and audio source separation
• A Multi-View Approach Based on Naming Behavioral Modeling for Aligning Chinese User Accounts across Multiple Networks
• Markov-switching State Space Models for Uncovering Musical Interpretation
• Relative Comparison Kernel Learning with Auxiliary Kernels