Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification

Abstract

Multi-label image and video classification are fundamental yet challenging tasks in computer vision. The main challenges lie in capturing spatial or temporal dependencies between labels and discovering the locations of discriminative features for each class. In order to overcome these challenges, we propose to use cross-modality attention with semantic graph embedding for multi label classification. Based on the constructed label graph, we propose an adjacency-based similarity graph embedding method to learn semantic label embeddings, which explicitly exploit label relationships. Then our novel cross-modality attention maps are generated with the guidance of learned label embeddings. Experiments on two multi-label image classification datasets (MS-COCO and NUS-WIDE) show our method outperforms other existing state-of-the-arts. In addition, we validate our method on a large multi-label video classification dataset (YouTube-8M Segments) and the evaluation results demonstrate the generalization capability of our method.

Accepted by AAAI2020

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• FitNets: Hints for Thin Deep Nets
• YouTube-8M: A Large-Scale Video Classification Benchmark
• CNN: Single-label to Multi-label
• CNN-RNN: A Unified Framework for Multi-label Image Classification
• Multi-Label Image Recognition with Graph Convolutional Networks
• Multi-label Image Recognition by Recurrently Discovering Attentional Regions
• Order-Free RNN with Visual Attention for Multi-Label Classification
• Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks

Cited by in corpus (2)

• MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data
• GM-MLIC: Graph Matching based Multi-Label Image Classification