Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
arXiv:2103.16110
Abstract
We present a new vision-language (VL) pre-training model dubbed Kaleido-BERT, which introduces a novel kaleido strategy for fashion cross-modality representations from transformers. In contrast to random masking strategy of recent VL models, we design alignment guided masking to jointly focus more on image-text semantic relations. To this end, we carry out five novel tasks, i.e., rotation, jigsaw, camouflage, grey-to-color, and blank-to-color for self-supervised VL pre-training at patches of different scale. Kaleido-BERT is conceptually simple and easy to extend to the existing BERT framework, it attains new state-of-the-art results by large margins on four downstream tasks, including text retrieval (R@1: 4.03% absolute improvement), image retrieval (R@1: 7.13% abs imv.), category recognition (ACC: 3.28% abs imv.), and fashion captioning (Bleu4: 1.2 abs imv.). We validate the efficiency of Kaleido-BERT on a wide range of e-commerical websites, demonstrating its broader potential in real-world applications.
CVPR2021 Accepted. Code: https://github.com/mczhuge/Kaleido-BERT
References in corpus (12)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Language Models are Few-Shot Learners
- The Kinetics Human Action Video Dataset
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
- ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
- Modality-Agnostic Attention Fusion for visual search with text feedback
- XGPT: Cross-modal Generative Pre-Training for Image Captioning
- MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering