Large-Scale Adversarial Training for Vision-and-Language Representation Learning
arXiv:2006.06195
Abstract
We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-agnostic adversarial pre-training; followed by (ii) task-specific adversarial finetuning. Instead of adding adversarial perturbations on image pixels and textual tokens, we propose to perform adversarial training in the embedding space of each modality. To enable large-scale training, we adopt the "free" adversarial training strategy, and combine it with KL-divergence-based regularization to promote higher invariance in the embedding space. We apply VILLA to current best-performing V+L models, and achieve new state of the art on a wide range of tasks, including Visual Question Answering, Visual Commonsense Reasoning, Image-Text Retrieval, Referring Expression Comprehension, Visual Entailment, and NLVR2.
arXiv commentNeurIPS 2020 Spotlight paper
References in corpus (43)
- Ensemble Adversarial Training: Attacks and Defenses
- Unsupervised Data Augmentation for Consistency Training
- Towards Deep Learning Models Resistant to Adversarial Attacks
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
- Theoretically Principled Trade-off between Robustness and Accuracy
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- Fast is better than free: Revisiting adversarial training
- Using Pre-Training Can Improve Model Robustness and Uncertainty
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Learning Video Representations using Contrastive Bidirectional Transformer
- Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning
- Adversarial Training Methods for Semi-Supervised Text Classification
- UNITER: UNiversal Image-TExt Representation Learning
- Are Labels Required for Improving Adversarial Robustness?
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding
- VideoBERT: A Joint Model for Video and Language Representation Learning
- Learning Conditioned Graph Structures for Interpretable Visual Question Answering
- Compositional Attention Networks for Machine Reasoning
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
- Learning to Reason: End-to-End Module Networks for Visual Question Answering
- Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
- Deep Modular Co-Attention Networks for Visual Question Answering
- Adversarial Training for Large Neural Language Models
- Unified Vision-Language Pre-Training for Image Captioning and VQA
- Boosting Adversarial Training with Hypersphere Embedding
- Metric Learning for Adversarial Robustness
- Relation-Aware Graph Attention Network for Visual Question Answering
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
- Adversarial Examples Improve Image Recognition
- Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
- VD-BERT: A Unified Vision and Dialog Transformer with BERT
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
- MUREL: Multimodal Relational Reasoning for Visual Question Answering
- XGPT: Cross-modal Generative Pre-Training for Image Captioning
- Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
- Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
- Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
- Meta Module Network for Compositional Visual Reasoning
- Improving the Robustness of Deep Neural Networks via Adversarial Training with Triplet Loss
- Exact Adversarial Attack to Image Captioning via Structured Output Learning with Latent Variables
Cited by in corpus (53)
- Learning Transferable Visual Models From Natural Language Supervision
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
- Florence: A New Foundation Model for Computer Vision
- VLP: A Survey on Vision-Language Pre-training
- FILIP: Fine-grained Interactive Language-Image Pre-Training
- UNITER: UNiversal Image-TExt Representation Learning
- ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
- Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
- From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
- Disentangling Hate in Online Memes
- WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
- Referring Transformer: A One-step Approach to Multi-task Visual Grounding
- InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
- VinVL: Revisiting Visual Representations in Vision-Language Models
- InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
- MERLOT: Multimodal Neural Script Knowledge Models
- Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training
- HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
- Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
- Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution
- M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
- A Multimodal Memes Classification: A Survey and Open Research Issues
- UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
- UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
- Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
- An Empirical Study of Training End-to-End Vision-and-Language Transformers
- Self-supervised Pre-training with Hard Examples Improves Visual Representations
- Scaling Up Vision-Language Pre-training for Image Captioning
- Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
- Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
- Spatio-temporal Prompting Network for Robust Video Feature Extraction
- Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
- Meta Module Network for Compositional Visual Reasoning
- OmDet: Large-scale vision-language multi-dataset pre-training with multimodal detection network
- Adversarial Feature Augmentation and Normalization for Visual Recognition
- PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting
- HateProof: Are Hateful Meme Detection Systems really Robust?
- KVL-BERT: Knowledge Enhanced Visual-and-Linguistic BERT for Visual Commonsense Reasoning
- Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
- Boon: A Neural Search Engine for Cross-Modal Information Retrieval
- ROME: Testing Image Captioning Systems via Recursive Object Melting
- CAP: Co-Adversarial Perturbation on Weights and Features for Improving Generalization of Graph Neural Networks
- Cross-Modal Generative Augmentation for Visual Question Answering
- HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
- Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation
- A Transformer-based Cross-modal Fusion Model with Adversarial Training for VQA Challenge 2021
- VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition
- Domain-robust VQA with diverse datasets and methods but no target labels
- LAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation
- Adversarial Evaluation of Multimodal Models under Realistic Gray Box Assumption