290 citations · 1.9k across the 59 of their papers we have counts for
21 papers · 1 filter
Wasserstein Contrastive Representation Distillation
Liqun Chen, Dong Wang, Zhe Gan +3
The primary goal of knowledge distillation (KD) is to encapsulate the information of a model learned from a teacher network into a student network, with the latter being more compa…
A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
Linjie Li, Zhe Gan, Jingjing Liu
Large-scale pre-trained multimodal transformers, such as ViLBERT and UNITER, have propelled the state of the art in vision-and-language (V+L) research to a new level. Although achi…
Cross-Thought for Sentence Encoder Pre-training
Shuohang Wang, Yuwei Fang, Siqi Sun +4
In this paper, we propose Cross-Thought, a novel approach to pre-training sequence encoder, which is instrumental in building reusable sequence embeddings for large-scale NLP tasks…
Multi-Fact Correction in Abstractive Text Summarization
Yue Dong, Shuohang Wang, Zhe Gan +3
Pre-trained neural abstractive summarization systems have dominated extractive strategies on news summarization performance, at least in terms of ROUGE. However, system-generated a…
InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
Boxin Wang, Shuohang Wang, Yu Cheng +4
Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies, however, show that such BERT-based models are…
Efficient Robust Training via Backward Smoothing
Jinghui Chen, Yu Cheng, Zhe Gan +2
Adversarial training is so far the most effective strategy in defending against adversarial examples. However, it suffers from high computational costs due to the iterative adversa…
Contrastive Distillation on Intermediate Representations for Language Model Compression
Siqi Sun, Zhe Gan, Yu Cheng +3
Existing language model compression methods mostly use a simple L2 loss to distill knowledge in the intermediate representations of a large BERT model to a smaller one. Although wi…
Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
Shuohang Wang, Luowei Zhou, Zhe Gan +5
Transformer has become ubiquitous in the deep learning field. One of the key ingredients that destined its success is the self-attention mechanism, which allows fully-connected con…
Accelerating Real-Time Question Answering via Question Generation
Yuwei Fang, Shuohang Wang, Zhe Gan +3
Although deep neural networks have achieved tremendous success for question answering (QA), they are still suffering from heavy computational and energy cost for real product deplo…
FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding
Yuwei Fang, Shuohang Wang, Zhe Gan +2
Large-scale cross-lingual language models (LM), such as mBERT, Unicoder and XLM, have achieved great success in cross-lingual representation learning. However, when applied to zero…
Graph Optimal Transport for Cross-Domain Alignment
Liqun Chen, Zhe Gan, Yu Cheng +3
Cross-domain alignment between two sets of entities (e.g., objects in an image, words in a sentence) is fundamental to both computer vision and natural language processing. Existin…
MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients
Chen Zhu, Yu Cheng, Zhe Gan +3
Adaptive gradient methods such as RMSProp and Adam use exponential moving estimate of the squared gradient to compute adaptive step sizes, achieving better convergence than SGD in…
Large-Scale Adversarial Training for Vision-and-Language Representation Learning
Zhe Gan, Yen-Chun Chen, Linjie Li +3
We present VILLA, the first known effort on large-scale adversarial training for vision-and-language (V+L) representation learning. VILLA consists of two training stages: (i) task-…
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
Jize Cao, Zhe Gan, Yu Cheng +3
Recent Transformer-based large-scale pre-trained models have revolutionized vision-and-language (V+L) research. Models such as ViLBERT, LXMERT and UNITER have significantly lifted…
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Linjie Li, Yen-Chun Chen, Yu Cheng +3
We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a…
APo-VAE: Text Generation in Hyperbolic Space
Shuyang Dai, Zhe Gan, Yu Cheng +3
Natural language often exhibits inherent hierarchical structure ingrained with complex syntax and semantics. However, most state-of-the-art deep generative models learn embeddings…
Contextual Text Style Transfer
Yu Cheng, Zhe Gan, Yizhe Zhang +3
We introduce a new task, Contextual Text Style Transfer - translating a sentence into a desired style with its surrounding context taken into account. This brings two key challenge…
BachGAN: High-Resolution Image Synthesis from Salient Object Layout
Yandong Li, Yu Cheng, Zhe Gan +3
We propose a new task towards more practical application for image generation - high-quality image synthesis from salient object layout. This new setting allows users to provide th…
VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
Jingzhou Liu, Wenhu Chen, Yu Cheng +4
We introduce a new task, Video-and-Language Inference, for joint multimodal understanding of video and text. Given a video clip with aligned subtitles as premise, paired with a nat…
Self-Guided Adaptation: Progressive Representation Alignment for Domain Adaptive Object Detection
Zongxian Li, Qixiang Ye, Chong Zhang +3
Unsupervised domain adaptation (UDA) has achieved unprecedented success in improving the cross-domain robustness of object detection models. However, existing UDA methods largely i…
Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching
Shuohang Wang, Yunshi Lan, Yi Tay +2
Transformer has been successfully applied to many natural language processing tasks. However, for textual sequence matching, simple matching between the representation of a pair of…