2.2k citations · 10.9k across the 144 of their papers we have counts for
153 papers
Multimodal Foundation Models: From Specialists to General-Purpose Assistants
Chunyuan Li, Zhe Gan, Zhengyuan Yang +4
This paper presents a comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities, focusing on the t…
An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
Yadong Lu, Chunyuan Li, Haotian Liu +3
Visual instruction tuning has recently shown encouraging progress with open-source large multimodal models (LMM) such as LLaVA and MiniGPT-4. However, most existing studies of open…
K-VIL: Keypoints-based Visual Imitation Learning
Jianfeng Gao, Zhi Tao, Noémie Jaquier +1
Visual imitation learning provides efficient and intuitive solutions for robotic systems to acquire novel manipulation skills. However, simultaneously learning geometric task const…
Data Augmentation for Abstractive Query-Focused Multi-Document Summarization
Ramakanth Pasunuru, Asli Celikyilmaz, Michel Galley +4
The progress in Query-focused Multi-Document Summarization (QMDS) has been limited by the lack of sufficient largescale high-quality training datasets. We present two QMDS training…
Learning to Shift Attention for Motion Generation
You Zhou, Jianfeng Gao, Tamim Asfour
One challenge of motion generation using robot learning from demonstration techniques is that human demonstrations follow a distribution with multiple modes for one task query. Pre…
Self-supervised Pre-training with Hard Examples Improves Visual Representations
Chunyuan Li, Xiujun Li, Lei Zhang +3
Self-supervised pre-training (SSP) employs random image transformations to generate training data for visual representation learning. In this paper, we first present a modeling fra…
VinVL: Revisiting Visual Representations in Vision-Language Models
Pengchuan Zhang, Xiujun Li, Xiaowei Hu +5
This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric re…
Rider: Reader-Guided Passage Reranking for Open-Domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu +4
Current open-domain question answering systems often follow a Retriever-Reader architecture, where the retriever first retrieves relevant passages and the reader then reads the ret…
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Hao Cheng, Yelong Shen, Xiaodong Liu +3
To date, most of recent work under the retrieval-reader framework for open-domain QA focuses on either extractive or generative reader exclusively. In this paper, we study a hybrid…
Few-Shot Named Entity Recognition: A Comprehensive Study
Jiaxin Huang, Chunyuan Li, Krishan Subudhi +6
This paper presents a comprehensive study to efficiently build named entity recognition (NER) systems when a small number of in-domain labeled data is available. Based upon recent…
RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems
Baolin Peng, Chunyuan Li, Zhu Zhang +3
For task-oriented dialog systems to be maximally useful, it must be able to process conversations in a way that is (1) generalizable with a small number of training examples for ne…
MiniVLM: A Smaller and Faster Vision-Language Model
Jianfeng Wang, Xiaowei Hu, Pengchuan Zhang +5
Recent vision-language (VL) studies have shown remarkable progress by learning generic representations from massive image-text pairs with transformer models and then fine-tuning on…
Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language
Hassan Akbari, Hamid Palangi, Jianwei Yang +6
Neuro-symbolic representations have proved effective in learning structure information in vision and language. In this paper, we propose a new model architecture for learning multi…
Overview of the Ninth Dialog System Technology Challenge: DSTC9
Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D'Haro +36
This paper introduces the Ninth Dialog System Technology Challenge (DSTC-9). This edition of the DSTC focuses on applying end-to-end dialog technologies for four distinct tasks in…
CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search
Chenyan Xiong, Zhenghao Liu, Si Sun +7
Neural rankers based on deep pretrained language models (LMs) have been shown to improve many information retrieval benchmarks. However, these methods are affected by their the cor…
Text Editing by Command
Felix Faltings, Michel Galley, Gerold Hintz +4
A prevailing paradigm in neural text generation is one-shot generation, where text is produced in a single step. The one-shot setting is inadequate, however, when the constraints t…
MagGAN: High-Resolution Face Attribute Editing with Mask-Guided Generative Adversarial Network
Yi Wei, Zhe Gan, Wenbo Li +5
We present Mask-guided Generative Adversarial Network (MagGAN) for high-resolution face attribute editing, in which semantic facial masks from a pre-trained face parser are used to…
GO FIGURE: A Meta Evaluation of Factuality in Summarization
Saadia Gabriel, Asli Celikyilmaz, Rahul Jha +2
While neural language models can generate text with remarkable fluency and coherence, controlling for factual correctness in generation remains an open research question. This majo…
Posterior Differential Regularization with f-divergence for Improving Model Robustness
Hao Cheng, Xiaodong Liu, Lis Pereira +2
We address the problem of enhancing model robustness through regularization. Specifically, we focus on methods that regularize the model posterior difference between clean and nois…
Robust Conversational AI with Grounded Text Generation
Jianfeng Gao, Baolin Peng, Chunyuan Li +4
This article presents a hybrid approach based on a Grounded Text Generation (GTG) model to building robust task bots at scale. GTG is a hybrid model which uses a large-scale Transf…
VIVO: Visual Vocabulary Pre-Training for Novel Object Captioning
Xiaowei Hu, Xi Yin, Kevin Lin +4
It is highly desirable yet challenging to generate image captions that can describe novel objects which are unseen in caption-labeled training data, a capability that is evaluated…
Generation-Augmented Retrieval for Open-domain Question Answering
Yuning Mao, Pengcheng He, Xiaodong Liu +4
We propose Generation-Augmented Retrieval (GAR) for answering open-domain questions, which augments a query through text generation of heuristically discovered relevant contexts wi…
HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Sanxing Chen, Xiaodong Liu, Jianfeng Gao +3
This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph. We propose HittER, a Hierarchical T…
Very Deep Transformers for Neural Machine Translation
Xiaodong Liu, Kevin Duh, Liyuan Liu +1
We explore the application of very deep Transformer models for Neural Machine Translation (NMT). Using a simple yet effective initialization technique that stabilizes training, we…