425 citations · 757 across the 37 of their papers we have counts for
38 papers
Effective Long-Context Scaling of Foundation Models
Wenhan Xiong, Jingyu Liu, Igor Molybog +18
We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with l…
Code Llama: Open Foundation Models for Code
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle +23
We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for lar…
Prompting Large Language Models with Speech Recognition Abilities
Yassir Fathullah, Chunyang Wu, Egor Lakomkin +9
Large language models have proven themselves highly flexible, able to solve a wide range of generative tasks, such as abstractive summarization and open-ended question answering. I…
Multi-Head State Space Model for Speech Recognition
Yassir Fathullah, Chunyang Wu, Yuan Shangguan +8
State space models (SSMs) have recently shown promising results on small-scale sequence and language modelling tasks, rivalling and outperforming many attention-based approaches. I…
VideoOFA: Two-Stage Pre-Training for Video-to-Text Generation
Xilun Chen, Lili Yu, Wenhan Xiong +3
We propose a new two-stage pre-training framework for video-to-text generation tasks such as video captioning and video question answering: A generative encoder-decoder model is fi…
Text-guided 3D Human Generation from 2D Collections
Tsu-Jui Fu, Wenhan Xiong, Yixin Nie +3
3D human modeling has been widely used for engaging interaction in gaming, film, and animation. The customization of these characters is crucial for creativity and scalability, whi…
Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality
Harman Singh, Pengchuan Zhang, Qifan Wang +4
Contrastively trained vision-language models have achieved remarkable progress in vision and language representation learning, leading to state-of-the-art models for various downst…
CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Embedding
Jingyu Liu, Wenhan Xiong, Ian Jones +3
Indoor scene synthesis involves automatically picking and placing furniture appropriately on a floor plan, so that the scene looks realistic and is functionally plausible. Such sce…
Bridging the Training-Inference Gap for Dense Phrase Retrieval
Gyuwan Kim, Jinhyuk Lee, Barlas Oguz +4
Building dense retrievers requires a series of standard procedures, including training and validating neural models and creating indexes for efficient search. However, these proced…
Adapting Pretrained Text-to-Text Models for Long Text Sequences
Wenhan Xiong, Anchit Gupta, Shubham Toshniwal +2
We present an empirical study of adapting an existing pretrained text-to-text model for long-sequence inputs. Through a comprehensive study along three axes of the pretraining pipe…
SCROLLS: Standardized CompaRison Over Long Language Sequences
Uri Shaham, Elad Segal, Maor Ivgi +8
NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild. We intr…
Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
Tianyi Liu, Zuxuan Wu, Wenhan Xiong +2
Most existing vision-language pre-training methods focus on understanding tasks and use BERT-like objectives (masked language modeling and image-text matching) during pretraining.…
Boosted Dense Retriever
Patrick Lewis, Barlas Oğuz, Wenhan Xiong +3
We propose DrBoost, a dense retrieval ensemble inspired by boosting. DrBoost is trained in stages: each component model is learned sequentially and specialized by focusing only on…
Simple Local Attentions Remain Competitive for Long-Context Tasks
Wenhan Xiong, Barlas Oğuz, Anchit Gupta +5
Many NLP tasks require processing long contexts beyond the length limit of pretrained models. In order to scale these models to longer text sequences, many efficient long-range att…
Open-Domain Question-Answering for COVID-19 and Other Emergent Domains
Sharon Levy, Kevin Mo, Wenhan Xiong +1
Since late 2019, COVID-19 has quickly emerged as the newest biomedical domain, resulting in a surge of new information. As with other emergent domains, the discussion surrounding t…
Zero-shot Fact Verification by Claim Generation
Liangming Pan, Wenhu Chen, Wenhan Xiong +2
Neural models for automated fact verification have achieved promising results thanks to the availability of large, human-annotated datasets. However, for each new domain that requi…
Unsupervised Multi-hop Question Answering by Question Generation
Liangming Pan, Wenhu Chen, Wenhan Xiong +2
Obtaining training data for multi-hop question answering (QA) is time-consuming and resource-intensive. We explore the possibility to train a well-performed multi-hop QA model with…
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
Wenhan Xiong, Xiang Lorraine Li, Srini Iyer +8
We propose a simple and efficient multi-hop dense retrieval approach for answering complex open-domain questions, which achieves state-of-the-art performance on two multi-hop datas…
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering
Wenhan Xiong, Hong Wang, William Yang Wang
To extract answers from a large corpus, open-domain question answering (QA) systems usually rely on information retrieval (IR) techniques to narrow the search space. Standard inver…
Learning to Recover Reasoning Chains for Multi-Hop Question Answering via Cooperative Games
Yufei Feng, Mo Yu, Wenhan Xiong +6
We propose the new problem of learning to recover reasoning chains from weakly supervised signals, i.e., the question-answer pairs. We propose a cooperative game approach to deal w…
HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
Wenhu Chen, Hanwen Zha, Zhiyu Chen +3
Existing question answering datasets focus on dealing with homogeneous information, based either only on text or KB/Table information alone. However, as human knowledge is distribu…
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
Wenhan Xiong, Jingfei Du, William Yang Wang +1
Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition…
Simple yet Effective Bridge Reasoning for Open-Domain Multi-Hop Question Answering
Wenhan Xiong, Mo Yu, Xiaoxiao Guo +4
A key challenge of multi-hop question answering (QA) in the open-domain setting is to accurately retrieve the supporting passages from a large corpus. Existing work on open-domain…
Do Multi-hop Readers Dream of Reasoning Chains?
Haoyu Wang, Mo Yu, Xiaoxiao Guo +3
General Question Answering (QA) systems over texts require the multi-hop reasoning capability, i.e. the ability to reason with information collected from multiple passages to deriv…