142 citations · 512 across the 30 of their papers we have counts for
6 papers · 1 filter
BARTSmiles: Generative Masked Language Models for Molecular Representations
Gayane Chilingaryan, Hovhannes Tamoyan, Ani Tevosyan +6
We discover a robust self-supervised strategy tailored towards molecular representations for generative masked language models through a series of tailored, in-depth ablations. Usi…
Retrieval-Augmented Multimodal Language Modeling
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi +6
Recent multimodal models such as DALL-E and CM3 have achieved remarkable progress in text-to-image and image-to-text generation. However, these models store all learned knowledge (…
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer +1
Despite their wide adoption, the underlying training and memorization dynamics of very large language models is not well understood. We empirically study exact memorization in caus…
Improving Passage Retrieval with Zero-Shot Question Generation
Devendra Singh Sachan, Mike Lewis, Mandar Joshi +4
We propose a simple and effective re-ranking method for improving passage retrieval in open question answering. The re-ranker re-scores retrieved passages with a zero-shot question…
InCoder: A Generative Model for Code Infilling and Synthesis
Daniel Fried, Armen Aghajanyan, Jessy Lin +7
Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthe…
CM3: A Causal Masked Multimodal Model of the Internet
Armen Aghajanyan, Bernie Huang, Candace Ross +8
We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens. Our new…