1.5k citations · 2.6k across the 9 of their papers we have counts for
11 papers
Text and Code Embeddings by Contrastive Pre-Training
Arvind Neelakantan, Tao Xu, Raul Puri +22
Text embeddings are useful features in many applications such as semantic search and computing text similarity. Previous work typically trains models customized for different use c…
Evaluating Large Language Models Trained on Code
Mark Chen, Jerry Tworek, Heewoo Jun +55
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex p…
Local Knowledge Powered Conversational Agents
Sashank Santhanam, Wei Ping, Raul Puri +3
State-of-the-art conversational agents have advanced significantly in conjunction with the use of large transformer-based language models. However, even with these advancements, co…
BioMegatron: Larger Biomedical Domain Language Model
Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina +4
There has been an influx of biomedical domain-specific language models, showing language models pre-trained on biomedical text perform better on biomedical domain benchmarks than t…
MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models
Peng Xu, Mostofa Patwary, Mohammad Shoeybi +4
Existing pre-trained large language models have shown unparalleled generative capabilities. However, they are not controllable. In this paper, we propose MEGATRON-CNTRL, a novel fr…
Large Scale Multi-Actor Generative Dialog Modeling
Alex Boyd, Raul Puri, Mohammad Shoeybi +2
Non-goal oriented dialog agents (i.e. chatbots) aim to produce varying and engaging conversations with a user; however, they typically exhibit either inconsistent personality acros…
Training Question Answering Models From Synthetic Data
Raul Puri, Ryan Spring, Mostofa Patwary +2
Question and answer generation is a data augmentation method that aims to improve question answering (QA) models given the limited amount of human labeled data. However, a consider…
Zero-shot Text Classification With Generative Language Models
Raul Puri, Bryan Catanzaro
This work investigates the use of natural language to enable zero-shot model adaptation to new tasks. We use text and metadata from social commenting platforms as a source for a si…
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Mohammad Shoeybi, Mostofa Patwary, Raul Puri +3
Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large…
Practical Text Classification With Large Pre-Trained Language Models
Neel Kant, Raul Puri, Nikolai Yakovenko +1
Multi-emotion sentiment classification is a natural language processing (NLP) problem with valuable use cases on real-world data. We demonstrate that large-scale unsupervised langu…
Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
Raul Puri, Robert Kirby, Nikolai Yakovenko +1
Recent work has shown how to train Convolutional Neural Networks (CNNs) rapidly on large image datasets, then transfer the knowledge gained from these models to a variety of tasks.…