Publications (23)
Finetuned Language Models Are Zero-Shot Learners
Jason Wei, Maarten Bosma, Vincent Y. Zhao +6
This paper explores a simple method for improving the zero-shot learning abilities of language models. We show that instruction tuning -- finetuning language models on a collection…
The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text
Nikhil Kandpal, Brian Lester, Colin Raffel +24
Large language models (LLMs) are typically trained on enormous quantities of unlicensed text, a practice that has led to scrutiny due to possible intellectual property infringement…
Realistic Evaluation of Model Merging for Compositional Generalization
Derek Tam, Yash Kant, Brian Lester +2
Merging has become a widespread way to cheaply combine individual models into a single model that inherits their capabilities and attains better performance. This popularity has sp…
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
Tu Vu, Brian Lester, Noah Constant +2
There has been growing interest in parameter-efficient methods to apply pre-trained language models to downstream tasks. Building on the Prompt Tuning approach of Lester et al. (20…
Advancements in Constitutive Model Calibration: Leveraging the Power of Full-Field DIC Measurements and In-Situ Load Path Selection for Reliable Parameter Inference
Denielle Ricciardi, D. Tom Seidl, Brian Lester +2
Accurate material characterization and model calibration are essential for computationally-supported engineering decisions. Current characterization and calibration methods (1) use…
Dynamics of magnetization at infinite temperature in a Heisenberg spin chain
Eliott Rosenberg, Trond Andersen, Rhine Samajdar +178
Understanding universal aspects of quantum dynamics is an unresolved problem in statistical mechanics. In particular, the spin dynamics of the 1D Heisenberg model were conjectured…
iobes: A Library for Span-Level Processing
Brian Lester
Many tasks in natural language processing, such as named entity recognition and slot-filling, involve identifying and labeling specific spans of text. In order to leverage common m…
Leader: Prefixing a Length for Faster Word Vector Serialization
Brian Lester
Two competing file formats have become the de facto standards for distributing pre-trained word embeddings. Both are named after the most popular pre-trained embeddings that are di…
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation
Tu Vu, Aditya Barua, Brian Lester +3
In this paper, we explore the challenging problem of performing a generative task in a target language when labeled data is only available in English, using summarization as a case…
Constrained Decoding for Computationally Efficient Named Entity Recognition Taggers
Brian Lester, Daniel Pressel, Amy Hemmeter +2
Current state-of-the-art models for named entity recognition (NER) are neural models with a conditional random field (CRF) as the final layer. Entities are represented as per-token…
An Effective Label Noise Model for DNN Text Classification
Ishan Jindal, Daniel Pressel, Brian Lester +1
Because large, human-annotated datasets suffer from labeling errors, it is crucial to be able to train deep neural networks in the presence of label noise. While training image cla…
The Power of Scale for Parameter-Efficient Prompt Tuning
Brian Lester, Rami Al-Rfou, Noah Constant
In this work, we explore "prompt tuning", a simple yet effective mechanism for learning "soft prompts" to condition frozen language models to perform specific downstream tasks. Unl…
Reducing Retraining by Recycling Parameter-Efficient Prompts
Brian Lester, Joshua Yurtsever, Siamak Shakeri +1
Parameter-efficient methods are able to use a single frozen pre-trained large language model (LLM) to perform many tasks by learning task-specific soft prompts that modulate model…
Scaling Up Models and Data with and
Adam Roberts, Hyung Won Chung, Anselm Levskaya +40
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves. Scaling can…
Intent Features for Rich Natural Language Understanding
Brian Lester, Sagnik Ray Choudhury, Rashmi Prasad +1
Complex natural language understanding modules in dialog systems have a richer understanding of user utterances, and thus are critical in providing a better user experience. Howeve…
TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
Gül Sena AltıntaÅ, Malikeh Ehghaghi, Brian Lester +4
Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performanc…
Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models
Nikhil Kandpal, Brian Lester, Mohammed Muqeeth +6
Currently, most machine learning models are trained by centralized teams and are rarely updated. In contrast, open-source software development involves the iterative development of…
Training LLMs over Neurally Compressed Text
Brian Lester, Jaehoon Lee, Alex Alemi +4
In this paper, we explore the idea of training large language models (LLMs) over highly compressed text. While standard subword tokenizers compress text by a small factor, neural t…
Computationally Efficient NER Taggers with Combined Embeddings and Constrained Decoding
Brian Lester, Daniel Pressel, Amy Hemmeter +1
Current State-of-the-Art models in Named Entity Recognition (NER) are neural models with a Conditional Random Field (CRF) as the final network layer, and pre-trained "contextual em…
Search for plant biomagnetism with a sensitive atomic magnetometer
Eric Corsini, Victor Acosta, Nicolas Baddour +6
We report what we believe is the first experimental limit placed on plant biomagnetism. Measurements with a sensitive atomic magnetometer were performed on the Titan arum (Amorphop…
Multiple Word Embeddings for Increased Diversity of Representation
Brian Lester, Daniel Pressel, Amy Hemmeter +2
Most state-of-the-art models in natural language processing (NLP) are neural models built on top of large, pre-trained, contextual language models that generate representations of…
Precision quantum simulation of magnon spectra and interactions
Trond I. Andersen, Nikita Astrakhantsev, Jeronimo Martinez +329
Quantum simulation promises to advance materials discovery by accurately simulating complex states of matter, their microscopic excitations, and macroscopic response functions. The…
Bayesian Optimal Experimental Design for Constitutive Model Calibration
Denielle Ricciardi, Tom Seidl, Brian Lester +2
Computational simulation is increasingly relied upon for high-consequence engineering decisions, and a foundational element to solid mechanics simulations, such as finite element a…