8 citations · 17 across the 8 of their papers we have counts for
7 papers · 1 filter
Compute Optimal Tokenization
Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer +6
Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token, on this relationship remains underexplored. In this wo…
Tulu 3: Pushing Frontiers in Open Language Model Post-Training
Nathan Lambert, Jacob Morrison, Valentina Pyatkin +20
Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag…
Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models
Hila Gonen, Terra Blevins, Alisa Liu +2
Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. In this paper, we identify and characterize a phenomenon never discuss…
Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
Jonathan Hayase, Alisa Liu, Yejin Choi +2
The pretraining data of today's strongest language models is opaque; in particular, little is known about the proportions of various domains or languages represented. In this work,…
A Taxonomy of Ambiguity Types for NLP
Margaret Y. Li, Alisa Liu, Zhaofeng Wu +1
Ambiguity is an critical component of language that allows for more effective communication between speakers, but is often ignored in NLP. Recent work suggests that NLP systems may…
Tuning Language Models by Proxy
Alisa Liu, Xiaochuang Han, Yizhong Wang +3
Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors. However, tuning these m…