1.1k citations · 1.1k across the 7 of their papers we have counts for
6 papers · 1 filter
Formal Aspects of Language Modeling
Ryan Cotterell, Anej Svete, Clara Meister +2
Large language models have become one of the most commonly deployed NLP inventions. In the past half-decade, their integration into core natural language processing tools has drama…
On the Representational Capacity of Recurrent Neural Language Models
Franz Nowak, Anej Svete, Li Du +1
This work investigates the computational expressivity of language models (LMs) based on recurrent neural networks (RNNs). Siegelmann and Sontag (1992) famously showed that RNNs wit…
Tokenization and the Noiseless Channel
Vilém Zouhar, Clara Meister, Juan Luis Gastaldi +3
Subword tokenization is a key part of many NLP pipelines. However, little is known about why some tokenizer and hyperparameter combinations lead to better downstream model performa…
A Formal Perspective on Byte-Pair Encoding
Vilém Zouhar, Clara Meister, Juan Luis Gastaldi +4
Byte-Pair Encoding (BPE) is a popular algorithm used for tokenizing data in NLP, despite being devised initially as a compression method. BPE appears to be a greedy algorithm at fa…
Structured Voronoi Sampling
Afra Amini, Li Du, Ryan Cotterell
Gradient-based sampling algorithms have demonstrated their effectiveness in text generation, especially in the context of controlled text generation. However, there exists a lack o…
Autoregressive Modeling with Lookahead Attention
Li Du, Hongyuan Mei, Jason Eisner
To predict the next token, autoregressive models ordinarily examine the past. Could they also benefit from also examining hypothetical futures? We consider a novel Transformer-base…