7 papers
Bearing Syntactic Fruit with Stack-Augmented Neural Networks
Brian DuSell, Ryan Cotterell
When children learn language, they make syntactic generalizations based on hierarchical rules. A recent line of work has inquired as to whether common neural network architectures…
Language Models over Canonical Byte-Pair Encodings
Tim Vieira, Tianyu Liu, Clemente Pasti +7
Modern language models represent probability distributions over character strings as distributions over (shorter) token strings derived via a deterministic tokenizer, such as byte-…
Information Locality as an Inductive Bias for Neural Language Models
Taiga Someya, Anej Svete, Brian DuSell +3
Inductive biases are inherent in every machine learning system, shaping how models generalize from finite data. In the case of neural language models (LMs), debates persist as to w…
From Language Models over Tokens to Language Models over Characters
Tim Vieira, Ben LeBrun, Mario Giulianelli +5
Modern language models are internally -- and mathematically -- distributions over strings rather than strings, posing numerous challenges for programm…
On the Proper Treatment of Tokenization in Psycholinguistics
Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi +3
Language models are widely used in computational psycholinguistics to test theories that relate the negative log probability (the surprisal) of a region of interest (a substring of…
PILA: A Historical-Linguistic Dataset of Proto-Italic and Latin
Stephen Bothwell, Brian DuSell, David Chiang +1
Computational historical linguistics seeks to systematically understand processes of sound change, including during periods at which little to no formal recording of language is at…
Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns
Brian DuSell, David Chiang
Attention, specifically scaled dot-product attention, has proven effective for natural language, but it does not have a mechanism for handling hierarchical patterns of arbitrary ne…