1 citations · 4 across the 27 of their papers we have counts for
5 papers · 1 filter
Formal Aspects of Language Modeling
Ryan Cotterell, Anej Svete, Clara Meister +2
Large language models have become one of the most commonly deployed NLP inventions. In the past half-decade, their integration into core natural language processing tools has drama…
On the Representational Capacity of Recurrent Neural Language Models
Franz Nowak, Anej Svete, Li Du +1
This work investigates the computational expressivity of language models (LMs) based on recurrent neural networks (RNNs). Siegelmann and Sontag (1992) famously showed that RNNs wit…
Recurrent Neural Language Models as Probabilistic Finite-state Automata
Anej Svete, Ryan Cotterell
Studying language models (LMs) in terms of well-understood formalisms allows us to precisely characterize their abilities and limitations. Previous work has investigated the repres…
A Geometric Notion of Causal Probing
Clément Guerner, Tianyu Liu, Anej Svete +2
The linear subspace hypothesis (Bolukbasi et al., 2016) states that, in a language model's representation space, all information about a concept such as verbal number is encoded in…
Algorithms for Acyclic Weighted Finite-State Automata with Failure Arcs
Anej Svete, Benjamin Dayan, Tim Vieira +2
Weighted finite-state automata (WSFAs) are commonly used in NLP. Failure transitions are a useful extension for compactly representing backoffs or interpolation in -gram models…