1 citations · 4 across the 27 of their papers we have counts for
12 papers · 1 filter
An Algorithm for Deterministic Weighted Regular Languages
Clemente Pasti, Talu Karagöz, Anej Svete +3
Extracting finite state automata (FSAs) from black-box models offers a powerful approach to gaining interpretable insights into complex model behaviors. To support this pursuit, we…
Gumbel Counterfactual Generation From Language Models
Shauli Ravfogel, Anej Svete, Vésteinn Snæbjarnarson +1
Understanding and manipulating the causal generation mechanisms in language models is essential for controlling their behavior. Previous work has primarily relied on techniques suc…
Training Neural Networks as Recognizers of Formal Languages
Alexandra Butoi, Ghazal Khalighinejad, Anej Svete +3
Characterizing the computational power of neural network architectures in terms of formal language theory remains a crucial line of research, as it describes lower and upper bounds…
Can Transformers Learn -gram Language Models?
Anej Svete, Nadav Borenstein, Mike Zhou +2
Much theoretical work has described the ability of transformers to represent formal languages. However, linking theoretical results to empirical performance is not straightforward…
Lower Bounds on the Expressivity of Recurrent Neural Language Models
Anej Svete, Franz Nowak, Anisha Mohamed Sahabdeen +1
The recent successes and spread of large neural language models (LMs) call for a thorough understanding of their computational ability. Describing their computational abilities thr…
On the Representational Capacity of Neural Language Models with Chain-of-Thought Reasoning
Franz Nowak, Anej Svete, Alexandra Butoi +1
The performance of modern language models (LMs) has been improved by chain-of-thought (CoT) reasoning, i.e., the process of generating intermediate results that guide the model tow…
A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors
Naaman Tan, Josef Valvoda, Tianyu Liu +4
The relationship between the quality of a string, as judged by a human reader, and its probability, under a language model undergirds the development of better…
What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages
Nadav Borenstein, Anej Svete, Robin Chan +5
What can large language models learn? By definition, language models (LM) are distributions over strings. Therefore, an intuitive way of addressing the above question is to formali…
On Affine Homotopy between Language Encoders
Robin SM Chan, Reda Boumasmoud, Anej Svete +8
Pre-trained language encoders -- functions that represent text as vectors -- are an integral component of many NLP tasks. We tackle a natural question in language encoder analysis:…
The Role of -gram Smoothing in the Age of Neural Networks
Luca Malagutti, Andrius Buinovskij, Anej Svete +3
For nearly three decades, language models derived from the -gram assumption held the state of the art on the task. The key to their success lay in the application of various smo…
Transformers Can Represent -gram Language Models
Anej Svete, Ryan Cotterell
Existing work has analyzed the representational capacity of the transformer architecture by means of formal models of computation. However, the focus so far has been on analyzing t…
On Efficiently Representing Regular Languages as RNNs
Anej Svete, Robin Shing Moon Chan, Ryan Cotterell
Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). It shows that RNNs can efficie…