2 citations · 3 across the 6 of their papers we have counts for
3 papers · 1 filter
The Geometry of Language Model Logits: Generic Structure and Learned Departures
Swathi Shree Narashiman, Aravind Ramana Venkatramanan, Nambi Seshadri
Autoregressive language models produce, at every prediction step, a vocabulary-sized logit vector whose structure determines the conditional next-token distribution. We empirically…
Induction Heads Interpolate N-Grams
Francesco D'Angelo, Oguz Kaan Yuksel, Swathi Shree Narashiman +1
Induction heads are attention circuits believed to underlie in-context learning in transformers, yet a precise characterization of the estimators they implement remains elusive. We…
Semantic Smoothing for Language Models via Distribution Estimation and Embeddings
Haricharan Balasundaram, Swathi Shree Narashiman, Pranay Mathur +1
We propose semantic smoothing, a smoothing method for language models that uses embeddings to share statistical observations across semantically similar contexts. The starting poin…