13 citations · 16 across the 8 of their papers we have counts for
8 papers
Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
Tina Behnia, Puneesh Deora, Christos Thrampoulidis
Language models are pretrained on sequences that blend statistical regularities (making text fluent) with factual associations between specific tokens (knowledge of facts). While r…
In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly
Puneesh Deora, Bhavya Vasudeva, Tina Behnia +1
In-context learning (ICL) enables transformers to adapt to new tasks through contextual examples without parameter updates. While existing research has typically studied ICL in fix…
Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations
Yize Zhao, Tina Behnia, Vala Vakilian +1
Next-token prediction (NTP) over large text corpora has become the go-to paradigm to train large language models. Yet, it remains unclear how NTP influences the mapping of linguist…
Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features
Tina Behnia, Christos Thrampoulidis
Recent findings reveal that over-parameterized deep neural networks, trained beyond zero training-error, exhibit a distinctive structural pattern at the final layer, termed as Neur…
Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching
Ganesh Ramachandra Kini, Vala Vakilian, Tina Behnia +2
Supervised contrastive loss (SCL) is a competitive and often superior alternative to the cross-entropy loss for classification. While prior studies have demonstrated that both loss…
On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Data
Tina Behnia, Ganesh Ramachandra Kini, Vala Vakilian +1
Various logit-adjusted parameterizations of the cross-entropy (CE) loss have been proposed as alternatives to weighted CE for training large models on label-imbalanced data far bey…
Imbalance Trouble: Revisiting Neural-Collapse Geometry
Christos Thrampoulidis, Ganesh R. Kini, Vala Vakilian +1
Neural Collapse refers to the remarkable structural properties characterizing the geometry of class embeddings and classifier weights, found by deep nets when trained beyond zero t…
On how to avoid exacerbating spurious correlations when models are overparameterized
Tina Behnia, Ke Wang, Christos Thrampoulidis
Overparameterized models fail to generalize well in the presence of data imbalance even when combined with traditional techniques for mitigating imbalances. This paper focuses on i…