10 papers
From Found to Designed: Concepts as a Design Axis for Large Language Models
Chen Shani
The paper examines how large language models implicitly encode concept-like information and proposes a taxonomy of concept-aware interventions, advocating for designing LLMs with e…
Categorize Early, Integrate Late: Divergent Processing Strategies in Automatic Speech Recognition
Nathan Roll, Pranav Bhalerao, Martijn Bartelds +7
In speech language modeling, two architectures dominate the frontier: the Transformer and the Conformer. However, it remains unknown whether their comparable performance stems from…
Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Models
Christine Zhang, Dan Jurafsky, Chen Shani
The next-token prediction (NTP) objective trains language models to predict a single token at each step, even though many continuations can express the same meaning. For example, i…
When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Models
Haoran Zheng, Chen Shani
Recurrent-attention hybrids aim to combine the efficiency of recurrence with the expressivity of attention, but existing approaches typically apply attention uniformly across all p…
The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?
Chen Shani, Yuval Reif, Nathan Roll +2
Multilingual language models (LMs) promise broader NLP access, yet current systems deliver uneven performance across the world's languages. This survey examines why these gaps pers…
Beyond Tokens: Concept-Level Training Objectives for LLMs
Laya Iyer, Pranav Somani, Alice Guo +2
The next-token prediction (NTP) objective has been foundational in the development of modern large language models (LLMs), driving advances in fluency and generalization. However,…