1 citations · 1 across the 10 of their papers we have counts for
14 papers
Toward a Universal Local Speech Feature Extractor Through Distillation
Jessica Yang, Marcelo Beramendi Caballero, Ju-Chieh Chou +1
Self-supervised speech models commonly consist of a set of convolutional neural network (CNN) layers used as a local feature extractor, followed by transformer layers. The CNN laye…
Encoder-Free Multimodal LLMs Learn Implicit Encoders
David Harwath, Desh Raj, Xavier Velez +5
We empirically analyze two recently proposed encoder-free multimodal LLMs, Gemma-4 12B and Inkling-Small. Unlike previous multimodal LLMs, these models do not use modality-specific…
Distilling Text LMs into Speech-to-Speech LMs by Matching Internal Text Predictions
Ju-Chieh Chou, Jiawei Zhou, Karen Livescu
Text language models (LMs) acquire broad linguistic and factual capabilities through large-scale pretraining, while comparable capabilities remain difficult to learn from speech al…
Distilling Sequential Computation in Transformer Language Models
Zixuan Lan, Jessica Yang, Yanhong Li +2
Transformer language models process sequences token by token in an autoregressive manner, making growing contexts increasingly expensive. Yet many adjacent token spans are highly p…
Quantifying the Generation Modality Gap in Speech-Text Language Models
Ju-Chieh Chou, Jiawei Zhou, Karen Livescu
Pure speech language models often lag behind text and speech-text language models in generating coherent content, but this gap is difficult to quantify because speech and text syst…
MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models
Chung-Ming Chien, Manu Orsini, Eugene Kharitonov +3
Speech-to-speech language models have recently emerged to enhance the naturalness of conversational AI. In particular, full-duplex models are distinguished by their real-time inter…
Cross-Modal Taxonomic Generalization in (Vision-) Language Models
Tianyang Xu, Marcelo Sandoval-Castaneda, Karen Livescu +2
What is the interplay between semantic representations learned by language models (LM) from surface form alone to those learned from more grounded evidence? We study this question…
OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking
Yanhong Li, Tianyang Xu, Kenan Tang +3
Knowledge-intensive question answering is central to large language models (LLMs) and is typically assessed using static benchmarks derived from sources like Wikipedia and textbook…
Transcribe, Translate, or Transliterate: An Investigation of Intermediate Representations in Spoken Language Models
Tolúlopé Ògúnrèmí, Christopher D. Manning, Dan Jurafsky +1
Spoken language models (SLMs) that integrate speech with large language models (LMs) rely on modality adapters (MAs) to map the output of speech encoders to a representation that i…
The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties
William Chen, Chutong Meng, Jiatong Shi +10
Recent improvements in multilingual ASR have not been equally distributed across languages and language varieties. To advance state-of-the-art (SOTA) ASR models, we present the Int…
Flow-SLM: Joint Learning of Linguistic and Acoustic Information for Spoken Language Modeling
Ju-Chieh Chou, Jiawei Zhou, Karen Livescu
Textless spoken language models (SLMs) are generative models of speech that do not rely on text supervision. Most textless SLMs learn to predict the next semantic token, a discrete…
On the Predictive Power of Representation Dispersion in Language Models
Yanhong Li, Ming Li, Karen Livescu +1
We show that a language model's ability to predict text is tightly linked to the breadth of its embedding space: models that spread their contextual representations more widely ten…
Chunk-Distilled Language Modeling
Yanhong Li, Karen Livescu, Jiawei Zhou
We introduce Chunk-Distilled Language Modeling (CD-LM), an approach to text generation that addresses two challenges in current large language models (LLMs): the inefficiency of to…
Approaching Deep Learning through the Spectral Dynamics of Weights
David Yunis, Kumar Kshitij Patel, Samuel Wheeler +5
We propose an empirical approach centered on the spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to unify and clarify several phe…