2.1k citations · 3.3k across the 55 of their papers we have counts for
61 papers
MOV-AAD: A Large-Scale Multimodal Dataset for Auditory Attention Decoding During Moving Conversations
Xiaomin He, Vishal Choudhari, Tristan J. Spratt +3
Auditory attention decoding (AAD) is often evaluated on static, simplified speech scenes that poorly match everyday listening. We introduce MOV-AAD, a large-scale dataset for study…
Grammatical "grandmother neurons" are rare in LLMs
Linyang He, Nima Mesgarani
Understanding how Large Language Models (LLMs) encode linguistic structures remains a fundamental challenge in interpretability research. While diagnostic classifiers (or "probes")…
Beyond the Stability--Plasticity Frontier in Streaming Target Speaker Extraction
Yuesheng Ma, Linyang He, Nima Mesgarani
Streaming target speaker extraction must maintain a representation of whom to extract while the target may fall silent, be masked by interference, or drift acoustically away from e…
Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO
Xilin Jiang, Riki Shimizu, Sukru Samet Dindar +3
Spoken dialog systems are typically designed for clean, dyadic interactions in which a single user and an assistant take turns speaking. Real-world social conversations, however, a…
Sympatheia: Emotionally Adaptive Voice Assistant with Continuous Affect Conditioning
Sukru Samet Dindar, Riki Shimizu, Xilin Jiang +1
Empathetic spoken dialogue systems must infer a user's emotional state to respond appropriately, yet everyday speech often carries weak, neutral, or ambiguous affective cues. To ad…
Prune, Interpret, Evaluate: A Cross-Layer Transcoder-Native Framework for Efficient Circuit Discovery via Feature Attribution
Qinhao Chen, Linyang He, Nima Mesgarani
Existing feature-interpretation pipelines typically operate on uniformly sampled units or exhaustive feature sets, incurring massive costs on units irrelevant to target behaviors.…
LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches
Linyang He, Qiyao Yu, Hanze Dong +5
Mathematical reasoning is a hallmark of human intelligence, and whether large language models (LLMs) can meaningfully perform it remains a central question in artificial intelligen…
AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs' Contextual and Cultural Knowledge and Thinking
Xilin Jiang, Qiaolin Wang, Junkai Wu +30
Internet audio-visual clips convey meaning through time-varying sound and motion, which extend beyond what text alone can represent. To examine whether AI models can understand suc…
LLMs Think with DAGs: Probing the Latent Graph Structure of Reasoning
Tianjun Zhong, Linyang He, Ziyang Li +1
Large language models often externalize reasoning as linear chains, but many reasoning problems can be more naturally modeled as directed acyclic graphs (DAGs), where intermediate…
MeanFlow-TSE: One-Step Generative Target Speaker Extraction with Mean Flow
Riki Shimizu, Xilin Jiang, Nima Mesgarani
Target speaker extraction (TSE) aims to isolate a desired speaker's voice from a multi-speaker mixture using auxiliary information such as a reference utterance. Although recent ad…
A cross-species neural foundation model for end-to-end speech decoding
Yizi Zhang, Linyang He, Chaofei Fan +9
Speech brain-computer interfaces (BCIs) aim to restore communication for people with paralysis by translating neural activity into text. Most systems use cascaded frameworks that d…
Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement
Linyang He, Tianjun Zhong, Richard Antonello +3
Understanding how the human brain progresses from processing simple linguistic inputs to performing high-level reasoning is a fundamental challenge in neuroscience. While modern la…
SightSound-R1: Cross-Modal Reasoning Distillation from Vision to Audio Language Models
Qiaolin Wang, Xilin Jiang, Linyang He +2
While large audio-language models (LALMs) have demonstrated state-of-the-art audio understanding, their reasoning capability in complex soundscapes still falls behind large vision-…
Layer-wise Minimal Pair Probing Reveals Contextual Grammatical-Conceptual Hierarchy in Speech Representations
Linyang He, Qiaolin Wang, Xilin Jiang +1
Transformer-based speech language models (SLMs) have significantly improved neural speech recognition and understanding. While existing research has examined how well SLMs encode s…
DMOSpeech 2: Reinforcement Learning for Duration Prediction in Metric-Optimized Speech Synthesis
Yinghao Aaron Li, Xilin Jiang, Fei Tao +4
Diffusion-based text-to-speech (TTS) systems have made remarkable progress in zero-shot speech synthesis, yet optimizing all components for perceptual metrics remains challenging.…
Interpretable Embeddings of Speech Enhance and Explain Brain Encoding Performance of Audio Models
Riki Shimizu, Richard J. Antonello, Chandan Singh +1
Speech foundation models (SFMs) are increasingly hailed as powerful computational models of human speech perception. However, since their representations are inherently black-box,…
Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG
Siavash Shams, Richard Antonello, Gavin Mischler +3
Decoding continuous language from neural signals remains a significant challenge in the intersection of neuroscience and artificial intelligence. We introduce Neuro2Semantic, a nov…
ZeroSep: Separate Anything in Audio with Zero Training
Chao Huang, Yuesheng Ma, Junxuan Huang +6
Audio source separation is fundamental for machines to understand complex acoustic environments and underpins numerous audio applications. Current supervised deep learning approach…
Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation
Xilin Jiang, Junkai Wu, Vishal Choudhari +1
Audio large language models (LLMs) are considered experts at recognizing sound objects, yet their performance relative to LLMs in other sensory modalities, such as visual or audio-…
XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs
Linyang He, Ercong Nie, Sukru Samet Dindar +10
We introduce XCOMPS in this work, a multilingual conceptual minimal pair dataset covering 17 languages. Using this dataset, we evaluate LLMs' multilingual conceptual understanding…
AAD-LLM: Neural Attention-Driven Auditory Scene Understanding
Xilin Jiang, Sukru Samet Dindar, Vishal Choudhari +6
Auditory foundation models, including auditory large language models (LLMs), process all sound inputs equally, independent of listener perception. However, human auditory perceptio…
Exploring Finetuned Audio-LLM on Heart Murmur Features
Adrian Florea, Xilin Jiang, Nima Mesgarani +1
Large language models (LLMs) for audio have excelled in recognizing and analyzing human speech, music, and environmental sounds. However, their potential for understanding other ty…
Large Language Models as Neurolinguistic Subjects: Discrepancy between Performance and Competence
Linyang He, Ercong Nie, Helmut Schmid +3
This study investigates the linguistic understanding of Large Language Models (LLMs) regarding signifier (form) and signified (meaning) by distinguishing two LLM assessment paradig…
Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue
Junkai Wu, Xulin Fan, Bo-Ru Lu +4
In recent years, we have observed a rapid advancement in speech language models (SpeechLLMs), catching up with humans' listening and reasoning abilities. SpeechLLMs have demonstrat…