6 citations · 18 across the 37 of their papers we have counts for
38 papers
Teaching Vision-Language Models to Use the Scale They Are Given: Label-Free Equivariance Training for Metric Physical Reasoning
Kaizhen Tan, Yang Feng, Heqing Du +3
Metric questions about video, such as the speed of a moving object, require a vision-language model to convert visual measurements into physical units using a real-world reference…
Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS
Yan Zhou, Yun Hong, Yang Feng
Natural-language instructions enable flexible control of synthesized speech, yet emotional TTS systems primarily model a single utterance-level affect, leaving multi-emotion contro…
Beyond Multimodal Alignment: Shared Physical Representations Across Sensors and Action Orders
Kaizhen Tan, Xin Xu, Siru Tao +4
Multimodal world models are often evaluated by whether different sensors produce similar representations. However, similar representations do not necessarily imply that the models…
What Can Latent World Models Know? Physical Information in Multimodal Predictive Representations
Kaizhen Tan, Sizhe Xu, Xin Xu +6
A central premise of latent world models is that predicting the future encourages representations to internalize the physics of their environment. We ask which physical quantities…
How Much Can a LoRA Adapter Memorize? Measuring Adapter Capacity in Bits
Kaizhen Tan, Heqing Du, Yang Feng
LoRA adapters are often shared on their own, and the amount of information they can hold about their training data bounds what sharing them can reveal. Following Morris et al. (202…
BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM
Qingkai Fang, Shoutao Guo, Yang Feng
Real-time, full-duplex speech interaction is a key feature of next-generation spoken chatbots, allowing the model to listen and speak at the same time and to handle natural phenome…
FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs
Yun Hong, Yan Zhou, Yang Feng
Empathy is essential for fostering natural interactions in spoken dialogue systems, as it enables machines to recognize the emotional tone of human speech and deliver empathetic re…
Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality
Mengyu Bu, Yang Feng
Large language models (LLMs) exhibit strong general intelligence, yet their multilingual performance remains highly imbalanced. Although LLMs encode substantial cross-lingual knowl…
From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception
Jilong Zhu, Yang Feng
While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general visual understanding, they frequently falter in fine-grained perception tasks th…
SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
Zhuofan Wen, Yang Feng
Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models (LLMs). Self-draft methods, which leverage the base LLM its…
Efficient Training for Cross-lingual Speech Language Models
Yan Zhou, Qingkai Fang, Yun Hong +1
Currently, large language models (LLMs) predominantly focus on the text modality. To enable more natural human-AI interaction, speech LLMs are emerging, but building effective end-…
Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs
Kaizhen Tan, Yang Feng, Heqing Du +3
Visual token pruning speeds up multimodal large language models (MLLMs) by keeping a small subset of the visual tokens, and pruning methods are compared by the accuracy they retain…
CREG: Compass Relational Evidence Graph for Characterizing Directional Structure in VLM Spatial-Reasoning Attribution
Kaizhen Tan, Yang Feng, Heqing Du
Standard attribution heatmaps show where a vision-language model (VLM) focuses, but they do not reveal whether the recovered evidence is organized by the queried spatial relation o…
IG-Pruning: Input-Guided Block Pruning for Large Language Models
Kangyu Qiao, Shaolei Zhang, Yang Feng
With the growing computational demands of large language models (LLMs), efficient inference has become increasingly critical for practical deployment. Depth pruning has emerged as…
AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment
Mengyu Bu, Shaolei Zhang, Zhongjun He +2
Multilingual large language models (LLMs) possess impressive multilingual understanding and generation capabilities. However, their performance and cross-lingual alignment often la…
PSO-Merging: Merging Models Based on Particle Swarm Optimization
Kehao Zhang, Shaolei Zhang, Yang Feng
Model merging has emerged as an efficient strategy for constructing multitask models by integrating the strengths of multiple available expert models, thereby reducing the need to…
StreamUni: Achieving Streaming Speech Translation with a Unified Large Speech-Language Model
Shoutao Guo, Xiang Li, Mengge Liu +2
Streaming speech translation (StreamST) requires determining appropriate timing, known as policy, to generate translations while continuously receiving source speech inputs, balanc…
FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing
Shoutao Guo, Shaolei Zhang, Qingkai Fang +3
The rapid advancement of Large Language Models (LLMs) has spurred significant progress in Large Speech-Language Models (LSLMs), enhancing their capabilities in both speech understa…
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
Shaolei Zhang, Shoutao Guo, Qingkai Fang +2
The emergence of GPT-4o-like large multimodal models (LMMs) has raised the exploration of integrating text, vision, and speech modalities to support more flexible multimodal intera…
LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis
Qingkai Fang, Yan Zhou, Shoutao Guo +2
Real-time, intelligent, and natural speech interaction is an essential part of the next-generation human-computer interaction. Recent advancements have showcased the potential of b…
LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
Shaolei Zhang, Qingkai Fang, Zhe Yang +1
The advent of real-time large multimodal models (LMMs) like GPT-4o has sparked considerable interest in efficient LMMs. LMM frameworks typically encode visual inputs into vision to…
Large Language Models Are Read/Write Policy-Makers for Simultaneous Generation
Shoutao Guo, Shaolei Zhang, Zhengrui Ma +1
Simultaneous generation models write generation results while reading streaming inputs, necessitating a policy-maker to determine the appropriate output timing. Existing simultaneo…
BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment
Shaolei Zhang, Kehao Zhang, Qingkai Fang +4
Large language models (LLMs), with their powerful generative capabilities and vast knowledge, empower various tasks in everyday life. However, these abilities are primarily concent…
Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models
Tian Yu, Shaolei Zhang, Yang Feng
Iterative retrieval refers to the process in which the model continuously queries the retriever during generation to enhance the relevance of the retrieved knowledge, thereby impro…