1 citations · 2 across the 19 of their papers we have counts for
22 papers
Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios
Hongbo Liu, Peixian Chen, Sihan Liu +12
Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding. However, their ability to follow instructions in this domain remains under-explored.…
FIRM-Video: Check Before You Score for Reliable Text-to-Video Reward Modeling
Peiyuan Zhang, Xiangyu Zhao, Hongbo Liu +8
Reliable reward models are essential for text-to-video evaluation and alignment. However, the trade-off between evaluation accuracy and inference efficiency places high demands on…
Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activation
Hongbo Jiang, Jie Li, Yunhang Shen +2
Omni-Large Language Models (Omni-LLMs) power complex multi-modal reasoning in applications like World Action Models and autonomous agents. However, their strong performance often m…
Toward Native Multimodal Modeling: A Roadmap
Siyu An, Junru Lu, Junnan Dong +18
Multimodal modeling represents a vital step from modality-agnostic reasoning toward world modeling. While early approaches predominantly rely on late-fusion that assembles encoders…
GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistant
Zhuokang Shen, Yifan Wang, Hanyu Chen +6
Recent advances in large language models (LLMs) have enabled increasingly capable chatbots. However, most existing systems focus on single-user settings and do not generalize well…
Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
Chaoyou Fu, Haozhi Yuan, Yuhao Dong +16
With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and…
Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion
Lijiang Li, Zuwei Long, Yunhang Shen +6
While recent multimodal large language models (MLLMs) have made impressive strides, they predominantly employ a conventional autoregressive architecture as their backbone, leaving…
Can Unified Generation and Understanding Models Maintain Semantic Equivalence Across Different Output Modalities?
Hongbo Jiang, Jie Li, Yunhang Shen +4
Unified Multimodal Large Language Models (U-MLLMs) integrate understanding and generation within a single architecture. However, existing evaluations typically assess these capabil…
Unleashing MLLMs on the Edge: A Unified Framework for Cross-Modal ReID via Adaptive SVD Distillation
Hongbo Jiang, Jie Li, Xinqi Cai +4
Practical cloud-edge deployment of Cross-Modal Re-identification (CM-ReID) faces challenges due to maintaining a fragmented ecosystem of specialized cloud models for diverse modali…
SE-Search: Self-Evolving Search Agent via Memory and Dense Reward
Jian Li, Yizhang Jin, Dongqi Liu +9
Retrieval augmented generation (RAG) reduces hallucinations and factual errors in large language models (LLMs) by conditioning generation on retrieved external knowledge. Recent se…
SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM
An Yu, Weiheng Lu, Jian Li +4
Video Moment Retrieval is a task in video understanding that aims to localize a specific temporal segment in an untrimmed video based on a natural language query. Despite recent pr…
VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing, Speaking, and Acting
Xiaoyu Liu, Chaoyou Fu, Chi Yan +15
Current Vision-Language-Action (VLA) models are often constrained by a rigid, static interaction paradigm, which lacks the ability to see, hear, speak, and act concurrently as well…
AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy
Jinghang Shi, Xiaoyu Tang, Yang Huang +4
Astronomical image interpretation presents a significant challenge for applying multimodal large language models (MLLMs) to specialized scientific tasks. Existing benchmarks focus…
VITA-VLA: Efficiently Teaching Vision-Language Models to Act via Action Expert Distillation
Shaoqi Dong, Chaoyou Fu, Haihan Gao +12
Vision-Language Action (VLA) models significantly advance robotic manipulation by leveraging the strong perception capabilities of pretrained vision-language models (VLMs). By inte…
Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models
Yuansen Liu, Haiming Tang, Jinlong Peng +12
Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks. However, their capacity to comprehend human-centric scenes has rarely…
DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE
Hang Shao, Heting Gao, Yunhang Shen +5
Native multimodal large language models (MLLMs) restructure a single large language model (LLM) into a spoken language model (SLM) capable of both speech and text generation. Compa…
VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model
Zuwei Long, Yunhang Shen, Chaoyou Fu +11
With the growing requirement for natural human-computer interaction, speech-based systems receive increasing attention as speech is one of the most common forms of daily communicat…
Aligning Multimodal LLM with Human Preference: A Survey
Tao Yu, Yi-Fan Zhang, Chaoyou Fu +14
Large language models (LLMs) can handle a wide variety of general tasks with simple prompts, without the need for task-specific training. Multimodal Large Language Models (MLLMs),…
LUCY: Linguistic Understanding and Control Yielding Early Stage of Her
Heting Gao, Hang Shao, Xiong Wang +12
The film Her features Samantha, a sophisticated AI audio agent who is capable of understanding both linguistic and paralinguistic information in human speech and delivering real-ti…
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Chaoyou Fu, Haojia Lin, Xiong Wang +13
Recent Multimodal Large Language Models (MLLMs) have typically focused on integrating visual and textual modalities, with less emphasis placed on the role of speech in enhancing in…
Sparrow: Data-Efficient Video-LLM with Text-to-Image Augmentation
Shukang Yin, Chaoyou Fu, Sirui Zhao +7
Recent years have seen the success of Multimodal Large Language Models (MLLMs) in the domain of vision understanding. The success of these models can largely be attributed to the d…
Towards Artwork Explanation in Large-scale Vision Language Models
Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito +2
Large-scale Vision-Language Models (LVLMs) output text from images and instructions, demonstrating capabilities in text generation and comprehension. However, it has not been clari…