8 papers
Realtime-Venus: A full-duplex interaction system with asynchronous delegation
Ant Group, Ruixiang Zhao, Hualei Wang +25
Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video in…
Emotion Understanding in Streaming Video with Trajectory-Aware Reliability
Qingsong Wang, Qigong Lei, Zitong Wang +6
Video emotion understanding is commonly studied as an offline classification problem, where the complete video segment is available before prediction. Real-time interaction, howeve…
OneBar: An End-to-End Content-Grounded Generative Query Recommendation Framework for E-Commerce Video Feeds
Yao Tang, Ying Yang, Ben Chen +5
Short-video platforms now expose clickable search entries beneath the video player, enabling users to easily express content-induced search intent. However, conventional query reco…
ROAD: Adaptive Data Mixing for Offline-to-Online Reinforcement Learning via Bi-Level Optimization
Letian Yang, Xu Liu, Yiqiang Lu +3
Offline-to-online reinforcement learning harnesses the stability of offline pretraining and the flexibility of online fine-tuning. A key challenge lies in the non-stationary distri…
LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs
Keda Tao, Yuhua Zheng, Jia Xu +13
Recent advancements in omnimodal large language models (OmniLLMs) have significantly improved the comprehension of audio and video inputs. However, current evaluations primarily fo…
StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding
Xinqi Jin, Hanxun Yu, Bohan Yu +8
Online video understanding is essential for applications like public surveillance and AI glasses. However, applying Multimodal Large Language Models (MLLMs) to this domain is chall…
Active Perception Agent for Omnimodal Audio-Video Understanding
Keda Tao, Wenjie Du, Bohan Yu +3
Omnimodal large language models have made significant strides in unifying audio and visual modalities; however, they often face challenges in fine-grained cross-modal understanding…
OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models
Keda Tao, Kele Shao, Bohan Yu +3
Omnimodal large language models (OmniLLMs) have attracted increasing research attention of late towards unified audio-video understanding. However, the high computational cost of p…