10 papers
SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation
Kaisi Guan, Ying Ba, Wenjing Wang +4
Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated speech and sound often tempor…
FATE: Frame-Level Audio-Visual Temporal Embedding
Kaisi Guan, Bingzi Zhang, Xihua Wang +4
When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations…
AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression
Yijing Chen, Wenhui Tan, Xiaoyi Yu +7
Multimodal Large Language Models have achieved remarkable progress in short-form audio-video understanding, yet long-form audio-video comprehension remains challenged by limited co…
SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learning
Xin Cheng, Xihua Wang, Ying Ba +5
Recent advancements in video-audio joint generation have achieved remarkable success in semantic correspondence. However, achieving precise temporal synchronization, which requires…
HuM-Eval: A Coarse-to-Fine Framework for Human-Centric Video Evaluation
Bingzi Zhang, Kaisi Guan, Ruihua Song
Video generation models have developed rapidly in recent years, where generating natural human motion plays a pivotal role. However, accurately evaluating the quality of generated…
ChronusOmni: Improving Time Awareness of Omni Large Language Models
Yijing Chen, Yihan Wu, Kaisi Guan +4
Time awareness is a fundamental ability of omni large language models, especially for understanding long videos and answering complex questions. Previous approaches mainly target v…
Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction
Kaisi Guan, Xihua Wang, Zhengfeng Lai +5
This study focuses on a challenging yet promising task, Text-to-Sounding-Video (T2SV) generation, which aims to generate a video with synchronized audio from text conditions, meanw…
VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learning
Xin Cheng, Yuyue Wang, Xihua Wang +7
Video-conditioned audio generation, including Video-to-Sound (V2S) and Visual Text-to-Speech (VisualTTS), has traditionally been treated as distinct tasks, leaving the potential fo…
ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question Generation and Answering
Kaisi Guan, Zhengfeng Lai, Yuchong Sun +5
Precisely evaluating semantic alignment between text prompts and generated videos remains a challenge in Text-to-Video (T2V) Generation. Existing text-to-video alignment metrics li…
BSharedRAG: Backbone Shared Retrieval-Augmented Generation for the E-commerce Domain
Kaisi Guan, Qian Cao, Yuchong Sun +2
Retrieval Augmented Generation (RAG) system is important in domains such as e-commerce, which has many long-tail entities and frequently updated information. Most existing works ad…