6 papers
AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation
Kien T. Pham, I Chieh Chen, Qifeng Chen +1
Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic…
AnimationBench: Are Video Models Good at Character-Centric Animation?
Leyi Wu, Pengjun Fang, Kai Sun +8
Video generation has advanced rapidly, with recent methods producing increasingly convincing animated results. However, existing benchmarks-largely designed for realistic videos-st…
SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generation
Kien T. Pham, Yingqing He, Yazhou Xing +2
Audio-driven video generation aims to synthesize realistic videos that align with input audio recordings, akin to the human ability to visualize scenes from auditory input. However…
LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models
Qiang Wen, Zhefan Rao, Yazhou Xing +1
Enhancing a low-light noisy RAW image into a well-exposed and clean sRGB image is a significant challenge for modern digital cameras. Prior approaches have difficulties in recoveri…
ModelGrow: Continual Text-to-Video Pre-training with Model Expansion and Language Understanding Enhancement
Zhefan Rao, Liya Ji, Yazhou Xing +6
Text-to-video (T2V) generation has gained significant attention recently. However, the costs of training a T2V model from scratch remain persistently high, and there is considerabl…
Large Motion Video Autoencoding with Cross-modal Video VAE
Yazhou Xing, Yang Fei, Yingqing He +4
Learning a robust video Variational Autoencoder (VAE) is essential for reducing video redundancy and facilitating efficient video generation. Directly applying image VAEs to indivi…