53 citations · 161 across the 15 of their papers we have counts for
5 papers · 1 filter
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation
Ludan Ruan, Yiyang Ma, Huan Yang +6
We propose the first joint audio-video generation framework that brings engaging watching and listening experiences simultaneously, towards high-quality realistic videos. To genera…
Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learning
Yuchong Sun, Hongwei Xue, Ruihua Song +3
Large-scale video-language pre-training has shown significant improvement in video-language understanding tasks. Previous studies of video-language pretraining mainly focus on shor…
AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation
Yiyang Ma, Huan Yang, Bei Liu +2
AI illustrator aims to automatically design visually appealing images for books to provoke rich thoughts and emotions. To achieve this goal, we propose a framework for translating…
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment
Hongwei Xue, Yuchong Sun, Bei Liu +4
The pre-trained image-text models, like CLIP, have demonstrated the strong power of vision-language representation learned from a large scale of web-collected image-text data. In l…
Language-Guided Face Animation by Recurrent StyleGAN-based Generator
Tiankai Hang, Huan Yang, Bei Liu +3
Recent works on language-guided image manipulation have shown great power of language in providing rich semantics, especially for face images. However, the other natural informatio…