222 citations · 554 across the 23 of their papers we have counts for
10 papers · 1 filter
Cinematic Behavior Transfer via NeRF-based Differentiable Filming
Xuekun Jiang, Anyi Rao, Jingbo Wang +2
In the evolving landscape of digital media and video production, the precise manipulation and reproduction of visual elements like camera movements and character actions are highly…
SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models
Yuwei Guo, Ceyuan Yang, Anyi Rao +3
The development of text-to-video (T2V), i.e., generating videos with a given text prompt, has been significantly advanced in recent years. However, relying solely on text prompts o…
Automated Conversion of Music Videos into Lyric Videos
Jiaju Ma, Anyi Rao, Li-Yi Wei +3
Musicians and fans often produce lyric videos, a form of music videos that showcase the song's lyrics, for their favorite songs. However, making such videos can be challenging and…
Zero-shot Skeleton-based Action Recognition via Mutual Information Estimation and Maximization
Yujie Zhou, Wenwen Qiang, Anyi Rao +3
Zero-shot skeleton-based action recognition aims to recognize actions of unseen categories after training on data of seen categories. The key is to build the connection between vis…
AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
Yuwei Guo, Ceyuan Yang, Anyi Rao +6
With the advance of text-to-image (T2I) diffusion models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest th…
HireVAE: An Online and Adaptive Factor Model Based on Hierarchical and Regime-Switch VAE
Zikai Wei, Anyi Rao, Bo Dai +1
Factor model is a fundamental investment tool in quantitative investment, which can be empowered by deep learning to become more flexible and efficient in practical complicated inv…
CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers
Dachuan Shi, Chaofan Tao, Anyi Rao +3
Recent vision-language models have achieved tremendous advances. However, their computational costs are also escalating dramatically, making model acceleration exceedingly critical…
Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
Yujie Zhou, Haodong Duan, Anyi Rao +2
Self-supervised learning has demonstrated remarkable capability in representation learning for skeleton-based action recognition. Existing methods mainly focus on applying global d…
Adding Conditional Control to Text-to-Image Diffusion Models
Lvmin Zhang, Anyi Rao, Maneesh Agrawala
We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready…
Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production
Anyi Rao, Xuekun Jiang, Yuwei Guo +5
Amateurs working on mini-films and short-form videos usually spend lots of time and effort on the multi-round complicated process of setting and adjusting scenes, plots, and camera…