3 citations · 4 across the 19 of their papers we have counts for
22 papers
PAWA: Panoramic Action-conditioned World Modeling for Multiple Agents
Siqi Yang, Yimu Wang, Ziqi Cai +6
A multiplayer world model must predict how the independent actions of several players change a shared environment, including changes outside a player's current field of view. A poi…
StreamAV-Bench: A Comprehensive Benchmark for Streaming Audio-Video Generation
Kaiqi Liu, Haoxuan Zeng, Jingqi Liu +7
Recent advancements in generative models are pushing video generation toward unbounded streaming audio-video generation for real-time interactive worlds. However, existing benchmar…
MASS: Multiplayer World Models with Authoritative Shared State
Ziqi Cai, Siqi Yang, Yimu Wang +6
Current video world models struggle in multiplayer environments because they entangle world state with view-dependent visual latents, leading to redundant compute, view inconsisten…
MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Control
Kaiqi Liu, Yunyao Mao, Ziqi Cai +8
While recent generative models produce high-fidelity videos, they struggle with the complex narrative control required for coherent multi-shot audio-visual generation. Existing met…
Video Generation Models Are Inherent Lighting Estimators
Ziqi Cai, Shuchen Weng, Kaiqi Liu +5
Recovering dynamic environment maps from a single in-the-wild video is crucial for photorealistic rendering, yet remains a challenge. Recent video generation models can produce pho…
MotionVLA: Vision-Language-Action Model for Humanoid Motion
Nonghai Zhang, Siyu Zhai, Yanjun Li +5
Generating realistic humanoid motion from scene images and text involves both low-frequency pose semantics and high-frequency physical dynamics. However, many existing methods toke…
InstructAV2AV: Instruction-Guided Audio-Video Joint Editing
Haojie Zheng, Yixin Yang, Siqi Yang +2
Recent diffusion-based methods have achieved impressive progress in video content manipulation. However, they typically ignore the accompanying audio, leaving the audio disjointed…
A Benchmark and Multi-Agent System for Instruction-driven Cinematic Video Compilation
Peixuan Zhang, Chang Zhou, Ziyuan Zhang +8
The surging demand for adapting long-form cinematic content into short videos has motivated the need for versatile automatic video compilation systems. However, existing compilatio…
ReContraster: Making Your Posters Stand Out with Regional Contrast
Peixuan Zhang, Zijian Jia, Ziqi Cai +3
Effective poster design requires rapidly capturing attention and clearly conveying messages. Inspired by the ``contrast effects'' principle, we propose ReContraster, the first trai…
Lighting-grounded Video Generation with Renderer-based Agent Reasoning
Ziqi Cai, Taoyu Yang, Zheng Chang +4
Diffusion models have achieved remarkable progress in video generation, but their controllability remains a major limitation. Key scene factors such as layout, lighting, and camera…
Towards Deeper Emotional Reflection: Crafting Affective Image Filters with Generative Priors
Peixuan Zhang, Shuchen Weng, Jiajun Tang +2
Social media platforms enable users to express emotions by posting text with accompanying images. In this paper, we propose the Affective Image Filter (AIF) task, which aims to ref…
STAGE: Storyboard-Anchored Generation for Cinematic Multi-shot Narrative
Peixuan Zhang, Zijian Jia, Kaiqi Liu +3
While recent advancements in generative models have achieved remarkable visual fidelity in video synthesis, creating coherent multi-shot narratives remains a significant challenge.…
AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner
Haojie Zheng, Shuchen Weng, Jingqi Liu +3
Recent advancements in video generation highlight that realistic audio-visual synchronization is crucial for engaging content creation. However, existing video editing methods larg…
Personalized Image Filter: Mastering Your Photographic Style
Chengxuan Zhu, Shuchen Weng, Jiacong Fang +4
Photographic style, as a composition of certain photographic concepts, is the charm behind renowned photographers. But learning and transferring photographic style need a profound…
PanoWan: Lifting Diffusion Video Generation Models to 360° with Latitude/Longitude-aware Mechanisms
Yifei Xia, Shuchen Weng, Siqi Yang +6
Panoramic video generation enables immersive 360° content creation, valuable in applications that demand scene-consistent world exploration. However, existing panoramic video gener…
Audio-Sync Video Generation with Multi-Stream Temporal Control
Shuchen Weng, Haojie Zheng, Zheng Chang +3
Audio is inherently temporal and closely synchronized with the visual world, making it a naturally aligned and expressive control signal for controllable video generation (e.g., mo…
Affective Image Editing: Shaping Emotional Factors via Text Descriptions
Peixuan Zhang, Shuchen Weng, Chengxuan Zhu +4
In daily life, images as common affective stimuli have widespread applications. Despite significant progress in text-driven image editing, there is limited work focusing on underst…
L-C4: Language-Based Video Colorization for Creative and Consistent Color
Zheng Chang, Shuchen Weng, Huan Ouyang +3
Automatic video colorization is inherently an ill-posed problem because each monochrome frame has multiple optional color candidates. Previous exemplar-based video colorization met…
VIRES: Video Instance Repainting via Sketch and Text Guided Generation
Shuchen Weng, Haojie Zheng, Peixuan Zhang +4
We introduce VIRES, a video instance repainting method with sketch and text guidance, enabling video instance repainting, replacement, generation, and removal. Existing approaches…
Technique Report of CVPR 2024 PBDL Challenges
Ying Fu, Yu Li, Shaodi You +96
The intersection of physics-based vision and deep learning presents an exciting frontier for advancing computer vision technologies. By leveraging the principles of physics to info…
Colorizing Monochromatic Radiance Fields
Yean Cheng, Renjie Wan, Shuchen Weng +3
Though Neural Radiance Fields (NeRF) can produce colorful 3D representations of the world by using a set of 2D images, such ability becomes non-existent when only monochromatic ima…
Language-guided Image Reflection Separation
Haofeng Zhong, Yuchen Hong, Shuchen Weng +2
This paper studies the problem of language-guided reflection separation, which aims at addressing the ill-posed reflection separation problem by introducing language descriptions t…