Publications (32)
Go-with-the-Track: Video Compositing and Motion Control with Point Tracking
Koichi Namekata, Yash Kant, Zhizheng Liu +9
Filmmaking demands precise motion control and reference image compositing -- capabilities that existing methods treat separately. Point-track-conditioned image-to-video models rest…
Jointly Optimizing Color Rendition and In-Camera Backgrounds in an RGB Virtual Production Stage
Chloe LeGendre, Lukas Lepicovsky, Paul Debevec
While the LED panels used in virtual production systems can display vibrant imagery with a wide color gamut, they produce problematic color shifts when used as lighting due to thei…
ReAge3D: Re-Aging 3D Faces with View Consistency
Libing Zeng, Li Ma, Mingming He +3
We present a novel framework for realistic and controllable 3D face re-aging which produces highly detailed, identity-preserving results. Existing 3D editing methods, while effecti…
Detail Enhanced Gaussian Splatting for Large-Scale Volumetric Capture
Julien Philip, Li Ma, Pascal Clausen +8
We present a unique system for large-scale, multi-performer, high resolution 4D volumetric capture providing realistic free-viewpoint video up to and including 4K resolution facial…
Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise
Ryan Burgert, Yuancheng Xu, Wenqi Xian +10
Generative modeling aims to transform random noise into structured outputs. In this work, we enhance video diffusion models by allowing motion control via structured latent noise s…
Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset
Yiqun Mei, Mingming He, Li Ma +9
Video portrait relighting remains challenging because the results need to be both photorealistic and temporally stable. This typically requires a strong model design that can captu…
MAViS: A Multi-Agent Framework for Long-Sequence Video Storytelling
Qian Wang, Ziqi Huang, Ruoxi Jia +2
Despite recent advances, long-sequence video generation frameworks still suffer from significant limitations: poor assistive capability, suboptimal visual quality, and limited expr…
NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng +3
We address the problem of recovering the shape and spatially-varying reflectance of an object from multi-view images (and their camera poses) of an object illuminated by one unknow…
ID-V2V: Identity-Preserving Video Restylization
Yuancheng Xu, Mingming He, Pablo Salamanca +5
In visual storytelling, human performances are central to creative intent and narrative meaning. However, preserving human identity and performance while enabling flexible visual e…
CineScale: Free Lunch in High-Resolution Cinematic Visual Generation
Haonan Qiu, Ning Yu, Ziqi Huang +2
Visual diffusion models achieve remarkable progress, yet they are typically trained at limited resolutions due to the lack of high-resolution data and constrained computation resou…
Fitting Spherical Gaussians to Dynamic HDRI Sequences
Pascal Clausen, Li Ma, Mingming He +3
We present a technique for fitting high dynamic range illumination (HDRI) sequences using anisotropic spherical Gaussians (ASGs) while preserving temporal consistency in the compre…
Neural Light Transport for Relighting and View Synthesis
Xiuming Zhang, Sean Fanello, Yun-Ta Tsai +10
The light transport (LT) of a scene describes how it appears under different lighting and viewing directions, and complete knowledge of a scene's LT enables the synthesis of novel…
Light Stage Super-Resolution: Continuous High-Frequency Relighting
Tiancheng Sun, Zexiang Xu, Xiuming Zhang +6
The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subj…
BodyReLux: Temporally Consistent Full-Body Video Relighting
Li Ma, Mingming He, Xueming Yu +4
Being able to relight human performance is a fundamental task for post production and content creation. We present BodyReLux, a subject-specific video diffusion-based framework for…
Magenta Green Screen: Spectrally Multiplexed Alpha Matting with Deep Colorization
Dmitriy Smirnov, Chloe LeGendre, Xueming Yu +1
We introduce Magenta Green Screen, a novel machine learning--enabled matting technique for recording the color image of a foreground actor and a simultaneous high-quality alpha cha…
DiffHDR: Re-Exposing LDR Videos with Video Diffusion Models
Zhengming Yu, Li Ma, Mingming He +11
Most digital videos are stored in 8-bit low dynamic range (LDR) formats, where much of the original high dynamic range (HDR) scene radiance is lost due to saturation and quantizati…
HDR Lighting Dilation for Dynamic Range Reduction on Virtual Production Stages
Paul Debevec, Chloe LeGendre
We present a technique to reduce the dynamic range of an HDRI lighting environment map in an efficient, energy-preserving manner by spreading out the light of concentrated light so…
DeepView: View Synthesis with Learned Gradient Descent
John Flynn, Michael Broxton, Paul Debevec +5
We present a novel approach to view synthesis using multiplane images (MPIs). Building on recent advances in learned gradient descent, our algorithm generates an MPI from a set of…
DifFRelight: Diffusion-Based Facial Performance Relighting
Mingming He, Pascal Clausen, Ahmet Levent TaÅel +8
We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. Leveraging a subject-specific dataset containing div…
A System for Acquiring, Processing, and Rendering Panoramic Light Field Stills for Virtual Reality
Ryan S. Overbeck, Daniel Erickson, Daniel Evangelakos +2
We present a system for acquiring, processing, and rendering panoramic light field still photography for display in Virtual Reality (VR). We acquire spherical light field datasets…
Vista4D: Video Reshooting with 4D Point Clouds
Kuan Heng Lin, Zhizheng Liu, Pablo Salamanca +9
We present Vista4D, a robust and flexible video reshooting framework that grounds the input video and target cameras in a 4D point cloud. Specifically, given an input video, our me…
Learning Illumination from Diverse Portraits
Chloe LeGendre, Wan-Chun Ma, Rohit Pandey +5
We present a learning-based technique for estimating high dynamic range (HDR), omnidirectional illumination from a single low dynamic range (LDR) portrait image captured under arbi…
A New Dimension in Testimony: Relighting Video with Reflectance Field Exemplars
Loc Huynh, Bipin Kishore, Paul Debevec
We present a learning-based method for estimating 4D reflectance field of a person given video footage illuminated under a flat-lit environment of the same subject. For training da…
Single Image Portrait Relighting
Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai +7
Lighting plays a central role in conveying the essence and depth of the subject in a portrait photograph. Professional photographers will carefully control the lighting in their st…
Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Captures
Yuancheng Xu, Wenqi Xian, Li Ma +10
We introduce a framework that enables both multi-view character consistency and 3D camera control in video diffusion models through a novel customization data pipeline. We train th…
VChain: Chain-of-Visual-Thought for Reasoning in Video Generation
Ziqi Huang, Ning Yu, Gordon Chen +3
Recent video generation models can produce smooth and visually appealing clips, but they often struggle to synthesize complex dynamics with a coherent chain of consequences. Accura…
Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction
Youming Deng, Wenqi Xian, Guandao Yang +4
In this paper, we present a self-calibrating framework that jointly optimizes camera parameters, lens distortion and 3D Gaussian representations, enabling accurate and efficient sc…
FlashDepth: Real-time Streaming Video Depth Estimation at 2K Resolution
Gene Chou, Wenqi Xian, Guandao Yang +5
A versatile video depth estimation model should (1) be accurate and consistent across frames, (2) produce high-resolution depth maps, and (3) support real-time streaming. We propos…
Survey of Video Diffusion Models: Foundations, Implementations, and Applications
Yimu Wang, Xuye Liu, Wei Pang +4
Recent advances in diffusion models have revolutionized video generation, offering superior temporal consistency and visual quality compared to traditional generative adversarial n…
Baking Neural Radiance Fields for Real-Time View Synthesis
Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall +2
Neural volumetric representations such as Neural Radiance Fields (NeRF) have emerged as a compelling technique for learning to represent 3D scenes from images with the goal of rend…
Lighting in Motion: Spatiotemporal HDR Lighting Estimation
Christophe Bolduc, Julien Philip, Li Ma +3
We present Lighting in Motion (LiMo), a diffusion-based approach to spatiotemporal lighting estimation. LiMo targets both realistic high-frequency detail prediction and accurate il…
DeepLight: Learning Illumination for Unconstrained Mobile Mixed Reality
Chloe LeGendre, Wan-Chun Ma, Graham Fyffe +4
We present a learning-based method to infer plausible high dynamic range (HDR), omnidirectional illumination given an unconstrained, low dynamic range (LDR) image from a mobile pho…