61 citations · 133 across the 52 of their papers we have counts for
25 papers · 1 filter
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
Zhen Li, Mingdeng Cao, Xintao Wang +3
Recent advances in text-to-image generation have made remarkable progress in synthesizing realistic human photos conditioned on given text prompts. However, existing personalized g…
AFL-Net: Integrating Audio, Facial, and Lip Modalities with a Two-step Cross-attention for Robust Speaker Diarization in the Wild
Yongkang Yin, Xu Li, Ying Shan +1
Speaker diarization in real-world videos presents significant challenges due to varying acoustic conditions, diverse scenes, the presence of off-screen speakers, etc. This paper bu…
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation
Zhouxia Wang, Ziyang Yuan, Xintao Wang +4
Motions in a video primarily consist of camera motion, induced by camera movement, and object motion, resulting from object movement. Accurate control of both camera and object mot…
ConTex-Human: Free-View Rendering of Human from a Single Image with Texture-Consistent Synthesis
Xiangjun Gao, Xiaoyu Li, Chaopeng Zhang +4
In this work, we propose a method to address the challenge of rendering a 3D human from a single image in a free-view manner. Some existing approaches could achieve this by using g…
GS-IR: 3D Gaussian Splatting for Inverse Rendering
Zhihao Liang, Qi Zhang, Ying Feng +2
We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian Splatting (GS) that leverages forward mapping volume rendering to achieve photorealistic novel view synthe…
TapMo: Shape-aware Motion Generation of Skeleton-free Characters
Jiaxu Zhang, Shaoli Huang, Zhigang Tu +4
Previous motion generation methods are limited to the pre-rigged 3D human model, hindering their applications in the animation of various non-rigged characters. In this work, we pr…
HumTrans: A Novel Open-Source Dataset for Humming Melody Transcription and Beyond
Shansong Liu, Xu Li, Dian Li +1
This paper introduces the HumTrans dataset, which is publicly available and primarily designed for humming melody transcription. The dataset can also serve as a foundation for down…
CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models
Ziyang Yuan, Mingdeng Cao, Xintao Wang +3
Incorporating a customized object into image generation presents an attractive feature in text-to-image generation. However, existing optimization-based and encoder-based methods a…
FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling
Haonan Qiu, Menghan Xia, Yong Zhang +4
With the availability of large-scale video datasets and the advances of diffusion models, text-driven video generation has achieved substantial progress. However, existing video ge…
HiFi-123: Towards High-fidelity One Image to 3D Content Generation
Wangbo Yu, Li Yuan, Yan-Pei Cao +6
Recent advances in diffusion models have enabled 3D generation from a single image. However, current methods often produce suboptimal results for novel views, with blurred textures…
Anti-Aliased Neural Implicit Surfaces with Encoding Level of Detail
Yiyu Zhuang, Qi Zhang, Ying Feng +6
We present LoD-NeuS, an efficient neural representation for high-frequency geometry detail recovery and anti-aliased novel view rendering. Drawing inspiration from voxel-based repr…
Unified Pretraining Target Based Video-music Retrieval With Music Rhythm And Video Optical Flow Information
Tianjun Mao, Shansong Liu, Yunxuan Zhang +2
Background music (BGM) can enhance the video's emotion. However, selecting an appropriate BGM often requires domain knowledge. This has led to the development of video-music retrie…
StyleAdapter: A Unified Stylized Image Generation Model
Zhouxia Wang, Xintao Wang, Liangbin Xie +4
This work focuses on generating high-quality images with specific style of reference images and content of provided textual descriptions. Current leading algorithms, i.e., DreamBoo…
Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning
Shansong Liu, Atin Sakkeer Hussain, Chenshuo Sun +1
Text-to-music generation (T2M-Gen) faces a major obstacle due to the scarcity of large-scale publicly available music datasets with natural language captions. To address this, we p…
OmniZoomer: Learning to Move and Zoom in on Sphere at High-Resolution
Zidong Cao, Hao Ai, Yan-Pei Cao +3
Omnidirectional images (ODIs) have become increasingly popular, as their large field-of-view (FoV) can offer viewers the chance to freely choose the view directions in immersive en…
Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views
Zi-Xin Zou, Weihao Cheng, Yan-Pei Cao +3
Reconstructing 3D objects from extremely sparse views is a long-standing and challenging problem. While recent techniques employ image diffusion models for generating plausible ima…
ViT-Lens: Initiating Omni-Modal Exploration through 3D Insights
Weixian Lei, Yixiao Ge, Jianfeng Zhang +4
Though the success of CLIP-based training recipes in vision-language models, their scalability to more modalities (e.g., 3D, audio, etc.) is limited to large-scale data, which is e…
SC-NeuS: Consistent Neural Surface Reconstruction from Sparse and Noisy Views
Shi-Sheng Huang, Zi-Xin Zou, Yi-Chi Zhang +1
The recent neural surface reconstruction by volume rendering approaches have made much progress by achieving impressive surface reconstruction quality, but are still limited to den…
Towards Unseen Triples: Effective Text-Image-joint Learning for Scene Graph Generation
Qianji Di, Wenxi Ma, Zhongang Qi +3
Scene Graph Generation (SGG) aims to structurally and comprehensively represent objects and their connections in images, it can significantly benefit scene understanding and other…
ID-Pose: Sparse-view Camera Pose Estimation by Inverting Diffusion Models
Weihao Cheng, Yan-Pei Cao, Ying Shan
Given sparse views of a 3D object, estimating their camera poses is a long-standing and intractable problem. Toward this goal, we consider harnessing the pre-trained diffusion mode…
SparseGNV: Generating Novel Views of Indoor Scenes with Sparse Input Views
Weihao Cheng, Yan-Pei Cao, Ying Shan
We study to generate novel views of indoor scenes given sparse input views. The challenge is to achieve both photorealism and view consistency. We present SparseGNV: a learning fra…
SurfelNeRF: Neural Surfel Radiance Fields for Online Photorealistic Reconstruction of Indoor Scenes
Yiming Gao, Yan-Pei Cao, Ying Shan
Online reconstructing and rendering of large-scale indoor scenes is a long-standing challenge. SLAM-based methods can reconstruct 3D scene geometry progressively in real time but c…
NeAI: A Pre-convoluted Representation for Plug-and-Play Neural Ambient Illumination
Yiyu Zhuang, Qi Zhang, Xuan Wang +5
Recent advances in implicit neural representation have demonstrated the ability to recover detailed geometry and material from multi-view images. However, the use of simplified lig…
LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation
Guangcong Zheng, Xianpan Zhou, Xuewei Li +3
Recently, diffusion models have achieved great success in image synthesis. However, when it comes to the layout-to-image generation where an image often has a complex scene of mult…