Publications (13)
DyaDiT: A Multi-Modal Diffusion Transformer for Socially Favorable Dyadic Gesture Generation
Yichen Peng, Jyun-Ting Song, Siyeol Jung +7
Generating realistic conversational gestures are essential for achieving natural, socially engaging interactions with digital humans. However, existing methods typically map a sing…
UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speaking
Xuangeng Chu, Ruicong Liu, Yifei Huang +3
Generating lifelike conversational avatars requires modeling not just isolated speakers, but the dynamic, reciprocal interaction of speaking and listening. However, modeling the li…
LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model
Xi Wang, Hongzhen Li, Heng Fang +4
Image rendering from line drawings is vital in design and image generation technologies reduce costs, yet professional line drawings demand preserving complex details. Text prompts…
DiffFaceSketch: High-Fidelity Face Image Synthesis with Sketch-Guided Latent Diffusion Model
Yichen Peng, Chunqi Zhao, Haoran Xie +2
Synthesizing face images from monochrome sketches is one of the most fundamental tasks in the field of image-to-image translation. However, it is still challenging to (1)~make mode…
SketchMeHow: Interactive Projection Guided Task Instruction with User Sketches
Haoran Xie, Yichen Peng, Hange Wang +1
In this work, we propose an interactive general instruction framework SketchMeHow to guidance the common users to complete the daily tasks in real-time. In contrast to the conventi…
Sensing Your Vocals: Exploring the Activity of Vocal Cord Muscles for Pitch Assessment Using Electromyography and Ultrasonography
Kanyu Chen, Rebecca Panskus, Erwin Wu +9
Vocal training is difficult because the muscles that control pitch, resonance, and phonation are internal and invisible to learners. This paper investigates how Electromyography (E…
BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis
Haiyang Liu, Zihao Zhu, Naoya Iwamoto +5
Achieving realistic, vivid, and human-like synthesized conversational gestures conditioned on multi-modal data is still an unsolved problem due to the lack of available datasets, m…
InterPet4D: A Multimodal 4D Human-Pet Interaction Dataset for Pet Motion Generation
Yichen Peng, Jyun-Ting Song, Chen-Chieh Liao +3
Human-pet interaction estimation and generation remain underexplored due to the absence of a high-quality large-scale dataset. We present InterPet4D, the first multimodal dataset c…
EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
Haiyang Liu, Zihao Zhu, Giorgio Becherini +7
We propose EMAGE, a framework to generate full-body human gestures from audio and masked gestures, encompassing facial, local body, hands, and global movements. To achieve this, we…
FilterPrompt: A Simple yet Efficient Approach to Guide Image Appearance Transfer in Diffusion Models
Xi Wang, Yichen Peng, Heng Fang +4
In controllable generation tasks, flexibly manipulating the generated images to attain a desired appearance or structure based on a single input image cue remains a critical and lo…
DualMotion: Global-to-Local Casual Motion Design for Character Animations
Yichen Peng, Chunqi Zhao, Haoran Xie +3
Animating 3D characters using motion capture data requires basic expertise and manual labor. To support the creativity of animation design and make it easier for common users, we p…
dualFace:Two-Stage Drawing Guidance for Freehand Portrait Sketching
Zhengyu Huang, Yichen Peng, Tomohiro Hibino +4
In this paper, we propose dualFace, a portrait drawing interface to assist users with different levels of drawing skills to complete recognizable and authentic face sketches. dualF…
SimDiff: Simulator-constrained Diffusion Model for Physically Plausible Motion Generation
Akihisa Watanabe, Jiawei Ren, Li Siyao +3
Generating physically plausible human motion is crucial for applications such as character animation and virtual reality. Existing approaches often incorporate a simulator-based mo…