1 citations · 2 across the 27 of their papers we have counts for
30 papers
FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference
Chaeyoung Jung, Youngjoon Jang, Seungwoo Lee +1
In this work, we present FastAV, the first token pruning framework tailored for audio-visual large language models (AV-LLMs). While token pruning has been actively explored in stan…
UNMIXX: Untangling Highly Correlated Singing Voices Mixtures
Jihoo Jung, Ji-Hoon Kim, Doyeop Kwak +3
We introduce UNMIXX, a novel framework for multiple singing voices separation (MSVS). While related to speech separation, MSVS faces unique challenges: data scarcity and the highly…
LP-CFM: Perceptual Invariance-Aware Conditional Flow Matching for Speech Modeling
Doyeop Kwak, Youngjoon Jang, Joon Son Chung
The goal of this paper is to provide a new perspective on speech modeling by incorporating perceptual invariances such as amplitude scaling and temporal shifts. Conventional genera…
TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation
Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak +2
The objective of this paper is to jointly synthesize interactive videos and conversational speech from text and reference images. With the ultimate goal of building human-like conv…
Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment
Youngjoon Jang, Liliane Momeni, Zifan Jiang +3
Our aim is to develop a unified model for sign language understanding, that performs sign language translation (SLT) and sign-subtitle alignment (SSA). Together, these two tasks en…
InfiniteAudio: Infinite-Length Audio Generation with Consistency
Chaeyoung Jung, Hojoon Ki, Ji-Hoon Kim +2
This paper presents InfiniteAudio, a simple yet effective strategy for generating infinite-length audio using diffusion-based text-to-audio methods. Current approaches face memory…
EDNet: A Versatile Speech Enhancement Framework with Gating Mamba Mechanism and Phase Shift-Invariant Training
Doyeop Kwak, Youngjoon Jang, Seongyu Kim +1
Speech signals in real-world environments are frequently affected by various distortions such as additive noise, reverberation, and bandwidth limitation, which may appear individua…
Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment
Jeongsoo Choi, Zhikang Niu, Ji-Hoon Kim +3
The goal of this paper is to optimize the training process of diffusion-based text-to-speech models. While recent studies have achieved remarkable advancements, their training dema…
Test-Time Augmentation for Pose-invariant Face Recognition
Jaemin Jung, Youngjoon Jang, Joon Son Chung
The goal of this paper is to enhance face recognition performance by augmenting head poses during the testing phase. Existing methods often rely on training on frontalised images o…
Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing
Jeongsoo Choi, Jaehun Kim, Joon Son Chung
This paper introduces a cross-lingual dubbing system that translates speech from one language to another while preserving key characteristics such as duration, speaker identity, an…
Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models
Chaeyoung Jung, Youngjoon Jang, Jongmin Choi +1
The goal of this work is to enhance balanced multimodal understanding in audio-visual large language models (AV-LLMs) by addressing modality bias without additional training. In cu…
AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding
Chaeyoung Jung, Youngjoon Jang, Joon Son Chung
Hallucination remains a major challenge in multimodal large language models (MLLMs). To address this, various contrastive decoding (CD) methods have been proposed that contrasts or…
VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models
Kim Sung-Bin, Jeongsoo Choi, Puyuan Peng +3
We present VoiceCraft-Dub, a novel approach for automated video dubbing that synthesizes high-quality speech from text and facial cues. This task has broad applications in filmmaki…
AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation
Jeongsoo Choi, Ji-Hoon Kim, Kim Sung-Bin +2
In this paper, we address the task of multimodal-to-speech generation, which aims to synthesize high-quality speech from multiple input modalities: text, video, and reference audio…
From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech
Ji-Hoon Kim, Jeongsoo Choi, Jaehun Kim +2
The objective of this study is to generate high-quality speech from silent talking face videos, a task also known as video-to-speech synthesis. A significant challenge in video-to-…
Deep Understanding of Sign Language for Sign to Subtitle Alignment
Youngjoon Jang, Jeongsoo Choi, Junseok Ahn +1
The objective of this work is to align asynchronous subtitles in sign language videos with limited labelled data. To achieve this goal, we propose a novel framework with the follow…
AdaptVC: High Quality Voice Conversion with Adaptive Learning
Jaehun Kim, Ji-Hoon Kim, Yeunju Choi +3
The goal of voice conversion is to transform the speech of a source speaker to sound like that of a reference speaker while preserving the original content. A key challenge is to e…
CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation
Ji-Hoon Kim, Hong-Sun Yang, Yoon-Cheol Ju +3
The goal of this work is to generate natural speech in multiple languages while maintaining the same speaker identity, a task known as cross-lingual speech synthesis. A key challen…
VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis
Jaemin Jung, Junseok Ahn, Chaeyoung Jung +3
We present VoiceDiT, a multi-modal generative model for producing environment-aware speech and audio from text and visual prompts. While aligning speech with text is crucial for in…
V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow
Jeongsoo Choi, Ji-Hoon Kim, Jinyu Li +2
In this paper, we introduce V2SFlow, a novel Video-to-Speech (V2S) framework designed to generate natural and intelligible speech directly from silent talking face videos. While re…
Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding
Tan Dat Nguyen, Ji-Hoon Kim, Jeongsoo Choi +4
The goal of this paper is to accelerate codec-based speech synthesis systems with minimum sacrifice to speech quality. We propose an enhanced inference method that allows for flexi…
Let Me Finish My Sentence: Video Temporal Grounding with Holistic Text Understanding
Jongbhin Woo, Hyeonggon Ryu, Youngjoon Jang +2
Video Temporal Grounding (VTG) aims to identify visual frames in a video clip that match text queries. Recent studies in VTG employ cross-attention to correlate visual frames and t…
Text-To-Speech Synthesis In The Wild
Jee-weon Jung, Wangyou Zhang, Soumi Maiti +11
Traditional Text-to-Speech (TTS) systems rely on studio-quality speech recorded in controlled settings.a Recently, an effort known as noisy-TTS training has emerged, aiming to util…
Bridging the Gap between Audio and Text using Parallel-attention for User-defined Keyword Spotting
Youkyum Kim, Jaemin Jung, Jihwan Park +2
This paper proposes a novel user-defined keyword spotting framework that accurately detects audio keywords based on text enrollment. Since audio data possesses additional acoustic…