1 citations · 2 across the 16 of their papers we have counts for
19 papers
FLOMSS: Generative Music Source Separation by Flow Matching
Shuyang Cui, Zhi Zhong, Keisuke Toyama +10
Music source separation (MSS) has been conventionally addressed by discriminative methods. Recently, flow matching has emerged as a strong generative paradigm for modeling complex…
Break-the-Beat! Controllable MIDI-to-Drum Audio Synthesis
Shuyang Cui, Zhi Zhong, Qiyu Wu +9
Current methods for creating drum loop audio in digital music production, such as using one-shot samples or resampling, often demand non-trivial efforts of creators. While recent g…
Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
Christian Simon, Masato Ishii, Wei-Yao Wang +8
Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information.…
Do Foundational Audio Encoders Understand Music Structure?
Keisuke Toyama, Zhi Zhong, Akira Takahashi +2
In music information retrieval (MIR) research, the use of pretrained foundational audio encoders (FAEs) has recently become a trend. FAEs pretrained on large amounts of music and a…
FoleyBench: A Benchmark For Video-to-Audio Models
Satvik Dixit, Koichi Saito, Zhi Zhong +2
Video-to-audio generation (V2A) is of increasing importance in domains such as film post-production, AR/VR, and sound design, particularly for the creation of Foley sound effects s…
'Studies for': A Human-AI Co-Creative Sound Artwork Using a Real-time Multi-channel Sound Generation Model
Chihiro Nagashima, Akira Takahashi, Zhi Zhong +2
This paper explores the integration of AI technologies into the artistic workflow through the creation of Studies for, a generative sound installation developed in collaboration wi…
SoundReactor: Frame-level Online Video-to-Audio Generation
Koichi Saito, Julian Tanke, Christian Simon +7
Prevailing Video-to-Audio (V2A) generation models operate offline, assuming an entire video sequence or chunks of frames are available beforehand. This critically limits their use…
TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models
Christian Simon, Masato Ishii, Akio Hayakawa +4
In the recent development of conditional diffusion models still require heavy supervised fine-tuning for performing control on a category of tasks. Training-free conditioning via g…
SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet
Zhi Zhong, Akira Takahashi, Shuyang Cui +3
Foley synthesis aims to synthesize high-quality audio that is both semantically and temporally aligned with video frames. Given its broad application in creative industries, the ta…
Cross-Modal Learning for Music-to-Music-Video Description Generation
Zhuoyuan Mao, Mengjie Zhao, Qiyu Wu +4
Music-to-music-video generation is a challenging task due to the intrinsic differences between the music and video modalities. The advent of powerful text-to-video diffusion models…
OpenMU: Your Swiss Army Knife for Music Understanding
Mengjie Zhao, Zhi Zhong, Zhuoyuan Mao +5
We present OpenMU-Bench, a large-scale benchmark suite for addressing the data scarcity issue in training multimodal language models to understand music. To construct OpenMU-Bench,…
Music Foundation Model as Generic Booster for Music Downstream Tasks
WeiHsiang Liao, Yuhta Takida, Yukara Ikemiya +13
We demonstrate the efficacy of using intermediate representations from a single foundation model to enhance various music downstream tasks. We introduce SoniDo, a music foundation…
Variable Bitrate Residual Vector Quantization for Audio Coding
Yunkee Chae, Woosung Choi, Yuhta Takida +8
Recent state-of-the-art neural audio compression models have progressively adopted residual vector quantization (RVQ). Despite this success, these models employ a fixed number of c…
SpecMaskGIT: Masked Generative Modeling of Audio Spectrograms for Efficient Audio Synthesis and Beyond
Marco Comunità, Zhi Zhong, Akira Takahashi +7
Recent advances in generative models that iteratively synthesize audio clips sparked great success to text-to-audio synthesis (TTA), but with the cost of slow synthesis speed and h…
Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation
Shiqi Yang, Zhi Zhong, Mengjie Zhao +4
In recent years, with the realistic generation results and a wide range of personalized applications, diffusion-based generative models gain huge attention in both visual and audio…
SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation
Koichi Saito, Dongjun Kim, Takashi Shibuya +4
Sound content creation, essential for multimedia works such as video games and films, often involves extensive trial-and-error, enabling creators to semantically reflect their arti…
On the Language Encoder of Contrastive Cross-modal Models
Mengjie Zhao, Junya Ono, Zhi Zhong +7
Contrastive cross-modal models such as CLIP and CLAP aid various vision-language (VL) and audio-language (AL) tasks. However, there has been limited investigation of and improvemen…
Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders
Hao Shi, Kazuki Shimada, Masato Hirano +6
Diffusion-based generative speech enhancement (SE) has recently received attention, but reverse diffusion remains time-consuming. One solution is to initialize the reverse diffusio…
Extending Audio Masked Autoencoders Toward Audio Restoration
Zhi Zhong, Hao Shi, Masato Hirano +5
Audio classification and restoration are among major downstream tasks in audio signal processing. However, restoration derives less of a benefit from pretrained models compared to…