52 citations · 333 across the 67 of their papers we have counts for
72 papers
AutoSynth: Learning to Generate Editable Synthesizer Programs from Audio and Text
Tristan Wu, Daniel Chin, Liwei Lin +2
Audio generation models can translate natural-language descriptions into sound, but their outputs are typically waveforms. Their audio quality is constrained by audio compression,…
SheetSage2: Coherent Lead-Sheet Transcription with Synthetic Supervision
Junyan Jiang, Ruibin Yuan, Jiahao Pan +4
Transcribing music into a human-readable score requires a coherent understanding of rhythm, harmony, melody, and form. Two obstacles limit this goal: annotated recordings are scarc…
From Isolated Feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment
Liwei Lin, Gus Xia
How can we understand what a music foundation model has learned \textit{internally}? Most interpretability approaches, such as probing and Sparse Autoencoders (SAEs), focus on iden…
YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality
Ruibin Yuan, Jiahao Pan, Junyan Jiang +32
Symbolic models make melody, harmony, rhythm, and form explicit but typically stop before a finished recording; audio models produce complete songs while leaving composition implic…
Learning Music Style for Piano Arrangement Through Cross-Modal Bootstrapping
Jingwei Zhao, Gus Xia, Ziyu Wang +1
What is music style? Though often described using text labels such as "swing," "classical," or "emotional," the real style remains implicit and hidden in concrete music examples. I…
Calliphony: A Calligraphy-Driven Interface for Real-Time Generative Music Performance
Tristan Wu, Ruiji Yu, Gus Xia
While music generative models have recently gained significant attention, how they can be effectively integrated into live music performances still requires further exploration. Th…
DDSynth-RL: Audio Synthesizer Inversion via Discrete Diffusion with Reinforcement Learning
Tristan Wu, Daniel Chin, Junan Zhang +3
Synthesizer inversion is challenging for two main reasons: 1) Distinct parameter configurations can produce perceptually similar sounds. 2) Parameter-space losses often fail to ref…
Turning music identification into a neural forward pass
Muhammad Taimoor Haseeb, Ahmad Hammoudeh, Gus Xia
Search, a foundational operation in computer science, maps a query to a matching item in a collection. It is typically implemented as a System-2 like, rule-based pipeline in which…
RenCon 2025: Revival of the Expressive Performance Rendering Competition
Huan Zhang, Taegyun Kwon, Anders Friberg +7
This paper presents a comprehensive documentation of RenCon 2025, the revival of the expressive performance rendering competition which took place at ISMIR 2025 in Daejeon, Korea.…
ViTex: Visual Texture Control for Multi-Track Symbolic Music Generation via Discrete Diffusion Models
Xiaoyu Yi, Qi He, Gus Xia +1
In automatic music generation, a central challenge is to design controls that enable meaningful human-machine interaction. Existing systems often rely on extrinsic inputs such as t…
Who Gets Heard? Rethinking Fairness in AI for Music Systems
Atharva Mehta, Shivam Chauhan, Megha Sharma +5
In recent years, the music research community has examined risks of AI models for music, with generative AI models in particular, raised concerns about copyright, deepfakes, and tr…
Automatic Melody Reduction via Shortest Path Finding
Ziyu Wang, Yuxuan Wu, Roger B. Dannenberg +1
Melody reduction, as an abstract representation of musical compositions, serves not only as a tool for music analysis but also as an intermediate representation for structured musi…
MusiScene: Leveraging MU-LLaMA for Scene Imagination and Enhanced Video Background Music Generation
Fathinah Izzati, Xinyue Li, Yuxuan Wu +1
Humans can imagine various atmospheres and settings when listening to music, envisioning movie scenes that complement each piece. For example, slow, melancholic music might evoke s…
EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation
Fathinah Izzati, Xinyue Li, Gus Xia
We propose Expotion (Facial Expression and Motion Control for Multimodal Music Generation), a generative model leveraging multimodal visual controls - specifically, human facial ex…
TOMI: Transforming and Organizing Music Ideas for Multi-Track Compositions with Full-Song Structure
Qi He, Gus Xia, Ziyu Wang
Hierarchical planning is a powerful approach to model long sequences structurally. Aside from considering hierarchies in the temporal structure of music, this paper explores an eve…
Versatile Symbolic Music-for-Music Modeling via Function Alignment
Junyan Jiang, Daniel Chin, Liwei Lin +2
Many music AI models learn a map between music content and human-defined labels. However, many annotations, such as chords, can be naturally expressed within the music modality its…
YuE: Scaling Open Foundation Models for Long-Form Music Generation
Ruibin Yuan, Hanfeng Lin, Shuyue Guo +55
We tackle the task of long-form music generation--particularly the challenging \textbf{lyrics-to-song} problem--by introducing YuE, a family of open foundation models based on the…
Language Model Mapping in Multimodal Music Learning: A Grand Challenge Proposal
Daniel Chin, Gus Xia
We have seen remarkable success in representation learning and language models (LMs) using deep neural networks. Many studies aim to build the underlying connections among differen…
CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages
Shangda Wu, Zhancheng Guo, Ruibin Yuan +7
CLaMP 3 is a unified framework developed to address challenges of cross-modal and cross-lingual generalization in music information retrieval. Using contrastive learning, it aligns…
Music for All: Representational Bias and Cross-Cultural Adaptability of Music Generation Models
Atharva Mehta, Shivam Chauhan, Amirbek Djanibekov +3
The advent of Music-Language Models has greatly enhanced the automatic music generation capability of AI systems, but they are also limited in their coverage of the musical genres…
Exploring GPT's Ability as a Judge in Music Understanding
Kun Fang, Ziyu Wang, Gus Xia +1
Recent progress in text-based Large Language Models (LLMs) and their extended ability to process multi-modal sensory data have led us to explore their applicability in addressing m…
M2M-Gen: A Multimodal Framework for Automated Background Music Generation in Japanese Manga Using Large Language Models
Megha Sharma, Muhammad Taimoor Haseeb, Gus Xia +1
This paper introduces M2M Gen, a multi modal framework for generating background music tailored to Japanese manga. The key challenges in this task are the lack of an available data…
CalliffusionV2: Personalized Natural Calligraphy Generation with Flexible Multi-modal Control
Qisheng Liao, Liang Li, Yulang Fei +1
In this paper, we introduce CalliffusionV2, a novel system designed to produce natural Chinese calligraphy with flexible multi-modal control. Unlike previous approaches that rely s…
Foundation Models for Music: A Survey
Yinghao Ma, Anders Øland, Anton Ragni +39
In recent years, foundation models (FMs) such as large language models (LLMs) and latent diffusion models (LDMs) have profoundly impacted diverse sectors, including music. This com…