11 papers
Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections
Haven Kim, Zachary Novack, Julian McAuley +1
Video-to-music generation has drawn growing interest for its role in conveying the emotion of visual media, including film. Progress in the field, however, is hampered by a reprodu…
Equilibrium Forcing: Adaptive Video Generation Without Noise Conditioning
Hansen Jin Lillemark, Alex Rojas, Zachary Novack +5
Standard autoregressive video generation algorithms based on Diffusion and Flow Matching rely on rigid training objectives and static sampling schedules, limiting inference procedu…
Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators
Zachary Novack, Stephen Brade, Haven Kim +8
Interactive streaming music generation promises the use of generative models for live performance and co-creation that is impossible with offline models. However, SOTA models exist…
Break-the-Beat! Controllable MIDI-to-Drum Audio Synthesis
Shuyang Cui, Zhi Zhong, Qiyu Wu +9
Current methods for creating drum loop audio in digital music production, such as using one-shot samples or resampling, often demand non-trivial efforts of creators. While recent g…
Low-Resource Guidance for Controllable Latent Audio Diffusion
Zachary Novack, Zack Zukowski, CJ Carr +6
Generative audio requires fine-grained controllable outputs, yet most existing methods require model retraining on specific controls or inference-time controls (\textit{e.g.}, guid…
Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio
Phillip Long, Zachary Novack, Chris Donahue
Autoregressive "language" models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to 8-bit audio, leaving open whether suc…
CSyMR: Benchmarking Compositional Music Information Retrieval in Symbolic Music Reasoning
Boyang Wang, Yash Vishe, Xin Xu +4
Natural language information needs over symbolic music scores rarely reduce to a single step lookup. Many queries require compositional Music Information Retrieval (MIR) that extra…
Evaluating Music Context Preservation: A Multi-facet Framework for Music Editing Systems
Yash Vishe, Eric Xue, Xunyi Jiang +4
Music editing plays a vital role in modern music production, with applications in film, broadcasting, and game development. Recent advances in music editing systems have enabled di…
SoundReactor: Frame-level Online Video-to-Audio Generation
Koichi Saito, Julian Tanke, Christian Simon +7
Prevailing Video-to-Audio (V2A) generation models operate offline, assuming an entire video sequence or chunks of frames are available beforehand. This critically limits their use…
WildFX: A DAW-Powered Pipeline for In-the-Wild Audio FX Graph Modeling
Qihui Yang, Taylor Berg-Kirkpatrick, Julian McAuley +1
Despite rapid progress in end-to-end AI music generation, AI-driven modeling of professional Digital Signal Processing (DSP) workflows remains challenging. In particular, while the…
Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generation
Jaechul Roh, Zachary Novack, Yuefeng Peng +3
Generative AI systems for music and video commonly use text-based filters to prevent regurgitation of copyrighted material. We expose a significant vulnerability in this approach b…