30 citations · 98 across the 28 of their papers we have counts for
37 papers
CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
William Chen, Prem Seetharaman, Ke Chen +10
Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond e…
Smorph: Playable Sound Morphing with Diffusion Models
Annie Chu, Hugo Flores García, Johannes Imort +5
Sound morphing, generating intermediate sounds that transition from one sonic identity to another, can be a powerful tool for musical sound design. Existing diffusion-based morphin…
Taming Audio VAEs via Target-KL Regularization
Prem Seetharaman, Rithesh Kumar
Latent diffusion models have emerged as the dominant paradigm for many generation tasks including audio generation such as text-to-audio, text-to-music and text-to-speech. A key co…
Audio Hallucination Attacks: Probing the Reliability of Large Audio Language Models
Ashish Seth, Sonal Kumar, Ramaneswaran Selvakumar +5
Large Audio Language Models (LALMs) achieve strong performance on audio-language tasks; however, their reliability in real-world settings remains underexplored. We introduce Audio…
AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing
William Chen, Prem Seetharaman, Rithesh Kumar +4
Despite recent breakthroughs, audio foundation models struggle in processing complex multi-source acoustic scenes. We refer to this challenging domain as audio stories, which can h…
Generative Audio Extension and Morphing
Prem Seetharaman, Oriol Nieto, Justin Salamon
In audio-related creative tasks, sound designers often seek to extend and morph different sounds from their libraries. Generative audio models, capable of creating audio using exam…
TAC: Timestamped Audio Captioning
Sonal Kumar, Prem Seetharaman, Ke Chen +8
Large Audio Language Models struggle to disentangle overlapping events in complex acoustic scenes, yielding temporally inconsistent captions and frequent hallucinations. We introdu…
Audiocards: Structured Metadata Improves Audio Language Models For Sound Design
Sripathi Sridhar, Prem Seetharaman, Oriol Nieto +2
Sound designers search for sounds in large sound effects libraries using aspects such as sound class or visual context. However, the metadata needed for such search is often missin…
Mix2Morph: Learning Sound Morphing from Noisy Mixes
Annie Chu, Hugo Flores García, Oriol Nieto +3
We introduce Mix2Morph, a text-to-audio diffusion model fine-tuned to perform sound morphing without a dedicated dataset of morphs. By finetuning on noisy surrogate mixes at higher…
PromptSep: Generative Audio Separation via Multimodal Prompting
Yutong Wen, Ke Chen, Prem Seetharaman +7
Recent breakthroughs in language-queried audio source separation (LASS) have shown that generative models can achieve higher separation audio quality than traditional masking-based…
The Rhythm In Anything: Audio-Prompted Drums Generation with Masked Language Modeling
Patrick O'Reilly, Julia Barnett, Hugo Flores García +4
Musicians and nonmusicians alike use rhythmic sound gestures, such as tapping and beatboxing, to express drum patterns. While these gestures effectively communicate musical ideas,…
FLAM: Frame-Wise Language-Audio Modeling
Yusong Wu, Christos Tsirigotis, Ke Chen +5
Recent multi-modal audio-language models (ALMs) excel at text-audio retrieval but struggle with frame-wise audio understanding. Prior works use temporal-aware labels or unsupervise…
SILA: Signal-to-Language Augmentation for Enhanced Control in Text-to-Audio Generation
Sonal Kumar, Prem Seetharaman, Justin Salamon +2
The field of text-to-audio generation has seen significant advancements, and yet the ability to finely control the acoustic characteristics of generated audio remains under-explore…
Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitations
Hugo Flores García, Oriol Nieto, Justin Salamon +2
We present Sketch2Sound, a generative audio model capable of creating high-quality sounds from a set of interpretable time-varying control signals: loudness, brightness, and pitch,…
Video-Guided Foley Sound Generation with Multimodal Controls
Ziyang Chen, Prem Seetharaman, Bryan Russell +4
Generating sound effects for videos often requires creating artistic sound effects that diverge significantly from real-life sources and flexible control in the sound design. To ad…
Code Drift: Towards Idempotent Neural Audio Codecs
Patrick O'Reilly, Prem Seetharaman, Jiaqi Su +2
Neural codecs have demonstrated strong performance in high-fidelity compression of audio signals at low bitrates. The token-based representations produced by these codecs have prov…
VampNet: Music Generation via Masked Acoustic Token Modeling
Hugo Flores Garcia, Prem Seetharaman, Rithesh Kumar +1
We introduce VampNet, a masked acoustic token modeling approach to music synthesis, compression, inpainting, and variation. We use a variable masking schedule during training which…
High-Fidelity Audio Compression with Improved RVQGAN
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs +2
Language models have been successfully used to model natural signals, such as images, speech, and music. A key component of these models is a high quality neural compression model…
Music Separation Enhancement with Generative Modeling
Noah Schaffer, Boaz Cogan, Ethan Manilow +3
Despite phenomenal progress in recent years, state-of-the-art music separation systems produce source estimates with significant perceptual shortcomings, such as adding extraneous…
How to Listen? Rethinking Visual Sound Localization
Ho-Hsiang Wu, Magdalena Fuentes, Prem Seetharaman +1
Localizing visual sounds consists on locating the position of objects that emit sound within an image. It is a growing research area with potential applications in monitoring natur…
Unsupervised Source Separation By Steering Pretrained Music Models
Ethan Manilow, Patrick O'Reilly, Prem Seetharaman +1
We showcase an unsupervised method that repurposes deep models trained for music generation and music tagging for audio source separation, without any retraining. An audio generati…
Wav2CLIP: Learning Robust Audio Representations From CLIP
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar +1
We propose Wav2CLIP, a robust audio representation learning method by distilling from Contrastive Language-Image Pre-training (CLIP). We systematically evaluate Wav2CLIP on a varie…
Chunked Autoregressive GAN for Conditional Waveform Synthesis
Max Morrison, Rithesh Kumar, Kundan Kumar +3
Conditional waveform synthesis models learn a distribution of audio waveforms given conditioning such as text, mel-spectrograms, or MIDI. These systems employ deep generative model…
What's All the FUSS About Free Universal Sound Separation Data?
Scott Wisdom, Hakan Erdogan, Daniel Ellis +6
We introduce the Free Universal Sound Separation (FUSS) dataset, a new corpus for experiments in separating mixtures of an unknown number of sounds from an open domain of sound typ…