138 citations · 368 across the 28 of their papers we have counts for
30 papers
SongCraft: Unified Song Generation and Editing with Reconstructive Learning
Haohe Liu, Varun Nagaraja, Gael Le Lan +5
Song generation and editing have mostly been treated as separate tasks. Existing editing methods often require noise injection and regeneration or curated paired training data. We…
Exploring Audio Hallucination in Egocentric Video Understanding
Ashish Seth, Xinhao Mei, Changsheng Zhao +9
Egocentric videos provide a distinctive setting in which sound serves as crucial cues to understand user activities and surroundings, particularly when visual information is unstab…
EgoAVU: Egocentric Audio-Visual Understanding
Ashish Seth, Xinhao Mei, Changsheng Zhao +9
Understanding egocentric videos plays a vital role for embodied intelligence. Recent multi-modal large language models (MLLMs) can accept both visual and audio inputs. However, due…
SLAP: Scalable Language-Audio Pretraining with Variable-Duration Audio and Multi-Objective Training
Xinhao Mei, Gael Le Lan, Haohe Liu +5
Contrastive language-audio pretraining (CLAP) has achieved notable success in learning semantically rich audio representations and is widely adopted for various audio-related tasks…
Lightweight and Generalizable Acoustic Scene Representations via Contrastive Fine-Tuning and Distillation
Kuang Yuan, Yang Gao, Xilin Li +4
Acoustic scene classification (ASC) models on edge devices typically operate under fixed class assumptions, lacking the transferability needed for real-world applications that requ…
MASV: Speaker Verification with Global and Local Context Mamba
Yang Liu, Li Wan, Yiteng Huang +3
Deep learning models like Convolutional Neural Networks and transformers have shown impressive capabilities in speech verification, gaining considerable attention in the research c…
SyncFlow: Toward Temporally Aligned Joint Audio-Video Generation from Text
Haohe Liu, Gael Le Lan, Xinhao Mei +7
Video and audio are closely correlated modalities that humans naturally perceive together. While recent advancements have enabled the generation of audio or video from text, produc…
First-Shot Unsupervised Anomalous Sound Detection With Unknown Anomalies Estimated by Metadata-Assisted Audio Generation
Hejing Zhang, Qiaoxi Zhu, Jian Guan +6
First-shot (FS) unsupervised anomalous sound detection (ASD) is a brand-new task introduced in DCASE 2023 Challenge Task 2, where the anomalous sounds for the target machine types…
FoleyGen: Visually-Guided Audio Generation
Xinhao Mei, Varun Nagaraja, Gael Le Lan +4
Recent advancements in audio generation have been spurred by the evolution of large-scale deep learning models and expansive datasets. However, the task of video-to-audio (V2A) gen…
Enhance audio generation controllability through representation similarity regularization
Yangyang Shi, Gael Le Lan, Varun Nagaraja +6
This paper presents an innovative approach to enhance control over audio generation by emphasizing the alignment between audio and text representations during model training. In th…
AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining
Haohe Liu, Yi Yuan, Xubo Liu +7
Although audio generation shares commonalities across different types of audio, such as speech, music, and sound effects, designing models for each type requires careful considerat…
Dual Transformer Decoder based Features Fusion Network for Automated Audio Captioning
Jianyuan Sun, Xubo Liu, Xinhao Mei +3
Automated audio captioning (AAC) which generates textual descriptions of audio content. Existing AAC models achieve good results but only use the high-dimensional representation of…
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
Xinhao Mei, Chutong Meng, Haohe Liu +6
The advancement of audio-language (AL) multimodal learning tasks has been significant in recent years. However, researchers face challenges due to the costly and time-consuming col…
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
Haohe Liu, Zehua Chen, Yi Yuan +5
Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited gener…
Towards Generating Diverse Audio Captions via Adversarial Training
Xinhao Mei, Xubo Liu, Jianyuan Sun +2
Automated audio captioning is a cross-modal translation task for describing the content of audio clips with natural language sentences. This task has attracted increasing attention…
Ontology-aware Learning and Evaluation for Audio Tagging
Haohe Liu, Qiuqiang Kong, Xubo Liu +3
This study defines a new evaluation metric for audio tagging tasks to overcome the limitation of the conventional mean average precision (mAP) metric, which treats different kinds…
Automated Audio Captioning via Fusion of Low- and High- Dimensional Features
Jianyuan Sun, Xubo Liu, Xinhao Mei +3
Automated audio captioning (AAC) aims to describe the content of an audio clip using simple sentences. Existing AAC methods are developed based on an encoder-decoder architecture t…
Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention
Xubo Liu, Qiushi Huang, Xinhao Mei +10
Audio captioning aims to generate text descriptions of audio clips. In the real world, many objects produce similar sounds. How to accurately recognize ambiguous sounds is a major…
Simple Pooling Front-ends For Efficient Audio Classification
Xubo Liu, Haohe Liu, Qiuqiang Kong +3
Recently, there has been increasing interest in building efficient audio neural networks for on-device scenarios. Most existing approaches are designed to reduce the size of audio…
Surrey System for DCASE 2022 Task 5: Few-shot Bioacoustic Event Detection with Segment-level Metric Learning
Haohe Liu, Xubo Liu, Xinhao Mei +3
Few-shot audio event detection is a task that detects the occurrence time of a novel sound class given a few examples. In this work, we propose a system based on segment-level metr…
Segment-level Metric Learning for Few-shot Bioacoustic Event Detection
Haohe Liu, Xubo Liu, Xinhao Mei +3
Few-shot bioacoustic event detection is a task that detects the occurrence time of a novel sound given a few examples. Previous methods employ metric learning to build a latent spa…
Automated Audio Captioning: An Overview of Recent Progress and New Challenges
Xinhao Mei, Xubo Liu, Mark D. Plumbley +1
Automated audio captioning is a cross-modal translation task that aims to generate natural language descriptions for given audio clips. This task has received increasing attention…
Separate What You Describe: Language-Queried Audio Source Separation
Xubo Liu, Haohe Liu, Qiuqiang Kong +5
In this paper, we introduce the task of language-queried audio source separation (LASS), which aims to separate a target source from an audio mixture based on a natural language qu…
Leveraging Pre-trained BERT for Audio Captioning
Xubo Liu, Xinhao Mei, Qiushi Huang +6
Audio captioning aims at using natural language to describe the content of an audio clip. Existing audio captioning systems are generally based on an encoder-decoder architecture,…