303 citations · 697 across the 37 of their papers we have counts for
38 papers
The USTC-NERCSLIP Systems for the CHiME-7 DASR Challenge
Ruoyu Wang, Maokui He, Jun Du +16
This technical report details our submission system to the CHiME-7 DASR Challenge, which focuses on speaker diarization and speech recognition under complex multi-speaker scenarios…
Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder
Yusheng Dai, Hang Chen, Jun Du +4
In recent research, slight performance improvement is observed from automatic speech recognition systems to audio-visual speech recognition systems in the end-to-end framework with…
Semi-supervised multi-channel speaker diarization with cross-channel attention
Shilong Wu, Jun Du, Maokui He +4
Most neural speaker diarization systems rely on sufficient manual training data labels, which are hard to collect under real-world scenarios. This paper proposes a semi-supervised…
Variance-Preserving-Based Interpolation Diffusion Models for Speech Enhancement
Zilu Guo, Jun Du, Chin-Hui Lee +2
The goal of this study is to implement diffusion models for speech enhancement (SE). The first step is to emphasize the theoretical foundation of variance-preserving (VP)-based int…
A Multi-dimensional Deep Structured State Space Approach to Speech Enhancement Using Small-footprint Models
Pin-Jui Ku, Chao-Han Huck Yang, Sabato Marco Siniscalchi +1
We propose a multi-dimensional structured state space (S4) approach to speech enhancement. To better capture the spectral dependencies across the frequency axis, we focus on modify…
The Multimodal Information based Speech Processing (MISP) 2022 Challenge: Audio-Visual Diarization and Recognition
Zhe Wang, Shilong Wu, Hang Chen +12
The Multi-modal Information based Speech Processing (MISP) challenge aims to extend the application of signal processing technology in specific scenarios by promoting the research…
A Quantum Kernel Learning Approach to Acoustic Modeling for Spoken Command Recognition
Chao-Han Huck Yang, Bo Li, Yu Zhang +4
We propose a quantum kernel learning (QKL) framework to address the inherent data sparsity issues often encountered in training large-scare acoustic models in low-resource scenario…
An Experimental Study on Private Aggregation of Teacher Ensemble Learning for End-to-End Speech Recognition
Chao-Han Huck Yang, I-Fan Chen, Andreas Stolcke +2
Differential privacy (DP) is one data protection avenue to safeguard user information used for training deep models by imposing noisy distortion on privacy data. Such a noise pertu…
An Ensemble Teacher-Student Learning Approach with Poisson Sub-sampling to Differential Privacy Preserving Speech Recognition
Chao-Han Huck Yang, Jun Qi, Sabato Marco Siniscalchi +1
We propose an ensemble learning framework with Poisson sub-sampling to effectively train a collection of teacher models to issue some differential privacy (DP) guarantee for traini…
A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification
Qing Wang, Jun Du, Siyuan Zheng +8
In this paper, we propose two techniques, namely joint modeling and data augmentation, to improve system performances for audio-visual scene classification (AVSC). We employ pre-tr…
A Study of Designing Compact Audio-Visual Wake Word Spotting System Based on Iterative Fine-Tuning in Neural Network Pruning
Hengshun Zhou, Jun Du, Chao-Han Huck Yang +2
Audio-only-based wake word spotting (WWS) is challenging under noisy conditions due to environmental interference in signal transmission. In this paper, we investigate on designing…
The USTC-Ximalaya system for the ICASSP 2022 multi-channel multi-party meeting transcription (M2MeT) challenge
Maokui He, Xiang Lv, Weilin Zhou +8
We propose two improvements to target-speaker voice activity detection (TS-VAD), the core component in our proposed speaker diarization system that was submitted to the 2022 Multi-…
Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition
Hengshun Zhou, Jun Du, Yuanyuan Zhang +3
Multimodal emotion recognition is a challenging task in emotion computing as it is quite difficult to extract discriminative features to identify the subtle differences in human em…
A Variational Bayesian Approach to Learning Latent Variables for Acoustic Knowledge Transfer
Hu Hu, Sabato Marco Siniscalchi, Chao-Han Huck Yang +1
We propose a variational Bayesian (VB) approach to learning distributions of latent variables in deep neural network (DNN) models for cross-domain knowledge transfer, to address ac…
Separation Guided Speaker Diarization in Realistic Mismatched Conditions
Shu-Tong Niu, Jun Du, Lei Sun +1
We propose a separation guided speaker diarization (SGSD) approach by fully utilizing a complementarity of speech separation and speaker clustering. Since the conventional clusteri…
A Lottery Ticket Hypothesis Framework for Low-Complexity Device-Robust Neural Acoustic Scene Classification
Hao Yen, Chao-Han Huck Yang, Hu Hu +9
We propose a novel neural model compression strategy combining data augmentation, knowledge transfer, pruning, and quantization for device-robust acoustic scene classification (ASC…
PATE-AAE: Incorporating Adversarial Autoencoder into Private Aggregation of Teacher Ensembles for Spoken Command Classification
Chao-Han Huck Yang, Sabato Marco Siniscalchi, Chin-Hui Lee
We propose using an adversarial autoencoder (AAE) to replace generative adversarial network (GAN) in the private aggregation of teacher ensembles (PATE), a solution for ensuring di…
USTC-NELSLIP System Description for DIHARD-III Challenge
Yuxuan Wang, Maokui He, Shutong Niu +6
This system description describes our submission system to the Third DIHARD Speech Diarization Challenge. Besides the traditional clustering based system, the innovation of our sys…
A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection
Qing Wang, Jun Du, Hua-Xin Wu +3
In this paper, we propose a novel four-stage data augmentation approach to ResNet-Conformer based acoustic modeling for sound event localization and detection (SELD). First, we exp…
Lip-reading with Hierarchical Pyramidal Convolution and Self-Attention
Hang Chen, Jun Du, Yu Hu +3
In this paper, we propose a novel deep learning architecture to improving word-level lip-reading. On the one hand, we first introduce the multi-scale processing into the spatial fe…
A Two-Stage Approach to Device-Robust Acoustic Scene Classification
Hu Hu, Chao-Han Huck Yang, Xianjun Xia +13
To improve device robustness, a highly desirable key feature of a competitive data-driven acoustic scene classification (ASC) system, a novel two-stage system based on fully convol…
Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition
Chao-Han Huck Yang, Jun Qi, Samuel Yen-Chi Chen +4
We propose a novel decentralized feature extraction approach in federated learning to address privacy-preservation issues for speech recognition. It is built upon a quantum convolu…
Correlating Subword Articulation with Lip Shapes for Embedding Aware Audio-Visual Speech Enhancement
Hang Chen, Jun Du, Yu Hu +3
In this paper, we propose a visual embedding approach to improving embedding aware speech enhancement (EASE) by synchronizing visual lip frames at the phone and place of articulati…
Device-Robust Acoustic Scene Classification Based on Two-Stage Categorization and Data Augmentation
Hu Hu, Chao-Han Huck Yang, Xianjun Xia +13
In this technical report, we present a joint effort of four groups, namely GT, USTC, Tencent, and UKE, to tackle Task 1 - Acoustic Scene Classification (ASC) in the DCASE 2020 Chal…