Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition

Abstract

Given recent advances in generative AI technology, a key question is how large language models (LLMs) can enhance acoustic modeling tasks using text decoding results from a frozen, pretrained automatic speech recognition (ASR) model. To explore new capabilities in language modeling for speech processing, we introduce the generative speech transcription error correction (GenSEC) challenge. This challenge comprises three post-ASR language modeling tasks: (i) post-ASR transcription correction, (ii) speaker tagging, and (iii) emotion recognition. These tasks aim to emulate future LLM-based agents handling voice-based interfaces while remaining accessible to a broad audience by utilizing open pretrained language models or agent-based APIs. We also discuss insights from baseline evaluations, as well as lessons learned for designing future evaluations.

IEEE SLT 2024. The initial draft version has been done in December 2023. Post-ASR Text Processing and Understanding Community and LlaMA-7B pre-training correction model: https://huggingface.co/GenSEC-LLM/SLT-Task1-Llama2-7b-HyPo-baseline

References in corpus (11)

• PaLM: Scaling Language Modeling with Pathways
• NeMo: a toolkit for building AI applications using Neural Modules
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
• Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting
• Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
• Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition
• Speaker Diarization with Lexical Information
• N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space
• DiarizationLM: Speaker Diarization Post-Processing with Large Language Models
• ASR-Aware End-to-end Neural Diarization
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models