116 citations · 334 across the 26 of their papers we have counts for
27 papers
Toward American Sign Language Processing in the Real World: Data, Tasks, and Methods
Bowen Shi
Sign language, which conveys meaning through gestures, is the chief means of communication among deaf people. Recognizing sign language in natural settings presents significant cha…
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Tu Anh Nguyen, Wei-Ning Hsu, Antony D'Avirro +10
Recent work has shown that it is possible to resynthesize high-quality speech based, not on text, but on low bitrate discrete units that have been learned in a self-supervised fash…
Prompt to GPT-3: Step-by-Step Thinking Instructions for Humor Generation
Yuetian Chen, Bowen Shi, Mei Si
Artificial intelligence has made significant progress in natural language processing, with models like GPT-3 demonstrating impressive capabilities. However, these models still have…
Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale
Matthew Le, Apoorv Vyas, Bowen Shi +8
Large-scale generative models such as GPT and DALL-E have revolutionized the research community. These models not only generate high fidelity outputs, but are also generalists whic…
Scaling Speech Technology to 1,000+ Languages
Vineel Pratap, Andros Tjandra, Bowen Shi +13
Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to ab…
MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
Mohamed Anwar, Bowen Shi, Vedanuj Goswami +3
We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languag…
Visual Story Generation Based on Emotion and Keywords
Yuetian Chen, Ruohua Li, Bowen Shi +2
Automated visual story generation aims to produce stories with corresponding illustrations that exhibit coherence, progression, and adherence to characters' emotional development.…
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement
Wei-Ning Hsu, Tal Remez, Bowen Shi +2
Prior works on improving speech quality with visual input typically study each type of auditory distortion separately (e.g., separation, inpainting, video-to-speech) and present ta…
Comparative layer-wise analysis of self-supervised speech models
Ankita Pasad, Bowen Shi, Karen Livescu
Many self-supervised speech models, varying in their pre-training objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive suc…
u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality
Wei-Ning Hsu, Bowen Shi
While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and…
Open-Domain Sign Language Translation Learned from Online Video
Bowen Shi, Diane Brentari, Greg Shakhnarovich +1
Existing work on sign language translation - that is, translation from sign language videos into sentences in a written language - has focused mainly on (1) data collected in a con…
Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT
Bowen Shi, Abdelrahman Mohamed, Wei-Ning Hsu
This paper investigates self-supervised pre-training for audio-visual speaker representation learning where a visual stream showing the speaker's mouth area is used alongside speec…
Searching for fingerspelled content in American Sign Language
Bowen Shi, Diane Brentari, Greg Shakhnarovich +1
Natural language processing for sign language video - including tasks like recognition, translation, and search - is crucial for making artificial intelligence technologies accessi…
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia +1
Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the pr…
Robust Self-Supervised Audio-Visual Speech Recognition
Bowen Shi, Wei-Ning Hsu, Abdelrahman Mohamed
Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine whic…
Fingerspelling Detection in American Sign Language
Bowen Shi, Diane Brentari, Greg Shakhnarovich +1
Fingerspelling, in which words are signed letter by letter, is an important component of American Sign Language. Most previous work on automatic fingerspelling recognition has assu…
A Joint Framework for Audio Tagging and Weakly Supervised Acoustic Event Detection Using DenseNet with Global Average Pooling
Chieh-Chi Kao, Bowen Shi, Ming Sun +1
This paper proposes a network architecture mainly designed for audio tagging, which can also be used for weakly supervised acoustic event detection (AED). The proposed network cons…
Whole-Word Segmental Speech Recognition with Acoustic Word Embeddings
Bowen Shi, Shane Settle, Karen Livescu
Segmental models are sequence prediction models in which scores of hypotheses are based on entire variable-length segments of frames. We consider segmental models for whole-word ("…
A Cross-Task Analysis of Text Span Representations
Shubham Toshniwal, Haoyue Shi, Bowen Shi +3
Many natural language processing (NLP) tasks involve reasoning with textual spans, including question answering, entity recognition, and coreference resolution. While extensive res…
Few-shot acoustic event detection via meta-learning
Bowen Shi, Ming Sun, Krishna C. Puvvada +3
We study few-shot acoustic event detection (AED) in this paper. Few-shot learning enables detection of new events with very limited labeled data. Compared to other research areas l…
Fingerspelling recognition in the wild with iterative visual attention
Bowen Shi, Aurora Martinez Del Rio, Jonathan Keane +3
Sign language recognition is a challenging gesture sequence recognition problem, characterized by quick and highly coarticulated motion. In this paper we focus on recognition of fi…
Compression of Acoustic Event Detection Models With Quantized Distillation
Bowen Shi, Ming Sun, Chieh-Chi Kao +3
Acoustic Event Detection (AED), aiming at detecting categories of events based on audio signals, has found application in many intelligent systems. Recently deep neural network sig…
Compression of Acoustic Event Detection Models with Low-rank Matrix Factorization and Quantization Training
Bowen Shi, Ming Sun, Chieh-Chi Kao +3
In this paper, we present a compression approach based on the combination of low-rank matrix factorization and quantization training, to reduce complexity for neural network based…
Semi-supervised Acoustic Event Detection based on tri-training
Bowen Shi, Ming Sun, Chieh-Chi Kao +3
This paper presents our work of training acoustic event detection (AED) models using unlabeled dataset. Recent acoustic event detectors are based on large-scale neural networks, wh…