MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language

Abstract

Sign language recognition is a challenging and often underestimated problem comprising multi-modal articulators (handshape, orientation, movement, upper body and face) that integrate asynchronously on multiple streams. Learning powerful statistical models in such a scenario requires much data, particularly to apply recent advances of the field. However, labeled data is a scarce resource for sign language due to the enormous cost of transcribing these unwritten languages. We propose the first real-life large-scale sign language data set comprising over 25,000 annotated videos, which we thoroughly evaluate with state-of-the-art methods from sign and related action recognition. Unlike the current state-of-the-art, the data set allows to investigate the generalization to unseen individuals (signer-independent test) in a realistic setting with over 200 signers. Previous work mostly deals with limited vocabulary tasks, while here, we cover a large class count of 1000 signs in challenging and unconstrained real-life recording conditions. We further propose I3D, known from video classifications, as a powerful and suitable architecture for sign language recognition, outperforming the current state-of-the-art by a large margin. The data set is publicly available to the community.

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Co-occurrence Feature Learning for Skeleton based Action Recognition using Regularized Deep LSTM Networks

Cited by in corpus (15)

• AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods
• A Comprehensive Study on Deep Learning-based Methods for Sign Language Recognition
• SignBERT+: Hand-model-aware Self-supervised Pre-training for Sign Language Understanding
• Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective
• Using Motion History Images with 3D Convolutional Networks in Isolated Sign Language Recognition
• A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities
• Towards Zero-shot Sign Language Recognition
• Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison
• Survey on Hand Gesture Recognition from Visual Input
• Transferring Cross-domain Knowledge for Video Sign Language Recognition
• Scaling up sign spotting through sign language dictionaries
• Facial Expression Phoenix (FePh): An Annotated Sequenced Dataset for Facial and Emotion-Specified Expressions in Sign Language
• Weakly-supervised Fingerspelling Recognition in British Sign Language Videos
• Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field
• LIGAR: Lightweight General-purpose Action Recognition