AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods

Abstract

Sign language recognition is a challenging problem where signs are identified by simultaneous local and global articulations of multiple sources, i.e. hand shape and orientation, hand movements, body posture, and facial expressions. Solving this problem computationally for a large vocabulary of signs in real life settings is still a challenge, even with the state-of-the-art models. In this study, we present a new largescale multi-modal Turkish Sign Language dataset (AUTSL) with a benchmark and provide baseline models for performance evaluations. Our dataset consists of 226 signs performed by 43 different signers and 38,336 isolated sign video samples in total. Samples contain a wide variety of backgrounds recorded in indoor and outdoor environments. Moreover, spatial positions and the postures of signers also vary in the recordings. Each sample is recorded with Microsoft Kinect v2 and contains RGB, depth, and skeleton modalities. We prepared benchmark training and test sets for user independent assessments of the models. We trained several deep learning based models and provide empirical evaluations using the benchmark; we used CNNs to extract features, unidirectional and bidirectional LSTM models to characterize temporal information. We also incorporated feature pooling modules and temporal attention to our models to improve the performances. We evaluated our baseline models on AUTSL and Montalbano datasets. Our models achieved competitive results with the state-of-the-art methods on Montalbano dataset, i.e. 96.11% accuracy. In AUTSL random train-test splits, our models performed up to 95.95% accuracy. In the proposed user-independent benchmark dataset our best baseline model achieved 62.02% accuracy. The gaps in the performances of the same baseline models show the challenges inherent in our benchmark dataset. AUTSL benchmark dataset is publicly available at https://cvml.ankara.edu.tr.

arXiv commentPreprint of the accepted paper at IEEE Access Journal. The revised version contains empirical results with Montalbano dataset, in addition to AUTSL. The abstract is revised accordingly

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Action Recognition using Visual Attention
• Foreground Segmentation Using a Triplet Convolutional Neural Network for Multiscale Feature Encoding
• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
• MS-ASL: A Large-Scale Data Set and Benchmark for Understanding American Sign Language
• Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective
• Dynamic Gesture Recognition by Using CNNs and Star RGB: a Temporal Information Condensation
• LSA64: An Argentinian Sign Language Dataset
• BosphorusSign22k Sign Language Recognition Dataset

Cited by in corpus (15)

• Using Motion History Images with 3D Convolutional Networks in Isolated Sign Language Recognition
• A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities
• From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation
• Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble
• Skeleton Aware Multi-modal Sign Language Recognition
• Slovo: Russian Sign Language Dataset
• Survey on Hand Gesture Recognition from Visual Input
• Resources for Turkish Natural Language Processing: A critical survey
• BdSLW60: A Word-Level Bangla Sign Language Dataset
• Including Signed Languages in Natural Language Processing
• E-TSL: A Continuous Educational Turkish Sign Language Dataset with Baseline Methods
• Spotter+GPT: Turning Sign Spottings into Sentences with LLMs
• Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field
• SkelCap: Automated Generation of Descriptive Text from Skeleton Keypoint Sequences
• Sign and Search: Sign Search Functionality for Sign Language Lexica