Gloss-Free Sign Language Translation: An Unbiased Evaluation of Progress in the Field

Abstract

Sign Language Translation (SLT) aims to automatically convert visual sign language videos into spoken language text and vice versa. While recent years have seen rapid progress, the true sources of performance improvements often remain unclear. Do reported performance gains come from methodological novelty, or from the choice of a different backbone, training optimizations, hyperparameter tuning, or even differences in the calculation of evaluation metrics? This paper presents a comprehensive study of recent gloss-free SLT models by re-implementing key contributions in a unified codebase. We ensure fair comparison by standardizing preprocessing, video encoders, and training setups across all methods. Our analysis shows that many of the performance gains reported in the literature often diminish when models are evaluated under consistent conditions, suggesting that implementation details and evaluation setups play a significant role in determining results. We make the codebase publicly available here (https://github.com/ozgemercanoglu/sltbaselines) to support transparency and reproducibility in SLT research.

This is a preprint of an article published in Computer Vision and Image Understanding (CVIU)

References in corpus (10)

• DINOv2: Learning Robust Visual Features without Supervision
• AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods
• Two-Stream Network for Sign Language Recognition and Translation
• ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition
• SLTUNET: A Simple Unified Model for Sign Language Translation
• YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus
• LLaVA-SLT: Visual Language Tuning for Sign Language Translation
• Spotter+GPT: Turning Sign Spottings into Sentences with LLMs
• C${}^2$RL: Content and Context Representation Learning for Gloss-free Sign Language Translation and Retrieval
• An Efficient Sign Language Translation Using Spatial Configuration and Motion Dynamics with LLMs