A Discussion on Building Practical NLP Leaderboards: The Case of Machine Translation

Abstract

Recent advances in AI and ML applications have benefited from rapid progress in NLP research. Leaderboards have emerged as a popular mechanism to track and accelerate progress in NLP through competitive model development. While this has increased interest and participation, the over-reliance on single, and accuracy-based metrics have shifted focus from other important metrics that might be equally pertinent to consider in real-world contexts. In this paper, we offer a preliminary discussion of the risks associated with focusing exclusively on accuracy metrics and draw on recent discussions to highlight prescriptive suggestions on how to develop more practical and effective leaderboards that can better reflect the real-world utility of models.

pre-print

References in corpus (4)

• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
• Thinking Slow about Latency Evaluation for Simultaneous Machine Translation
• Unsung Challenges of Building and Deploying Language Technologies for Low Resource Language Communities
• How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation