LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
arXiv:2301.13126 · doi:10.18653/v1/2023.findings-emnlp.200
Abstract
Lately, propelled by the phenomenal advances around the transformer architecture, the legal NLP field has enjoyed spectacular growth. To measure progress, well curated and challenging benchmarks are crucial. However, most benchmarks are English only and in legal NLP specifically there is no multilingual benchmark available yet. Additionally, many benchmarks are saturated, with the best models clearly outperforming the best humans and achieving near perfect scores. We survey the legal NLP literature and select 11 datasets covering 24 languages, creating LEXTREME. To provide a fair comparison, we propose two aggregate scores, one based on the datasets and one on the languages. The best baseline (XLM-R large) achieves both a dataset aggregate score a language aggregate score of 61.3. This indicates that LEXTREME is still very challenging and leaves ample room for improvement. To make it easy for researchers and practitioners to use, we release LEXTREME on huggingface together with all the code required to evaluate models and a public Weights and Biases project with all the runs.
Published at EMNLP Findings 2023
References in corpus (16)
- LLaMA: Open and Efficient Foundation Language Models
- DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
- BERTje: A Dutch BERT Model
- Holistic Evaluation of Language Models
- Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
- BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling
- Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities
- A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction
- LegalBench: Prototyping a Collaborative Benchmark for Legal Reasoning
- Privacy-Preserving Models for Legal Natural Language Processing
- One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
- An Empirical Study on Cross-X Transfer for Legal Judgment Prediction
- LegalRelectra: Mixed-domain Language Modeling for Long-range Legal Text Comprehension
- Spanish Legalese Language Model and Corpora
- How not to Lie with a Benchmark: Rearranging NLP Leaderboards
- Automatic Anonymization of Swiss Federal Supreme Court Rulings