Transformer-based Approaches for Legal Text Processing

Abstract

In this paper, we introduce our approaches using Transformer-based models for different problems of the COLIEE 2021 automatic legal text processing competition. Automated processing of legal documents is a challenging task because of the characteristics of legal documents as well as the limitation of the amount of data. With our detailed experiments, we found that Transformer-based pretrained language models can perform well with automated legal text processing problems with appropriate approaches. We describe in detail the processing steps for each task such as problem formulation, data processing and augmentation, pretraining, finetuning. In addition, we introduce to the community two pretrained models that take advantage of parallel translations in legal domain, NFSP and NMSP. In which, NFSP achieves the state-of-the-art result in Task 5 of the competition. Although the paper focuses on technical reporting, the novelty of its approaches can also be an useful reference in automated legal document processing using Transformer-based models.

arXiv admin note: substantial text overlap with arXiv:2106.13405

References in corpus (4)

• Building Legal Case Retrieval Systems with Lexical Matching and Summarization using A Pre-Trained Phrase Scoring Model
• JNLP Team: Deep Learning for Legal Processing in COLIEE 2020
• ParaLaw Nets -- Cross-lingual Sentence-level Pretraining for Legal Text Processing
• Threshold-Based Retrieval and Textual Entailment Detection on Legal Bar Exam Questions