A Hierarchical Neural Framework for Classification and its Explanation in Large Unstructured Legal Documents

Abstract

Automatic legal judgment prediction and its explanation suffer from the problem of long case documents exceeding tens of thousands of words, in general, and having a non-uniform structure. Predicting judgments from such documents and extracting their explanation becomes a challenging task, more so on documents with no structural annotation. We define this problem as "scarce annotated legal documents" and explore their lack of structural information and their long lengths with a deep-learning-based classification framework which we call MESc; "Multi-stage Encoder-based Supervised with-clustering"; for judgment prediction. We explore the adaptability of LLMs with multi-billion parameters (GPT-Neo, and GPT-J) to legal texts and their intra-domain(legal) transfer learning capacity. Alongside this, we compare their performance and adaptability with MESc and the impact of combining embeddings from their last layers. For such hierarchical models, we also propose an explanation extraction algorithm named ORSE; Occlusion sensitivity-based Relevant Sentence Extractor; based on the input-occlusion sensitivity of the model, to explain the predictions with the most relevant sentences from the document. We explore these methods and test their effectiveness with extensive experiments and ablation studies on legal documents from India, the European Union, and the United States with the ILDC dataset and a subset of the LexGLUE dataset. MESc achieves a minimum total performance gain of approximately 2 points over previous state-of-the-art proposed methods, while ORSE applied on MESc achieves a total average gain of 50% over the baseline explainability scores.

arXiv commentPublished as non archival paper in the The 3rd International Workshop on Mining and Learning in the Legal Domain (MLLD-2023) at CIKM 2023, Birmingham, United Kingdom. (https://sites.google.com/view/mlld2023/)

References in corpus (11)

• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• LLaMA: Open and Efficient Foundation Language Models
• Longformer: The Long-Document Transformer
• LaMDA: Language Models for Dialog Applications
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction
• Legal Prompting: Teaching a Language Model to Think Like a Lawyer
• Legal Prompt Engineering for Multilingual Legal Judgement Prediction
• Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law
• An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification
• Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?