(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice

Abstract

Large language models (LLMs) are increasingly capable of providing users with advice in a wide range of professional domains, including legal advice. However, relying on LLMs for legal queries raises concerns due to the significant expertise required and the potential real-world consequences of the advice. To explore \textit{when} and \textit{why} LLMs should or should not provide advice to users, we conducted workshops with 20 legal experts using methods inspired by case-based reasoning. The provided realistic queries ("cases") allowed experts to examine granular, situation-specific concerns and overarching technical and legal constraints, producing a concrete set of contextual considerations for LLM developers. By synthesizing the factors that impacted LLM response appropriateness, we present a 4-dimension framework: (1) User attributes and behaviors, (2) Nature of queries, (3) AI capabilities, and (4) Social impacts. We share experts' recommendations for LLM response strategies, which center around helping users identify `right questions to ask' and relevant information rather than providing definitive legal judgments. Our findings reveal novel legal considerations, such as unauthorized practice of law, confidentiality, and liability for inaccurate advice, that have been overlooked in the literature. The case-based deliberation method enabled us to elicit fine-grained, practice-informed insights that surpass those from de-contextualized surveys or speculative principles. These findings underscore the applicability of our method for translating domain-specific professional knowledge and practices into policies that can guide LLM behavior in a more responsible direction.

14 pages

References in corpus (7)

• Gender bias and stereotypes in Large Language Models
• Queer In AI: A Case Study in Community-Led Participatory AI
• Augmenting Interpretable Models with LLMs during Training
• Humans, AI, and Context: Understanding End-Users' Trust in a Real-World Computer Vision Application
• Evaluating Biased Attitude Associations of Language Models in an Intersectional Context
• Promises and pitfalls of artificial intelligence for legal applications
• Case Repositories: Towards Case-Based Reasoning for AI Alignment

Cited by in corpus (10)

• Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
• Pre-Trained Language Models for Keyphrase Prediction: A Review
• Objection Overruled! Lay People can Distinguish Large Language Models from Lawyers, but still Favour Advice from an LLM
• PolicyCraft: Supporting Collaborative and Participatory Policy Design through Case-Grounded Deliberation
• The Epistemological Consequences of Large Language Models: Rethinking collective intelligence and institutional knowledge
• CUIfy the XR: An Open-Source Package to Embed LLM-powered Conversational Agents in XR
• Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judge
• Designing Human-AI System for Legal Research: A Case Study of Precedent Search in Chinese Law
• ORFuzz: Fuzzing the "Other Side" of LLM Safety -- Testing Over-Refusal
• How Can AI Augment Access to Justice? Public Defenders' Perspectives on AI Adoption