Understanding and Mitigating Risks of Generative AI in Financial Services
arXiv:2504.20086 · doi:10.1145/3715275.3732168
Abstract
To responsibly develop Generative AI (GenAI) products, it is critical to define the scope of acceptable inputs and outputs. What constitutes a "safe" response is an actively debated question. Academic work puts an outsized focus on evaluating models by themselves for general purpose aspects such as toxicity, bias, and fairness, especially in conversational applications being used by a broad audience. In contrast, less focus is put on considering sociotechnical systems in specialized domains. Yet, those specialized systems can be subject to extensive and well-understood legal and regulatory scrutiny. These product-specific considerations need to be set in industry-specific laws, regulations, and corporate governance requirements. In this paper, we aim to highlight AI content safety considerations specific to the financial services domain and outline an associated AI content risk taxonomy. We compare this taxonomy to existing work in this space and discuss implications of risk category violations on various stakeholders. We evaluate how existing open-source technical guardrail solutions cover this taxonomy by assessing them on data collected via red-teaming activities. Our results demonstrate that these guardrails fail to detect most of the content risks we discuss.
Accepted to FAccT 2025
References in corpus (13)
- Constitutional AI: Harmlessness from AI Feedback
- Challenges and Applications of Large Language Models
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
- Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
- Model evaluation for extreme risks
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Sociotechnical Safety Evaluation of Generative AI Systems
- A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges
- A Hazard Analysis Framework for Code Synthesis Large Language Models
- Introducing v0.5 of the AI Safety Benchmark from MLCommons
- WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs
- AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts
- ShieldGemma: Generative AI Content Moderation Based on Gemma