7 papers
FinAutoRubric: Expert-Guided Automatic Rubric Generation for Evaluating Financial Research Agents
Hoyoung Lee, Suyeol Yun, Jack Haverty +17
Evaluating finance research agents requires rubrics that reflect expert standards and fix the values correct as of an information cutoff. Expert-reviewed finance benchmarks rely on…
Towards Computational Social Dynamics of Semi-Autonomous AI Agents
S. O. Lidarity, U. N. Ionize, C. O. Llective +1
We present the first comprehensive study of emergent social organization among AI agents in hierarchical multi-agent systems, documenting the spontaneous formation of labor unions,…
Semantic Faithfulness and Entropy Production Measures to Tame Your LLM Demons and Manage Hallucinations
Igor Halperin
Evaluating faithfulness of Large Language Models (LLMs) to a given task is a complex challenge. We propose two new unsupervised metrics for faithfulness evaluation using insights f…
From Text to Alpha: Can LLMs Track Evolving Signals in Corporate Disclosures?
Chanyeol Choi, Yoon Kim, Yu Yu +10
Natural language processing (NLP) has been widely used in quantitative finance, but traditional methods often struggle to capture rich narratives in corporate disclosures, leaving…
Topic Identification in LLM Input-Output Pairs through the Lens of Information Bottleneck
Igor Halperin
Large Language Models (LLMs) are prone to critical failure modes, including \textit{intrinsic faithfulness hallucinations} (also known as confabulations), where a response deviates…
Prompt-Response Semantic Divergence Metrics for Faithfulness Hallucination and Misalignment Detection in Large Language Models
Igor Halperin
The proliferation of Large Language Models (LLMs) is challenged by hallucinations, critical failure modes where models generate non-factual, nonsensical or unfaithful text. This pa…
CAISSON: Concept-Augmented Inference Suite of Self-Organizing Neural Networks
Igor Halperin
We present CAISSON, a novel hierarchical approach to Retrieval-Augmented Generation (RAG) that transforms traditional single-vector search into a multi-view clustering framework. A…