Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research

Abstract

General-purpose LLMs pose misinformation risks for development and policy experts, lacking epistemic humility for verifiable outputs. We present AVA (AI + Verified Analysis), a GenAI platform built on a curated library of over 4,000 World Bank Reports with multilingual capabilities. AVA's multi-agent pipeline enables users to query and receive evidence-based syntheses. It operationalizes epistemic humility through two mechanisms: citation verifiability (tracing claims to sources) and reasoned abstention (declining unsupported queries with justification and redirection). We conducted an in-the-wild evaluation with over 2,200 individuals from heterogeneous organisations and roles in 116 countries, via log analysis, surveys, and 20 interviews. Difference-in-Differences estimates associate sustained engagement with 2.4-3.9 hours saved weekly. Qualitatively, participants used AVA as a specialized "evidence engine"; reasoned abstention clarified scope boundaries, and trust was calibrated through institutional provenance and page-anchored citations. We contribute design guidelines for specialized AI and articulate a vision for "ecosystem-aware" Humble AI.

Accepted at ACM CHI'26

References in corpus (13)

• To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processes
• The egalitarian effect of search engines
• HILL: A Hallucination Identifier for Large Language Models
• RELIC: Investigating Large Language Model Responses using Self-Consistency
• Task Supportive and Personalized Human-Large Language Model Interaction: A User Study
• PaperWeaver: Enriching Topical Paper Alerts by Contextualizing Recommended Papers with User-collected Papers
• MMTEB: Massive Multilingual Text Embedding Benchmark
• Toward an Evaluation Science for Generative AI Systems
• ACAI for SBOs: AI Co-creation for Advertising and Inspiration for Small Business Owners
• Teaching Large Language Models to Express Knowledge Boundary from Their Own Signals
• Evaluation Framework for AI Systems in "the Wild"