7 papers
FactLens: Benchmarking Fine-Grained Fact Verification
Kushan Mitra, Dan Zhang, Sajjadur Rahman +1
Large Language Models (LLMs) have shown impressive capability in language generation and understanding, but their tendency to hallucinate and produce factually incorrect informatio…
A Blueprint Architecture of Compound AI Systems for Enterprise
Eser Kandogan, Sajjadur Rahman, Nikita Bhutani +11
Large Language Models (LLMs) have showcased remarkable capabilities surpassing conventional NLP challenges, creating opportunities for use in production use cases. Towards this goa…
CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI Systems
Yanlin Feng, Sajjadur Rahman, Aaron Feng +2
Compound AI systems (CASs) that employ LLMs as agents to accomplish knowledge-intensive tasks via interactions with tools and data retrievers have garnered significant interest wit…
MEGAnno+: A Human-LLM Collaborative Annotation System
Hannah Kim, Kushan Mitra, Rafael Li Chen +2
Large language models (LLMs) can label data faster and cheaper than humans for various NLP tasks. Despite their prowess, LLMs may fall short in understanding of complex, sociocultu…
Knowledge Acquisition and Integration with Expert-in-the-loop
Sajjadur Rahman, Frederick Choi, Hannah Kim +2
Constructing and serving knowledge graphs (KGs) is an iterative and human-centered process involving on-demand programming and analysis. In this paper, we present Kyurem, a program…
Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutions
Pouya Pezeshkpour, Eser Kandogan, Nikita Bhutani +3
Remarkable performance of large language models (LLMs) in a variety of tasks brings forth many opportunities as well as challenges of utilizing them in production settings. Towards…