11 papers
Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions
Avisha Das, Mihir Parmar, Mohana Ramnath +1
Romanized Code Mixing (RCM), where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of communication across multiling…
Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark
Himanshu Gupta, Shreyas Verma, Ujjwala Anantheswaran +4
Multi-modal Large Language Models (MLLMs) exhibit impressive problem-solving abilities in various domains, but their visual comprehension and abstract reasoning skills remain under…
CoAct: Co-Active LLM Preference Learning with Human-AI Synergy
Ruiyao Xu, Mihir Parmar, Tiankai Yang +3
Learning from preference-based feedback has become an effective approach for aligning LLMs across diverse tasks. However, high-quality human-annotated preference data remains expen…
GuidedSampling: Steering LLMs Towards Diverse Candidate Solutions at Inference-Time
Divij Handa, Mihir Parmar, Aswin RRV +3
Repeated Sampling (RS) is a simple inference-time algorithm that has been shown to improve model performance on complex tasks. Although it is an effective way of scaling inference…
LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science
Alireza Salemi, Mihir Parmar, Palash Goyal +5
Advances in large language models (LLMs) have created new opportunities in data science, but their deployment is often limited by the challenge of finding relevant data in large da…
Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes
Matthew W. Kenaston, Umair Ayub, Mihir Parmar +14
Despite high performance on clinical benchmarks, large language models may reach correct conclusions through faulty reasoning, a failure mode with safety implications for oncology…