5 papers · 1 filter
Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable
Abhinav Rajeev Kumar, Paras Chopra
Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on their merits rather than deferr…
ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?
Ayush Nangia, Shikhar Mishra, Aman Gokrani +1
We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the…
Language Models Entangle Language and Culture
Shourya Jain, Paras Chopra
Users should not be systemically disadvantaged by the language they use for interacting with LLMs; i.e. users across languages should get responses of similar quality irrespective…
METIS: Mentoring Engine for Thoughtful Inquiry & Solutions
Abhinav Rajeev Kumar, Dhruv Trehan, Paras Chopra
Many students lack access to expert research mentorship. We ask whether an AI mentor can move undergraduates from an idea to a paper. We build METIS, a tool-augmented, stage-aware…
Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts
Dhruv Trehan, Paras Chopra
We report a case study of four end-to-end attempts to autonomously generate ML research papers using a pipeline of six LLM agents mapped to stages of the scientific workflow. Of th…