5 papers
ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?
Ayush Nangia, Shikhar Mishra, Aman Gokrani +1
We introduce ISO-Bench, a benchmark for coding agents to test their capabilities on real-world inference optimization tasks. These tasks were taken from vLLM and SGLang, two of the…
Making Large Language Models Speak Tulu: Structured Prompting for an Extremely Low-Resource Language
Prathamesh Devadiga, Paras Chopra
Can large language models converse in languages virtually absent from their training data? We investigate this question through a case study on Tulu, a Dravidian language with over…
Language Models Entangle Language and Culture
Shourya Jain, Paras Chopra
Users should not be systemically disadvantaged by the language they use for interacting with LLMs; i.e. users across languages should get responses of similar quality irrespective…
METIS: Mentoring Engine for Thoughtful Inquiry & Solutions
Abhinav Rajeev Kumar, Dhruv Trehan, Paras Chopra
Many students lack access to expert research mentorship. We ask whether an AI mentor can move undergraduates from an idea to a paper. We build METIS, a tool-augmented, stage-aware…
Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts
Dhruv Trehan, Paras Chopra
We report a case study of four end-to-end attempts to autonomously generate ML research papers using a pipeline of six LLM agents mapped to stages of the scientific workflow. Of th…