4 papers
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi +3
Recent advancements in Large Language Models (LLMs) have sparked interest in their formal reasoning capabilities, particularly in mathematics. The GSM8K benchmark is widely used to…
Conformal Prediction Sets for Deep Generative Models via Reduction to Conformal Regression
Hooman Shahrokhi, Devjeet Raj Roy, Yan Yan +2
We consider the problem of generating valid and small prediction sets by sampling outputs (e.g., software code and natural language text) from a black-box deep generative model for…
Direct Prediction Set Minimization via Bilevel Conformal Classifier Training
Yuanjie Shi, Hooman Shahrokhi, Xuesong Jia +3
Conformal prediction (CP) is a promising uncertainty quantification framework which works as a wrapper around a black-box classifier to construct prediction sets (i.e., subset of c…
Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models
Keivan Alizadeh, Iman Mirzadeh, Hooman Shahrokhi +6
Large Language Models (LLMs) typically generate outputs token by token using a fixed compute budget, leading to inefficient resource utilization. To address this shortcoming, recen…