6 papers
The Wording Effect: Quantifying Two-Way Drift in LLM Benchmark Performance
Shailja Thakur, Sungeun An, Chad DeLuca +1
A benchmark score comes from a single phrasing of each problem. That single phrasing is treated as if it stood for the whole space of ways the same problem could be asked, but it d…
Generating Verifiable Chain of Thoughts from Exection-Traces
Shailja Thakur, Vaibhav Saxena, Rohan Kulkarni +4
Getting language models to reason correctly about code requires training on data where each reasoning step can be checked. Current synthetic Chain-of-Thought (CoT) training data of…
A Systematic Approach for Large Language Models Debugging
Basel Shbita, Anna Lisa Gentile, Bing Zhang +10
Large language models (LLMs) have become central to modern AI workflows, powering applications from open-ended text generation to complex agent-based reasoning. However, debugging…
STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs
Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur +2
Benchmarks are often used as a standard to understand LLM capabilities in different domains. However, aggregate benchmark scores provide limited insight into compositional skill ga…
CIFE: Code Instruction-Following Evaluation
Sravani Gunnu, Shanmukha Guttula, Hima Patel
Large Language Models (LLMs) are increasingly applied to real-world code generation, where functional correctness alone is insufficient for reliable deployment, developers also exp…
STRIDE: A Systematic Framework for Selecting AI Modalities -- Agentic AI, AI Assistants, or LLM Calls
Shubhi Asthana, Bing Zhang, Chad DeLuca +2
The rapid shift from stateless large language models (LLMs) to autonomous, goal-driven agents raises a central question: When is agentic AI truly necessary? While agents enable mul…