9 papers
TCS-BENCH: Benchmarking State-of-the-Art Generative AI Theoretical Computer Science Research Ability
Vincent Cohen-Addad, Dimitris Paparas, Ernest van Wijland +13
We introduce TCS-Bench, a benchmark for evaluating Large Language Models (LLMs) on research-level Theoretical Computer Science (TCS) proof generation. TCS-Bench consists of theorem…
Decodable but Not Faithful: Coupling Natural-Language Rationales to Programmatic Verifiers
Vatsal Ananthula, Adarsh Kumarappan
Language models can generate plausible rationales for their predictions, but these explanations may not faithfully represent the model's internal reasoning. We propose verifier-cou…
Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation
Bruce Changlong Xu, Adarsh Kumarappan, Mu Zhou
Key-value (KV) cache quantization is widely used to reduce Large Language Model (LLM) inference memory, yet existing evaluations solely focus on measuring perplexity and accuracy w…
Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy
Adarsh Kumarappan, Ananya Mujoo
LLM-based multi-agent pipelines flip from correct to incorrect answers under simulated peer disagreement at rates we term yield, a vulnerability widely attributed to RLHF-induced s…
DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models
Adarsh Kumarappan, Pareesa Ameneh Golnari, Wen Wen +5
DevBench is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,800 evaluation instances across six pro…
Automating Deception: Scalable Multi-Turn LLM Jailbreaks
Adarsh Kumarappan, Ananya Mujoo
Multi-turn conversational attacks, which leverage psychological principles like Foot-in-the-Door (FITD), where a small initial request paves the way for a more significant one, to…