6 papers
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Wang Bill Zhu, Miaosen Chai, Shangshang Wang +5
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during de…
Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
Song Bian, Tao Yu, Shivaram Venkataraman +1
Scaling the number of parameters and the size of training data has proven to be an effective strategy for improving large language model (LLM) performance. Yet, as these models gro…
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini +82
AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not…
What Limits Agentic Systems Efficiency?
Song Bian, Minghao Yan, Anand Jayarajan +2
Large Language Models (LLMs), such as OpenAI-o1 and DeepSeek-R1, have demonstrated strong reasoning capabilities. To further enhance LLM capabilities, recent agentic systems, such…
Tesserae: Scalable Placement Policies for Deep Learning Workloads
Song Bian, Saurabh Agarwal, Md. Tareq Mahmood +1
Training deep learning (DL) models has become a dominant workload in data-centers and improving resource utilization is a key goal of DL cluster schedulers. In order to do this, sc…
Scaling Inference-Efficient Language Models
Song Bian, Minghao Yan, Shivaram Venkataraman
Scaling laws are powerful tools to predict the performance of large language models. However, current scaling laws fall short of accounting for inference costs. In this work, we fi…