172 citations · 205 across the 22 of their papers we have counts for
11 papers · 1 filter
LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess
Sai Kolasani, Maxim Saplin, Nicholas Crispino +5
We introduce LLM CHESS, an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs) through extend…
SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent
Shiyi Cao, Dacheng Li, Fangzhou Zhao +12
We introduce SkyRL-Agent, a framework for efficient, multi-turn, long-horizon agent training and evaluation. It provides efficient asynchronous dispatching, lightweight tool integr…
Barbarians at the Gate: How AI is Upending Systems Research
Audrey Cheng, Shu Liu, Melissa Pan +14
Artificial Intelligence (AI) is starting to transform the research process as we know it by automating the discovery of new solutions. Given a task, the typical AI-driven approach…
Supporting Our AI Overlords: Redesigning Data Systems to be Agent-First
Shu Liu, Soujanya Ponnapalli, Shreya Shankar +12
Large Language Model (LLM) agents, acting on their users' behalf to manipulate and analyze data, are likely to become the dominant workload for data systems in the future. When wor…
Establishing Best Practices for Building Rigorous Agentic Benchmarks
Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun +22
Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to e…
Reasoning Models Can Be Effective Without Thinking
Wenjie Ma, Jingxuan He, Charlie Snell +3
Recent LLMs have significantly improved reasoning capabilities, primarily by including an explicit, lengthy Thinking process as part of generation. In this paper, we question wheth…
Why Do Multi-Agent LLM Systems Fail?
Mert Cemri, Melissa Z. Pan, Shuyi Yang +10
Despite enthusiasm for Multi-Agent LLM Systems (MAS), their performance gains on popular benchmarks are often minimal. This gap highlights a critical need for a principled understa…
Optimizing Model Selection for Compound AI Systems
Lingjiao Chen, Jared Quincy Davis, Boris Hanin +4
Compound AI systems that combine multiple LLM calls, such as self-refine and multi-agent-debate, achieve strong performance on many AI tasks. We address a core question in optimizi…
LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!
Dacheng Li, Shiyi Cao, Tyler Griggs +9
Large reasoning models (LRMs) tackle complex reasoning problems by following long chain-of-thoughts (Long CoT) that incorporate reflection, backtracking, and self-validation. Howev…
Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design
Jared Quincy Davis, Boris Hanin, Lingjiao Chen +3
As practitioners seek to surpass the current reliability and quality frontier of monolithic models, Compound AI Systems consisting of many language model inference calls are increa…
Data Acquisition: A New Frontier in Data-centric AI
Lingjiao Chen, Bilge Acun, Newsha Ardalani +8
As Machine Learning (ML) systems continue to grow, the demand for relevant and comprehensive datasets becomes imperative. There is limited study on the challenges of data acquisiti…