7 papers
RubricCraft: Where Solutions Diverge, Criteria Emerge
Hosein Hasanbeig, Priyanshu Gupta, Ashish Tiwari +2
Evaluating language agents on open-ended, artifact-producing tasks is difficult: model-based graders can be inconsistent, while exhaustive programmatic checks are often impractical…
fxBench: Evaluating and Understanding Formula Suggestions in Spreadsheets
Sanket Mhatre, Sumit Gulwani, Vu Le +2
Spreadsheets are among the most widely used tools for computations, and formulas are their core building blocks. Besides agentic spreadsheet tools, proactive formula suggestions en…
Whose turn is it? Context-guided model routing for stateful agents
Anikesh Parashar, Avik Dutta, Vashu Chauhan +6
Model routing can reduce the cost of LLM applications by assigning simpler requests to cheaper models without sacrificing the quality of the response. However, most routers treat r…
A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets
Tejas Agrawal, Vu Le, Sumit Gulwani +1
Predictive code completion greatly accelerates how quickly developers work. In spreadsheets, despite being much more common, such auto-completion features are virtually non-existen…
The Invisible Mentor: Inferring User Actions from Screen Recordings to Recommend Better Workflows
Litao Yan, Andrew Head, Ken Milne +4
Many users struggle to notice when a more efficient workflow exists in feature-rich tools like Excel. Existing AI assistants offer help only after users describe their goals or pro…
LLM-Guided Compositional Program Synthesis
Ruhma Khan, Sumit Gulwani, Vu Le +3
Program synthesis from input-output examples, also called programming by example (PBE), has had tremendous impact on automating end-user tasks. Large language models (LLMs) have th…
TableTalk: Scaffolding Spreadsheet Development with a Language Agent
Jenny T. Liang, Aayush Kumar, Yasharth Bajpai +7
Spreadsheet programming is challenging. Programmers use spreadsheet programming knowledge (e.g., formulas) and problem-solving skills to combine actions into complex tasks. Advance…