5 papers
The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages
Zixuan Wu, Carolyn Jane Anderson, Arjun Guha
Although coding agents are now very effective in a variety of programming languages, this paper first shows that the cost (in tokens) can very significantly by programming language…
Agnostics: Learning to Code in Any Programming Language via Reinforcement with a Universal Learning Environment
Aleksander Boruch-Gruszecki, Yangtian Zi, Zixuan Wu +4
Large language models (LLMs) already excel at writing code in high-resource languages such as Python and JavaScript, yet stumble on low-resource languages that remain essential to…
ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models
Zixuan Wu, Francesca Lucchetti, Aleksander Boruch-Gruszecki +5
Existing benchmarks for frontier models often test specialized, "PhD-level" knowledge that is difficult for non-experts to grasp. In contrast, we present a benchmark with 613 probl…
GlyphPattern: An Abstract Pattern Recognition Benchmark for Vision-Language Models
Zixuan Wu, Yoolim Kim, Carolyn Jane Anderson
Vision-Language Models (VLMs) building upon the foundation of powerful large language models have made rapid progress in reasoning across visual and textual data. While VLMs perfor…
"I Would Have Written My Code Differently'': Beginners Struggle to Understand LLM-Generated Code
Yangtian Zi, Luisa Li, Arjun Guha +2
Large language models (LLMs) are being increasingly adopted for programming work. Prior work shows that while LLMs accelerate task completion for professional programmers, beginnin…