4 papers
Putting Transformer Theories to the Test
Awni Altabaa, Tobias Schnabel, Kiran Tomlinson
Several formal theories have been developed to characterize the capabilities and limitations of the transformer architecture, each employing different abstractions that capture dif…
Code Understanding is a Bottleneck for Coding Agents
Nishant Balepur, Kiran Tomlinson, Tobias Schnabel
Repository benchmarks (e.g., SWE-bench) for coding agents often assume that lines of code edited can predict task difficulty, but such datasets' poor control over code and task typ…
Reasoning about Reasoning: BAPO Bounds on Chain-of-Thought Token Complexity in LLMs
Kiran Tomlinson, Tobias Schnabel, Adith Swaminathan +1
Inference-time scaling via chain-of-thought (CoT) reasoning is a major driver of state-of-the-art LLM performance, but it comes with substantial latency and compute costs. We addre…
Lost in Transmission: When and Why LLMs Fail to Reason Globally
Tobias Schnabel, Kiran Tomlinson, Adith Swaminathan +1
Despite their many successes, transformer-based large language models (LLMs) continue to struggle with tasks that require complex reasoning over large parts of their input. We argu…