12 papers · 1 filter
Towards Evaluation of Implicit Software World Models in Coding LLMs
Egor Bogomolov, Yaroslav Zharov
Software engineering, whether performed by humans or by AI agents, requires reasoning about how software behaves. We call the internal model that supports such reasoning the softwa…
LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models
Daniele Cipollone, Sergey Titov, Maliheh Izadi +2
Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledg…
On Problems of Implicit Context Compression for Software Engineering Agents
Kirill Gelvan, Igor Slinko, Felix Steinbauer +3
LLM-based Software Engineering agents face a critical bottleneck: context length limitations cause failures on complex, long-horizon tasks. One promising solution is to encode cont…
Diff-XYZ: A Benchmark for Evaluating Diff Understanding
Evgeniy Glukhov, Michele Conti, Egor Bogomolov +2
Reliable handling of code diffs is central to agents that edit and refactor repositories at scale. We introduce Diff-XYZ, a compact benchmark for code-diff understanding with three…
The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management
Tobias Lindenbauer, Igor Slinko, Ludwig Felder +2
Large Language Model (LLM)-based agents solve complex tasks through iterative reasoning, exploration, and tool-use, a process that can result in long, expensive context histories.…
Challenge on Optimization of Context Collection for Code Completion
Dmitry Ustalov, Egor Bogomolov, Alexander Bezzubov +4
The rapid advancement of workflows and methods for software engineering using AI emphasizes the need for a systematic evaluation and analysis of their ability to leverage informati…