10 papers
GRASP: Deterministic argument ranking in interaction graphs
Diganta Misra, Antonio Orvieto, Rediet Abebe +1
Large language models are increasingly deployed as automated judges to evaluate the strength of arguments. As this role expands, their legitimacy depends on consistency, transparen…
Agents Learn Their Runtime: Interpreter Persistence as Training-Time Semantics
Victor May, Aaditya Salgarkar, Yishan Wang +2
Tool-augmented LLMs are increasingly deployed as agents that interleave natural-language reasoning with executable Python actions, as in CodeAct-style frameworks. In deployment, th…
Explaining Grokking in Transformers through the Lens of Inductive Bias
Jaisidh Singh, Diganta Misra, Antonio Orvieto
We investigate grokking in transformers through the lens of inductive bias: dispositions arising from architecture or optimization that let the network prefer one solution over ano…
MMTEB: Massive Multilingual Text Embedding Benchmark
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua +83
Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more co…
FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration
Victor May, Diganta Misra, Yanqi Luo +3
AI coding assistants are rapidly becoming integral to modern software development. A key challenge in this space is the continual need to migrate and modernize codebases in respons…
GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities
Diganta Misra, Nizar Islah, Victor May +9
The rapid evolution of software libraries poses a considerable hurdle for code generation, necessitating continuous adaptation to frequent version updates while preserving backward…