134 citations · 776 across the 124 of their papers we have counts for
135 papers
TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications
Kazuki Egashira, Niels Mündler-Sasahara, Mark Niklas Müller +1
Coding agents are increasingly used to create large software projects. While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks rem…
AutoMark: Enabling Autoresearch to Discover Better LLM Watermarks
Thibaud Gloaguen, Robin Staab, Martin Vechev
With LLM watermarking being deployed commercially and now required by regulations, improving its reliability and effectiveness has become crucial. Yet, recent progress in the field…
Leveraging Instruction Tuning and Merging for Reasoning Model Adaptation
Yu-Du Feng, Niels Mündler-Sasahara, Mark Vero +1
Reasoning language models (RLMs) demonstrate impressive performance by leveraging test-time compute in the form of reasoning tokens. However, this behavior makes adapting RLMs to n…
Generative Compilation: On-the-Fly Compiler Feedback as AI Generates Code
Niels Mündler-Sasahara, Hristo Venev, Dawn Song +2
Languages with rich static semantics, such as Rust, provide stronger guarantees for AI-generated code, but their strictness makes generation more difficult. Off-the-shelf compilers…
Marking the Wrong Symptoms: Evaluating LLM Watermarks in Medical Texts
Melanie Rieff, Robin Staab, Thibaud Gloaguen +2
Large language models (LLMs) are increasingly integrated into clinical workflows, stressing the need for reliable traceability of model-generated output with watermarking. Yet, mos…
Making Open-Source Text LLM Watermarks Durable Against Merging
Luisa Scharff, Thibaud Gloaguen, Robin Staab +1
Open-source LLMs (OSMs)arereaching near state-of-the-art performance, prompting prior works to trace the text they generate by embedding text watermarking algorithms directly into…
Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness
Ivo Petrov, Jasper Dekoninck, Dimitar I. Dimitrov +1
Large language models (LLMs) have become capable mathematical problem-solvers, often producing correct proofs for challenging problems. However, correctness alone is not sufficient…
Optimizing the Cost-Quality Tradeoff of Agentic Theorem Provers in Lean
Kári Rögnvaldsson, Chenhao Sun, Jasper Dekoninck +1
Large language models (LLMs) are increasingly used in workflows for generating formal proofs in Lean. These workflows often decompose problems into smaller lemmas, sample many proo…
TIGER: Inverting Transformer Gradients via Embedding-Subspace Distance Optimization
William Kalikman, Ivo Petrov, Dimitar I. Dimitrov +1
Federated learning allows multiple clients to jointly train a shared model by sending gradient updates to a central server while keeping raw inputs local. However, prior gradient i…
Widening the Gap: Exploiting LLM Quantization via Outlier Injection
Xiaohua Zhan, Kazuki Egashira, Robin Staab +2
LLM quantization has become essential for memory-efficient deployment. Recent work has shown that quantization schemes can pose critical security risks: an adversary may release a…
SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics
Nikolay Georgiev, Maria Drencheva, Kseniia Ibragimova +3
As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational r…
Learning from Saturated Data: Signals Beyond Correctness for LLM Training
Hanno Hiss, Jasper Dekoninck, Martin Vechev
The growing capabilities of large language models (LLMs) have led to the saturation of many benchmarks and training datasets used to improve them. Motivated by this, we investigate…
Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots
Mark Vero, Fabian Kaczmarczyck, Ivan Petrov +6
Honeypots are decoy systems mimicking real system components designed to defend against cyber attacks. Recently, LLMs increasingly serve as simulation backbones for honeypots. They…
Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs
Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger +4
Large language models (LLMs) are becoming increasingly capable mathematical collaborators, but static benchmarks are no longer sufficient for evaluating progress: they are often na…
Every Bit, Everywhere, All at Once: A Binomial Multibit LLM Watermark
Thibaud Gloaguen, Robin Staab, Mark Vero +1
With LLM watermarking already being deployed commercially, practical applications increasingly require multibit watermarks that encode more complex payloads, such as user IDs or ti…
Coding Agents Don't Know When to Act
Thibaud Gloaguen, Niels Mündler, Mark Müller +2
Coding agents are increasingly deployed to autonomously maintain software, including to resolve user-reported issues: a bug report comes in and the agent creates a patch to address…
Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
Kazuki Egashira, Mark Vero, Jasper Dekoninck +3
Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs). While RLVR is designe…
SecPI: Secure Code Generation with Reasoning Models via Security Reasoning Internalization
Hao Wang, Niels Mündler, Mark Vero +3
Reasoning language models (RLMs) are increasingly used in programming. Yet, even state-of-the-art RLMs frequently introduce critical security vulnerabilities in generated code. Pri…
CodeTaste: Can LLMs Generate Human-Level Code Refactorings?
Alex Thillen, Niels Mündler, Veselin Raychev +1
LLM coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt. Human developers address such issues through refa…
Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets
Hanna Yukhymenko, Anton Alexandrov, Martin Vechev
The reliability of multilingual Large Language Model (LLM) evaluation is currently compromised by the inconsistent quality of translated benchmarks. Existing resources often suffer…
A Unified Framework for LLM Watermarks
Thibaud Gloaguen, Robin Staab, Nikola Jovanović +1
LLM watermarks allow tracing AI-generated texts by inserting a detectable signal into their generated content. Recent works have proposed a wide range of watermarking algorithms, e…
Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
Thibaud Gloaguen, Niels Mündler-Sasahara, Niels Mündler +4
A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md. Although this practice is strongly encouraged by ag…
Learning Compact Boolean Networks
Shengpu Wang, Yuhao Mao, Yani Zhang +1
Floating-point neural networks dominate modern machine learning but incur substantial inference costs, motivating emerging interest in Boolean networks for resource-constrained dep…
AutoBaxBuilder: Bootstrapping Code Security Benchmarking
Tobias von Arx, Niels Mündler, Mark Vero +2
As large language models (LLMs) see wide adoption in software engineering, the reliable assessment of the correctness and security of LLM-generated code is crucial. Notably, prior…