11 papers
Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?
Ankit Sonthalia, Haritz Puerto, Alexander Rubinstein +2
Large language models (LLMs) can solve many narrow tasks, but querying them separately for millions of related instances can be prohibitively expensive. Can LLM agents autonomously…
MEME: Multi-entity & Evolving Memory Evaluation
Seokwon Jung, Alexander Rubinstein, Arnas Uselis +2
LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate on…
MASEval: Extending Multi-Agent Evaluation from Models to Systems
Cornelius Emde, Alexander Rubinstein, Anmol Goel +4
The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are m…
DISCO: Diversifying Sample Condensation for Efficient Model Evaluation
Alexander Rubinstein, Benjamin Raible, Martin Gubri +1
Evaluating modern machine learning models has become prohibitively expensive. Benchmarks such as LMMs-Eval and HELM demand thousands of GPU hours per model. Costly evaluation reduc…
Are We Done with Object-Centric Learning?
Alexander Rubinstein, Ameya Prabhu, Matthias Bethge +1
Object-centric learning (OCL) seeks to learn representations that only encode an object, isolated from other objects or background cues in a scene. This approach underpins various…
Scalable Ensemble Diversification for OOD Generalization and Detection
Alexander Rubinstein, Luca Scimeca, Damien Teney +1
Training a diverse ensemble of models has several practical applications such as providing candidates for model selection with better out-of-distribution (OOD) generalization, and…
Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents
Evgenii Kortukov, Alexander Rubinstein, Elisa Nguyen +1
Retrieval-augmented generation (RAG) mitigates many problems of fully parametric language models, such as temporal degradation, hallucinations, and lack of grounding. In RAG, the m…
Do Deep Neural Network Solutions Form a Star Domain?
Ankit Sonthalia, Alexander Rubinstein, Ehsan Abbasnejad +1
It has recently been conjectured that neural network solution sets reachable via stochastic gradient descent (SGD) are convex, considering permutation invariances (Entezari et al.,…
Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks
Luca Scimeca, Alexander Rubinstein, Armand Mihai Nicolicioiu +2
Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to shortcut learning phenomena, where a model may rely on erroneous, easy-to-…
Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles
Luca Scimeca, Alexander Rubinstein, Damien Teney +2
Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to a phenomenon known as shortcut learning, where a model relies on erroneous…
Trustworthy Machine Learning
Bálint Mucsányi, Michael Kirchhof, Elisa Nguyen +2
As machine learning technology gets applied to actual products and solutions, new challenges have emerged. Models unexpectedly fail to generalize to small changes in the distributi…