8 citations · 14 across the 20 of their papers we have counts for
22 papers
AgentPersonaBench: Benchmarking Persona-Driven User Simulation
Jintao Huang, Yifan Wang, Hongyu Shen +43
We introduce AgentPersonaBench (APB), a benchmark evaluating whether persona conditioning faithfully steers downstream agent behavior. While language models are increasingly deploy…
OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software
Dingyuan Dai, Heli Qi, Lei Liu +28
Scientific software presents a demanding test for computer-using agents based on visual language models (VLMs): completing a research workflow requires interpreting specialized int…
Lens Modeling and Cosmological Inference from an Impure Sample of Galaxy-Galaxy Strong Lenses
Philip Holloway, Aprajita Verma, Philip J. Marshall +7
The start of the Legacy Survey of Space and Time marks a new era for strong lensing science, where the number of strong lenses identified is expected to increase to $\mathcal{O}(10…
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Lin Shi, Haowei Lin, Zixuan Zhu +123
Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a…
Bringing analytic rigor to agentic AI for science: The Brain Researcher platform for neuroimaging data analysis
Zijiao Chen, Nicholas Lu, Xinhui Li +14
AI agents can execute scientific analyses, but an analytic output becomes a defensible claim only after alternatives are weighed and the claim is limited to what the evidence suppo…
Early Exploration of the Scientific Discovery Space for the Habitable Worlds Observatory
Courtney D. Dressing, Danica Adams, Evelyne Alecian +324
The Habitable Worlds Observatory (HWO) is a future NASA flagship mission concept identified by the Astro2020 Decadal Survey as the highest priority for large space missions. HWO sh…
OpenThoughts-Agent: Data Recipes for Agentic Models
Negin Raoof, Richard Zhuang, Marianna Nezhurina +47
Agentic language models dramatically expand the applications of AI yet little is publicly known about how to curate training data for broadly capable agents. Existing open efforts…
Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
Jan Batzner, Sree Harsha Nelaturu, Damian Stachura +45
AI evaluations are widely used for testing and understanding progress. However, the diverse evaluators bring with them inconsistencies that challenge analysis and comparison. First…
SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?
Rishi Desai, Jesse Hu, Joan Cabezas +23
AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments. Yet current agent b…
Toward decision-aware AI for LSST-scale time-domain astronomy
C. R. Bom, A. Mahabal, F. Bianco +27
The Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) will generate approximately (10^7) alerts per night, pushing time-domain astronomy beyond pipelines that trea…
AsymmetryZero: A Framework for Operationalizing Human Expert Preferences as Semantic Evals
Tadhg Looram, Lucas Nuzzi, Kyle Waters +1
Much of the focus in RL today is on evaluation design: building meaningful evals that serve simultaneously as benchmarks and as well-defined reward signals for post-training. Yet,…
SLSim: a strong lensing population simulation package
Narayan Khadka, Simon Birrer, Henry Best +45
Gravitational lensing offers unique insights into cosmology by bending light around massive objects. Strong gravitational lensing, in particular, produces magnified and often multi…
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
Xiangyi Li, Yimin Liu, Wenbo Chen +75
Agent Skills are structured packages of procedural knowledge that augment large language model (LLM) agents at inference time. Despite rapid adoption, there is no standard way to m…
Opportunities in AI/ML for the Rubin LSST Dark Energy Science Collaboration
LSST Dark Energy Science Collaboration, Eric Aubourg, Camille Avestruz +63
The Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST) will produce unprecedented volumes of heterogeneous astronomical data (images, catalogs, and alerts) that cha…
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini +82
AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not…
ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?
Christine Ye, Sihan Yuan, Suchetha Cooray +10
Frontier AI agents show increasing promise as scientific research assistants, and may eventually be useful for extended, open-ended research workflows. However, in order to use age…
Investigating the Dark Energy Constraint from Strongly Lensed AGN at LSST-Scale
Sydney Erickson, Martin Millon, Padmavathi Venkatraman +12
Strongly lensed Active Galactic Nuclei (AGN) with an observable time delay can be used to constrain the expansion history of the Universe through time-delay cosmography (TDC). As t…
The Advanced X-ray Imaging Satellite (AXIS) Community Science Book
Michael Koss, Nafisa Aftab, Steven W. Allen +395
The AXIS Community Science Book represents the collective effort of 592 scientists worldwide to define the transformative science enabled by the Advanced X-ray Imaging Satellite (A…
Lens Model Accuracy in the Expected LSST Lensed AGN Sample
Padmavathi Venkatraman, Sydney Erickson, Phil Marshall +13
Strong gravitational lensing of active galactic nuclei (AGN) enables measurements of cosmological parameters through time-delay cosmography (TDC). With data from the upcoming LSST…
Learning Representations of Event Time Series with Sparse Autoencoders for Anomaly Detection, Similarity Search, and Unsupervised Classification
Steven Dillmann, Juan Rafael Martínez-Galarza
Event time series are sequences of discrete events occurring at irregular time intervals, each associated with a domain-specific observational modality. They are common in domains…
Building Machine Learning Challenges for Anomaly Detection in Science
Elizabeth G. Campolongo, Yuan-Tang Chou, Ekaterina Govorkova +148
Scientific discoveries are often made by finding a pattern or object that was not predicted by the known rules of science. Oftentimes, these anomalous events or objects that do not…
Representation Learning for Time-Domain High-Energy Astrophysics: Discovery of Extragalactic Fast X-ray Transient XRT 200515
Steven Dillmann, Juan Rafael Martínez-Galarza, Roberto Soria +2
We present a novel representation learning method for downstream tasks like anomaly detection, unsupervised classification, and similarity searches in high-energy data sets. This e…