7 papers
Evaluating Nova 2.0 Lite model under Amazon's Frontier Model Safety Framework
Satyapriya Krishna, Matteo Memelli, Tong Wang +5
Amazon published its Frontier Model Safety Framework (FMSF) as part of the Paris AI summit, following which we presented a report on Amazon's Premier model. In this report, we pres…
LIMI: Less is More for Agency
Yang Xiao, Mohan Jiang, Jie Sun +18
We define Agency as the emergent capacity of AI systems to function as autonomous agents actively discovering problems, formulating hypotheses, and executing solutions through self…
Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments
Amir Homayounirad, Enrico Liscio, Tong Wang +2
Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial r…
Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs
Jingcheng Niu, Xingdi Yuan, Tong Wang +2
We observe a novel phenomenon, contextual entrainment, across a wide range of language models (LMs) and prompt settings, providing a new mechanistic perspective on how LMs become d…
WGSR-Bench: Wargame-based Game-theoretic Strategic Reasoning Benchmark for Large Language Models
Qiyue Yin, Pei Xu, Qiaozhe Li +15
Recent breakthroughs in Large Language Models (LLMs) have led to a qualitative leap in artificial intelligence' s performance on reasoning tasks, particularly demonstrating remarka…
DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environments
Chiyu Zhang, Marc-Alexandre Cote, Michael Albada +6
Large language model (LLM) agents have shown impressive capabilities in human language comprehension and reasoning, yet their potential in cybersecurity remains underexplored. We i…