8 papers
Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMs
Charbel El Feghali, Arkil Patel, Nicholas Meade +3
Open-weight Large Language Models (LLMs) enable scientific progress and broad deployment. However, they make it difficult to control access to sensitive capabilities. Current pract…
Forecasting Downstream Performance of LLMs With Proxy Metrics
Arkil Patel, Siva Reddy, Marius Mosbach +1
Progress in language model development is often driven by comparative decisions: which architecture to adopt, which pretraining corpus to use, or which training recipe to apply. Ma…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1144
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…
DeepSeek-R1 Thoughtology: Let's think about LLM Reasoning
Sara Vera MarjanoviÄ, Arkil Patel, Vaibhav Adlakha +14
Large Reasoning Models like DeepSeek-R1 mark a fundamental shift in how LLMs approach complex problems. Instead of directly producing an answer for a given input, DeepSeek-R1 creat…
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade +7
Web agents enable users to perform tasks on web browsers through natural language interaction. Evaluating web agents trajectories is an important problem, since it helps us determi…
Investigating Adversarial Trigger Transfer in Large Language Models
Nicholas Meade, Arkil Patel, Siva Reddy
Recent work has developed optimization procedures to find token sequences, called adversarial triggers, which can elicit unsafe responses from aligned language models. These trigge…