23 citations · 35 across the 40 of their papers we have counts for
10 papers · 1 filter
LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess
Sai Kolasani, Maxim Saplin, Nicholas Crispino +5
We introduce LLM CHESS, an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs) through extend…
VMDT: Decoding the Trustworthiness of Video Foundation Models
Yujin Potter, Zhun Wang, Nicholas Crispino +11
As foundation models become more sophisticated, ensuring their trustworthiness becomes increasingly critical; yet, unlike text and image, the video modality still lacks comprehensi…
Predicting Task Performance with Context-aware Scaling Laws
Kyle Montgomery, David Park, Jianhong Tu +4
Scaling laws have transformed our understanding of large language models by linking upstream metrics like cross-entropy loss to design factors such as model size, training data, an…
Budget-aware Test-time Scaling via Discriminative Verification
Kyle Montgomery, Sijun Tan, Yuqi Chen +4
Test-time scaling is a powerful strategy for boosting the performance of large language models on complex reasoning tasks. While state-of-the-art approaches often employ generative…
SteeringSafety: Benchmarking Representation Steering in LLMs Across Safety Perspectives
Vincent Siu, Nicholas Crispino, David Park +5
We introduce SteeringSafety, a benchmark for evaluating representation steering methods across nine safety perspectives spanning 18 datasets. While prior work highlights the genera…
RepIt: Steering Language Models with Concept-Specific Refusal Vectors
Vincent Siu, Nathan W. Henry, Nicholas Crispino +3
Current safety evaluations of language models rely on benchmark-based assessments that may miss localized vulnerabilities. We present RepIt, a simple and data-efficient framework f…
VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding
Jian Chen, Ming Li, Jihyung Kil +6
Most organizational data in this world are stored as documents, and visual retrieval plays a crucial role in unlocking the collective intelligence from all these documents. However…
AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents
Zhun Wang, Vincent Siu, Zhe Ye +6
The strong planning and reasoning capabilities of Large Language Models (LLMs) have fostered the development of agent-based systems capable of leveraging external tools and interac…
COSMIC: Generalized Refusal Direction Identification in LLM Activations
Vincent Siu, Nicholas Crispino, Zihao Yu +5
Large Language Models (LLMs) encode behaviors such as refusal within their activation space, yet identifying these behaviors remains a significant challenge. Existing methods often…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1168
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…