54 citations · 150 across the 28 of their papers we have counts for
32 papers
FinancialAuditBench: Benchmark Construction under Differential Privacy Using Real-World Priors
Jerry Huang, Sarvesh Babu, Matt Van Buren +5
As AI agents are becoming widely adopted in the financial services industry, careful measurement is essential to understand where they can be reliably deployed and where oversight…
A Survey on the Linear Representation Hypothesis
Sewoong Lee, Marc E. Canby, Ikhyun Cho +1
The term "linear representation hypothesis" (LRH) has appeared across diverse subfields of artificial intelligence, neuroscience, and cognitive science. But previous works have not…
MineCEraft: Evaluating Language Models as Construction Engineers in the World of Minecraft
Sewoong Lee, Risham Sidhu, Julia Hockenmaier +1
We introduce MineCEraft (Minecraft Construction Engineering Benchmark, pronounced mine-see-ee-raft), an easy-to-use, open-source benchmark designed to systematically evaluate the r…
Steering Instruction Hierarchies at Inference Time
Siqi Zeng, Sewoong Lee, Han Zhao +1
Instruction hierarchies are a core safety assumption of language model deployment: higher priority inputs, such as system prompts, should override conflicting lower priority inputs…
ReasoningFlow: Discourse Structures for Understanding LLM Reasoning Traces
Jinu Lee, Shivam Agarwal, Amruta Parulekar +3
Large reasoning models (LRMs) produce reasoning traces with non-linear structures, such as backtracking and self-correction, that complicate the evaluation and monitoring of the re…
SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors
Yekaterina Yegorova, Argyrios Gerogiannis, Haolong Zheng +3
Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight…
Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures
Risham Sidhu, Julia Hockenmaier
We introduce GSU, a text-only grid dataset to evaluate the spatial reasoning capabilities of LLMs over 3 core tasks: navigation, object localization, and structure composition. By…
Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics
Jinu Lee, Kyoung-Woon On, Simeng Han +2
Evaluating the quality of LLM-generated reasoning traces in expert domains (e.g., law) is essential for ensuring credibility and explainability, yet remains challenging due to the…
Contextual Relevance and Adaptive Sampling for LLM-Based Document Reranking
Jerry Huang, Siddarth Madala, Cheng Niu +2
Reranking algorithms have made progress in improving document retrieval quality by efficiently aggregating relevance judgments generated by large language models (LLMs). However, i…
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
Rajarshi Haldar, Julia Hockenmaier
As Natural Language Generation (NLG) continues to be widely adopted, properly assessing it has become quite difficult. Lately, using large language models (LLMs) for evaluating the…
ReasoningFlow: Semantic Structure of Complex Reasoning Traces
Jinu Lee, Sagnik Mukherjee, Dilek Hakkani-Tur +1
Large reasoning models (LRMs) generate complex reasoning traces with planning, reflection, verification, and backtracking. In this work, we introduce ReasoningFlow, a unified schem…
Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality
Sewoong Lee, Adam Davies, Marc E. Canby +1
Sparse autoencoders (SAEs) are widely used in mechanistic interpretability research for large language models; however, the state-of-the-art method of using -sparse autoencoders…
Scaling Evaluation-time Compute with Reasoning Models as Evaluators
Seungone Kim, Ian Wu, Jinu Lee +8
As language model (LM) outputs get more and more natural, it is becoming more difficult than ever to evaluate their quality. Simultaneously, increasing LMs' "thinking" time through…
RAG-RL: Advancing Retrieval-Augmented Generation via RL and Curriculum Learning
Jerry Huang, Siddarth Madala, Risham Sidhu +4
Retrieval-augmented generation (RAG) systems rely on retrieval models for identifying relevant contexts and answer generation models for utilizing those contexts. However, retrieve…
Entailment-Preserving First-order Logic Representations in Natural Language Entailment
Jinu Lee, Qi Liu, Runzhi Ma +4
First-order logic (FOL) can represent the logical entailment semantics of natural language (NL) sentences, but determining natural language entailment using FOL remains a challenge…
Evaluating Step-by-step Reasoning Traces: A Survey
Jinu Lee, Julia Hockenmaier
Step-by-step reasoning is widely used to enhance the reasoning ability of large language models (LLMs) in complex problems. Evaluating the quality of reasoning traces is crucial fo…
BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues
Prashant Jayannavar, Liliang Ren, Marisa Hudspeth +6
Developing interactive agents that can understand language, perceive their surroundings, and act within the physical world is a long-standing goal of AI research. The Minecraft Col…
How Reliable are Causal Probing Interventions?
Marc Canby, Adam Davies, Chirag Rastogi +1
Causal probing aims to analyze foundation models by examining how intervening on their representation of various latent properties impacts their outputs. Recent works have cast dou…
Analyzing the Performance of Large Language Models on Code Summarization
Rajarshi Haldar, Julia Hockenmaier
Large language models (LLMs) such as Llama 2 perform very well on tasks that involve both natural language and source code, particularly code summarization and code generation. We…
ViT-MUL: A Baseline Study on Recent Machine Unlearning Methods Applied to Vision Transformers
Ikhyun Cho, Changyeon Park, Julia Hockenmaier
Machine unlearning (MUL) is an arising field in machine learning that seeks to erase the learned information of specific training data points from a trained model. Despite the rece…
Attack and Reset for Unlearning: Exploiting Adversarial Noise toward Machine Unlearning through Parameter Re-initialization
Yoonhwa Jung, Ikhyun Cho, Shun-Hsiang Hsu +1
With growing concerns surrounding privacy and regulatory compliance, the concept of machine unlearning has gained prominence, aiming to selectively forget or erase specific learned…
A Framework for Bidirectional Decoding: Case Study in Morphological Inflection
Marc E. Canby, Julia Hockenmaier
Transformer-based encoder-decoder models that generate outputs in a left-to-right fashion have become standard for sequence-to-sequence tasks. In this paper, we propose a framework…
Multimedia Generative Script Learning for Task Planning
Qingyun Wang, Manling Li, Hou Pong Chan +4
Goal-oriented generative script learning aims to generate subsequent steps to reach a particular goal, which is an essential task to assist robots or humans in performing stereotyp…
Human-guided Collaborative Problem Solving: A Natural Language based Framework
Harsha Kokel, Mayukh Das, Rakibul Islam +10
We consider the problem of human-machine collaborative problem solving as a planning task coupled with natural language communication. Our framework consists of three components --…