13 papers
TaxoSAE: Rethinking Sparse Autoencoders through Taxonomic Logic
Xinyu Guo, Minglai Yang, Clayton T. Morrison +2
Recent Sparse Autoencoders (SAEs) decompose the representations of large language models (LLMs) into interpretable features, enabling precise readout and control of encoded knowled…
Evidence-First Agentic Verification
Reza Sanayei, Razvan-Gabriel Dumitru, Sunjie Hou +7
Long-horizon, semi-verifiable agentic tasks often rely on rubric-based evaluation, but the evidence needed to assess these tasks can be spread across the environment state and the…
Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers
Haris Riaz, Hyungji Kim, Mihai Surdeanu
Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}. We introduce \textbf{S}yntax-\textbf{i}nformed…
Understanding Cultural Alignment in Multilingual LLMs via Natural Debate Statements
Vlad-Andrei Negru, Camelia Lemnaru, Mihai Surdeanu +1
In this work we investigate the sociocultural values learned by large language models (LLMs). We introduce a novel open-access dataset, Sociocultural Statements, constructed from n…
AlignSAE: Concept-Aligned Sparse Autoencoders
Minglai Yang, Xinyu Guo, Zhengliang Shi +4
Large Language Models (LLMs) encode factual knowledge within hidden parametric spaces that are difficult to inspect or control. While Sparse Autoencoders (SAEs) can decompose hidde…
The Answer Lies Within: Self-Derived Rewards Enable Explainable Relation Extraction
Xinyu Guo, Zhengliang Shi, Minglai Yang +2
Despite the remarkable reasoning capabilities of large language models, they still struggle with one-shot relation extraction without predefined relation labels. We identify two pi…
How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark
Minglai Yang, Ethan Huang, Liang Zhang +3
We introduce Grade School Math with Distracting Context (GSM-DC), a synthetic benchmark to evaluate Large Language Models' (LLMs) reasoning robustness against systematically contro…
Say Less, Mean More: Leveraging Pragmatics in Retrieval-Augmented Generation
Haris Riaz, Ellen Riloff, Mihai Surdeanu
We propose a simple, unsupervised method that injects pragmatic principles in retrieval-augmented generation (RAG) frameworks such as Dense Passage Retrieval to enhance the utility…
MorphNLI: A Stepwise Approach to Natural Language Inference Using Text Morphing
Vlad Andrei Negru, Robert Vacareanu, Camelia Lemnaru +2
We introduce MorphNLI, a modular step-by-step approach to natural language inference (NLI). When classifying the premise-hypothesis pairs into {entailment, contradiction, neutral},…
CopySpec: Accelerating LLMs with Speculative Copy-and-Paste Without Compromising Quality
Razvan-Gabriel Dumitru, Minglai Yang, Vikas Yadav +1
We introduce CopySpec, a simple yet effective technique to tackle the inefficiencies LLMs face when generating responses that closely resemble previous outputs or responses that ca…
Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy
Razvan-Gabriel Dumitru, Paul-Ioan Clotan, Vikas Yadav +2
This paper introduces a novel model compression approach through dynamic layer-specific pruning in Large Language Models (LLMs), enhancing the traditional methodology established b…
Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels
Razvan-Gabriel Dumitru, Vikas Yadav, Rishabh Maheshwary +3
We present a simple meta quantization approach that quantizes different layers of a large language model (LLM) at different bit levels, and is independent of the underlying quantiz…
ELLEN: Extremely Lightly Supervised Learning For Efficient Named Entity Recognition
Haris Riaz, Razvan-Gabriel Dumitru, Mihai Surdeanu
In this work, we revisit the problem of semi-supervised named entity recognition (NER) focusing on extremely light supervision, consisting of a lexicon containing only 10 examples…