10 papers
Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers
Haris Riaz, Hyungji Kim, Mihai Surdeanu
Positional embeddings (PE) in Transformers encode token distance and order but are largely agnostic to \textit{syntactic structure}. We introduce \textbf{S}yntax-\textbf{i}nformed…
The Answer Lies Within: Self-Derived Rewards Enable Explainable Relation Extraction
Xinyu Guo, Zhengliang Shi, Minglai Yang +1
Despite the remarkable reasoning capabilities of large language models, they still struggle with one-shot relation extraction without predefined relation labels. We identify two pi…
Understanding Cultural Alignment in Multilingual LLMs via Natural Debate Statements
Vlad-Andrei Negru, Camelia Lemnaru, Mihai Surdeanu +1
In this work we investigate the sociocultural values learned by large language models (LLMs). We introduce a novel open-access dataset, Sociocultural Statements, constructed from n…
AlignSAE: Concept-Aligned Sparse Autoencoders
Minglai Yang, Xinyu Guo, Zhengliang Shi +4
Large Language Models (LLMs) encode factual knowledge within hidden parametric spaces that are difficult to inspect or control. While Sparse Autoencoders (SAEs) can decompose hidde…
How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark
Minglai Yang, Ethan Huang, Liang Zhang +3
We introduce Grade School Math with Distracting Context (GSM-DC), a synthetic benchmark to evaluate Large Language Models' (LLMs) reasoning robustness against systematically contro…
CopySpec: Accelerating LLMs with Speculative Copy-and-Paste Without Compromising Quality
Razvan-Gabriel Dumitru, Minglai Yang, Vikas Yadav +1
We introduce CopySpec, a simple yet effective technique to tackle the inefficiencies LLMs face when generating responses that closely resemble previous outputs or responses that ca…