10 papers
The Sharp Directions Are Against You: Curvature Analysis of Activation Steering
Seyedarmin Azizi, Arya Fayyazi, Parsa Razmara +3
Activation steering adds a vector to a language model's internal representations at inference time. It is a lightweight alternative to fine-tuning for behavioral control. The stand…
Running Fewer Loops Removes Refusal Behaviour in Recurrent-Depth Language Models
Seyedarmin Azizi, Erfan Baghaei Potraghloo, Arya Fayyazi +2
A recurrent-depth language model reuses one block of weights several times in a row. The number of repetitions, which we call loops, is chosen when the model runs, not when it is t…
Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution
Erfan Baghaei Potraghloo, Seyedarmin Azizi, Arya Fayyazi +4
A language model can give a correct answer more probability than any single incorrect answer and still usually sample an incorrect one, because the incorrect answers together hold…
Labels Override Definitions in Jev-Style Typed Decision Models
Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram
A typed decision model answers a fixed question about an input by returning a probability for each of several caller-defined options. Each option carries a short label and a writte…
Chopthin-Consensus Power Sampling: A Diversity-Preserving Approach to LLM Decoding
Minoo Ahmadi, Seyedarmin Azizi, Erfan Baghaei Potraghloo +2
Inference-time power sampling via Sequential Monte Carlo (SMC) can substantially improve large language model (LLM) reasoning without requiring post-training. However, many existin…
One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness
Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu +1
Instruction-tuned large language models produce helpful, structured responses, but how robust is this helpfulness under trivial constraints? We show that simple lexical constraints…
Power-SMC: Low-Latency Sequence-Level Power Sampling for Training-Free LLM Reasoning
Seyedarmin Azizi, Erfan Baghaei Potraghloo, Minoo Ahmadi +2
Reasoning ability in large language models is often attributed to \emph{distribution sharpening}: concentrating output probability on high-likelihood sequences. Recent works show t…
SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models
Jiayi Tian, Seyedarmin Azizi, Yequan Zhao +7
Large reasoning models (LRMs) often incur significant key-value (KV) cache overhead, due to their linear growth with the verbose chain-of-thought (CoT) reasoning. This incurs both…
Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation
Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu +1
Large language models (LLMs), despite their impressive performance across a wide range of tasks, often struggle to balance two competing objectives in open-ended text generation: f…
Activation Steering for Chain-of-Thought Compression
Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram
Large language models (LLMs) excel at complex reasoning when they include intermediate steps, known as "chains of thought" (CoTs). However, these rationales are often overly verbos…