9 papers
CircuitSteer: Geometrically Aligned Multi-Layer Steering via Sparse Autoencoder Circuits
Mehrshad Saadatinia, Parsa Razmara, Ardalan Aryashad +2
Controlling the behavior of large language models (LLMs) remains a critical challenge for AI alignment. Existing steering methods, such as Contrastive Activation Addition (CAA), ty…
IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
Ali Abbasi, Chayne Thrash, Haoran Qin +2
Large language models deliver strong performance across language and reasoning tasks, but their storage and compute costs remain major barriers to deployment in resource-constraine…
ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs
Chayne Thrash, Ali Abbasi, Soheil Kolouri
Large language models (LLMs) are costly to deploy due to their large memory footprint and high inference cost. Weight-activation quantization can reduce these costs, but low-bit ac…
Vector-Quantized Soft Label Compression for Dataset Distillation
Ali Abbasi, Ashkan Shahbazi, Hamed Pirsiavash +1
Dataset distillation is an emerging technique for reducing the computational and storage costs of training machine learning models by synthesizing a small, informative subset of da…
Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression
Ali Abbasi, Chayne Thrash, Haoran Qin +3
Advances in large language models have driven strong performance across many tasks, but their memory and compute costs still hinder deployment. SVD-based compression reduces storag…
Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
Luyang Fang, Xiaowei Yu, Jiazhang Cai +23
The exponential growth of Large Language Models (LLMs) continues to highlight the need for efficient strategies to meet ever-expanding computational and data demands. This survey p…