6 papers
ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
Jiaxi Li, Lu Yin, Li Shen +5
Large Language Models (LLMs) have achieved remarkable capabilities, but their immense computational demands during training remain a critical bottleneck for widespread adoption. Lo…
Long Chain-of-Thought Compression via Fine-Grained Group Policy Optimization
Xinchen Han, Hossam Afifi, Michel Marot +2
Large Language Models (LLMs) often generate unnecessarily verbose Chain-of-Thought (CoT) reasoning that increases computational costs and latency without proportional performance g…
Search or Accelerate: Confidence-Switched Position Beam Search for Diffusion Language Models
Mingyu Cao, Alvaro H. C. Correia, Christos Louizos +2
Diffusion Language Models (DLMs) generate text by iteratively denoising a masked sequence, repeatedly deciding which positions to commit at each step. Standard decoding follows a g…
Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning
Andy Li, Aiden Durrant, Milan Markovic +5
Pruning of deep neural networks has been an effective technique for reducing model size while preserving most of the performance of dense networks, crucial for deploying models on…
LOST: Low-rank and Sparse Pre-training for Large Language Models
Jiaxi Li, Lu Yin, Li Shen +6
While large language models (LLMs) have achieved remarkable performance across a wide range of tasks, their massive scale incurs prohibitive computational and memory costs for pre-…
OWLed: Outlier-weighed Layerwise Pruning for Efficient Autonomous Driving Framework
Jiaxi Li, Lu Yin, Xilu Wang
The integration of Large Language Models (LLMs) into autonomous driving systems offers promising enhancements in environmental understanding and decision-making. However, the subst…