1 citations · 1 across the 18 of their papers we have counts for
6 papers · 1 filter
Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scaling
Giorgio Giannone, Guangxuan Xu, Nikhil Shivakumar Nayak +4
Inference-Time Scaling (ITS) improves language models by allocating more computation at generation time. Particle Filtering (PF) has emerged as a strong ITS method for complex math…
Inference-Time Scaling of Diffusion Language Models via Trajectory Refinement
Meihua Dang, Jiaqi Han, Minkai Xu +3
Discrete diffusion models have recently emerged as strong alternatives to autoregressive language models, matching their performance through large-scale training. However, inferenc…
Hopscotch: Discovering and Skipping Redundancies in Language Models
Mustafa Eyceoz, Nikhil Shivakumar Nayak, Hao Wang +2
Modern causal language models stack many attention blocks to improve performance, but not all blocks are necessary for every task. We propose Hopscotch, a simple yet effective meth…
Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning
Nikhil Shivakumar Nayak, Krishnateja Killamsetty, Ligong Han +8
Continual learning in large language models (LLMs) is prone to catastrophic forgetting, where adapting to new tasks significantly degrades performance on previously learned ones. E…
SQuat: Subspace-orthogonal KV Cache Quantization
Hao Wang, Ligong Han, Kai Xu +1
The key-value (KV) cache accelerates LLMs decoding by storing KV tensors from previously generated tokens. It reduces redundant computation at the cost of increased memory usage. T…
Activation-Informed Merging of Large Language Models
Amin Heyrani Nobari, Kaveh Alim, Ali ArjomandBigdeli +3
Model merging, a method that combines the parameters and embeddings of multiple fine-tuned large language models (LLMs), offers a promising approach to enhance model performance ac…