5 citations · 9 across the 18 of their papers we have counts for
5 papers · 1 filter
Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty
Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implicitly assuming the teacher to…
Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization
Palaash Goel, Ayan Sengupta, Akshay Nambi +1
Structured pruning is a promising approach for compressing large language models (LLMs), yet existing methods rely heavily on greedy heuristics that produce myopic decisions, and o…
Latent Performance Profiling of Large Language Models
Tanmoy Chakraborty, Ayan Sengupta, Suparna Bhattacharya +7
Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of their capabilities. Evaluating open-source…
From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers
Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar +1
Knowledge distillation (KD) methods are pivotal in compressing large pre-trained language models into smaller models, ensuring computational efficiency without significantly droppi…
Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics
Samhruth Ananthanarayanan, Ayan Sengupta, Tanmoy Chakraborty
As context windows in LLMs scale to 100K+ tokens, the key-value (KV) cache becomes the dominant memory bottleneck, with recent methods claiming 80-90% savings and minimal benchmark…