5 citations · 9 across the 18 of their papers we have counts for
22 papers
Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
Ayan Sengupta, Vaibhav Seth, Tanmoy Chakraborty
Knowledge Distillation (KD) trains a smaller-capacity student model to imitate a larger-capacity teacher model by matching output distributions, implicitly assuming the teacher to…
Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization
Palaash Goel, Ayan Sengupta, Akshay Nambi +1
Structured pruning is a promising approach for compressing large language models (LLMs), yet existing methods rely heavily on greedy heuristics that produce myopic decisions, and o…
Latent Performance Profiling of Large Language Models
Tanmoy Chakraborty, Ayan Sengupta, Suparna Bhattacharya +7
Large language models (LLMs) frequently achieve impressive scores on standardized benchmarks, yet accuracy alone offers a limited view of their capabilities. Evaluating open-source…
From Images to Words: Efficient Cross-Modal Knowledge Distillation to Language Models from Black-box Teachers
Ayan Sengupta, Shantanu Dixit, Md Shad Akhtar +1
Knowledge distillation (KD) methods are pivotal in compressing large pre-trained language models into smaller models, ensuring computational efficiency without significantly droppi…
Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics
Samhruth Ananthanarayanan, Ayan Sengupta, Tanmoy Chakraborty
As context windows in LLMs scale to 100K+ tokens, the key-value (KV) cache becomes the dominant memory bottleneck, with recent methods claiming 80-90% savings and minimal benchmark…
The Art of Scaling Test-Time Compute for Large Language Models
Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty
Test-time scaling (TTS) -- the dynamic allocation of compute during inference -- is a promising direction for improving reasoning in large language models (LLMs). However, a system…
Value-Guided KV Compression for LLMs via Approximated CUR Decomposition
Ayan Sengupta, Siddhant Chaudhary, Tanmoy Chakraborty
Key-value (KV) cache compression has emerged as a critical technique for reducing the memory and latency overhead of autoregressive language models during inference. Prior approach…
Position: Enough of Scaling LLMs! Lets Focus on Downscaling
Yash Goel, Ayan Sengupta, Tanmoy Chakraborty
We challenge the dominant focus on neural scaling laws and advocate for a paradigm shift toward downscaling in the development of large language models (LLMs). While scaling laws h…
First Finish Search: Efficient Test-Time Scaling in Large Language Models
Aradhye Agarwal, Ayan Sengupta, Tanmoy Chakraborty
Test-time scaling (TTS), which involves dynamic allocation of compute during inference, offers a promising way to improve reasoning in large language models. While existing TTS met…
On the Generalization vs Fidelity Paradox in Knowledge Distillation
Suhas Kamasetty Ramesh, Ayan Sengupta, Tanmoy Chakraborty
Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance. Despite the recent traction of KD research, its…
Order-preserving unique Hahn-Banach extensions
Tanmoy Paul, T. S. S. R. K. Rao
Let be a real Banach lattice with a unit, and let be a closed subspace containing the unit. In this paper, we study the order-theoretic (also isometric) structu…
You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning
Ayan Sengupta, Siddhant Chaudhary, Tanmoy Chakraborty
The ever-increasing size of large language models (LLMs) presents significant challenges for deployment due to their heavy computational and memory requirements. Current model prun…
How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines
Ayan Sengupta, Yash Goel, Tanmoy Chakraborty
Neural scaling laws have revolutionized the design and optimization of large-scale AI models by revealing predictable relationships between model size, dataset volume, and computat…
Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation
Ayan Sengupta, Vaibhav Seth, Arinjay Pathak +5
Large Language Models (LLMs) are highly resource-intensive to fine-tune due to their enormous size. While low-rank adaptation is a prominent parameter-efficient fine-tuning approac…
Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models
Aradhye Agarwal, Suhas K Ramesh, Ayan Sengupta +1
Fine-tuning large language models (LLMs) on downstream tasks requires substantial computational resources. Selective PEFT, a class of parameter-efficient fine-tuning (PEFT) methodo…
Manifold-Preserving Transformers are Effective for Short-Long Range Encoding
Ayan Sengupta, Md Shad Akhtar, Tanmoy Chakraborty
Multi-head self-attention-based Transformers have shown promise in different learning tasks. Albeit these models exhibit significant improvement in understanding short-term and lon…
Persona-aware Generative Model for Code-mixed Language
Ayan Sengupta, Md Shad Akhtar, Tanmoy Chakraborty
Code-mixing and script-mixing are prevalent across online social networks and multilingual societies. However, a user's preference toward code-mixing depends on the socioeconomic s…
A Comprehensive Understanding of Code-mixed Language Semantics using Hierarchical Transformer
Ayan Sengupta, Tharun Suresh, Md Shad Akhtar +1
Being a popular mode of text-based communication in multilingual communities, code-mixing in online social media has became an important subject to study. Learning the semantics an…
HIT: A Hierarchically Fused Deep Attention Network for Robust Code-mixed Language Representation
Ayan Sengupta, Sourabh Kumar Bhattacharjee, Tanmoy Chakraborty +1
Understanding linguistics and morphology of resource-scarce code-mixed texts remains a key challenge in text processing. Although word embedding comes in handy to support downstrea…
An Embedding-based Joint Sentiment-Topic Model for Short Texts
Ayan Sengupta, William Scott Paka, Suman Roy +2
Short text is a popular avenue of sharing feedback, opinions and reviews on social media, e-commerce platforms, etc. Many companies need to extract meaningful information (which ma…
Fault Detection Engine in Intelligent Predictive Analytics Platform for DCIM
Bodhisattwa Prasad Majumder, Ayan Sengupta, Sajal jain +1
With the advancement of huge data generation and data handling capability, Machine Learning and Probabilistic modelling enables an immense opportunity to employ predictive analytic…
A Large-scale Analysis of the Marketplace Characteristics in Fiverr
Suman Kalyan Maity, Chandra Bhanu Jha, Avinash Kumar +3
Crowdsourcing platforms have become quite popular due to the increasing demand of human computation-based tasks. Though the crowdsourcing systems are primarily demand-driven like M…