22 citations · 47 across the 13 of their papers we have counts for
9 papers · 1 filter
Memorilla: Latent Semantic Memory for LLMs
Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli +4
Memory enables large language models (LLMs) to use information beyond their parameters, such as past user conversations, long documents, or an agent's earlier actions. For a frozen…
RT-J: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions
Rishabh Ranjan, Vignesh Kothapalli, Harshvardhan Agarwal +5
Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. However, adapting existing models…
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
Tianyu Pang, Vignesh Kothapalli, Shenyang Deng +3
We study optimal learning-rate selection in two-layer and three-layer linear neural networks trained to learn linear target functions. In particular, we derive the exact closed-for…
Liger Kernel: Efficient Triton Kernels for LLM Training
Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli +7
Training Large Language Models (LLMs) efficiently at scale presents a formidable challenge, driven by their ever-increasing computational demands and the need for enhanced performa…
From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networks
Vignesh Kothapalli, Tianyu Pang, Shenyang Deng +2
Training strategies for modern deep neural networks (NNs) tend to induce a heavy-tailed (HT) empirical spectral density (ESD) in the layer weights. While previous efforts have show…
Can Kernel Methods Explain How the Data Affects Neural Collapse?
Vignesh Kothapalli, Tom Tirer
A vast amount of literature has recently focused on the "Neural Collapse" (NC) phenomenon, which emerges when training neural network (NN) classifiers beyond the zero training erro…
A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks
Vignesh Kothapalli, Tom Tirer, Joan Bruna
Graph neural networks (GNNs) have become increasingly popular for classification tasks on graph-structured data. Yet, the interplay between graph topology and feature evolution in…
Randomized Schur Complement Views for Graph Contrastive Learning
Vignesh Kothapalli
We introduce a randomized topological augmentor based on Schur complements for Graph Contrastive Learning (GCL). Given a graph laplacian matrix, the technique generates unbiased ap…
Neural Collapse: A Review on Modelling Principles and Generalization
Vignesh Kothapalli
Deep classifier neural networks enter the terminal phase of training (TPT) when training error reaches zero and tend to exhibit intriguing Neural Collapse (NC) properties. Neural c…