87 citations · 144 across the 17 of their papers we have counts for
17 papers
Faster Query-Key Learning Sharpens Attention in Self-Attention Models
Rahul Vashisht, Harish G. Ramaswamy
A standard self-attention layer consists of two interacting circuits: the query-key circuit that governs attention allocation, and the output-value circuit that maps attended repre…
Optimizing Shortfall Risk Metric for Learning Regression Models
Harish G. Ramaswamy, L. A. Prashanth
We consider the problem of estimating and optimizing utility-based shortfall risk (UBSR) of a loss, say , in the context of a regression problem. Empirical risk min…
Impact of Label Noise on Learning Complex Features
Rahul Vashisht, P. Krishna Kumar, Harsha Vardhan Govind +1
Neural networks trained with stochastic gradient descent exhibit an inductive bias towards simpler decision boundaries, typically converging to a narrow family of functions, and of…
Graph Classification with GNNs: Optimisation, Representation and Inductive Bias
P. Krishna Kumar a, Harish G. Ramaswamy
Theoretical studies on the representation power of GNNs have been centered around understanding the equivalence of GNNs, using WL-Tests for detecting graph isomorphism. In this pap…
Half-Space Feature Learning in Neural Networks
Mahesh Lorik Yadav, Harish Guruprasad Ramaswamy, Chandrashekar Lakshminarayanan
There currently exist two extreme viewpoints for neural network feature learning -- (i) Neural networks simply implement a kernel method (a la NTK) and hence no features are learne…
On the Learning Dynamics of Attention Networks
Rahul Vashisht, Harish G. Ramaswamy
Attention models are typically learned by optimizing one of three standard loss functions that are variously called -- soft attention, hard attention, and latent variable marginal…
On the Interpretability of Attention Networks
Lakshmi Narayan Pandey, Rahul Vashisht, Harish G. Ramaswamy
Attention mechanisms form a core component of several successful deep learning architectures, and are based on one key idea: ''The output depends only on a small (but unknown) segm…
Consistent Multiclass Algorithms for Complex Metrics and Constraints
Harikrishna Narasimhan, Harish G. Ramaswamy, Shiv Kumar Tavker +3
We present consistent algorithms for multiclass learning with complex performance metrics and constraints, where the objective and constraints are defined by arbitrary functions of…
Predicting the success of Gradient Descent for a particular Dataset-Architecture-Initialization (DAI)
Umangi Jain, Harish G. Ramaswamy
Despite their massive success, training successful deep neural networks still largely relies on experimentally choosing an architecture, hyper-parameters, initialization, and train…
Using noise resilience for ranking generalization of deep neural networks
Depen Morwani, Rahul Vashisht, Harish G. Ramaswamy
Recent papers have shown that sufficiently overparameterized neural networks can perfectly fit even random labels. Thus, it is crucial to understand the underlying reason behind th…
Inductive Bias of Gradient Descent for Weight Normalized Smooth Homogeneous Neural Nets
Depen Morwani, Harish G. Ramaswamy
We analyze the inductive bias of gradient descent for weight normalized smooth homogeneous neural nets, when trained on exponential or cross-entropy loss. We analyse both standard…
Convex Calibrated Surrogates for the Multi-Label F-Measure
Mingyuan Zhang, Harish G. Ramaswamy, Shivani Agarwal
The F-measure is a widely used performance measure for multi-label classification, where multiple labels can be active in an instance simultaneously (e.g. in image tagging, multipl…
On Controllable Sparse Alternatives to Softmax
Anirban Laha, Saneem A. Chemmengath, Priyanka Agrawal +3
Converting an n-dimensional vector to a probability distribution over n objects is a commonly used component in many machine learning tasks like multiclass classification, multilab…
Mixture Proportion Estimation via Kernel Embedding of Distributions
Harish G. Ramaswamy, Clayton Scott, Ambuj Tewari
Mixture proportion estimation (MPE) is the problem of estimating the weight of a component distribution in a mixture, given samples from the mixture and component. This problem con…
Consistent Algorithms for Multiclass Classification with a Reject Option
Harish G. Ramaswamy, Ambuj Tewari, Shivani Agarwal
We consider the problem of -class classification (), where the classifier can choose to abstain from making predictions at a given cost, say, a factor of the cost o…
Consistent Classification Algorithms for Multi-class Non-Decomposable Performance Metrics
Harish G. Ramaswamy, Harikrishna Narasimhan, Shivani Agarwal
We study consistency of learning algorithms for a multi-class performance metric that is a non-decomposable function of the confusion matrix of a classifier and cannot be expressed…
Convex Calibration Dimension for Multiclass Loss Matrices
Harish G. Ramaswamy, Shivani Agarwal
We study consistency properties of surrogate loss functions for general multiclass learning problems, defined by a general multiclass loss matrix. We extend the notion of classific…