6 papers
Stochastic gradient descent with discontinuity across a manifold
Vivek S. Borkar
Stochastic gradient descent for a loss function discontinuous across lower dimensional manifolds is analyzed by studying its differential equation limit.
Exploding and vanishing gradients in deep neural networks: the effect of residual connections
Vivek S Borkar
The well known phenomenon of exploding and vanishing gradients in deep neural networks is analyzed using multiplicative ergodic theory. The effect of adding a residual connection i…
Adynamical systems view of training generativemodels and the memorization phenomenon
Siva Athreya, Chiranjib Bhattacharya, Vivek S. Borkar
Using recent works of one of the authors (VSB) on collapse in generative models and two time scale dynamics in stochastic gradient descent in high dimensions, we give a system theo…
Stochastic approximation in non-markovian environments revisited
Vivek Shripad Borkar
Based on some recent work of the author on stochastic approximation in non-markovian environments, the situation when the driving random process is non-ergodic in addition to being…
Small noise asymptotics for a class of jump-diffusions with heavy tails for large times
Sumith Reddy Anugu, Siva R. Athreya, Vivek S. Borkar
In this work, we investigate positive recurrent Lévy diffusions driven by appropriately scaled Brownian motion and -stable process (with ) in the small noise regime. S…
Asymptotic convexity of wide and shallow neural networks
Vivek Borkar, Parthe Pandit
For a simple model of shallow and wide neural networks, we show that the epigraph of its input-output map as a function of the network parameters approximates epigraph of a. convex…