292 citations · 770 across the 46 of their papers we have counts for
53 papers
Generalization Bounds for Transformer-Based Next-Token Prediction in a Language Model
Insung Kong, Niklas Dexheimer, Johannes Schmidt-Hieber
A refined statistical understanding of LLM pre-training requires the analysis of the transformer architecture for data distributions that encapsulate key characteristics of text da…
Hyper Input Convex Neural Networks for Shape Constrained Learning and Optimal Transport
Shayan Hundrieser, Insung Kong, Johannes Schmidt-Hieber
We introduce Hyper Input Convex Neural Networks (HyCNNs), a novel neural network architecture designed for learning convex functions. HyCNNs combine the principles of Maxout networ…
Central limit theorems for the outputs of fully convolutional neural networks with time series input
Annika Betken, Giorgio Micali, Johannes Schmidt-Hieber
Deep learning is widely deployed for time series learning tasks such as classification and forecasting. Despite the empirical successes, only little theory has been developed so fa…
Semi-Supervised Learning on Graphs using Graph Neural Networks
Juntong Chen, Claire Donnat, Olga Klopp +1
Graph neural networks (GNNs) work remarkably well in semi-supervised node regression, yet a rigorous theory explaining when and why they succeed remains lacking. To address this ga…
On the Universal Representation Property of Spiking Neural Networks
Shayan Hundrieser, Philipp Tuchel, Insung Kong +1
Inspired by biology, spiking neural networks (SNNs) process information via discrete spikes over time, offering an energy-efficient alternative to the classical computing paradigm…
Statistical Guarantees for High-Dimensional Stochastic Gradient Descent
Jiaqi Li, Zhipeng Lou, Johannes Schmidt-Hieber +1
Stochastic Gradient Descent (SGD) and its Ruppert-Polyak averaged variant (ASGD) lie at the heart of modern large-scale learning, yet their theoretical properties in high-dimension…
Spike-timing-dependent Hebbian learning as noisy gradient descent
Niklas Dexheimer, Sascha Gaudlitz, Johannes Schmidt-Hieber
Hebbian learning is a key principle underlying learning in biological neural networks. We relate a Hebbian spike-timing-dependent plasticity rule to noisy gradient descent with res…
On the expressivity of deep Heaviside networks
Insung Kong, Juntong Chen, Sophie Langer +1
We show that deep Heaviside networks (DHNs) have limited expressiveness but that this can be overcome by including either skip connections or neurons with linear activation. We pro…
Ordinal Patterns Based Change Points Detection
Annika Betken, Giorgio Micali, Johannes Schmidt-Hieber
The ordinal patterns of a fixed number of consecutive values in a time series is the spatial ordering of these values. Counting how often a specific ordinal pattern occurs in a tim…
Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling
Niklas Dexheimer, Johannes Schmidt-Hieber
Forward gradient descent (FGD) has been proposed as a biologically more plausible alternative of gradient descent as it can be computed without backward pass. Considering the linea…
On the VC dimension of deep group convolutional neural networks
Anna Sepliarskaia, Sophie Langer, Johannes Schmidt-Hieber
We study the generalization capabilities of Group Convolutional Neural Networks (GCNNs) with ReLU activation function by deriving upper and lower bounds for their Vapnik-Chervonenk…
Understanding the Effect of GCN Convolutions in Regression Tasks
Juntong Chen, Johannes Schmidt-Hieber, Claire Donnat +1
Graph Convolutional Networks (GCNs) have become a pivotal method in machine learning for modeling functions over graphs. Despite their widespread success across various application…
Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Models
Jiaqi Li, Johannes Schmidt-Hieber, Wei Biao Wu
This paper proposes an asymptotic theory for online inference of the stochastic gradient descent (SGD) iterates with dropout regularization in linear regression. Specifically, we e…
Generative Modelling via Quantile Regression
Johannes Schmidt-Hieber, Petr Zamolodtchikov
We link conditional generative modelling to quantile regression. We propose a suitable loss function and derive minimax convergence rates for the associated risk under smoothness a…
Hebbian learning inspired estimation of the linear regression parameters from queries
Johannes Schmidt-Hieber, Wouter M Koolen
Local learning rules in biological neural networks (BNNs) are commonly referred to as Hebbian learning. [26] links a biologically motivated Hebbian learning rule to a specific zero…
Convergence guarantees for forward gradient descent in the linear regression model
Thijs Bos, Johannes Schmidt-Hieber
Renewed interest in the relationship between artificial and biological neural networks motivates the study of gradient-free methods. Considering the linear regression model with ra…
Dropout Regularization Versus -Penalization in the Linear Model
Gabriel Clara, Sophie Langer, Johannes Schmidt-Hieber
We investigate the statistical behavior of gradient descent iterates with dropout in the linear regression model. In particular, non-asymptotic bounds for the convergence of expect…
A supervised deep learning method for nonparametric density estimation
Thijs Bos, Johannes Schmidt-Hieber
Nonparametric density estimation is an unsupervised learning problem. In this work we propose a two-step procedure that casts the density estimation problem in the first step into…
Lower bounds for the trade-off between bias and mean absolute deviation
Alexis Derumigny, Johannes Schmidt-Hieber
In nonparametric statistics, rate-optimal estimators typically balance bias and stochastic error. The recent work on overparametrization raises the question whether rate-optimal es…
Codivergences and information matrices
Alexis Derumigny, Johannes Schmidt-Hieber
We propose a new concept of codivergence, which quantifies the similarity between two probability measures relative to a reference probability measure . In the neig…
Interpreting learning in biological neural networks as zero-order optimization method
Johannes Schmidt-Hieber
Recently, significant progress has been made regarding the statistical understanding of artificial neural networks (ANNs). ANNs are motivated by the functioning of the brain, but d…
A novel statistical approach to analyze image classification
Juntong Chen, Sophie Langer, Johannes Schmidt-Hieber
The recent statistical theory of neural networks focuses on nonparametric denoising problems that treat randomness as additive noise. Variability in image classification datasets d…
On the inability of Gaussian process regression to optimally learn compositional functions
Matteo Giordano, Kolyan Ray, Johannes Schmidt-Hieber
We rigorously prove that deep Gaussian process priors can outperform Gaussian process priors if the target function has a compositional structure. To this end, we study information…
Local convergence rates of the nonparametric least squares estimator with applications to transfer learning
Johannes Schmidt-Hieber, Petr Zamolodtchikov
Convergence properties of empirical risk minimizers can be conveniently expressed in terms of the associated population risk. To derive bounds for the performance of the estimator…