170 citations · 909 across the 66 of their papers we have counts for
81 papers
Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering
Yijun Dong, Kevin Miller, Qi Lei +1
Despite the empirical success and practical significance of (relational) knowledge distillation that matches (the relations of) features between teacher and student models, the cor…
Robust Implicit Regularization via Weight Normalization
Hung-Hsu Chou, Holger Rauhut, Rachel Ward
Overparameterized models may have many interpolating solutions; implicit regularization refers to the hidden preference of a particular optimization method towards a certain interp…
Concentration Inequalities for Sums of Markov Dependent Random Matrices
Joe Neeman, Bobby Shi, Rachel Ward
We give Hoeffding and Bernstein-type concentration inequalities for the largest eigenvalue of sums of random matrices arising from a Markov chain. We consider time-dependent matrix…
On the fast convergence of minibatch heavy ball momentum
Raghu Bollapragada, Tyler Chen, Rachel Ward
Simple stochastic momentum methods are widely used in machine learning optimization, but their good practical performance is at odds with an absence of theoretical guarantees of ac…
How catastrophic can catastrophic forgetting be in linear regression?
Itay Evron, Edward Moroshko, Rachel Ward +2
To better understand catastrophic forgetting, we study fitting an overparameterized linear model to a sequence of tasks with different input distributions. We analyze how much the…
An Exponentially Increasing Step-size for Parameter Estimation in Statistical Models
Nhat Ho, Tongzheng Ren, Sujay Sanghavi +2
Using gradient descent (GD) with fixed or decaying step-size is a standard practice in unconstrained optimization problems. However, when the loss function is only locally convex,…
Scalable symmetric Tucker tensor decomposition
Ruhui Jin, Joe Kileel, Tamara G. Kolda +1
We study the best low-rank Tucker decomposition of symmetric tensors. The motivating application is decomposing higher-order multivariate moments. Moment tensors have special struc…
Concentration of Random Feature Matrices in High-Dimensions
Zhijun Chen, Hayden Schaeffer, Rachel Ward
The spectra of random feature matrices provide essential information on the conditioning of the linear system used in random feature regression problems and are thus connected to t…
Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Space
Juncai He, Richard Tsai, Rachel Ward
The low-dimensional manifold hypothesis posits that the data found in many applications, such as those involving natural images, lie (approximately) on low-dimensional manifolds em…
Sample Efficiency of Data Augmentation Consistency Regularization
Shuo Yang, Yijun Dong, Rachel Ward +3
Data augmentation is popular in the training of large neural networks; currently, however, there is no clear theoretical comparison between different algorithmic choices on how to…
The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variance
Matthew Faw, Isidoros Tziotis, Constantine Caramanis +3
We study convergence rates of AdaGrad-Norm as an exemplar of adaptive stochastic gradient methods (SGD), where the step sizes change based on observed stochastic gradients, for min…
SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruning
Yuege Xie, Bobby Shi, Hayden Schaeffer +1
Sparse shrunk additive models and sparse random feature models have been developed separately as methods to learn low-order functions, where there are few interactions between vari…
Learning to Forecast Dynamical Systems from Streaming Data
Dimitris Giannakis, Amelia Henriksen, Joel A. Tropp +1
Kernel analog forecasting (KAF) is a powerful methodology for data-driven, non-parametric forecasting of dynamically generated time series data. This approach has a rigorous founda…
AdaLoss: A computationally-efficient and provably convergent adaptive gradient method
Xiaoxia Wu, Yuege Xie, Simon Du +1
We propose a computationally-friendly adaptive learning rate schedule, "AdaLoss", which directly uses the information of the loss function to adjust the stepsize in gradient descen…
Arbitrary-length analogs to de Bruijn sequences
Abhinav Nellore, Rachel Ward
Let be a length- cyclic sequence of characters from a size- alphabet such that the number of occurrences of any length- string on …
Johnson-Lindenstrauss Embeddings with Kronecker Structure
Stefan Bamberger, Felix Krahmer, Rachel Ward
We prove the Johnson-Lindenstrauss property for matrices where has the restricted isometry property and is a diagonal matrix containing the entries of a Kronecker…
The Hanson-Wright Inequality for Random Tensors
Stefan Bamberger, Felix Krahmer, Rachel Ward
We provide moment bounds for expressions of the type where denotes the Kronecker pro…
Bootstrapping the error of Oja's algorithm
Robert Lunde, Purnamrita Sarkar, Rachel Ward
We consider the problem of quantifying uncertainty for the estimation error of the leading eigenvector from Oja's algorithm for streaming principal component analysis, where the da…
Generalization Bounds for Sparse Random Feature Expansions
Abolfazl Hashemi, Hayden Schaeffer, Robert Shi +3
Random feature methods have been successful in various machine learning tasks, are easy to compute, and come with theoretical accuracy bounds. They serve as an alternative approach…
Streaming k-PCA: Efficient guarantees for Oja's algorithm, beyond rank-one updates
De Huang, Jonathan Niles-Weed, Rachel Ward
We analyze Oja's algorithm for streaming -PCA and prove that it achieves performance nearly matching that of an optimal offline algorithm. Given access to a sequence of i.i.d. $…
Overparameterization and generalization error: weighted trigonometric interpolation
Yuege Xie, Hung-Hsu Chou, Holger Rauhut +1
Motivated by surprisingly good generalization properties of learned deep neural networks in overparameterized scenarios and by the related double descent phenomenon, this paper ana…
Matrix Concentration for Products
De Huang, Jonathan Niles-Weed, Joel A. Tropp +1
This paper develops nonasymptotic growth and concentration bounds for a product of independent random matrices. These results sharpen and generalize recent work of Henriksen-Ward,…
Implicit Regularization and Convergence for Weight Normalization
Xiaoxia Wu, Edgar Dobriban, Tongzheng Ren +5
Normalization methods such as batch [Ioffe and Szegedy, 2015], weight [Salimansand Kingma, 2016], instance [Ulyanov et al., 2016], and layer normalization [Baet al., 2016] have bee…
Faster Johnson-Lindenstrauss Transforms via Kronecker Products
Ruhui Jin, Tamara G. Kolda, Rachel Ward
The Kronecker product is an important matrix operation with a wide range of applications in supporting fast linear transforms, including signal processing, graph theory, quantum co…