202 citations · 638 across the 18 of their papers we have counts for
18 papers
Self-Evaluation Guided Beam Search for Reasoning
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao +4
Breaking down a problem into intermediate steps has demonstrated impressive performance in Large Language Model (LLM) reasoning. However, the growth of the reasoning chain introduc…
An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularization
Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi +2
Variance-Invariance-Covariance Regularization (VICReg) is a self-supervised learning (SSL) method that has shown promising results on a variety of tasks. However, the fundamental m…
Auxiliary Learning as an Asymmetric Bargaining Game
Aviv Shamsian, Aviv Navon, Neta Glazer +3
Auxiliary learning is an effective method for enhancing the generalization capabilities of trained models, particularly when dealing with small datasets. However, this approach may…
Robustness Implies Generalization via Data-Dependent Generalization Bounds
Kenji Kawaguchi, Zhun Deng, Kyle Luh +1
This paper proves that robustness implies generalization via data-dependent generalization bounds. As a result, robustness and generalization are shown to be connected closely in a…
Eliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit
Jascha Sohl-Dickstein, Kenji Kawaguchi
Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization. We show that the cor…
Elimination of All Bad Local Minima in Deep Learning
Kenji Kawaguchi, Leslie Pack Kaelbling
In this paper, we theoretically prove that adding one special neuron per output unit eliminates all suboptimal local minima of any deep neural network, for multi-class classificati…
Effect of Depth and Width on Local Minima in Deep Learning
Kenji Kawaguchi, Jiaoyang Huang, Leslie Pack Kaelbling
In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. With…
Depth with Nonlinearity Creates No Bad Local Minima in ResNets
Kenji Kawaguchi, Yoshua Bengio
In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense t…
Generalization in Machine Learning via Analytical Learning Theory
Kenji Kawaguchi, Yoshua Bengio, Vikas Verma +1
This paper introduces a novel measure-theoretic theory for machine learning that does not require statistical assumptions. Based on this theory, a new regularization method in deep…
Theory of Deep Learning III: explaining the non-overfitting puzzle
Tomaso Poggio, Kenji Kawaguchi, Qianli Liao +5
A main puzzle of deep networks revolves around the absence of overfitting despite large overparametrization and despite the large capacity demonstrated by zero training error on ra…
Generalization in Deep Learning
Kenji Kawaguchi, Leslie Pack Kaelbling, Yoshua Bengio
This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness…
Deep Semi-Random Features for Nonlinear Function Approximation
Kenji Kawaguchi, Bo Xie, Vikas Verma +1
We propose semi-random features for nonlinear function approximation. The flexibility of semi-random feature lies between the fully adjustable units in deep learning and the random…
Depth Creates No Bad Local Minima
Haihao Lu, Kenji Kawaguchi
In deep learning, \textit{depth}, as well as \textit{nonlinearity}, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that wi…
Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
Qianli Liao, Kenji Kawaguchi, Tomaso Poggio
We systematically explored a spectrum of normalization algorithms related to Batch Normalization (BN) and propose a generalized formulation that simultaneously solves two major lim…
Global Continuous Optimization with Error Bound and Fast Convergence
Kenji Kawaguchi, Yu Maruyama, Xiaoyu Zheng
This paper considers global optimization with a black-box unknown objective function that can be non-convex and non-differentiable. Such a difficult optimization problem arises in…
Deep Learning without Poor Local Minima
Kenji Kawaguchi
In this paper, we prove a conjecture published in 1989 and also partially address an open problem announced at the Conference on Learning Theory (COLT) 2015. With no unrealistic as…
Bounded Optimal Exploration in MDP
Kenji Kawaguchi
Within the framework of probably approximately correct Markov decision processes (PAC-MDP), much theoretical work has focused on methods to attain near optimality after a relativel…
Bayesian Optimization with Exponential Convergence
Kenji Kawaguchi, Leslie Pack Kaelbling, Tomás Lozano-Pérez
This paper presents a Bayesian optimization method with exponential convergence without the need of auxiliary optimization and without the delta-cover sampling. Most Bayesian optim…