56 citations · 174 across the 12 of their papers we have counts for
17 papers
Principled Approximation Methods for Efficient and Scalable Deep Learning
Pedro Savarese
Recent progress in deep learning has been driven by increasingly larger models. However, their computational and energy demands have grown proportionally, creating significant barr…
Approaching Deep Learning through the Spectral Dynamics of Weights
David Yunis, Kumar Kshitij Patel, Samuel Wheeler +5
We propose an empirical approach centered on the spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to unify and clarify several phe…
Drop-Connect as a Fault-Tolerance Approach for RRAM-based Deep Neural Network Accelerators
Mingyuan Xiang, Xuhan Xie, Pedro Savarese +3
Resistive random-access memory (RRAM) is widely recognized as a promising emerging hardware platform for deep neural networks (DNNs). Yet, due to manufacturing limitations, current…
SONIQ: System-Optimized Noise-Injected Ultra-Low-Precision Quantization with Full-Precision Parity
Cyrus Zhou, Pedro Savarese, Zack Hassman +4
Ultra-low-precision inference can sharply reduce memory and latency but often degrades accuracy and relies on specialized hardware. We present SONIQ, a system-optimized, noise-inje…
Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation
Xin Yuan, Pedro Savarese, Michael Maire
We develop an approach to efficiently grow neural networks, within which parameterization and optimization strategies are designed by considering their effects on the training dyna…
Information-Theoretic Segmentation by Inpainting Error Maximization
Pedro Savarese, Sunnie S. Y. Kim, Michael Maire +2
We study image segmentation from an information-theoretic perspective, proposing a novel adversarial method that performs unsupervised segmentation by partitioning images into maxi…
Growing Efficient Deep Networks by Structured Continuous Sparsification
Xin Yuan, Pedro Savarese, Michael Maire
We develop an approach to growing deep network architectures over the course of training, driven by a principled combination of accuracy and sparsity objectives. Unlike existing pr…
Kernel and Rich Regimes in Overparametrized Models
Blake Woodworth, Suriya Gunasekar, Jason D. Lee +5
A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus trai…
Winning the Lottery with Continuous Sparsification
Pedro Savarese, Hugo Silva, Michael Maire
The search for efficient, sparse deep neural network models is most prominently performed by pruning: training a dense, overparameterized network and removing parameters, usually v…
Domain-independent Dominance of Adaptive Methods
Pedro Savarese, David McAllester, Sudarshan Babu +1
From a simplified analysis of adaptive methods, we derive AvaGrad, a new optimizer which outperforms SGD on vision tasks when its adaptability is properly tuned. We observe that th…
Building a Massive Corpus for Named Entity Recognition using Free Open Data Sources
Daniel Specht Menezes, Pedro Savarese, Ruy Luiz Milidiú
With the recent progress in machine learning, boosted by techniques such as deep learning, many tasks can be successfully solved once a large enough dataset is available for traini…
On the Convergence of AdaBound and its Connection to SGD
Pedro Savarese
Adaptive gradient methods such as Adam have gained extreme popularity due to their success in training complex neural networks and less sensitivity to hyperparameter tuning compare…
Kernel and Rich Regimes in Overparametrized Models
Blake Woodworth, Suriya Gunasekar, Pedro Savarese +5
A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus trai…
Learning Implicitly Recurrent CNNs Through Parameter Sharing
Pedro Savarese, Michael Maire
We introduce a parameter sharing scheme, in which different layers of a convolutional neural network (CNN) are defined by a learned linear combination of parameter tensors from a g…
How do infinite width bounded norm networks look in function space?
Pedro Savarese, Itay Evron, Daniel Soudry +1
We consider the question of what functions can be captured by ReLU networks with an unbounded number of units (infinite width), but where the overall network Euclidean norm (sum of…
From Monte Carlo to Las Vegas: Improving Restricted Boltzmann Machine Training Through Stopping Sets
Pedro H. P. Savarese, Mayank Kakodkar, Bruno Ribeiro
We propose a Las Vegas transformation of Markov Chain Monte Carlo (MCMC) estimators of Restricted Boltzmann Machines (RBMs). We denote our approach Markov Chain Las Vegas (MCLV). M…
Learning Identity Mappings with Residual Gates
Pedro H. P. Savarese, Leonardo O. Mazza, Daniel R. Figueiredo
We propose a new layer design by adding a linear gating mechanism to shortcut connections. By using a scalar parameter to control each gate, we provide a way to learn identity mapp…