391 citations · 553 across the 18 of their papers we have counts for
19 papers
A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features
Emi Zeger, Yifei Wang, Aaron Mishkin +3
We prove that training neural networks on 1-D data is equivalent to solving convex Lasso problems with discrete, explicitly defined dictionary matrices. We consider neural networks…
The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Models
Tolga Ergen, Mert Pilanci
Due to the non-convex nature of training Deep Neural Network (DNN) models, their effectiveness relies on the use of non-convex optimization heuristics. Traditional methods for trai…
Fixing the NTK: From Neural Network Linearizations to Exact Convex Programs
Rajat Vadiraj Dwaraknath, Tolga Ergen, Mert Pilanci
Recently, theoretical analyses of deep neural networks have broadly focused on two directions: 1) Providing insight into neural network training by SGD in the limit of infinite hid…
Globally Optimal Training of Neural Networks with Threshold Activation Functions
Tolga Ergen, Halil Ibrahim Gulluk, Jonathan Lacotte +1
Threshold activation functions are highly preferable in neural networks due to their efficiency in hardware implementations. Moreover, their mode of operation is more interpretable…
Convexifying Transformers: Improving optimization and understanding of transformer networks
Tolga Ergen, Behnam Neyshabur, Harsh Mehta
Understanding the fundamental mechanism behind the success of transformer networks is still an open problem in the deep learning literature. Although their remarkable performance h…
GLEAM: Greedy Learning for Large-Scale Accelerated MRI Reconstruction
Batu Ozturkler, Arda Sahiner, Tolga Ergen +6
Unrolled neural networks have recently achieved state-of-the-art accelerated MRI reconstruction. These networks unroll iterative optimization algorithms by alternating between phys…
Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers
Arda Sahiner, Tolga Ergen, Batu Ozturkler +3
Vision transformers using self-attention or its proposed alternatives have demonstrated promising results in many image related tasks. However, the underpinning inductive bias of a…
Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks
Tolga Ergen, Mert Pilanci
Understanding the fundamental principles behind the success of deep neural networks is one of the most important open questions in the current literature. To this end, we study the…
Parallel Deep Neural Networks Have Zero Duality Gap
Yifei Wang, Tolga Ergen, Mert Pilanci
Training deep neural networks is a challenging non-convex optimization problem. Recent work has proven that the strong duality holds (which means zero duality gap) for regularized…
Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs
Tolga Ergen, Mert Pilanci
Understanding the fundamental mechanism behind the success of deep neural networks is one of the key challenges in the modern machine learning literature. Despite numerous attempts…
Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions
Arda Sahiner, Tolga Ergen, Batu Ozturkler +4
Generative Adversarial Networks (GANs) are commonly used for modeling complex distributions of data. Both the generators and discriminators of GANs are often modeled by neural netw…
Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization
Tolga Ergen, Arda Sahiner, Batu Ozturkler +3
Batch Normalization (BN) is a commonly used technique to accelerate and stabilize training of deep neural networks. Despite its empirical success, a full theoretical understanding…
Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms
Arda Sahiner, Tolga Ergen, John Pauly +1
We describe the convex semi-infinite dual of the two-layer vector-output ReLU neural network training problem. This semi-infinite dual admits a finite dimensional representation, b…
Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time
Tolga Ergen, Mert Pilanci
We study training of Convolutional Neural Networks (CNNs) with ReLU activations and introduce exact convex optimization formulations with a polynomial complexity with respect to th…
Convex Geometry and Duality of Over-parameterized Neural Networks
Tolga Ergen, Mert Pilanci
We develop a convex analytic approach to analyze finite width two-layer ReLU networks. We first prove that an optimal solution to the regularized training problem can be characteri…
Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networks
Mert Pilanci, Tolga Ergen
We develop exact representations of training two-layer neural networks with rectified linear units (ReLUs) in terms of a single convex program with number of variables polynomial i…
Revealing the Structure of Deep Neural Networks via Convex Duality
Tolga Ergen, Mert Pilanci
We study regularized deep neural networks (DNNs) and introduce a convex analytic framework to characterize the structure of the hidden layers. We show that a set of optimal hidden…
Unsupervised and Semi-supervised Anomaly Detection with LSTM Neural Networks
Tolga Ergen, Ali Hassan Mirza, Suleyman Serdar Kozat
We investigate anomaly detection in an unsupervised framework and introduce Long Short Term Memory (LSTM) neural network based algorithms. In particular, given variable length data…
Online Training of LSTM Networks in Distributed Systems for Variable Length Data Sequences
Tolga Ergen, Suleyman Serdar Kozat
In this brief paper, we investigate online training of Long Short Term Memory (LSTM) architectures in a distributed network of nodes, where each node employs an LSTM based structur…