38 citations · 247 across the 108 of their papers we have counts for
56 papers · 1 filter
Convex Optimization for Alignment and Preference Learning on a Single GPU
Miria Feng, Mert Pilanci
Fine-tuning large language models (LLMs) to align with human preferences has driven the success of systems such as Gemini and ChatGPT. However, approaches like Reinforcement Learni…
Convex Low-resource Accent-Robust Language Detection in Speech Recognition
Miria Feng, William Tan, Mert Pilanci
Globalization and multiculturalism continue to produce increasingly diverse speech varieties. Yet current spoken dialogue systems frequently fail on under-represented dialects and…
Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspective
Emi Zeger, Mert Pilanci
Deep neural networks (DNNs), particularly those using Rectified Linear Unit (ReLU) activation functions, have achieved remarkable success across diverse machine learning tasks, inc…
A Recovery Guarantee for Sparse Neural Networks
Sara Fridovich-Keil, Mert Pilanci
We prove the first guarantees of sparse recovery for ReLU neural networks, where the sparse network weights constitute the signal to be recovered. Specifically, we study structural…
MatRL: Provably Generalizable Iterative Algorithm Discovery via Monte-Carlo Tree Search
Sungyoon Kim, Rajat Vadiraj Dwaraknath, Longling geng +1
Iterative methods for computing matrix functions have been extensively studied and their convergence speed can be significantly improved with the right tuning of parameters and by…
LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization
Erica Zhang, Ryunosuke Goto, Naomi Sagan +7
We introduce LLM-Lasso, a novel framework that leverages large language models (LLMs) to guide feature selection in Lasso regression. Unlike traditional methods that rely…
CRONOS: Enhancing Deep Learning with Scalable GPU Accelerated Convex Neural Networks
Miria Feng, Zachary Frangella, Mert Pilanci
We introduce the CRONOS algorithm for convex optimization of two-layer neural networks. CRONOS is the first algorithm capable of scaling to high-dimensional datasets such as ImageN…
Exploring the loss landscape of regularized neural networks via convex duality
Sungyoon Kim, Aaron Mishkin, Mert Pilanci
We discuss several aspects of the loss landscape of regularized neural networks: the structure of stationary points, connectivity of optimal solutions, path with nonincreasing loss…
Black Boxes and Looking Glasses: Multilevel Symmetries, Reflection Planes, and Convex Optimization in Deep Networks
Emi Zeger, Mert Pilanci
We show that training deep neural networks (DNNs) with absolute value activation and arbitrary input dimension can be formulated as equivalent convex Lasso problems with novel feat…
Convex Distillation: Efficient Compression of Deep Networks via Convex Optimization
Prateek Varshney, Mert Pilanci
Deploying large and complex deep neural networks on resource-constrained edge devices poses significant challenges due to their computational demands and the complexities of non-co…
Active Learning of Deep Neural Networks via Gradient-Free Cutting Planes
Erica Zhang, Fangzhao Zhang, Mert Pilanci
Active learning methods aim to improve sample complexity in machine learning. In this work, we investigate an active learning scheme via a novel gradient-free cutting-plane trainin…
ConvexECG: Lightweight and Explainable Neural Networks for Personalized, Continuous Cardiac Monitoring
Rayan Ansari, John Cao, Sabyasachi Bandyopadhyay +3
We present ConvexECG, an explainable and resource-efficient method for reconstructing six-lead electrocardiograms (ECG) from single-lead data, aimed at advancing personalized and c…
AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers
Emil Biju, Anirudh Sriram, Mert Pilanci
While large transformer-based models have exhibited remarkable performance in speaker-independent speech recognition, their large size and computational requirements make them expe…
Randomized Geometric Algebra Methods for Convex Neural Networks
Yifei Wang, Sungyoon Kim, Paul Chu +2
We introduce randomized algorithms to Clifford's Geometric Algebra, generalizing randomized linear algebra to hypercomplex vector spaces. This novel approach has many implications…
Compressing Large Language Models using Low Rank and Low Precision Decomposition
Rajarshi Saha, Naomi Sagan, Varun Srivastava +2
The prohibitive sizes of Large Language Models (LLMs) today make it difficult to deploy them on memory-constrained edge devices. This work introduces -- a new post-tr…
Adversarial Training of Two-Layer Polynomial and ReLU Activation Networks via Convex Optimization
Daniel Kuelbs, Sanjay Lall, Mert Pilanci
Training neural networks which are robust to adversarial attacks remains an important problem in deep learning, especially as heavily overparameterized models are adopted in safety…
Spectral Adapter: Fine-Tuning in Spectral Space
Fangzhao Zhang, Mert Pilanci
Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for pretrained deep neural networks have captured widespread interest. In this work, we study the enhancement…
A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features
Emi Zeger, Yifei Wang, Aaron Mishkin +3
We prove that training neural networks on 1-D data is equivalent to solving convex Lasso problems with discrete, explicitly defined dictionary matrices. We consider neural networks…
Adaptive Inference: Theoretical Limits and Unexplored Opportunities
Soheil Hor, Ying Qian, Mert Pilanci +1
This paper introduces the first theoretical framework for quantifying the efficiency and performance gain opportunity size of adaptive inference algorithms. We provide new approxim…
Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time
Sungyoon Kim, Mert Pilanci
In this paper, we study the optimality gap between two-layer ReLU networks regularized with weight decay and their convex relaxations. We show that when the training data is random…
Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models
Fangzhao Zhang, Mert Pilanci
Low-Rank Adaptation (LoRA) emerges as a popular parameter-efficient fine-tuning (PEFT) method, which proposes to freeze pretrained model weights and update an additive low-rank tra…
Analyzing Neural Network-Based Generative Diffusion Models through Convex Optimization
Fangzhao Zhang, Mert Pilanci
Diffusion models are gaining widespread use in cutting-edge image, video, and audio generation. Score-based diffusion models stand out among these methods, necessitating the estima…
The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Models
Tolga Ergen, Mert Pilanci
Due to the non-convex nature of training Deep Neural Network (DNN) models, their effectiveness relies on the use of non-convex optimization heuristics. Traditional methods for trai…
Polynomial-Time Solutions for ReLU Network Training: A Complexity Classification via Max-Cut and Zonotopes
Yifei Wang, Mert Pilanci
We investigate the complexity of training a two-layer ReLU neural network with weight decay regularization. Previous research has shown that the optimal solution of this problem ca…