438 citations · 1k across the 57 of their papers we have counts for
64 papers
Dynamic Privacy Allocation for Locally Differentially Private Federated Learning with Composite Objectives
Jiaojiao Zhang, Dominik Fay, Mikael Johansson
This paper proposes a locally differentially private federated learning algorithm for strongly convex but possibly nonsmooth problems that protects the gradients of each worker aga…
Delay-agnostic Asynchronous Coordinate Update Algorithm
Xuyang Wu, Changxin Liu, Sindri Magnusson +1
We propose a delay-agnostic asynchronous coordinate update algorithm (DEGAS) for computing operator fixed points, with applications to asynchronous optimization. DEGAS includes nov…
Generalized Polyak Step Size for First Order Optimization with Momentum
Xiaoyu Wang, Mikael Johansson, Tong Zhang
In machine learning applications, it is well known that carefully designed learning rate (step size) schedules can significantly improve the convergence of commonly used first-orde…
Delay-agnostic Asynchronous Distributed Optimization
Xuyang Wu, Changxin Liu, Sindri Magnusson +1
Existing asynchronous distributed optimization algorithms often use diminishing step-sizes that cause slow practical convergence, or fixed step-sizes that depend on an assumed uppe…
Over-the-Air Computation for Distributed Systems: Something Old and Something New
Zheng Chen, Erik G. Larsson, Carlo Fischione +2
Facing the upcoming era of Internet-of-Things and connected intelligence, efficient information processing, computation, and communication design becomes a key challenge in large-s…
Distributed Safe Resource Allocation using Barrier Functions
Xuyang Wu, Sindri Magnusson, Mikael Johansson
Resource allocation plays a central role in many networked systems such as smart grids, communication networks and urban transportation systems. In these systems, many constraints…
Optimal convergence rates of totally asynchronous optimization
Xuyang Wu, Sindri Magnusson, Hamid Reza Feyzmahdavian +1
Asynchronous optimization algorithms are at the core of modern machine learning and resource allocation systems. However, most convergence results consider bounded information dela…
Delay-adaptive step-sizes for asynchronous learning
Xuyang Wu, Sindri Magnusson, Hamid Reza Feyzmahdavian +1
In scalable machine learning systems, model training is often parallelized over multiple nodes that run without tight synchronization. Most analysis results for the related asynchr…
On Uniform Boundedness Properties of SGD and its Momentum Variants
Xiaoyu Wang, Mikael Johansson
A theoretical, and potentially also practical, problem with stochastic gradient descent is that trajectories may escape to infinity. In this note, we investigate uniform boundednes…
A Fast and Accurate Splitting Method for Optimal Transport: Analysis and Implementation
Vien V. Mai, Jacob Lindbäck, Mikael Johansson
We develop a fast and reliable method for solving large-scale optimal transport (OT) problems at an unprecedented combination of speed and accuracy. Built on the celebrated Douglas…
Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guarantees
Hamid Reza Feyzmahdavian, Mikael Johansson
We introduce novel convergence results for asynchronous iterations that appear in the analysis of parallel and distributed optimization algorithms. The results are simple to apply…
Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimization
Xiaoyu Wang, Mikael Johansson
Many popular learning-rate schedules for deep neural networks combine a decaying trend with local perturbations that attempt to escape saddle points and bad local minima. We derive…
A New Family of Feasible Methods for Distributed Resource Allocation
Xuyang Wu, Sindri Magnusson, Mikael Johansson
Distributed resource allocation is a central task in network systems such as smart grids, water distribution networks, and urban transportation systems. When solving such problems…
On the Convergence of Step Decay Step-Size for Stochastic Optimization
Xiaoyu Wang, Sindri Magnússon, Mikael Johansson
The convergence of stochastic gradient descent is highly dependent on the step-size, especially on non-convex problems such as neural network training. Step decay step-size schedul…
Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness
Vien V. Mai, Mikael Johansson
Stochastic gradient algorithms are often unstable when applied to functions that do not have Lipschitz-continuous and/or bounded gradients. Gradient clipping is a simple and effect…
Advances in Asynchronous Parallel and Distributed Optimization
Mahmoud Assran, Arda Aytekin, Hamid Feyzmahdavian +2
Motivated by large-scale optimization problems arising in the context of machine learning, there have been several advances in the study of asynchronous parallel and distributed op…
A flexible framework for communication-efficient machine learning: from HPC to IoT
Sarit Khirirat, Sindri Magnússon, Arda Aytekin +1
With the increasing scale of machine learning tasks, it has become essential to reduce the communication between computing nodes. Early work on gradient compression focused on the…
Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimization
Vien V. Mai, Mikael Johansson
Stochastic gradient methods with momentum are widely used in applications and at the core of optimization subroutines in many popular machine learning libraries. However, their sam…
Dynamic cut aggregation in L-shaped algorithms
Martin Biel, Mikael Johansson
We present a novel framework for dynamic cut aggregation in L-shaped algorithms. The aim is to improve the parallel performance of distributed L-shaped algorithms through reduced c…
Anderson Acceleration of Proximal Gradient Methods
Vien V. Mai, Mikael Johansson
Anderson acceleration is a well-established and simple technique for speeding up fixed-point computations with countless applications. Previous studies of Anderson acceleration in…
Efficient Stochastic Programming in Julia
Martin Biel, Mikael Johansson
We present StochasticPrograms.jl, a user-friendly and powerful open-source framework for stochastic programming written in the Julia language. The framework includes both modeling…
Compressed Gradient Methods with Hessian-Aided Error Compensation
Sarit Khirirat, Sindri Magnússon, Mikael Johansson
The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. The performance bottleneck, which used to be computations, is now often c…
Noisy Accelerated Power Method for Eigenproblems with Applications
Vien V. Mai, Mikael Johansson
This paper introduces an efficient algorithm for finding the dominant generalized eigenvectors of a pair of symmetric matrices. Combining tools from approximation theory and convex…
Curvature-Exploiting Acceleration of Elastic Net Computations
Vien V. Mai, Mikael Johansson
This paper introduces an efficient second-order method for solving the elastic net problem. Its key innovation is a computationally efficient technique for injecting curvature info…