19 citations · 59 across the 22 of their papers we have counts for
26 papers
Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
Cristian McGee, El Houcine Bergou, Aritra Dutta
Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combi…
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
Elijah Cadenhead, Cristian McGee, Xin Li +2
Low-rank adaptation (LoRA) and its variants provide a memory- and compute-efficient alternative to full fine-tuning of pre-trained models. However, questions remain about the compa…
MFedAQI: Multimodal Federated Learning for Air Quality Prediction on Heterogeneous Edge Devices
Manjil Nepal, Kimsie Phan, Tamoghna Ojha +2
Accurate air quality prediction is essential for public health, environmental monitoring, and industrial safety. However, most existing approaches rely on centralized learning para…
Stabilizing Policy Gradient Methods via Reward Profiling
Shihab Ahmed, El Houcine Bergou, Aritra Dutta +1
Policy gradient methods, which have been extensively studied in the last decade, offer an effective and efficient framework for reinforcement learning problems. However, their perf…
Where Have All the Kaczmarz Iterates Gone?
El Houcine Bergou, Soumia Boucherouite, Aritra Dutta +2
The randomized Kaczmarz (RK) algorithm is one of the most computationally and memory-efficient iterative algorithms for solving large-scale linear systems. However, practical appli…
Kolmogorov-Arnold Attention: Is Learnable Attention Better For Vision Transformers?
Subhajit Maity, Killian Hitsman, Xin Li +1
Kolmogorov-Arnold networks (KANs) are a remarkable innovation that consists of learnable activation functions, with the potential to capture more complex relationships from data. P…
Towards Multi-modal Transformers in Federated Learning
Guangyu Sun, Matias Mendieta, Aritra Dutta +2
Multi-modal transformers mark significant progress in different domains, but siloed high-quality data hinders their further improvement. To remedy this, federated learning (FL) has…
Demystifying the Myths and Legends of Nonconvex Convergence of SGD
Aritra Dutta, El Houcine Bergou, Soumia Boucherouite +3
Stochastic gradient descent (SGD) and its variants are the main workhorses for solving large-scale optimization problems with nonconvex objective functions. Although the convergenc…
A Note on Randomized Kaczmarz Algorithm for Solving Doubly-Noisy Linear Systems
El Houcine Bergou, Soumia Boucherouite, Aritra Dutta +2
Large-scale linear systems, , frequently arise in practice and demand effective iterative solvers. Often, these systems are noisy due to operational errors or faulty data-col…
Personalized Federated Learning with Communication Compression
El Houcine Bergou, Konstantin Burlachenko, Aritra Dutta +1
In contrast to training traditional machine learning (ML) models in data centers, federated learning (FL) trains ML models over local datasets contained on resource-constrained het…
Rethinking gradient sparsification as total error minimization
Atal Narayan Sahu, Aritra Dutta, Ahmed M. Abdelmoniem +3
Gradient compression is a widely-established remedy to tackle the communication bottleneck in distributed training of large deep neural networks (DNNs). Under the error-feedback fr…
DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning
Kelly Kostopoulou, Hang Xu, Aritra Dutta +3
Sparse tensors appear frequently in distributed deep learning, either as a direct artifact of the deep neural network's gradients, or as a result of an explicit sparsification proc…
An Adaptive Rank Continuation Algorithm for General Weighted Low-rank Recovery
Aritra Dutta, Jingwei Liang, Xin Li
This paper is devoted to proposing a general weighted low-rank recovery model and designing a fast SVD-free computational scheme to solve it. First, our generic weighted low-rank r…
On the Convergence Analysis of Asynchronous SGD for Solving Consistent Linear Systems
Atal Narayan Sahu, Aritra Dutta, Aashutosh Tiwari +1
In the realm of big data and machine learning, data-parallel, distributed stochastic algorithms have drawn significant attention in the present days.~While the synchronous versions…
On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learning
Aritra Dutta, El Houcine Bergou, Ahmed M. Abdelmoniem +4
Compressed communication, in the form of sparsification or quantization of stochastic gradients, is employed to reduce communication costs in distributed data-parallel training of…
Direct Nonlinear Acceleration
Aritra Dutta, El Houcine Bergou, Yunming Xiao +2
Optimization acceleration techniques such as momentum play a key role in state-of-the-art machine learning algorithms. Recently, generic vector sequence extrapolation techniques, s…
Best Pair Formulation & Accelerated Scheme for Non-convex Principal Component Pursuit
Aritra Dutta, Filip Hanzely, Jingwei Liang +1
The best pair problem aims to find a pair of points that minimize the distance between two disjoint sets. In this paper, we formulate the classical robust principal component analy…
A Nonconvex Projection Method for Robust PCA
Aritra Dutta, Filip Hanzely, Peter Richtárik
Robust principal component analysis (RPCA) is a well-studied problem with the goal of decomposing a matrix into the sum of low-rank and sparse components. In this paper, we propose…
Weighted Low-Rank Approximation of Matrices and Background Modeling
Aritra Dutta, Xin Li, Peter Richtarik
We primarily study a special a weighted low-rank approximation of matrices and then apply it to solve the background modeling problem. We propose two algorithms for this purpose: o…
Online and Batch Supervised Background Estimation via L1 Regression
Aritra Dutta, Peter Richtarik
We propose a surprisingly simple model for supervised video background estimation. Our model is based on regression. As existing methods for regression do not sca…
Weighted Low Rank Approximation for Background Estimation Problems
Aritra Dutta, Xin Li
Classical principal component analysis (PCA) is not robust to the presence of sparse outliers in the data. The use of the norm in the Robust PCA (RPCA) method successfully…
A Batch-Incremental Video Background Estimation Model using Weighted Low-Rank Approximation of Matrices
Aritra Dutta, Xin Li, Peter Richtárik
Principal component pursuit (PCP) is a state-of-the-art approach for background estimation problems. Due to their higher computational cost, PCP algorithms, such as robust principa…
Weighted Singular Value Thresholding and its Application to Background Estimation
Aritra Dutta, Boqing Gong, Xin Li +1
Singular value thresholding (SVT) plays an important role in the well-known robust principal component analysis (RPCA) algorithms which have many applications in computer vision an…
A Fast Algorithm for a Weighted Low Rank Approximation
Aritra Dutta, Xin Li
Matrix low rank approximation including the classical PCA and the robust PCA (RPCA) method have been applied to solve the background modeling problem in video analysis. Recently, i…