117 citations · 411 across the 25 of their papers we have counts for
11 papers · 1 filter
Query-based Knowledge Transfer for Heterogeneous Learning Environments
Norah Alballa, Wenxuan Zhang, Ziquan Liu +3
Decentralized collaborative learning under data heterogeneity and privacy constraints has rapidly advanced. However, existing solutions like federated learning, ensembles, and tran…
Flashback: Understanding and Mitigating Forgetting in Federated Learning
Mohammed Aljahdali, Ahmed M. Abdelmoniem, Marco Canini +1
In Federated Learning (FL), forgetting, or the loss of knowledge across rounds, hampers algorithm convergence, particularly in the presence of severe data heterogeneity among clien…
Practical Insights into Knowledge Distillation for Pre-Trained Models
Norah Alballa, Ahmed M. Abdelmoniem, Marco Canini
This research investigates the enhancement of knowledge distillation (KD) processes in pre-trained models, an emerging field in knowledge transfer with significant implications for…
Quantize Once, Train Fast: Allreduce-Compatible Compression with Provable Guarantees
Jihao Xin, Marco Canini, Peter Richtárik +1
Distributed training enables large-scale deep learning, but suffers from high communication overhead, especially as models and datasets grow. Gradient compression, particularly qua…
FilFL: Client Filtering for Optimized Client Participation in Federated Learning
Fares Fourati, Salma Kharrat, Vaneet Aggarwal +2
Federated learning, an emerging machine learning paradigm, enables clients to collaboratively train a model without exchanging local data. Clients participating in the training pro…
Resource-Efficient Federated Learning
Ahmed M. Abdelmoniem, Atal Narayan Sahu, Marco Canini +1
Federated Learning (FL) enables distributed training by learners using local data, thereby enhancing privacy and reducing communication. However, it presents numerous challenges re…
Rethinking gradient sparsification as total error minimization
Atal Narayan Sahu, Aritra Dutta, Ahmed M. Abdelmoniem +3
Gradient compression is a widely-established remedy to tackle the communication bottleneck in distributed training of large deep neural networks (DNNs). Under the error-feedback fr…
AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly
Yuchen Jin, Tianyi Zhou, Liangyu Zhao +4
The learning rate (LR) schedule is one of the most important hyper-parameters needing careful tuning in training DNNs. However, it is also one of the least automated parts of machi…
On the Impact of Device and Behavioral Heterogeneity in Federated Learning
Ahmed M. Abdelmoniem, Chen-Yu Ho, Pantelis Papageorgiou +2
Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. FL has seen successful deployme…
An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
Ahmed M. Abdelmoniem, Ahmed Elzanaty, Mohamed-Slim Alouini +1
The recent many-fold increase in the size of deep neural networks makes efficient distributed training challenging. Many proposals exploit the compressibility of the gradients and…
Natural Compression for Distributed Deep Learning
Samuel Horvath, Chen-Yu Ho, Ludovit Horvath +3
Modern deep learning models are often trained in parallel over a collection of distributed machines to reduce training time. In such settings, communication of model updates among…