49 citations · 101 across the 20 of their papers we have counts for
9 papers · 1 filter
The Magic Correlations: Understanding Knowledge Transfer from Pretraining to Supervised Fine-Tuning
Simin Fan, Dimitris Paparas, Natasha Noy +3
Understanding how language model capabilities transfer from pretraining to supervised fine-tuning (SFT) is fundamental to efficient model development and data curation. In this wor…
Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representations
Rylan Schaeffer, Victor Lecomte, Dhruv Bhandarkar Pai +10
Maximum Manifold Capacity Representations (MMCR) is a recent multi-view self-supervised learning (MVSSL) method that matches or surpasses other leading MVSSL methods. MMCR is intri…
On Fairness of Low-Rank Adaptation of Large Models
Zhoujie Ding, Ken Ziyu Liu, Pura Peetathawatchai +2
Low-rank adaptation of large models, particularly LoRA, has gained traction due to its computational efficiency. This efficiency, contrasted with the prohibitive costs of full-mode…
Adaptive Compression in Federated Learning via Side Information
Berivan Isik, Francesco Pase, Deniz Gunduz +3
The high communication cost of sending model updates from the clients to the server is a significant bottleneck for scalable federated learning (FL). Among existing approaches, sta…
Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimation
Berivan Isik, Wei-Ning Chen, Ayfer Ozgur +2
We study the mean estimation problem under communication and local differential privacy constraints. While previous work has proposed \emph{order}-optimal algorithms for the same p…
Sparse Random Networks for Communication-Efficient Federated Learning
Berivan Isik, Francesco Pase, Deniz Gunduz +2
One main challenge in federated learning is the large communication cost of exchanging weight updates from clients to the server at each round. While prior work has made great prog…
An Information-Theoretic Justification for Model Pruning
Berivan Isik, Tsachy Weissman, Albert No
We study the neural network (NN) compression problem, viewing the tension between the compression ratio and NN performance through the lens of rate-distortion theory. We choose a d…
Neural Network Compression for Noisy Storage Devices
Berivan Isik, Kristy Choi, Xin Zheng +4
Compression and efficient storage of neural network (NN) parameters is critical for applications that run on resource-constrained devices. Despite the significant progress in NN mo…
rTop-k: A Statistical Estimation Approach to Distributed SGD
Leighton Pate Barnes, Huseyin A. Inan, Berivan Isik +1
The large communication cost for exchanging gradients between different nodes significantly limits the scalability of distributed training for large-scale learning models. Motivate…