411 citations · 756 across the 34 of their papers we have counts for
9 papers · 1 filter
Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5
Modern neural-network training uses mixed recipes, assigning AdamW-like or sign-based updates to embedding and output layers and spectral updates to hidden matrices. These assignme…
Shuffling-Aware Optimization for Private Vector Mean Estimation
Shun Takagi, Seng Pei Liew
We study -dimensional unbiased mean estimation in the single-message shuffle model, where each user sends a single privatized message and the analyzer only observes the shuffled…
FedDuA: Doubly Adaptive Federated Learning
Shokichi Takakura, Seng Pei Liew, Satoshi Hasegawa
Federated learning is a distributed learning framework where clients collaboratively train a global model without sharing their raw data. FedAvg is a popular algorithm for federate…
Accelerating Differentially Private Federated Learning via Adaptive Extrapolation
Shokichi Takakura, Seng Pei Liew, Satoshi Hasegawa
The federated learning (FL) framework enables multiple clients to collaboratively train machine learning models without sharing their raw data, but it remains vulnerable to privacy…
Scaling Laws for Upcycling Mixture-of-Experts Language Models
Seng Pei Liew, Takuya Kato, Sho Takase
Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters. There are two approaches of mitigating such…
Scaling Private Deep Learning with Low-Rank and Sparse Gradients
Ryuichi Ito, Seng Pei Liew, Tsubasa Takahashi +2
Applying Differentially Private Stochastic Gradient Descent (DPSGD) to training modern, large-scale neural networks such as transformer-based models is a challenging task, as the m…