411 citations · 756 across the 34 of their papers we have counts for
36 papers
Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5
Modern neural-network training uses mixed recipes, assigning AdamW-like or sign-based updates to embedding and output layers and spectral updates to hidden matrices. These assignme…
Differentially Private Sampling from Distributions via Wasserstein Projection
Shokichi Takakura, Seng Pei Liew, Satoshi Hasegawa
In this paper, we study the problem of sampling from a distribution under the constraint of differential privacy (DP). Prior works measure the utility of DP sampling with density r…
Shuffling-Aware Optimization for Private Vector Mean Estimation
Shun Takagi, Seng Pei Liew
We study -dimensional unbiased mean estimation in the single-message shuffle model, where each user sends a single privatized message and the analyzer only observes the shuffled…
Analysis of Shuffling Beyond Pure Local Differential Privacy
Shun Takagi, Seng Pei Liew
Shuffling is a powerful way to amplify privacy of a local randomizer in private distributed data analysis. Most existing analyses of how shuffling amplifies privacy are based on th…
Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
Seng Pei Liew, Kenta Shinzato, Yuyang Dong
Modern Mixture-of-Experts (MoE) language models are designed based on total parameters (memory footprint) and active parameters (inference cost). However, we find these two factors…
Reusing Overtrained Language Models Saturates Scaling
Seng Pei Liew, Takuya Kato
Reusing pretrained base models for further pretraining, such as continual pretraining or model growth, is promising at reducing the cost of training language models from scratch. H…