10 papers
Differentially Private Sampling from Distributions via Wasserstein Projection
Shokichi Takakura, Seng Pei Liew, Satoshi Hasegawa
In this paper, we study the problem of sampling from a distribution under the constraint of differential privacy (DP). Prior works measure the utility of DP sampling with density r…
Shuffling-Aware Optimization for Private Vector Mean Estimation
Shun Takagi, Seng Pei Liew
We study -dimensional unbiased mean estimation in the single-message shuffle model, where each user sends a single privatized message and the analyzer only observes the shuffled…
Analysis of Shuffling Beyond Pure Local Differential Privacy
Shun Takagi, Seng Pei Liew
Shuffling is a powerful way to amplify privacy of a local randomizer in private distributed data analysis. Most existing analyses of how shuffling amplifies privacy are based on th…
Reusing Overtrained Language Models Saturates Scaling
Seng Pei Liew, Takuya Kato
Reusing pretrained base models for further pretraining, such as continual pretraining or model growth, is promising at reducing the cost of training language models from scratch. H…
Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
Seng Pei Liew, Kenta Shinzato, Yuyang Dong
Modern Mixture-of-Experts (MoE) language models are designed based on total parameters (memory footprint) and active parameters (inference cost). However, we find these two factors…
Optimal Variance and Covariance Estimation under Differential Privacy in the Add-Remove Model and Beyond
Shokichi Takakura, Seng Pei Liew, Satoshi Hasegawa
In this paper, we study the problem of estimating the variance and covariance of datasets under differential privacy in the add-remove model. While estimation in the swap model has…