411 citations · 756 across the 34 of their papers we have counts for
5 papers · 1 filter
Reusing Overtrained Language Models Saturates Scaling
Seng Pei Liew, Takuya Kato
Reusing pretrained base models for further pretraining, such as continual pretraining or model growth, is promising at reducing the cost of training language models from scratch. H…
Optimal Variance and Covariance Estimation under Differential Privacy in the Add-Remove Model and Beyond
Shokichi Takakura, Seng Pei Liew, Satoshi Hasegawa
In this paper, we study the problem of estimating the variance and covariance of datasets under differential privacy in the add-remove model. While estimation in the swap model has…
FedDuA: Doubly Adaptive Federated Learning
Shokichi Takakura, Seng Pei Liew, Satoshi Hasegawa
Federated learning is a distributed learning framework where clients collaboratively train a global model without sharing their raw data. FedAvg is a popular algorithm for federate…
Accelerating Differentially Private Federated Learning via Adaptive Extrapolation
Shokichi Takakura, Seng Pei Liew, Satoshi Hasegawa
The federated learning (FL) framework enables multiple clients to collaboratively train machine learning models without sharing their raw data, but it remains vulnerable to privacy…
Scaling Laws for Upcycling Mixture-of-Experts Language Models
Seng Pei Liew, Takuya Kato, Sho Takase
Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters. There are two approaches of mitigating such…