6 papers
Black-Box Inference of LLM Architectural Properties with Restrictive API Access
Christopher Ellis, Shreyas Chaudhari, Mei-Yu Wang +3
In practice, most commercial LLM providers do not publicly release details of underlying LLM architectures. However, prior work has shown that given limited API access to an LLM (n…
Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining
Shuqi Ke, Giulia Fanti
Continued pretraining is optimized with fixed self-supervised tasks but selected by downstream performance, creating a coarse feedback loop in which practitioners evaluate checkpoi…
Private Evolution Converges
Tomás González, Giulia Fanti, Aaditya Ramdas
Private Evolution (PE) is a promising training-free method for differentially private (DP) synthetic data generation. While it achieves strong performance in some domains (e.g., im…
Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusion
Shuqi Ke, Charlie Hou, Sewoong Oh +1
We show that differentially private full fine-tuning (DP-FFT) can distort pre-trained backbone features based on both theoretical and empirical results. We identify the cause of th…
POPri: Private Federated Learning using Preference-Optimized Synthetic Data
Charlie Hou, Mei-Yu Wang, Yige Zhu +2
In practical settings, differentially private Federated learning (DP-FL) is the dominant method for training models from private, on-device client data. Recent work has suggested t…
PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs
Charlie Hou, Akshat Shrivastava, Hongyuan Zhan +5
On-device training is currently the most common approach for training machine learning (ML) models on private, distributed user data. Despite this, on-device training has several d…