7 papers
Information-Theoretic Analysis of Next-Token Prediction under Markovian Data
Masoud Kavian, Abdellatif Zaidi, Milad Sefidgaran
We develop an information-theoretic framework for generalization in next-token prediction under temporally dependent data. We consider independent trajectories generated by finite-…
Tighter CMI-Based Generalization Bounds via Stochastic Projection and Quantization
Milad Sefidgaran, Kimia Nadjahi, Abdellatif Zaidi
In this paper, we leverage stochastic projection and lossy compression to establish new conditional mutual information (CMI) bounds on the generalization error of statistical learn…
Multiview Representation Learning via Distributed Joint Latent Space Structuring
Milad Sefidgaran, Piotr Krasnowski, Abdellatif Zaidi
We study distributed multiview representation learning, a problem in which clients each observe a distinct but possibly statistically correlated view. The clients independently…
Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors
Milad Sefidgaran, Abdellatif Zaidi, Piotr Krasnowski
We establish in-expectation and tail bounds on the generalization error of representation learning type algorithms. The bounds are in terms of the relative entropy between the dist…
Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspective
Masoud Kavian, Romain Chor, Milad Sefidgaran +1
In this paper, we investigate the effect of data heterogeneity across clients on the performance of distributed learning systems, i.e., one-round Federated Learning, as measured by…
Minimal Communication-Cost Statistical Learning
Milad Sefidgaran, Abdellatif Zaidi, Piotr Krasnowski
A client device which has access to training data samples needs to obtain a statistical hypothesis or model and then to send it to a remote server. The client and the serve…
Minimum Description Length and Generalization Guarantees for Representation Learning
Milad Sefidgaran, Abdellatif Zaidi, Piotr Krasnowski
A major challenge in designing efficient statistical supervised learning algorithms is finding representations that perform well not only on available training samples but also on…