361 citations
- Carnegie Mellon UniversityUS12 papers
- Stanford UniversityUS6 papers
- Google (United States)US5 papers
- University of WashingtonUS5 papers
- Amazon (United States)US4 papers
- Apple (Germany)DE4 papers
- Apple (Israel)IL4 papers
- Arizona State UniversityUS4 papers
- Centre National de la Recherche ScientifiqueFR4 papers
- Jet Propulsion LaboratoryUS4 papers
- Duke UniversityUS3 papers
- École Polytechnique Fédérale de LausanneCH3 papers
16 papers · 1 filter
ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration
Mengting Ai, Tianxin Wei, Yifan Chen +7
Mixture-of-Experts (MoE) Transformer, the backbone architecture of multiple phenomenal language models, leverages sparsity by activating only a fraction of model parameters for eac…
On the Minimal Degree Bias in Generalization on the Unseen for non-Boolean Functions
Denys Pushkin, Raphaël Berthier, Emmanuel Abbe
We investigate the out-of-domain generalization of random feature (RF) models and Transformers. We first prove that in the `generalization on the unseen (GOTU)' setting, where trai…
Enhancing Hypergradients Estimation: A Study of Preconditioning and Reparameterization
Zhenzhang Ye, Gabriel Peyré, Daniel Cremers +1
Bilevel optimization aims to optimize an outer objective function that depends on the solution to an inner optimization problem. It is routinely used in Machine Learning, notably f…
MultiView Independent Component Analysis with Delays
Ambroise Heurtebise, Pierre Ablin, Alexandre Gramfort
Linear Independent Component Analysis (ICA) is a blind source separation technique that has been used in various domains to identify independent latent sources from observed signal…
Minimax Demographic Group Fairness in Federated Learning
Afroditi Papadaki, Natalia Martinez, Martin Bertran +2
Federated learning is an increasingly popular paradigm that enables a large number of entities to collaboratively learn better models. In this work, we study minimax group fairness…
When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?
Gavin Brown, Mark Bun, Vitaly Feldman +2
Modern machine learning models are complex and frequently encode surprising amounts of information about individual inputs. In extreme cases, complex models appear to memorize enti…