7 citations · 11 across the 13 of their papers we have counts for
7 papers · 1 filter
Learning Mixtures of Experts with EM: A Mirror Descent Perspective
Quentin Fruytier, Aryan Mokhtari, Sujay Sanghavi
Classical Mixtures of Experts (MoE) are Machine Learning models that involve partitioning the input space, with a separate "expert" model trained on each partition. Recently, MoE-b…
Understanding Self-Supervised Learning via Gaussian Mixture Models
Parikshit Bansal, Ali Kavis, Sujay Sanghavi
Self-supervised learning attempts to learn representations from un-labeled data; it does so via a loss function that encourages the embedding of a point to be close to that of its…
RARe: Retrieval Augmented Retrieval with In-Context Examples
Atula Tejaswi, Yoonsang Lee, Sujay Sanghavi +1
While in-context learning is well-studied with decoder-only language models (LLMs), its utility for encoder-only models remains underexplored. We study in-context learning for enco…
DataComp-LM: In search of the next generation of training sets for language models
Jeffrey Li, Alex Fang, Georgios Smyrnis +56
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardize…
Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimization
Ruichen Jiang, Ali Kavis, Qiujiang Jin +2
We propose adaptive, line search-free second-order methods with optimal rate of convergence for solving convex-concave min-max problems. By means of an adaptive step size, our algo…
SVFT: Parameter-Efficient Fine-Tuning with Singular Vectors
Vijay Lingam, Atula Tejaswi, Aditya Vavre +7
Popular parameter-efficient fine-tuning (PEFT) methods, such as LoRA and its variants, freeze pre-trained model weights \(W\) and inject learnable matrices \(ΔW\). These \(ΔW\) mat…
Time Weaver: A Conditional Time Series Generation Model
Sai Shankar Narasimhan, Shubhankar Agarwal, Oguzhan Akcin +2
Imagine generating a city's electricity demand pattern based on weather, the presence of an electric vehicle, and location, which could be used for capacity planning during a winte…