2 citations · 3 across the 7 of their papers we have counts for
7 papers
Essential-Web v1.0: 24T tokens of organized web data
Essential AI, :, Andrew Hojel +22
Data plays the most prominent role in how language models acquire skills and knowledge. The lack of massive, well-organized pre-training datasets results in costly and inaccessible…
Practical Efficiency of Muon for Pretraining
Essential AI, :, Ishaan Shah +22
We demonstrate that Muon, the simplest instantiation of a second-order optimizer, explicitly expands the Pareto frontier over AdamW on the compute-time tradeoff. We find that Muon…
Rethinking Reflection in Pre-Training
Essential AI, :, Darsh J Shah +26
A language model's ability to reflect on its own reasoning provides a key advantage for solving complex problems. While most recent research has focused on how this ability develop…
CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous models
Abhishek Singh, Gauri Gupta, Ritvik Kapila +5
Federated Learning (FL) enables collaborative optimization of machine learning models across decentralized data by aggregating model parameters. Our approach extends this concept b…
First 100 days of pandemic; an interplay of pharmaceutical, behavioral and digital interventions -- A study using agent based modeling
Gauri Gupta, Ritvik Kapila, Ayush Chopra +1
Pandemics, notably the recent COVID-19 outbreak, have impacted both public health and the global economy. A profound understanding of disease progression and efficient response str…
ZKROWNN: Zero Knowledge Right of Ownership for Neural Networks
Nojan Sheybani, Zahra Ghodsi, Ritvik Kapila +1
Training contemporary AI models requires investment in procuring learning data and computing resources, making the models intellectual property of the owners. Popular model waterma…
Domain Generalization In Robust Invariant Representation
Gauri Gupta, Ritvik Kapila, Keshav Gupta +1
Unsupervised approaches for learning representations invariant to common transformations are used quite often for object recognition. Learning invariances makes models more robust…