4 papers
Essential-Web v1.0: 24T tokens of organized web data
Essential AI, :, Andrew Hojel +22
Data plays the most prominent role in how language models acquire skills and knowledge. The lack of massive, well-organized pre-training datasets results in costly and inaccessible…
Practical Efficiency of Muon for Pretraining
Essential AI, :, Ishaan Shah +22
We demonstrate that Muon, the simplest instantiation of a second-order optimizer, explicitly expands the Pareto frontier over AdamW on the compute-time tradeoff. We find that Muon…
Rethinking Reflection in Pre-Training
Essential AI, :, Darsh J Shah +26
A language model's ability to reflect on its own reasoning provides a key advantage for solving complex problems. While most recent research has focused on how this ability develop…
DeFog: Fog Computing Benchmarks
Jonathan McChesney, Nan Wang, Ashish Tanwer +2
Fog computing envisions that deploying services of an application across resources in the cloud and those located at the edge of the network may improve the overall performance of…