310 citations · 436 across the 15 of their papers we have counts for
18 papers
Small-Scale Experiments: Are We There Yet?
Nicholas Lourie, Kyunghyun Cho, Karen Ullrich +1
Scaling laws promised cost-effective experiments; six years later, they have yet to fully deliver. Instead, researchers have found them unreliable at small scales (starting at 4M p…
Neural Neural Scaling Laws
Michael Y. Hu, Jane Pan, Ayush Rajesh Jhaveri +2
Neural scaling laws predict how language model performance improves with increased training inputs. While aggregate metrics like validation loss can follow smooth power-law curves,…
Universal Spectral Tokenization via Self-Supervised Panchromatic Representation Learning
Jeff Shen, Francois Lanusse, Liam Holden Parker +24
Sequential scientific data span many resolutions and domains, and unifying them into a common representation is a key step toward developing foundation models for the sciences. Ast…
AION-1: Omnimodal Foundation Model for Astronomical Sciences
Liam Parker, Francois Lanusse, Jeff Shen +24
While foundation models have shown promise across a variety of fields, astronomy still lacks a unified framework for joint modeling across its highly diverse data modalities. In th…
Hyperparameter Loss Surfaces Are Simple Near their Optima
Nicholas Lourie, He He, Kyunghyun Cho
Hyperparameters greatly impact models' capabilities; however, modern models are too large for extensive search. Instead, researchers design recipes that train well across scales ba…
Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check
Nicholas Lourie, Michael Y. Hu, Kyunghyun Cho
Downstream scaling laws aim to predict task performance at larger scales from the model's performance at smaller scales. Whether such prediction should be possible is unclear: some…
What's In Your Field? Mapping Scientific Research with Knowledge Graphs and Large Language Models
Abhipsha Das, Nicholas Lourie, Siavash Golkar +1
The scientific literature's exponential growth makes it increasingly challenging to navigate and synthesize knowledge across disciplines. Large language models (LLMs) are powerful…
Aioli: A Unified Optimization Framework for Language Model Data Mixing
Mayee F. Chen, Michael Y. Hu, Nicholas Lourie +2
Language model performance depends on identifying the optimal mixture of data groups to train on (e.g., law, code, math). Prior work has proposed a diverse set of methods to effici…
Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Task
Siavash Golkar, Alberto Bietti, Mariel Pettee +12
Transformers have revolutionized machine learning across diverse domains, yet understanding their behavior remains crucial, particularly in high-stakes applications. This paper int…
Show Your Work with Confidence: Confidence Bands for Tuning Curves
Nicholas Lourie, Kyunghyun Cho, He He
The choice of hyperparameters greatly impacts performance in natural language processing. Often, it is hard to tell if a method is better than another or just better tuned. Tuning…
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark
Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula +1
Commonsense AI has long been seen as a near impossible goal -- until recently. Now, research interest has sharply increased with an influx of new benchmarks and models. We propose…
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation
Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg +5
While often assumed a gold standard, effective human evaluation of text generation remains an important, open area for research. We revisit this problem with a focus on producing c…
Learning from Task Descriptions
Orion Weller, Nicholas Lourie, Matt Gardner +1
Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by reading some instructions, with perhaps an exam…
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie +4
Large datasets have become commonplace in NLP research. However, the increased emphasis on data quantity has made it challenging to assess the quality of data. We introduce Data Ma…
Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes
Nicholas Lourie, Ronan Le Bras, Yejin Choi
As AI systems become an increasing part of people's everyday lives, it becomes ever more important that they understand people's ethical norms. Motivated by descriptive ethics, a f…
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Alon Talmor, Jonathan Herzig, Nicholas Lourie +1
When answering a question, people often draw upon their rich world knowledge in addition to the particular context. Recent work has focused primarily on answering questions given s…
Extracting Scientific Figures with Distantly Supervised Neural Networks
Noah Siegel, Nicholas Lourie, Russell Power +1
Non-textual components such as charts, diagrams and tables provide key information in many scientific documents, but the lack of large labeled datasets has impeded the development…
Tripod configurations of curves
Eric Chen, Nick Lourie
Tripod configurations of plane curves, formed by certain triples of normal lines coinciding at a point, were introduced by Tabachnikov, who showed that closed convex curves p…