Publications (15)
OLMoASR: Open Models and Data for Training Robust Speech Recognition Models
Huong Ngo, Matt Deitke, Martijn Bartelds +4
Improvements in training data scale and quality have led to significant advances, yet its influence in speech recognition remains underexplored. In this paper, we present a large-s…
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens
Anas Awadalla, Le Xue, Oscar Lo +11
Multimodal interleaved datasets featuring free-form interleaved sequences of images and text are crucial for training frontier large multimodal models (LMMs). Despite the rapid pro…
Inverse Problems Leveraging Pre-trained Contrastive Representations
Sriram Ravula, Georgios Smyrnis, Matt Jordan +1
We study a new family of inverse problems for recovering representations of corrupted data. We assume access to a pre-trained representation learning network R(x) that operates on…
Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality
Alex Fang, Hadi Pouransari, Matt Jordan +4
Data filtering has become a powerful tool for improving model performance while reducing computational cost. However, as large language model compute budgets continue to grow, the…
Zonotope Domains for Lagrangian Neural Network Verification
Matt Jordan, Jonathan Hayase, Alexandros G. Dimakis +1
Neural network verification aims to provide provable bounds for the output of a neural network for a given input range. Notable prior works in this domain have either generated bou…
Provable Lipschitz Certification for Generative Models
Matt Jordan, Alexandros G. Dimakis
We present a scalable technique for upper bounding the Lipschitz constant of generative models. We relate this quantity to the maximal norm over the set of attainable vector-Jacobi…
Olmo 3
Team Olmo, :, Allyson Ettinger +66
We introduce Olmo 3, a family of state-of-the-art, fully-open language models at the 7B and 32B parameter scales. Olmo 3 model construction targets long-context reasoning, function…
Provable Certificates for Adversarial Examples: Fitting a Ball in the Union of Polytopes
Matt Jordan, Justin Lewis, Alexandros G. Dimakis
We propose a novel method for computing exact pointwise robustness of deep neural networks for all convex norms. Our algorithm, GeoCert, finds the largest ball ce…
Quarantines as a Targeted Immunization Strategy
Jessica Hoffmann, Matt Jordan, Constantine Caramanis
In the context of the recent COVID-19 outbreak, quarantine has been used to "flatten the curve" and slow the spread of the disease. In this paper, we show that this is not the only…
Exactly Computing the Local Lipschitz Constant of ReLU Networks
Matt Jordan, Alexandros G. Dimakis
The local Lipschitz constant of a neural network is a useful metric with applications in robustness, generalization, and fairness evaluation. We provide novel analytic results rela…
Quantifying Perceptual Distortion of Adversarial Examples
Matt Jordan, Naren Manoj, Surbhi Goel +1
Recent work has shown that additive threat models, which only permit the addition of bounded noise to the pixels of an image, are insufficient for fully capturing the space of impe…
DataComp-LM: In search of the next generation of training sets for language models
Jeffrey Li, Alex Fang, Georgios Smyrnis +56
We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardize…
2 OLMo 2 Furious
Team OLMo, Pete Walsh, Luca Soldaini +40
We present OLMo 2, the next generation of our fully open language models. OLMo 2 includes a family of dense autoregressive language models at 7B, 13B and 32B scales with fully rele…
Conditional Generative Models are Sufficient to Sample from Any Causal Effect Estimand
Md Musfiqur Rahman, Matt Jordan, Murat Kocaoglu
Causal inference from observational data plays critical role in many applications in trustworthy machine learning. While sound and complete algorithms exist to compute causal effec…
Olmix: A Framework for Data Mixing Throughout LM Development
Mayee F. Chen, Tyler Murray, David Heineman +5
Data mixing -- determining the ratios of data from different domains -- is a first-order concern for training language models (LMs). While existing mixing methods show promise, the…