357 citations · 992 across the 28 of their papers we have counts for
28 papers
Muse Spark Safety & Preparedness Report
Cristina Menghini, Peter Ney, Hamza Kwisaba +117
Muse Spark is the latest large language model developed by Meta. In this report, we first present evaluations for catastrophic risk domains under Meta's Advanced AI Scaling Framewo…
Online Distribution Shift Detection via Recency Prediction
Rachel Luo, Rohan Sinha, Yixiao Sun +5
When deploying modern machine learning-enabled robotic systems in high-stakes applications, detecting distribution shift is critical. However, most existing methods for detecting d…
Generalizing Bayesian Optimization with Decision-theoretic Entropies
Willie Neiswanger, Lantao Yu, Shengjia Zhao +2
Bayesian optimization (BO) is a popular method for efficiently inferring optima of an expensive black-box function via a sequence of queries. Existing information-theoretic BO proc…
Modular Conformal Calibration
Charles Marx, Shengjia Zhao, Willie Neiswanger +1
Uncertainty estimates must be calibrated (i.e., accurate) and sharp (i.e., informative) in order to be useful. This has motivated a variety of methods for recalibration, which use…
Low-Degree Multicalibration
Parikshit Gopalan, Michael P. Kim, Mihir Singhal +1
Introduced as a notion of algorithmic fairness, multicalibration has proved to be a powerful and versatile concept with implications far beyond its original intent. This stringent…
Sample-Efficient Safety Assurances using Conformal Prediction
Rachel Luo, Shengjia Zhao, Jonathan Kuck +4
When deploying machine learning models in high-stakes robotics applications, the ability to detect unsafe situations is crucial. Early warning systems can provide alerts when an un…
Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration
Shengjia Zhao, Michael P. Kim, Roshni Sahoo +2
When facing uncertainty, decision-makers want predictions they can trust. A machine learning provider can convey confidence to decision-makers by guaranteeing their predictions are…
Improved Autoregressive Modeling with Distribution Smoothing
Chenlin Meng, Jiaming Song, Yang Song +2
While autoregressive models excel at image compression, their sample quality is often lacking. Although not realistic, generated images often have high likelihood according to the…
Local Calibration: Metrics and Recalibration
Rachel Luo, Aadyot Bhatnagar, Yu Bai +7
Probabilistic classifiers output confidence scores along with their predictions, and these confidence scores should be calibrated, i.e., they should reflect the reliability of the…
Right Decisions from Wrong Predictions: A Mechanism Design Alternative to Individual Calibration
Shengjia Zhao, Stefano Ermon
Decision makers often need to rely on imperfect probabilistic forecasts. While average performance metrics are typically available, it is difficult to assess the quality of individ…
Privacy Preserving Recalibration under Domain Shift
Rachel Luo, Shengjia Zhao, Jiaming Song +3
Classifiers deployed in high-stakes real-world applications must output calibrated confidence scores, i.e. their predicted probabilities should reflect empirical frequencies. Recal…
A Framework for Sample Efficient Interval Estimation with Control Variates
Shengjia Zhao, Christopher Yeh, Stefano Ermon
We consider the problem of estimating confidence intervals for the mean of a random variable, where the goal is to produce the smallest possible interval for a given number of samp…
Individual Calibration with Randomized Forecasting
Shengjia Zhao, Tengyu Ma, Stefano Ermon
Machine learning applications often require calibrated predictions, e.g. a 90\% credible interval should contain the true outcome 90\% of the times. However, typical definitions of…
Permutation Invariant Graph Generation via Score-Based Generative Modeling
Chenhao Niu, Yang Song, Jiaming Song +3
Learning generative models for graph-structured data is challenging because graphs are discrete, combinatorial, and the underlying data distribution is invariant to the ordering of…
A Theory of Usable Information Under Computational Constraints
Yilun Xu, Shengjia Zhao, Jiaming Song +2
We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into ac…
Approximating Human Judgment of Generated Image Quality
Y. Alex Kolchinski, Sharon Zhou, Shengjia Zhao +2
Generative models have made immense progress in recent years, particularly in their ability to generate high quality images. However, that quality has been difficult to evaluate ri…
Domain Adaptive Imitation Learning
Kuno Kim, Yihong Gu, Jiaming Song +2
We study the question of how to imitate tasks across domains with discrepancies such as embodiment, viewpoint, and dynamics mismatch. Many prior works require paired, aligned demon…
Learning Neural PDE Solvers with Convergence Guarantees
Jun-Ting Hsieh, Shengjia Zhao, Stephan Eismann +2
Partial differential equations (PDEs) are widely used across the physical and computational sciences. Decades of research and engineering went into designing fast iterative solutio…
Learning Controllable Fair Representations
Jiaming Song, Pratyusha Kalluri, Aditya Grover +2
Learning data representations that are transferable and are fair with respect to certain protected attributes is crucial to reducing unfair decisions while preserving the utility o…
Bias and Generalization in Deep Generative Models: An Empirical Study
Shengjia Zhao, Hongyu Ren, Arianna Yuan +3
In high dimensional settings, density estimation algorithms rely crucially on their inductive bias. Despite recent empirical success, the inductive bias of deep generative models i…
The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models
Shengjia Zhao, Jiaming Song, Stefano Ermon
A large number of objectives have been proposed to train latent variable generative models. We show that many of them are Lagrangian dual functions of the same primal optimization…
Amortized Inference Regularization
Rui Shu, Hung H. Bui, Shengjia Zhao +2
The variational autoencoder (VAE) is a popular model for density estimation and representation learning. Canonically, the variational principle suggests to prefer an expressive inf…
A-NICE-MC: Adversarial Training for MCMC
Jiaming Song, Shengjia Zhao, Stefano Ermon
Existing Markov Chain Monte Carlo (MCMC) methods are either based on general-purpose and domain-agnostic schemes which can lead to slow convergence, or hand-crafting of problem-spe…
InfoVAE: Information Maximizing Variational Autoencoders
Shengjia Zhao, Jiaming Song, Stefano Ermon
A key advance in learning generative models is the use of amortized inference distributions that are jointly trained with the models. We find that existing training objectives for…