Publications (54)
From seeing to remembering: Images with harder-to-reconstruct representations leave stronger memory traces
Qi Lin, Zifan Li, John Lafferty +1
Much of what we remember is not due to intentional selection, but simply a by-product of perceiving. This raises a foundational question about the architecture of the mind: How doe…
Expectation-Propogation for the Generative Aspect Model
Thomas P. Minka, John Lafferty
The generative aspect model is an extension of the multinomial model for text that allows word probabilities to vary stochastically across documents. Previous results with aspect m…
Approximation of relation functions and attention mechanisms
Awni Altabaa, John Lafferty
Inner products of neural network feature maps arise in a wide variety of machine learning frameworks as a method of modeling relations between inputs. This work studies the approxi…
Iterative Markov Chain Monte Carlo Computation of Reference Priors and Minimax Risk
John Lafferty, Larry A. Wasserman
We present an iterative Markov chainMonte Carlo algorithm for computingreference priors and minimax risk forgeneral parametric families. Ourapproach uses MCMC techniques based onth…
Testing for Global Network Structure Using Small Subgraph Statistics
Chao Gao, John Lafferty
We study the problem of testing for community structure in networks using relations between the observed frequencies of small subgraphs. We propose a simple test for the existence…
The huge Package for High-dimensional Undirected Graph Estimation in R
Tuo Zhao, Han Liu, Kathryn Roeder +2
We describe an R package named huge which provides easy-to-use functions for estimating high dimensional undirected graphs from data. This package implements recent results in the…
Rodeo: Sparse, greedy nonparametric regression
John Lafferty, Larry Wasserman
We present a greedy method for simultaneously performing local bandwidth selection and variable selection in nonparametric regression. The method starts with a local linear estimat…
Quantized Nonparametric Estimation over Sobolev Ellipsoids
Yuancheng Zhu, John Lafferty
We formulate the notion of minimax estimation under storage or communication constraints, and prove an extension to Pinsker's theorem for nonparametric estimation over Sobolev elli…
TopicEq: A Joint Topic and Mathematical Equation Model for Scientific Texts
Michihiro Yasunaga, John Lafferty
Scientific documents rely on both mathematics and text to communicate ideas. Inspired by the topical correspondence between mathematical equations and word contexts observed in sci…
Testing Network Structure Using Relations Between Small Subgraph Probabilities
Chao Gao, John Lafferty
We study the problem of testing for structure in networks using relations between the observed frequencies of small subgraphs. We consider the statistics \begin{align*} T_3 & =(\te…
Sparse Nonparametric Graphical Models
John Lafferty, Han Liu, Larry Wasserman
We present some nonparametric methods for graphical modeling. In the discrete case, where the data are binary or drawn from a finite alphabet, Markov random fields are already esse…
Adaptive Risk Bounds in Unimodal Regression
Sabyasachi Chatterjee, John Lafferty
We study the statistical properties of the least squares estimator in unimodal sequence estimation. Although closely related to isotonic regression, unimodal regression has not bee…
Forest Density Estimation
Han Liu, Min Xu, Haijie Gu +3
We study graph estimation and density estimation in high dimensions, using a family of density estimators based on forest structured undirected graphical models. For density estima…
Rodeo: Sparse Nonparametric Regression in High Dimensions
John Lafferty, Larry Wasserman
We present a greedy method for simultaneously performing local bandwidth selection and variable selection in nonparametric regression. The method starts with a local linear estimat…
A Convergent Gradient Descent Algorithm for Rank Minimization and Semidefinite Programming from Random Linear Measurements
Qinqing Zheng, John Lafferty
We propose a simple, scalable, and fast gradient descent algorithm to optimize a nonconvex objective for the rank minimization problem and a closely related family of semidefinite…
The Nonparanormal: Semiparametric Estimation of High Dimensional Undirected Graphs
Han Liu, John Lafferty, Larry Wasserman
Recent methods for estimating sparse undirected graphs for real-valued data in high dimensional problems rely heavily on the assumption of normality. We show how to use a semiparam…
Distributed Nonparametric Regression under Communication Constraints
Yuancheng Zhu, John Lafferty
This paper studies the problem of nonparametric estimation of a smooth function with data distributed across multiple machines. We assume an independent sample from a white noise m…
Fair quantile regression
Dana Yang, John Lafferty, David Pollard
Quantile regression is a tool for learning conditional distributions. In this paper we study quantile regression in the setting where a protected attribute is unavailable when fitt…
Selective Inference for Group-Sparse Linear Models
Fan Yang, Rina Foygel Barber, Prateek Jain +1
We develop tools for selective inference in the setting of group sparsity, including the construction of confidence intervals and p-values for testing selected groups of variables.…
The Relational Bottleneck as an Inductive Bias for Efficient Abstraction
Taylor W. Webb, Steven M. Frankland, Awni Altabaa +9
A central challenge for cognitive science is to explain how abstract concepts are acquired from limited experience. This has often been framed in terms of a dichotomy between conne…
A Robust Parsing Algorithm For Link Grammars
Dennis Grinberg, John Lafferty, Daniel Sleator
In this paper we present a robust parsing algorithm based on the link grammar formalism for parsing natural languages. Our algorithm is a natural extension of the original dynamic…
Convergence Analysis for Rectangular Matrix Completion Using Burer-Monteiro Factorization and Gradient Descent
Qinqing Zheng, John Lafferty
We address the rectangular matrix completion problem by lifting the unknown matrix to a positive semidefinite matrix in higher dimension, and optimizing a nonconvex objective over…
Information-Computation Tradeoffs for Noiseless Linear Regression with Oblivious Contamination
Ilias Diakonikolas, Chao Gao, Daniel M. Kane +2
We study the task of noiseless linear regression under Gaussian covariates in the presence of additive oblivious contamination. Specifically, we are given i.i.d.\ samples from a di…
Prediction Rule Reshaping
Matt Bonakdarpour, Sabyasachi Chatterjee, Rina Foygel Barber +1
Two methods are proposed for high-dimensional shape-constrained regression and classification. These methods reshape pre-trained prediction rules to satisfy shape constraints like…
A Model of Lexical Attraction and Repulsion
Doug Beeferman, Adam Berger, John Lafferty
This paper introduces new methods based on exponential families for modeling the correlations between words in text and speech. While previous work assumed the effects of word co-o…
Quantized Estimation of Gaussian Sequence Models in Euclidean Balls
Yuancheng Zhu, John Lafferty
A central result in statistical theory is Pinsker's theorem, which characterizes the minimax rate in the normal means model of nonparametric estimation. In this paper, we present a…
Local Minimax Complexity of Stochastic Convex Optimization
Yuancheng Zhu, Sabyasachi Chatterjee, John Duchi +1
We extend the traditional worst-case, minimax analysis of stochastic convex optimization by introducing a localized form of minimax complexity for individual functions. Our main re…
Sparse Additive Functional and Kernel CCA
Sivaraman Balakrishnan, Kriti Puniyani, John Lafferty
Canonical Correlation Analysis (CCA) is a classical tool for finding correlations among the components of two random vectors. In recent years, CCA has been widely applied to the an…
Surfing: Iterative optimization over incrementally trained deep networks
Ganlin Song, Zhou Fan, John Lafferty
We investigate a sequential optimization procedure to minimize the empirical risk functional for certain families of deep netwo…
Faithful Variable Screening for High-Dimensional Convex Regression
Min Xu, Minhua Chen, John Lafferty
We study the problem of variable selection in convex nonparametric regression. Under the assumption that the true regression function is convex and sparse, we develop a screening p…
Uncovering Latent Reasoning Strategies in Language Models
Awni Altabaa, John Lafferty
A language model trained on reasoning tasks learns to solve problems via multiple distinct strategies, yet these strategies are implicit and entangled within the m…
Compressed Regression
Shuheng Zhou, John Lafferty, Larry Wasserman
Recent research has studied the role of sparsity in high dimensional regression and signal reconstruction, establishing theoretical limits for recovering sparse models from sparse…
Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoning
Awni Altabaa, Siyu Chen, John Lafferty +1
Systematic, compositional generalization beyond the training distribution remains a core challenge in machine learning -- and a critical bottleneck for the emergent reasoning abili…
Nonparametric Reduced Rank Regression
Rina Foygel, Michael Horrell, Mathias Drton +1
We propose an approach to multivariate nonparametric regression that generalizes reduced rank regression for linear models. An additive model is estimated for each dimension of a $…
The Nonparanormal SKEPTIC
Han Liu, Fang Han, Ming Yuan +2
We propose a semiparametric approach, named nonparanormal skeptic, for estimating high dimensional undirected graphical models. In terms of modeling, we consider the nonparanormal…
Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
Ganlin Song, Ruitu Xu, John Lafferty
Stochastic gradient descent with backpropagation is the workhorse of artificial neural networks. It has long been recognized that backpropagation fails to be a biologically plausib…
Conditional Sparse Coding and Grouped Multivariate Regression
Min Xu, John Lafferty
We study the problem of multivariate regression where the data are naturally grouped, and a regression matrix is to be estimated for each group. We propose an approach in which a d…
Sparse Additive Models
Pradeep Ravikumar, John Lafferty, Han Liu +1
We present a new class of methods for high-dimensional nonparametric regression and classification called sparse additive models (SpAM). Our methods combine ideas from sparse linea…
Sequential Nonparametric Regression
Haijie Gu, John Lafferty
We present algorithms for nonparametric regression in settings where the data are obtained sequentially. While traditional estimators select bandwidths that depend upon the sample…
Denoising Flows on Trees
Sabyasachi Chatterjee, John Lafferty
We study the estimation of flows on trees, a structured generalization of isotonic regression. A tree flow is defined recursively as a positive flow value into a node that is parti…
Variational Chernoff Bounds for Graphical Models
Pradeep Ravikumar, John Lafferty
Recent research has made significant progress on the problem of bounding log partition functions for exponential family graphical models. Such bounds have associated dual parameter…
Union Support Recovery in Multi-task Learning
Mladen Kolar, John Lafferty, Larry Wasserman
We sharply characterize the performance of different penalization schemes for the problem of selecting the relevant variables in the multi-task setting. Previous work focuses on th…
Graph-Valued Regression
Han Liu, Xi Chen, John Lafferty +1
Undirected graphical models encode in a graph the dependency structure of a random vector . In many applications, it is of interest to model given another random vector…
Confidence Intervals for Linear Models with Arbitrary Noise Contamination
Dong Xie, Chao Gao, John Lafferty
We study confidence interval construction for linear regression under Huber's contamination model, where an unknown fraction of noise variables is arbitrarily corrupted. While robu…
Learning Hierarchical Relational Representations through Relational Convolutions
Awni Altabaa, John Lafferty
An evolving area of research in deep learning is the study of architectures and inductive biases that support the learning of relational feature representations. In this paper, we…
Emergent organization of receptive fields in networks of excitatory and inhibitory neurons
Leon Lufkin, Ashish Puri, Ganlin Song +2
Local patterns of excitation and inhibition that can generate neural waves are studied as a computational mechanism underlying the organization of neuronal tunings. Sparse coding a…
Disentangling and Integrating Relational and Sensory Information in Transformer Architectures
Awni Altabaa, John Lafferty
Relational reasoning is a central component of generally intelligent systems, enabling robust and data-efficient inductive generalization. Recent empirical evidence shows that many…
Time Varying Undirected Graphs
Shuheng Zhou, John Lafferty, Larry Wasserman
Undirected graphs are often used to describe high dimensional distributions. Under sparsity conditions, the graph can be estimated using penalization methods. However, cur…
Model Repair: Robust Recovery of Over-Parameterized Statistical Models
Chao Gao, John Lafferty
A new type of robust estimation problem is introduced where the goal is to recover a statistical model that has been corrupted after it has been estimated from data. Methods are pr…
Text Segmentation Using Exponential Models
Doug Beeferman, Adam Berger, John Lafferty
This paper introduces a new statistical approach to partitioning text automatically into coherent segments. Our approach enlists both short-range and long-range language models to…
CoT Information: Improved Sample Complexity under Chain-of-Thought Supervision
Awni Altabaa, Omar Montasser, John Lafferty
Learning complex functions that involve multi-step reasoning poses a significant challenge for standard supervised learning from input-output examples. Chain-of-thought (CoT) super…
Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in Transformers
Awni Altabaa, Taylor Webb, Jonathan Cohen +1
An extension of Transformers is proposed that enables explicit relational reasoning through a novel module called the Abstractor. At the core of the Abstractor is a variant of atte…
Towards History-based Grammars: Using Richer Models for Probabilistic Parsing
Ezra Black, Fred Jelinek, John Lafferty +3
We describe a generative probabilistic model of natural language, which we call HBG, that takes advantage of detailed linguistic information to resolve ambiguity. HBG incorporates…
High Dimensional Semiparametric Gaussian Copula Graphical Models
Han Liu, Fang Han, Ming Yuan +2
In this paper, we propose a semiparametric approach, named nonparanormal skeptic, for efficiently and robustly estimating high dimensional undirected graphical models. To achieve m…