688 citations · 985 across the 18 of their papers we have counts for
15 papers · 1 filter
Continuous Chain of Thought Enables Parallel Exploration and Reasoning
Halil Alperen Gozeten, M. Emrullah Ildiz, Xuechen Zhang +3
Modern language models generate chain-of-thought traces by autoregressively sampling tokens from a finite vocabulary. While this discrete sampling has achieved remarkable success,…
A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs
Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh +12
A primary challenge in large language model (LLM) development is their onerous pre-training cost. Typically, such pre-training involves optimizing a self-supervised objective (such…
Mimetic Initialization Helps State Space Models Learn to Recall
Asher Trockman, Hrayr Harutyunyan, J. Zico Kolter +2
Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with r…
In-context Learning in Presence of Spurious Correlations
Hrayr Harutyunyan, Rafayel Darbinyan, Samvel Karapetyan +1
Large language models exhibit a remarkable capacity for in-context learning, where they learn to solve tasks given a few examples. Recent work has shown that transformers can be tr…
On information captured by neural networks: connections with memorization and generalization
Hrayr Harutyunyan
Despite the popularity and success of deep learning, there is limited understanding of when, how, and why neural networks generalize to unseen examples. Since learning can be seen…
Identifying and Disentangling Spurious Features in Pretrained Image Representations
Rafayel Darbinyan, Hrayr Harutyunyan, Aram H. Markosyan +1
Neural networks employ spurious correlations in their predictions, resulting in decreased performance when these correlations do not hold. Recent works suggest fixing pretrained re…
Supervision Complexity and its Role in Knowledge Distillation
Hrayr Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon +2
Despite the popularity and efficacy of knowledge distillation, there is limited understanding of why it helps. In order to study the generalization behavior of a distilled student,…
Formal limitations of sample-wise information-theoretic generalization bounds
Hrayr Harutyunyan, Greg Ver Steeg, Aram Galstyan
Some of the tightest information-theoretic generalization bounds depend on the average information between the learned hypothesis and a single training example. However, these samp…
Failure Modes of Domain Generalization Algorithms
Tigran Galstyan, Hrayr Harutyunyan, Hrant Khachatrian +2
Domain generalization algorithms use training data from multiple domains to learn models that generalize well to unseen domains. While recently proposed benchmarks demonstrate that…
Information-theoretic generalization bounds for black-box learning algorithms
Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg +1
We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training…
Estimating informativeness of samples with Smooth Unique Information
Hrayr Harutyunyan, Alessandro Achille, Giovanni Paolini +4
We define a notion of information that an individual sample provides to the training of a neural network, and we specialize it to measure both how much a sample informs the final w…
Improving Generalization by Controlling Label-Noise Information in Neural Network Weights
Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg +1
In the presence of noisy or incorrect labels, neural networks have the undesirable tendency to memorize information about the noise. Standard regularization techniques such as drop…
MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing
Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor +5
Existing popular methods for semi-supervised learning with Graph Neural Networks (such as the Graph Convolutional Network) provably cannot learn a general class of neighborhood mix…
Efficient Covariance Estimation from Temporal Data
Hrayr Harutyunyan, Daniel Moyer, Hrant Khachatrian +2
Estimating the covariance structure of multivariate time series is a fundamental problem with a wide-range of real-world applications -- from financial modeling to fMRI analysis. D…
Disentangled Representations via Synergy Minimization
Greg Ver Steeg, Rob Brekelmans, Hrayr Harutyunyan +1
Scientists often seek simplified representations of complex systems to facilitate prediction and understanding. If the factors comprising a representation allow us to make accurate…