3.5k citations · 7.7k across the 28 of their papers we have counts for
31 papers
PolyAlign: Conditional Human-Distribution Alignment
L. D. M. S. Sai Teja, Ufaq Khan, Sathira Silva +2
Post-training methods such as supervised fine-tuning (SFT) and preference optimization typically align language models toward a single global assistant behavior. While effective fo…
Training LLMs for Honesty via Confessions
Manas Joglekar, Jeremy Chen, Gabriel Wu +4
Large language models (LLMs) can be dishonest when reporting on their actions and beliefs -- for example, they may overstate their confidence in factual claims or cover up evidence…
RxRx1: A Dataset for Evaluating Experimental Batch Correction Methods
Maciej Sypetkowski, Morteza Rezanejad, Saber Saberian +9
High-throughput screening techniques are commonly used to obtain large quantities of data in many fields of biology. It is well known that artifacts arising from variability in the…
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao +448
Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabil…
Language Models are Few-shot Multilingual Learners
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin +3
General-purpose language models have demonstrated impressive capabilities, performing on par with state-of-the-art approaches on a range of downstream natural language processing (…
When does loss-based prioritization fail?
Niel Teng Hu, Xinyu Hu, Rosanne Liu +2
Not all examples are created equal, but standard deep neural network training protocols treat each training point uniformly. Each example is propagated forward and backward through…
Supermasks in Superposition
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu +4
We present the Supermasks in Superposition (SupSup) model, capable of sequentially learning thousands of tasks without catastrophic forgetting. Our approach uses a randomly initial…
Estimating Q(s,s') with Deep Deterministic Dynamics Gradients
Ashley D. Edwards, Himanshu Sahni, Rosanne Liu +7
In this paper, we introduce a novel form of value function, , that expresses the utility of transitioning from a state to a neighboring state and then acting opt…
Plug and Play Language Models: A Simple Approach to Controlled Text Generation
Sumanth Dathathri, Andrea Madotto, Janice Lan +5
Large transformer-based language models (LMs) trained on huge text corpora have shown unparalleled generation capabilities. However, controlling attributes of the generated languag…
First-Order Preconditioning via Hypergradient Descent
Ted Moskovitz, Rui Wang, Janice Lan +4
Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulti…
LCA: Loss Change Allocation for Neural Network Training
Janice Lan, Rosanne Liu, Hattie Zhou +1
Neural networks enjoy widespread use, but many aspects of their training, representation, and operation are poorly understood. In particular, our view into the training process is…
Hamiltonian Neural Networks
Sam Greydanus, Misko Dzamba, Jason Yosinski
Even though neural networks enjoy widespread use, they still struggle to learn the basic laws of physics. How might we endow them with better inductive biases? In this paper, we dr…
Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
Hattie Zhou, Janice Lan, Rosanne Liu +1
The recent "Lottery Ticket Hypothesis" paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keeping the large weights) results in models that are tr…
Understanding Neural Networks via Feature Visualization: A survey
Anh Nguyen, Jason Yosinski, Jeff Clune
A neuroscience method to understanding the brain is to find and study the preferred stimuli that highly activate an individual cell or groups of cells. Recent advances in machine l…
Metropolis-Hastings Generative Adversarial Networks
Ryan Turner, Jane Hung, Eric Frank +2
We introduce the Metropolis-Hastings generative adversarial network (MH-GAN), which combines aspects of Markov chain Monte Carlo and GANs. The MH-GAN draws samples from the distrib…
An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
Rosanne Liu, Joel Lehman, Piero Molino +4
Few ideas have enjoyed as large an impact on deep learning as convolution. For any problem involving pixels or spatial representations, common intuition holds that convolutional ne…
Measuring the Intrinsic Dimension of Objective Landscapes
Chunyuan Li, Heerad Farkhoor, Rosanne Liu +1
Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of…
Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning
Andrew Gordon Wilson, Jason Yosinski, Patrice Simard +2
This is the Proceedings of NIPS 2017 Symposium on Interpretable Machine Learning, held in Long Beach, California, USA on December 7, 2017
SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
Maithra Raghu, Justin Gilmer, Jason Yosinski +1
We propose a new technique, Singular Vector Canonical Correlation Analysis (SVCCA), a tool for quickly comparing two representations in a way that is both invariant to affine trans…
Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski +2
Deep neural networks (DNNs) have demonstrated state-of-the-art results on many pattern recognition tasks, especially vision classification problems. Understanding the inner working…
Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
Anh Nguyen, Jason Yosinski, Jeff Clune
We can better understand deep neural networks by identifying which features each of their neurons have learned to detect. To do so, researchers have created Deep Visualization tech…
Convergent Learning: Do different neural networks learn the same representations?
Yixuan Li, Jason Yosinski, Jeff Clune +2
Recent success in training deep neural networks have prompted active investigation into the features learned on their intermediate layers. Such research is difficult because it req…
Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation
Sina Honari, Jason Yosinski, Pascal Vincent +1
Deep neural networks with alternating convolutional, max-pooling and decimation layers are widely used in state of the art architectures for computer vision. Max-pooling purposeful…
Understanding Neural Networks Through Deep Visualization
Jason Yosinski, Jeff Clune, Anh Nguyen +2
Recent years have produced great advances in training large, deep neural networks (DNNs), including notable successes in training convolutional neural networks (convnets) to recogn…