9 citations · 15 across the 6 of their papers we have counts for
6 papers
Estimating the Probability of Sampling a Trained Neural Network at Random
Adam Scherlis, Nora Belrose
We present and analyze an algorithm for estimating the size, under a Gaussian or uniform measure, of a localized neighborhood in neural network parameter space with behavior simila…
Understanding Gradient Descent through the Training Jacobian
Nora Belrose, Adam Scherlis
We examine the geometry of neural network training using the Jacobian of trained network parameters with respect to their initial values. Our analysis reveals low-dimensional struc…
Refusal in LLMs is an Affine Function
Thomas Marshall, Adam Scherlis, Nora Belrose
We propose affine concept editing (ACE) as an approach for steering language models' behavior by intervening directly in activations. We begin with an affine decomposition of model…
Polysemanticity and Capacity in Neural Networks
Adam Scherlis, Kshitij Sachan, Adam S. Jermyn +2
Individual neurons in neural networks often represent a mixture of unrelated features. This phenomenon, called polysemanticity, can make interpreting neural networks more difficult…
Adversarial Training for High-Stakes Reliability
Daniel M. Ziegler, Seraphina Nix, Lawrence Chan +9
In the future, powerful AI systems may be deployed in high-stakes settings, where a single failure could be catastrophic. One technique for improving AI safety in high-stakes setti…
The Goldilocks zone: Towards better understanding of neural network loss landscapes
Stanislav Fort, Adam Scherlis
We explore the loss landscape of fully-connected and convolutional neural networks using random, low-dimensional hyperplanes and hyperspheres. Evaluating the Hessian, , of the l…