115 citations · 331 across the 37 of their papers we have counts for
25 papers · 1 filter
Can Capability Enhancement be Localized?
Anh Tuan Nguyen, Anna Bair, Zico Kolter
Transformer-based language models have shown strong capabilities across many tasks and domains. However, prior work shows that, for a given task, not all attention heads matter equ…
Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning
Benhao Huang, Zhengyang Geng, Zico Kolter
Scaling test-time compute by iteratively updating a latent state has emerged as a powerful paradigm for reasoning. Yet the internal mechanisms that enable these iterative models to…
Contextures: Representations from Contexts
Runtian Zhai, Kai Yang, Che-Ping Tsai +3
Despite the empirical success of foundation models, we do not have a systematic characterization of the representations that these models learn. In this paper, we establish the con…
Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
Yixuan Even Xu, Yash Savani, Fei Fang +1
Reinforcement learning with verifiable rewards (RLVR) has emerged as the leading approach for enhancing reasoning capabilities in large language models. However, it faces a fundame…
Weight Ensembling Improves Reasoning in Language Models
Xingyu Dang, Christina Baek, Kaiyue Wen +2
We investigate a failure mode that arises during the training of reasoning models, where the diversity of generations begins to collapse, leading to suboptimal test-time scaling. N…
Understanding Optimization in Deep Learning with Central Flows
Jeremy M. Cohen, Alex Damian, Ameet Talwalkar +2
Traditional theories of optimization cannot describe the dynamics of optimization in deep learning, even in the simple setting of deterministic training. The challenge is that opti…
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian +11
The robustness of LLMs to jailbreak attacks, where users design prompts to circumvent safety measures and misuse model capabilities, has been studied primarily for LLMs acting as s…
Improving Alignment and Robustness with Circuit Breakers
Andy Zou, Long Phan, Justin Wang +7
AI systems can take harmful actions and are highly vulnerable to adversarial attacks. We present an approach, inspired by recent advances in representation engineering, that interr…
Why is SAM Robust to Label Noise?
Christina Baek, Zico Kolter, Aditi Raghunathan
Sharpness-Aware Minimization (SAM) is most known for achieving state-of the-art performances on natural image and language tasks. However, its most pronounced improvements (of tens…
Neural Network Verification with Branch-and-Bound for General Nonlinearities
Zhouxing Shi, Qirui Jin, Zico Kolter +3
Branch-and-bound (BaB) is among the most effective techniques for neural network (NN) verification. However, existing works on BaB for NN verification have mostly focused on NNs wi…
Predicting the Performance of Foundation Models via Agreement-on-the-Line
Rahul Saxena, Taeyoun Kim, Aman Mehra +3
Estimating the out-of-distribution performance in regimes where labels are scarce is critical to safely deploy foundation models. Recently, it was shown that ensembles of neural ne…
Representation Engineering: A Top-Down Approach to AI Transparency
Andy Zou, Long Phan, Sarah Chen +18
In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights f…
Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression
Runtian Zhai, Bingbin Liu, Andrej Risteski +2
Data augmentation is critical to the empirical success of modern self-supervised representation learning, such as contrastive learning and masked language modeling. However, a theo…
Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Single
Paul Vicol, Zico Kolter, Kevin Swersky
We propose an evolution strategies-based algorithm for estimating gradients in unrolled computation graphs, called ES-Single. Similarly to the recently-proposed Persistent Evolutio…
Provably Bounding Neural Network Preimages
Suhas Kotha, Christopher Brix, Zico Kolter +2
Most work on the formal verification of neural networks has focused on bounding the set of outputs that correspond to a given set of inputs (for example, bounded perturbations of a…
Learning Options via Compression
Yiding Jiang, Evan Zheran Liu, Benjamin Eysenbach +2
Identifying statistical regularities in solutions to some tasks in multi-task reinforcement learning can accelerate the learning of new tasks. Skill learning offers one way of iden…
Path Independent Equilibrium Models Can Better Exploit Test-Time Computation
Cem Anil, Ashwini Pokle, Kaiqu Liang +5
Designing networks capable of attaining better performance with an increased inference budget is important to facilitate generalization to harder problem instances. Recent efforts…
Deep Equilibrium Approaches to Diffusion Models
Ashwini Pokle, Zhengyang Geng, Zico Kolter
Diffusion-based generative models are extremely effective in generating high-quality images, with generated samples often surpassing the quality of those produced by other models u…
Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagation
Zhouxing Shi, Yihan Wang, Huan Zhang +2
Lipschitz constants are connected to many properties of neural networks, such as robustness, fairness, and generalization. Existing methods for computing Lipschitz constants either…
Test-Time Adaptation via Conjugate Pseudo-labels
Sachin Goyal, Mingjie Sun, Aditi Raghunathan +1
Test-time adaptation (TTA) refers to adapting neural networks to distribution shifts, with access to only the unlabeled test samples from the new domain at test-time. Prior TTA met…
Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift
Christina Baek, Yiding Jiang, Aditi Raghunathan +1
Recently, Miller et al. showed that a model's in-distribution (ID) accuracy has a strong linear correlation with its out-of-distribution (OOD) accuracy on several OOD benchmarks --…
Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Robustness Verification
Shiqi Wang, Huan Zhang, Kaidi Xu +4
Bound propagation based incomplete neural network verifiers such as CROWN are very efficient and can significantly accelerate branch-and-bound (BaB) based complete verification of…
You Only Query Once: Effective Black Box Adversarial Attacks with Minimal Repeated Queries
Devin Willmott, Anit Kumar Sahu, Fatemeh Sheikholeslami +2
Researchers have repeatedly shown that it is possible to craft adversarial attacks on deep classifiers (small perturbations that significantly change the class label), even in the…
Provably robust deep generative models
Filipe Condessa, Zico Kolter
Recent work in adversarial attacks has developed provably robust methods for training deep neural network classifiers. However, although they are often mentioned in the context of…