164 citations · 552 across the 11 of their papers we have counts for
6 papers · 1 filter
Scaling Optimal LR Across Token Horizons
Johan Bjorck, Alon Benhaim, Vishrav Chaudhary +2
State-of-the-art LLMs are powered by scaling -- scaling model size, dataset size and cluster size. It is economically infeasible to extensively tune hyperparameter for the largest…
Is High Variance Unavoidable in RL? A Case Study in Continuous Control
Johan Bjorck, Carla P. Gomes, Kilian Q. Weinberger
Reinforcement learning (RL) experiments have notoriously high variance, and minor details can have disproportionately large effects on measured outcomes. This is problematic for cr…
Towards Deeper Deep Reinforcement Learning with Spectral Normalization
Johan Bjorck, Carla P. Gomes, Kilian Q. Weinberger
In computer vision and natural language processing, innovations in model architecture that increase model capacity have reliably translated into gains in performance. In stark cont…
Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
Johan Bjorck, Xiangyu Chen, Christopher De Sa +2
Low-precision training has become a popular approach to reduce compute requirements, memory footprint, and energy consumption in supervised learning. In contrast, this promising ap…
Understanding Decoupled and Early Weight Decay
Johan Bjorck, Kilian Weinberger, Carla Gomes
Weight decay (WD) is a traditional regularization technique in deep learning, but despite its ubiquity, its behavior is still an area of active research. Golatkar et al. have recen…
Understanding Batch Normalization
Johan Bjorck, Carla Gomes, Bart Selman +1
Batch normalization (BN) is a technique to normalize activations in intermediate layers of deep neural networks. Its tendency to improve accuracy and speed up training have establi…