9.4k citations · 11.7k across the 21 of their papers we have counts for
8 papers · 1 filter
Training nGPT
Ilya Loshchilov, Boris Ginsburg
The normalized Transformer (nGPT) realizes hyperspherical representation learning by constraining model parameter vectors and activation vectors to the unit hypersphere. In this pa…
Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
NVIDIA, :, Aakshita Chandiramani +544
We describe the pre-training, post-training, and quantization of Nemotron 3 Super, a 120 billion (active 12 billion) parameter hybrid Mamba-Attention Mixture-of-Experts model. Nemo…
nGPT: Normalized Transformer with Representation Learning on the Hypersphere
Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun +1
We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP,…
Weight Norm Control
Ilya Loshchilov
We note that decoupled weight decay regularization is a particular case of weight norm control where the target norm of weights is set to 0. Any optimization method (e.g., Adam) wh…
Decoupled Weight Decay Regularization
Ilya Loshchilov, Frank Hutter
L regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \em…
SGDR: Stochastic Gradient Descent with Warm Restarts
Ilya Loshchilov, Frank Hutter
Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to i…
Online Batch Selection for Faster Training of Neural Networks
Ilya Loshchilov, Frank Hutter
Deep neural networks are commonly trained using stochastic non-convex optimization procedures, which are driven by gradient information estimated on fractions (batches) of the data…
KL-based Control of the Learning Schedule for Surrogate Black-Box Optimization
Ilya Loshchilov, Marc Schoenauer, Michèle Sebag
This paper investigates the control of an ML component within the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) devoted to black-box optimization. The known CMA-ES weakn…