Batchless Normalization: How to Normalize Activations Across Instances with Minimal Memory Requirements

Benjamin Berger, Victor Uc Cetina

In training neural networks, batch normalization has many benefits, not all of them entirely understood. But it also has some drawbacks. Foremost is arguably memory consumption, as…