Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks

Minyoung Huh, Brian Cheung, Pulkit Agrawal +1

This work examines the challenges of training neural networks using vector quantization using straight-through estimation. We find that a primary cause of training instability is t…