4 papers
Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization
Wenhao Zhao, Qiran Zou, Rushi Shah +3
Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. It is widely employed in tokenizing data represen…
Representation Collapsing Problems in Vector Quantization
Wenhao Zhao, Qiran Zou, Rushi Shah +1
Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. It is widely employed in tokenizing data represen…
Gaussian Mixture Vector Quantization with Aggregated Categorical Posterior
Mingyuan Yan, Jiawei Wu, Rushi Shah +1
The vector quantization is a widely used method to map continuous representation to discrete space and has important application in tokenization for generative mode, bottlenecking…
Improving Discrete Optimisation Via Decoupled Straight-Through Estimator
Rushi Shah, Mingyuan Yan, Michael Curtis Mozer +1
The Straight-Through Estimator (STE) is the dominant method for training neural networks with discrete variables, enabling gradient-based optimisation by routing gradients through…