Matryoshka Quantization

Pranav Nair, Puranjay Datta, Jeff Dean +2

Quantizing model weights is critical for reducing the communication and inference costs of large models. However, quantizing models -- especially to low precisions like int4 or int…