AutoCompress: Critical Layer Isolation for Efficient Transformer Compression

Archit Thorat

We present AutoCompress, a transformer compression method motivated by an empirical finding: in small transformers, Layer 0 carries disproportionately high task-critical informatio…