A Dataflow Compiler for Efficient LLM Inference using Custom Microscaling Formats

Jianyi Cheng, Cheng Zhang, Zhewen Yu +3

Model quantization represents both parameters (weights) and intermediate values (activations) in a more compact format, thereby directly reducing both computational and memory cost…