An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2

Pepijn de Reus, Ana Oprescu, Jelle Zuidema

This study examines quantisation and pruning strategies to reduce energy consumption in code Large Language Models (LLMs) inference. Using StarCoder2, we observe increased energy d…