Characterizing High Bandwidth Flash for LLM Serving

Zack Yu, Chloe Wong, Coleman Hooper +7

Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth…