Power Law Guided Dynamic Sifting for Efficient Attention

Nirav Koley, Prajwal Singhania, Abhinav Bhatele

Efficient inference on GPUs using large language models remains challenging due to memory bandwidth limitations, particularly during data transfers between High Bandwidth Memory (H…