SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification

Pragaash Ponnusamy, Shivam Sahni, Jue Wang +1

Sampling in LLM inference comprises a combinatorial set of logit processing, token selection, and verification operations for speculative decoding. However, existing implementation…