Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs

Ryan Synk, Monte Hoover, John Kirchenbauer +6

There is growing demand for performing inference with hundreds of thousands of input tokens on trained transformer models. Inference at this extreme scale demands significant compu…