CREST: Effectively Compacting a Datastore For Retrieval-Based Speculative Decoding

Sophia Ho, Jinsol Park, Patrick Wang

We present CREST (Compact Retrieval-Based Speculative Decoding), a redesign of REST that allows it to be effectively "compacted". REST is a drafting technique for speculative decod…