Long-Context Drafting Does Not Require Long-Context Drafters

Aref Jafari, Vansh Bhatia, Matt Qin +5

Speculative decoding pairs a cheap drafter with parallel verification against the target. At ultra-long context it stops paying, because a drafter that attends over the full prefix…