RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training

Dor Tsur, Maor Ashkenazi, Maximilian Golub +7

Speculative decoding is a popular decode acceleration method. As such, it is particularly relevant to long-output generation, which requires an increasingly large KV cache. KV cach…