RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding

Yuichiro Hoshino, Hideyuki Tachibana, Muneyoshi Inahara +1

Hybrid models combining Transformers and State Space Models (SSMs) are promising for balancing performance and efficiency. However, optimizing these hybrid models, particularly by…