AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration

Bradley McDanel

Large language models typically generate tokens autoregressively, using each token as input for the next. Recent work on Speculative Decoding has sought to accelerate this process…