Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism

Jinhui Wei, Ye Huang, Yuhui Zhou +3

In-situ LLM inference on end-user devices has gained significant interest due to its privacy benefits and reduced dependency on external infrastructure. However, as the decoding pr…