Token Level Routing Inference System for Edge Devices

Jianshu She, Wenhao Zheng, Zhengzhong Liu +4

The computational complexity of large language model (LLM) inference significantly constrains their deployment efficiency on edge devices. In contrast, small language models offer…