On-device Semantic Selection Made Low Latency and Memory Efficient with Monolithic Forwarding

Jiahao Zhou, Chengliang Lin, Dingji Li +2

Semantic top-K selection with cross-encoder rerankers underpins on-device AI services, such as retrieval-augmented generation, agent memory, and personalized recommendation. Howeve…