FastQuery: Communication-efficient Embedding Table Query for Private LLM Inference

Chenqi Lin, Tianshi Xu, Zebin Yang +3

With the fast evolution of large language models (LLMs), privacy concerns with user queries arise as they may contain sensitive information. Private inference based on homomorphic…