Dynamic Semantic Compression for Efficient Latent-Space Inference in Large Language Models

Peipei Li, Dongsen Zhang, Yuchen Liu +1

Large Language Models (LLMs) primarily perform inference at the token level, resulting in substantial memory overhead and compromised computational efficiency. In this paper, we pr…