1 citations · 1 across the 2 of their papers we have counts for
5 papers
KVBuffer: IO-aware Serving for Linear Attention
Longwei Zou, Lin Zhong
Linear attention has recently gained significant attention for long-context inference due to its constant decoding cost with respect to context length. However, existing serving sy…
Confidential Prompting: Privacy-preserving LLM Inference on Cloud
Caihua Li, In Gim, Lin Zhong
This paper introduces a vision of confidential prompting: securing user prompts from an untrusted, cloud-hosted large language model (LLM) while preserving model confidentiality, o…
MappedTrace: Tracing Pointer Remotely with Compiler-generated Maps
Zhiyao Ma, Caihua Li, Lin Zhong
Existing precise pointer tracing methods introduce substantial runtime overhead to the program being traced and are applicable only at specific program execution points. We propose…
Asynchronous LLM Function Calling
In Gim, Seung-seob Lee, Lin Zhong
Large language models (LLMs) use function calls to interface with external tools and data source. However, the current approach to LLM function calling is inherently synchronous, w…
Blindfold: Confidential Memory Management by Untrusted Operating System
Caihua Li, Seung-seob Lee, Lin Zhong
Confidential Computing (CC) has received increasing attention in recent years as a mechanism to protect user data from untrusted operating systems (OSes). Existing CC solutions hid…