1 citations · 4 across the 19 of their papers we have counts for
22 papers
Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs
Vincent-Daniel Yun, Woosang Lim, Haneul Yoo +3
Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires each receiver to prefill shared con…
ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2
Long-context LLM inference is bottlenecked by attention, whose repeated KV-cache reads make decoding memory-bound. Self-speculative decoding alleviates this by drafting tokens with…
Output-Aware Rotation for INT2 KV-Cache Quantization
Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4
The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important…
OTRO: Oblivious Tokenization Path with Square-Root ORAM
Jonghyun Lee, Yongqin Wang, Rachit Rajat +3
The CPU-side large language model (LLM) tokenizer is a critical security gap in LLM serving through a confidential computing stack with CPU and GPU trusted execution environments (…
Infrastructure for Valuable, Tradable, and Verifiable Agent Memory
Mengyuan Li, Lei Gao, Haoxuan Xu +5
Every API token you spend is your accumulated wealth; once you can prove its value and the effort behind it, you can resell it. As autonomous agents repeatedly call models and tool…
LRD-MPC: Efficient MPC Inference through Low-rank Decomposition
Tingting Tang, Yongqin Wang, Murali Annavaram
Secure Multi-party Computation (MPC) enables untrusted parties to jointly compute a function without revealing their inputs. Its application to machine learning (ML) has gained sig…
Differentially Private Retrieval-Augmented Generation
Tingting Tang, James Flemings, Yongqin Wang +1
Retrieval-augmented generation (RAG) is a widely used framework for reducing hallucinations in large language models (LLMs) on domain-specific tasks by retrieving relevant document…
Striking the Right Balance between Compute and Copy: Improving LLM Inferencing Under Speculative Decoding
Arun Ramachandran, Ramaswamy Govindarajan, Murali Annavaram +4
With the skyrocketing costs of GPUs and their virtual instances in the cloud, there is a significant desire to use CPUs for large language model (LLM) inference. KV cache update, o…
DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing
Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch +3
Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode ph…
DistilLock: Safeguarding LLMs from Unauthorized Knowledge Distillation on the Edge
Asmita Mohanty, Gezheng Kang, Lei Gao +1
Large Language Models (LLMs) have demonstrated strong performance across diverse tasks, but fine-tuning them typically relies on cloud-based, centralized infrastructures. This requ…
MARché: Fast Masked Autoregressive Image Generation with Cache-Aware Attention
Chaoyi Jiang, Sungwoo Kim, Lei Gao +3
Masked autoregressive (MAR) models unify the strengths of masked and autoregressive generation by predicting tokens in a fixed order using bidirectional attention for image generat…
DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding
Hossein Entezari Zarch, Lei Gao, Chaoyi Jiang +1
Speculative Decoding (SD) is a widely used approach to accelerate the inference of large language models (LLMs) without reducing generation quality. It operates by first using a co…
Characterization of GPU TEE Overheads in Distributed Data Parallel ML Training
Jonghyun Lee, Yongqin Wang, Rachit Rajat +1
Confidential computing (CC) or trusted execution enclaves (TEEs) is now the most common approach to enable secure computing in the cloud. The recent introduction of GPU TEEs by NVI…
KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation
Chaoyi Jiang, Lei Gao, Hossein Entezari Zarch +1
Inference for Large Language Models (LLMs) is computationally demanding. To reduce the cost of auto-regressive decoding, Key-Value (KV) cache is used to store intermediate activati…
Fastrack: Fast IO for Secure ML using GPU TEEs
Yongqin Wang, Rachit Rajat, Jonghyun Lee +2
As cloud-based ML expands, ensuring data security during training and inference is critical. GPU-based Trusted Execution Environments (TEEs) offer secure, high-performance solution…
MobiZO: Enabling Efficient LLM Fine-Tuning at the Edge via Inference Engines
Lei Gao, Amir Ziashahabi, Yue Niu +2
Large Language Models (LLMs) are currently pre-trained and fine-tuned on large cloud servers. The next frontier is LLM personalization, where a foundation model can be fine-tuned w…
CADC: Encoding User-Item Interactions for Compressing Recommendation Model Training Data
Hossein Entezari Zarch, Abdulla Alshabanah, Chaoyi Jiang +1
Deep learning recommendation models (DLRMs) are at the heart of the current e-commerce industry. However, the amount of training data used to train these large models is growing ex…
Ethos: Rectifying Language Models in Orthogonal Parameter Space
Lei Gao, Yue Niu, Tingting Tang +2
Language models (LMs) have greatly propelled the research on natural language processing. However, LMs also raise concerns regarding the generation of biased or toxic content and t…
Edge Private Graph Neural Networks with Singular Value Perturbation
Tingting Tang, Yue Niu, Salman Avestimehr +1
Graph neural networks (GNNs) play a key role in learning representations from graph-structured data and are demonstrated to be useful in many applications. However, the GNN trainin…
CompactTag: Minimizing Computation Overheads in Actively-Secure MPC for Deep Neural Networks
Yongqin Wang, Pratik Sarkar, Nishat Koti +2
Secure Multiparty Computation (MPC) protocols enable secure evaluation of a circuit by several parties, even in the presence of an adversary who maliciously corrupts all but one of…
MPC-Pipe: an Efficient Pipeline Scheme for Secure Multi-party Machine Learning Inference
Yongqin Wang, Rachit Rajat, Murali Annavaram
Multi-party computing (MPC) has been gaining popularity as a secure computing model over the past few years. However, prior works have demonstrated that MPC protocols still pay sub…
High-Throughput Secure Multiparty Computation with an Honest Majority in Various Network Settings
Christopher Harth-Kitzerow, Ajith Suresh, Yongqin Wang +3
In this work, we present novel protocols over rings for semi-honest secure three-party computation (3PC) and malicious four-party computation (4PC) with one corruption. While most…