5 papers
SeDeM: Selective Decompression of Hidden-State Memories for Long-Context Question Answering
Maryam Haghifam, Jason Cong, Yizhou Sun
Long-context inference with large language models (LLMs) is costly: self-attention during prefill scales quadratically with sequence length, and the key-value (KV) cache grows with…
HMT: Semantic Hierarchy-Aware Hierarchical Memory Transformer
Maryam Haghifam, Zifan He, Jason Cong +1
Transformer-based LLMs achieve strong results on many language tasks; however, long inputs remain challenging because context windows are finite, and prefill latency and memory gro…
Structure-Aware RAG: Structured Retrieval Augmented Generation from Noisy Data for Conversational Agents
Kaiqiao Han, LuAn Tang, Renliang Sun +6
Large Language Models (LLMs) have been widely adopted in conversational applications. However, their reliance on parametric knowledge limits reliability in real-world scenarios tha…
Compressing Sequences in the Latent Embedding Space: -Token Merging for Large Language Models
Zihao Xu, John Harvill, Ziwei Fan +3
Large Language Models (LLMs) incur significant computational and memory costs when processing long prompts, as full self-attention scales quadratically with input length. Token com…
ConFu: Contemplate the Future for Better Speculative Sampling
Zongyue Qin, Raghavv Goel, Mukul Gagrani +3
Speculative decoding has emerged as a powerful approach to accelerate large language model (LLM) inference by employing lightweight draft models to propose candidate tokens that ar…