From the 1 of 7 linked papers with an AI index.
4 papers · 1 filter
HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models
Hei Yi Mak, Shadan Golestan, Hoang Le +10
The paper introduces HiFloat4, a 4-bit floating-point format and a Rollout Residual Quantization technique that enable end-to-end reinforcement learning post‑training of large lang…
Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets
Duc Duong, Hoang Anh Duy Le, Jianwen Xie +2
Long-context and agentic LLM workloads push the KV cache past any fixed memory budget, forcing the inference stack to permanently evict tokens at every step of a continuous-inferen…
SOCKET: SOft Collision Kernel EsTimator for Sparse Attention
Sahil Joshi, Agniva Chowdhury, Wyatt Bellinger +5
Exploiting sparsity during long-context inference is key to scaling large language models, as attention dominates the cost of autoregressive decoding. Sparse attention reduces this…
Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inference
Hoang Anh Duy Le, Sahil Joshi, Zeyu Yang +2
Self-attention dominates the computational and memory cost of long-context LLM inference across both prefill and decode phases. To address this challenge, we introduce Sketch&Walk…