1 citations · 2 across the 5 of their papers we have counts for
6 papers
LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation
Jinwoo Ahn, Ingyu Seong, Akhil Kedia +4
Transformer-based large language models (LLMs) rely on key-value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves…
Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs
Anshumann, Mohd Abbas Zaidi, Akhil Kedia +5
Knowledge distillation can be a cost-effective technique to distill knowledge in Large Language Models, if the teacher output logits can be pre-computed and cached. However, succes…
UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages
Bethel Melesse Tessema, Akhil Kedia, Tae-Sun Chung
Large language models (LLMs) under-perform on low-resource languages due to limited training data. We present a method to efficiently collect text data for low-resource languages f…
Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models
Akhil Kedia, Mohd Abbas Zaidi, Sushil Khyalia +3
In spite of their huge success, transformer models remain difficult to scale in depth. In this work, we develop a unified signal propagation theory and provide formulae that govern…
FiE: Building a Global Probability Space by Leveraging Early Fusion in Encoder for Open-Domain Question Answering
Akhil Kedia, Mohd Abbas Zaidi, Haejun Lee
Generative models have recently started to outperform extractive models in Open Domain Question Answering, largely by leveraging their decoder to attend over multiple encoded passa…
You Only Need One Model for Open-domain Question Answering
Haejun Lee, Akhil Kedia, Jongwon Lee +3
Recent approaches to Open-domain Question Answering refer to an external knowledge base using a retriever model, optionally rerank passages with a separate reranker model and gener…