11 citations · 35 across the 21 of their papers we have counts for
27 papers
SGD-KV: Summarization Guided KV Cache Compression
Zeyu Liu, Woomin Song, Xuandi Fu +5
Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techn…
RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
Yufeng Du, Phillip Harris, Minyang Tian +5
We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis abstracts away from the specifi…
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
Yufeng Du, Minyang Tian, Srikanth Ronanki +7
Large language models (LLMs) often fail to scale their performance on long-context tasks performance in line with the context lengths they support. This gap is commonly attributed…
LAWCAT: Efficient Distillation from Quadratic to Linear Attention with Convolution across Tokens for Long Context Modeling
Zeyu Liu, Souvik Kundu, Lianghao Jiang +5
Although transformer architectures have achieved state-of-the-art performance across diverse domains, their quadratic computational complexity with respect to sequence length remai…
Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
Goeric Huybrechts, Srikanth Ronanki, Sai Muralidhar Jayanthi +2
The proliferation of multimodal Large Language Models has significantly advanced the ability to analyze and understand complex data inputs from different modalities. However, the p…
Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers
Woomin Song, Sai Muralidhar Jayanthi, Srikanth Ronanki +5
As large language models increasingly gain popularity in real-world applications, processing extremely long contexts, often exceeding the model's pre-trained context limits, has em…