8 papers · 1 filter
Reasoning Language Model Inference Serving Unveiled: An Empirical Study
Qi Li, Junpan Wu, Xiang Liu +6
The reasoning large language model (RLLM) has been proven competitive in solving complex reasoning tasks such as mathematics, coding, compared to general LLM. However, the serving…
VeritasFi: An Adaptable, Multi-tiered RAG Framework for Multi-modal Financial Question Answering
Zhenghan Tai, Hanwei Wu, Qingchen Hu +24
Retrieval-Augmented Generation (RAG) is becoming increasingly essential for Question Answering (QA) in the financial sector, where accurate and contextually grounded insights from…
AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models
Zeyu Li, Chuanfu Xiao, Yang Wang +6
Quantization has emerged as an effective and lightweight solution to reduce the memory footprint of the KV cache in Large Language Models. Nevertheless, minimizing the accuracy deg…
Causes and Consequences of Representational Similarity in Machine Learning Models
Zeyu Michael Li, Hung Anh Vu, Damilola Awofisayo +1
Numerous works have noted similarities in how machine learning models represent the world, even across modalities. Although much effort has been devoted to uncovering properties an…
Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation
Zhuo-Yang Song, Zeyu Li, Qing-Hong Cao +2
The geometric evolution of token representations in large language models (LLMs) presents a fundamental paradox: while human language inherently organizes semantic information in l…
Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models
Teng Wang, Zhangyi Jiang, Zhenqi He +9
Recent studies show that Large Language Models (LLMs) achieve strong reasoning capabilities through supervised fine-tuning or reinforcement learning. However, a key approach, the P…
Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression
Xiang Liu, Zhenheng Tang, Hong Chen +6
While Key-Value (KV) cache compression is essential for efficient LLM inference, current evaluations disproportionately focus on sparse retrieval tasks, potentially masking the deg…
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
Xiang Liu, Zhenheng Tang, Peijie Dong +5
Large Language Models (LLMs) require significant GPU memory when processing long texts, with the key value (KV) cache consuming up to 70\% of total memory during inference. Althoug…