75 citations · 217 across the 23 of their papers we have counts for
13 papers · 1 filter
Does RAG Really Perform Bad For Long-Context Processing?
Kun Luo, Zheng Liu, Peitian Zhang +3
The efficient processing of long context poses a serious challenge for large language models (LLMs). Recently, retrieval-augmented generation (RAG) has emerged as a promising strat…
Boosting Long-Context Management via Query-Guided Activation Refilling
Hongjin Qian, Zheng Liu, Peitian Zhang +2
Processing long contexts poses a significant challenge for large language models (LLMs) due to their inherent context-window limitations and the computational burden of extensive k…
MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation
Hongjin Qian, Zheng Liu, Peitian Zhang +4
Processing long contexts presents a significant challenge for large language models (LLMs). While recent advancements allow LLMs to handle much longer contexts than before (e.g., 3…
Are Long-LLMs A Necessity For Long-Context Tasks?
Hongjin Qian, Zheng Liu, Peitian Zhang +4
The learning and deployment of long-LLMs remains a challenging problem despite recent progresses. In this work, we argue that the long-LLMs are not a necessity to solve long-contex…
Compressing Lengthy Context With UltraGist
Peitian Zhang, Zheng Liu, Shitao Xiao +3
Compressing lengthy context is a critical but technically challenging problem. In this paper, we propose a new method called UltraGist, which is distinguished for its high-quality…
Extending Llama-3's Context Ten-Fold Overnight
Peitian Zhang, Ninglu Shao, Zheng Liu +4
We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G) GPU…
Extensible Embedding: A Flexible Multipler For LLM's Context Length
Ninglu Shao, Shitao Xiao, Zheng Liu +1
Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality…
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
Jianlv Chen, Shitao Xiao, Peitian Zhang +3
In this paper, we introduce a new embedding model called M3-Embedding, which is distinguished for its versatility in \textit{Multi-Linguality}, \textit{Multi-Functionality}, and \t…
Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
Ninglu Shao, Shitao Xiao, Zheng Liu +1
Large language models (LLMs) are in need of sufficient contexts to handle many critical applications, such as retrieval augmented generation and few-shot learning. However, due to…
INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning
Yutao Zhu, Peitian Zhang, Chenghao Zhang +5
Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Despite this, their application to information retrieval (IR) t…
Long Context Compression with Activation Beacon
Peitian Zhang, Zheng Liu, Shitao Xiao +3
Long context compression is a critical research problem due to its significance in reducing the high computational and memory costs associated with LLMs. In this paper, we propose…
LM-Cocktail: Resilient Tuning of Language Models via Model Merging
Shitao Xiao, Zheng Liu, Peitian Zhang +1
The pre-trained language models are continually fine-tuned to better support downstream applications. However, this operation may result in significant performance degeneration on…
C-Pack: Packed Resources For General Chinese Embeddings
Shitao Xiao, Zheng Liu, Peitian Zhang +3
We introduce C-Pack, a package of resources that significantly advance the field of general Chinese embeddings. C-Pack includes three critical resources. 1) C-MTEB is a comprehensi…