75 citations · 311 across the 28 of their papers we have counts for
19 papers · 1 filter
Does RAG Really Perform Bad For Long-Context Processing?
Kun Luo, Zheng Liu, Peitian Zhang +3
The efficient processing of long context poses a serious challenge for large language models (LLMs). Recently, retrieval-augmented generation (RAG) has emerged as a promising strat…
MMTEB: Massive Multilingual Text Embedding Benchmark
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua +83
Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more co…
MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation
Hongjin Qian, Zheng Liu, Peitian Zhang +4
Processing long contexts presents a significant challenge for large language models (LLMs). While recent advancements allow LLMs to handle much longer contexts than before (e.g., 3…
Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment
Kun Luo, Minghao Qin, Zheng Liu +3
Pretrained language models like BERT and T5 serve as crucial backbone encoders for dense retrieval. However, these models often exhibit limited generalization capabilities and face…
Are Long-LLMs A Necessity For Long-Context Tasks?
Hongjin Qian, Zheng Liu, Peitian Zhang +4
The learning and deployment of long-LLMs remains a challenging problem despite recent progresses. In this work, we argue that the long-LLMs are not a necessity to solve long-contex…
Compressing Lengthy Context With UltraGist
Peitian Zhang, Zheng Liu, Shitao Xiao +3
Compressing lengthy context is a critical but technically challenging problem. In this paper, we propose a new method called UltraGist, which is distinguished for its high-quality…
Extending Llama-3's Context Ten-Fold Overnight
Peitian Zhang, Ninglu Shao, Zheng Liu +4
We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G) GPU…
Understanding Privacy Risks of Embeddings Induced by Large Language Models
Zhihao Zhu, Ninglu Shao, Defu Lian +4
Large language models (LLMs) show early signs of artificial general intelligence but struggle with hallucinations. One promising solution to mitigate these hallucinations is to sto…
Extensible Embedding: A Flexible Multipler For LLM's Context Length
Ninglu Shao, Shitao Xiao, Zheng Liu +1
Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality…
BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models
Kun Luo, Zheng Liu, Shitao Xiao +1
Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality…
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
Jianlv Chen, Shitao Xiao, Peitian Zhang +3
In this paper, we introduce a new embedding model called M3-Embedding, which is distinguished for its versatility in \textit{Multi-Linguality}, \textit{Multi-Functionality}, and \t…
Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
Ninglu Shao, Shitao Xiao, Zheng Liu +1
Large language models (LLMs) are in need of sufficient contexts to handle many critical applications, such as retrieval augmented generation and few-shot learning. However, due to…
INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning
Yutao Zhu, Peitian Zhang, Chenghao Zhang +5
Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Despite this, their application to information retrieval (IR) t…
Long Context Compression with Activation Beacon
Peitian Zhang, Zheng Liu, Shitao Xiao +3
Long context compression is a critical research problem due to its significance in reducing the high computational and memory costs associated with LLMs. In this paper, we propose…
LM-Cocktail: Resilient Tuning of Language Models via Model Merging
Shitao Xiao, Zheng Liu, Peitian Zhang +1
The pre-trained language models are continually fine-tuned to better support downstream applications. However, this operation may result in significant performance degeneration on…
Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval
Zheng Liu, Chaofan Li, Shitao Xiao +2
Dense retrieval calls for discriminative embeddings to represent the semantic relationship between query and document. It may benefit from the using of large language models (LLMs)…
C-Pack: Packed Resources For General Chinese Embeddings
Shitao Xiao, Zheng Liu, Peitian Zhang +3
We introduce C-Pack, a package of resources that significantly advance the field of general Chinese embeddings. C-Pack includes three critical resources. 1) C-MTEB is a comprehensi…
RetroMAE v2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models
Shitao Xiao, Zheng Liu
To better support retrieval applications such as web search and question answering, growing effort is made to develop retrieval-oriented language models. Most of the existing works…
GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph
Junhan Yang, Zheng Liu, Shitao Xiao +6
The representation learning on textual graph is to generate low-dimensional embeddings for the nodes based on the individual textual features and the neighbourhood information. Rec…