75 citations · 217 across the 23 of their papers we have counts for
23 papers
Task-Aware KV Compression For Cost-Effective Long Video Understanding
Minghao Qin, Yan Shu, Peitian Zhang +6
Long-video understanding (LVU) remains a severe challenge for existing multimodal large language models (MLLMs), primarily due to the prohibitive computational cost. Recent approac…
Does RAG Really Perform Bad For Long-Context Processing?
Kun Luo, Zheng Liu, Peitian Zhang +3
The efficient processing of long context poses a serious challenge for large language models (LLMs). Recently, retrieval-augmented generation (RAG) has emerged as a promising strat…
Search-o1: Agentic Search-Enhanced Large Reasoning Models
Xiaoxi Li, Guanting Dong, Jiajie Jin +5
Large reasoning models (LRMs) like OpenAI-o1 have demonstrated impressive long stepwise reasoning capabilities through large-scale reinforcement learning. However, their extended r…
Boosting Long-Context Management via Query-Guided Activation Refilling
Hongjin Qian, Zheng Liu, Peitian Zhang +2
Processing long contexts poses a significant challenge for large language models (LLMs) due to their inherent context-window limitations and the computational burden of extensive k…
Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding
Yan Shu, Zheng Liu, Peitian Zhang +5
Long video understanding poses a significant challenge for current Multi-modal Large Language Models (MLLMs). Notably, the MLLMs are constrained by their limited context lengths an…
MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation
Hongjin Qian, Zheng Liu, Peitian Zhang +4
Processing long contexts presents a significant challenge for large language models (LLMs). While recent advancements allow LLMs to handle much longer contexts than before (e.g., 3…
Are Long-LLMs A Necessity For Long-Context Tasks?
Hongjin Qian, Zheng Liu, Peitian Zhang +4
The learning and deployment of long-LLMs remains a challenging problem despite recent progresses. In this work, we argue that the long-LLMs are not a necessity to solve long-contex…
Compressing Lengthy Context With UltraGist
Peitian Zhang, Zheng Liu, Shitao Xiao +3
Compressing lengthy context is a critical but technically challenging problem. In this paper, we propose a new method called UltraGist, which is distinguished for its high-quality…
Extending Llama-3's Context Ten-Fold Overnight
Peitian Zhang, Ninglu Shao, Zheng Liu +4
We extend the context length of Llama-3-8B-Instruct from 8K to 80K via QLoRA fine-tuning. The entire training cycle is super efficient, which takes 8 hours on one 8xA800 (80G) GPU…
From Matching to Generation: A Survey on Generative Information Retrieval
Xiaoxi Li, Jiajie Jin, Yujia Zhou +4
Information Retrieval (IR) systems are crucial tools for users to access information, which have long been dominated by traditional methods relying on similarity matching. With the…
Extensible Embedding: A Flexible Multipler For LLM's Context Length
Ninglu Shao, Shitao Xiao, Zheng Liu +1
Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality…
M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
Jianlv Chen, Shitao Xiao, Peitian Zhang +3
In this paper, we introduce a new embedding model called M3-Embedding, which is distinguished for its versatility in \textit{Multi-Linguality}, \textit{Multi-Functionality}, and \t…
Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
Ninglu Shao, Shitao Xiao, Zheng Liu +1
Large language models (LLMs) are in need of sufficient contexts to handle many critical applications, such as retrieval augmented generation and few-shot learning. However, due to…
INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning
Yutao Zhu, Peitian Zhang, Chenghao Zhang +5
Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Despite this, their application to information retrieval (IR) t…
Long Context Compression with Activation Beacon
Peitian Zhang, Zheng Liu, Shitao Xiao +3
Long context compression is a critical research problem due to its significance in reducing the high computational and memory costs associated with LLMs. In this paper, we propose…
LM-Cocktail: Resilient Tuning of Language Models via Model Merging
Shitao Xiao, Zheng Liu, Peitian Zhang +1
The pre-trained language models are continually fine-tuned to better support downstream applications. However, this operation may result in significant performance degeneration on…
A Multi-Task Embedder For Retrieval Augmented LLMs
Peitian Zhang, Shitao Xiao, Zheng Liu +2
LLMs confront inherent limitations in terms of its knowledge, memory, and action. The retrieval augmentation stands as a vital mechanism to address these limitations, which brings…
C-Pack: Packed Resources For General Chinese Embeddings
Shitao Xiao, Zheng Liu, Peitian Zhang +3
We introduce C-Pack, a package of resources that significantly advance the field of general Chinese embeddings. C-Pack includes three critical resources. 1) C-MTEB is a comprehensi…
Generative Retrieval via Term Set Generation
Peitian Zhang, Zheng Liu, Yujia Zhou +3
Recently, generative retrieval emerges as a promising alternative to traditional retrieval paradigms. It assigns each document a unique identifier, known as DocID, and employs a ge…
Hybrid Inverted Index Is a Robust Accelerator for Dense Retrieval
Peitian Zhang, Zheng Liu, Shitao Xiao +2
Inverted file structure is a common technique for accelerating dense retrieval. It clusters documents based on their embeddings; during searching, it probes nearby clusters w.r.t.…
Ultron: An Ultimate Retriever on Corpus with a Model-based Indexer
Yujia Zhou, Jing Yao, Zhicheng Dou +3
Document retrieval has been extensively studied within the index-retrieve framework for decades, which has withstood the test of time. Unfortunately, such a pipelined framework lim…
GateFormer: Speeding Up News Feed Recommendation with Input Gated Transformers
Peitian Zhang, Zheng liu
News feed recommendation is an important web service. In recent years, pre-trained language models (PLMs) have been intensively applied to improve the recommendation quality. Howev…
Learning to Select Historical News Articles for Interaction based Neural News Recommendation
Peitian Zhang, Zhicheng Dou, Jing Yao
The key to personalized news recommendation is to match the user's interests with the candidate news precisely and efficiently. Most existing approaches embed user interests into a…