Continual Learning for Generative Retrieval over Dynamic Corpora
arXiv:2308.14968 · doi:10.1145/3583780.3614821
Abstract
Generative retrieval (GR) directly predicts the identifiers of relevant documents (i.e., docids) based on a parametric model. It has achieved solid performance on many ad-hoc retrieval tasks. So far, these tasks have assumed a static document collection. In many practical scenarios, however, document collections are dynamic, where new documents are continuously added to the corpus. The ability to incrementally index new documents while preserving the ability to answer queries with both previously and newly indexed relevant documents is vital to applying GR models. In this paper, we address this practical continual learning problem for GR. We put forward a novel Continual-LEarner for generatiVE Retrieval (CLEVER) model and make two major contributions to continual learning for GR: (i) To encode new documents into docids with low computational cost, we present Incremental Product Quantization, which updates a partial quantization codebook according to two adaptive thresholds; and (ii) To memorize new documents for querying without forgetting previous knowledge, we propose a memory-augmented learning mechanism, to form meaningful connections between old and new documents. Empirical results demonstrate the effectiveness and efficiency of the proposed model.
Accepted by CIKM 2023
References in corpus (23)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- Sequence to Sequence Learning with Neural Networks
- Overcoming catastrophic forgetting in neural networks
- A Simple Framework for Contrastive Learning of Visual Representations
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- A continual learning survey: Defying forgetting in classification tasks
- A Deep Relevance Matching Model for Ad-hoc Retrieval
- On Tiny Episodic Memories in Continual Learning
- Document Expansion by Query Prediction
- Autoregressive Entity Retrieval
- Rethinking Search: Making Domain Experts out of Dilettantes
- Transformer Memory as a Differentiable Search Index
- Autoregressive Search Engines: Generating Substrings as Document Identifiers
- GERE: Generative Evidence Retrieval for Fact Verification
- CorpusBrain: Pre-train a Generative Retrieval Model for Knowledge-Intensive Language Tasks
- A Neural Corpus Indexer for Document Retrieval
- Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
- A Unified Generative Retriever for Knowledge-Intensive Language Tasks via Prompt Learning
- Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation
- B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval
- Ultron: An Ultimate Retriever on Corpus with a Model-based Indexer
- Model Zoo: A Growing "Brain" That Learns Continually