Fine-tuning Strategies for Domain Specific Question Answering under Low Annotation Budget Constraints
arXiv:2401.09168 · doi:10.1109/ICTAI59109.2023.00032
Abstract
The progress introduced by pre-trained language models and their fine-tuning has resulted in significant improvements in most downstream NLP tasks. The unsupervised training of a language model combined with further target task fine-tuning has become the standard QA fine-tuning procedure. In this work, we demonstrate that this strategy is sub-optimal for fine-tuning QA models, especially under a low QA annotation budget, which is a usual setting in practice due to the extractive QA labeling cost. We draw our conclusions by conducting an exhaustive analysis of the performance of the alternatives of the sequential fine-tuning strategy on different QA datasets. Based on the experiments performed, we observed that the best strategy to fine-tune the QA model in low-budget settings is taking a pre-trained language model (PLM) and then fine-tuning PLM with a dataset composed of the target dataset and SQuAD dataset. With zero extra annotation effort, the best strategy outperforms the standard strategy by 2.28% to 6.48%. Our experiments provide one of the first investigations on how to best fine-tune a QA system under a low budget and are therefore of the utmost practical interest to the QA practitioners.
References in corpus (11)
- Decoupled Weight Decay Regularization
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- LLaMA: Open and Efficient Foundation Language Models
- LoRA: Low-Rank Adaptation of Large Language Models
- On the Opportunities and Risks of Foundation Models
- ChatGPT: Jack of all trades, master of none
- The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
- Leveraging BERT for Extractive Text Summarization on Lectures
- CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
- Putting Question-Answering Systems into Practice: Transfer Learning for Efficient Domain Customization
- Towards Domain Adaptation from Limited Data for Question Answering Using Deep Neural Networks