Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems

Abstract

Task-oriented dialogue systems use four connected modules, namely, Natural Language Understanding (NLU), a Dialogue State Tracking (DST), Dialogue Policy (DP) and Natural Language Generation (NLG). A research challenge is to learn each module with the least amount of samples (i.e., few-shots) given the high cost related to the data collection. The most common and effective technique to solve this problem is transfer learning, where large language models, either pre-trained on text or task-specific data, are fine-tuned on the few samples. These methods require fine-tuning steps and a set of parameters for each task. Differently, language models, such as GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), allow few-shot learning by priming the model with few examples. In this paper, we evaluate the priming few-shot ability of language models in the NLU, DST, DP and NLG tasks. Importantly, we highlight the current limitations of this approach, and we discuss the possible implication for future work.

Blog (https://andreamad8.github.io/few-shot-gpt/), Medium (https://medium.com/@madottoandrea/language-model-as-few-shot-learner-for-task-oriented-dialogue-systems-db4765796744) and Code (https://github.com/andreamad8/TASK-ORIENTED-LM-FEWSHOT)

References in corpus (13)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
• Big Bird: Transformers for Longer Sequences
• SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching
• Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
• Few-shot Natural Language Generation for Task-Oriented Dialog
• MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines
• Domain Adaptive Dialog Generation via Meta Learning
• Coach: A Coarse-to-Fine Approach for Cross-domain Slot Filling
• Meta Dialogue Policy Learning
• Template Guided Text Generation for Task-Oriented Dialogue

Cited by in corpus (6)

• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems
• Synthesizing Speech from Intracranial Depth Electrodes using an Encoder-Decoder Framework
• Show, Don't Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue
• Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
• Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Tracking
• N-Shot Learning for Augmenting Task-Oriented Dialogue State Tracking