A Financial Service Chatbot based on Deep Bidirectional Transformers

Abstract

We develop a chatbot using Deep Bidirectional Transformer models (BERT) to handle client questions in financial investment customer service. The bot can recognize 381 intents, and decides when to say "I don't know" and escalates irrelevant/uncertain questions to human operators. Our main novel contribution is the discussion about uncertainty measure for BERT, where three different approaches are systematically compared on real problems. We investigated two uncertainty metrics, information entropy and variance of dropout sampling in BERT, followed by mixed-integer programming to optimize decision thresholds. Another novel contribution is the usage of BERT as a language model in automatic spelling correction. Inputs with accidental spelling errors can significantly decrease intent classification performance. The proposed approach combines probabilities from masked language model and word edit distances to find the best corrections for misspelled words. The chatbot and the entire conversational AI system are developed using open-source tools, and deployed within our company's intranet. The proposed approach can be useful for industries seeking similar in-house solutions in their specific business domains. We share all our code and a sample chatbot built on a public dataset on Github.

References in corpus (18)

• Natural Language Processing (almost) from Scratch
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• On Calibration of Modern Neural Networks
• Cross-lingual Language Model Pretraining
• Weight Uncertainty in Neural Networks
• Character-Aware Neural Language Models
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Semi-supervised Sequence Learning
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Towards a Human-like Open-Domain Chatbot
• A Deep Reinforcement Learning Chatbot
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Rasa: Open Source Language Understanding and Dialogue Management
• Avoiding Echo-Responses in a Retrieval-Based Conversation System
• Towards calibrated and scalable uncertainty representations for neural networks
• Sampling-based Bayesian Inference with gradient uncertainty

Cited by in corpus (3)

• Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluations
• Exploring Fluent Query Reformulations with Text-to-Text Transformers and Reinforcement Learning
• Fingerprinting Fine-tuned Language Models in the Wild