Learning Slice-Aware Representations with Mixture of Attentions

Abstract

Real-world machine learning systems are achieving remarkable performance in terms of coarse-grained metrics like overall accuracy and F-1 score. However, model improvement and development often require fine-grained modeling on individual data subsets or slices, for instance, the data slices where the models have unsatisfactory results. In practice, it gives tangible values for developing such models that can pay extra attention to critical or interested slices while retaining the original overall performance. This work extends the recent slice-based learning (SBL)~\cite{chen2019slice} with a mixture of attentions (MoA) to learn slice-aware dual attentive representations. We empirically show that the MoA approach outperforms the baseline method as well as the original SBL approach on monitored slices with two natural language understanding (NLU) tasks.

arXiv commentFindings of the ACL: ACL-IJCNLP 2021

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Neural model robustness for skill routing in large-scale conversational AI systems: A design choice exploration
• Handling Long-Tail Queries with Slice-Aware Conversational Systems
• A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding

Cited by in corpus (1)

• A scalable framework for learning from implicit user feedback to improve natural language understanding in large-scale conversational AI systems