Dual-View Distilled BERT for Sentence Embedding
arXiv:2104.08675
Abstract
Recently, BERT realized significant progress for sentence matching via word-level cross sentence attention. However, the performance significantly drops when using siamese BERT-networks to derive two sentence embeddings, which fall short in capturing the global semantic since the word-level attention between two sentences is absent. In this paper, we propose a Dual-view distilled BERT~(DvBERT) for sentence matching with sentence embeddings. Our method deals with a sentence pair from two distinct views, i.e., Siamese View and Interaction View. Siamese View is the backbone where we generate sentence embeddings. Interaction View integrates the cross sentence interaction as multiple teachers to boost the representation ability of sentence embeddings. Experiments on six STS tasks show that our method outperforms the state-of-the-art sentence embedding methods significantly.
arXiv commentAccepted at SIGIR 2021
References in corpus (6)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- A Survey on Multi-view Learning
- Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
- Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning
- Making Neural Machine Reading Comprehension Faster