SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation

Abstract

Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data. The task obtained strong participation from 31 teams, with 17 participating in all language tracks. We summarize performance and review a selection of well performing methods. Analysis highlights common errors, providing insight into the limitations of existing models. To support ongoing work on semantic representations, the STS Benchmark is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012-2017).

arXiv commentTo appear in proceedings of the SemEval workshop at ACL 2017; 14 pages, 14 Tables, 1 Figure

Cited by in corpus (28)

• Evolution of Semantic Similarity -- A Survey
• Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
• Matching Patients to Clinical Trials with Large Language Models
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search
• SemMT: A Semantic-based Testing Approach for Machine Translation Systems
• Universal Multimodal Representation for Language Understanding
• From Free Text to Clusters of Content in Health Records: An Unsupervised Graph Partitioning Approach
• Extracting Sentence Embeddings from Pretrained Transformer Models
• Exploiting Transformer-based Multitask Learning for the Detection of Media Bias in News Articles
• Resources for Turkish Natural Language Processing: A critical survey
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• NewsEmbed: Modeling News through Pre-trained Document Representations
• Learning semantic sentence representations from visually grounded language without lexical knowledge
• Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives
• Aspect-Based Sentiment Analysis Techniques: A Comparative Study
• Infusing Finetuning with Semantic Dependencies
• Improving BERT with Self-Supervised Attention
• Multistage BiCross encoder for multilingual access to COVID-19 health information
• Predicting drug-gene relations via analogy tasks with word embeddings
• A Survey on Awesome Korean NLP Datasets
• Can persistent homology whiten Transformer-based black-box models? A case study on BERT compression
• COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Search
• Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations
• Semantic Search as Extractive Paraphrase Span Detection
• PESTS: Persian_English Cross Lingual Corpus for Semantic Textual Similarity
• A reproducible experimental survey on biomedical sentence similarity: a string-based method sets the state of the art
• Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer
• COPER: a Query-adaptable Semantics-based Search Engine for Persian COVID-19 Articles