34 citations · 38 across the 7 of their papers we have counts for
11 papers
Word Order and World Knowledge
Qinghua Zhao, Vinit Ravishankar, Nicolas Garneau +1
Word order is an important concept in natural language, and in this work, we study how word order affects the induction of world knowledge from raw text using language models. We u…
Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings
Isabelle Mohr, Markus Krimmel, Saba Sturua +16
We introduce a novel suite of state-of-the-art bilingual text embedding models that are designed to support English and another target language. These models are capable of process…
Word Order Does Matter (And Shuffled Language Models Know It)
Vinit Ravishankar, Mostafa Abdou, Artur Kulmizev +1
Recent studies have shown that language models pretrained and/or fine-tuned on randomly permuted sentences exhibit competitive performance on GLUE, putting into question the import…
The Impact of Positional Encodings on Multilingual Compression
Vinit Ravishankar, Anders Søgaard
In order to preserve word-order information in a non-autoregressive setting, transformer architectures tend to include positional knowledge, by (for instance) adding positional enc…
Attention Can Reflect Syntactic Structure (If You Let It)
Vinit Ravishankar, Artur Kulmizev, Mostafa Abdou +2
Since the popularization of the Transformer as a general-purpose feature encoder for NLP, many studies have attempted to decode linguistic structure from its novel multi-head atten…
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations
Mostafa Abdou, Vinit Ravishankar, Maria Barrett +3
Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge, a widely employed test of common sens…
From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers
Anne Lauscher, Vinit Ravishankar, Ivan Vulić +1
Massively multilingual transformers pretrained with language modeling objectives (e.g., mBERT, XLM-R) have become a de facto default transfer paradigm for zero-shot cross-lingual t…
Do Neural Language Models Show Preferences for Syntactic Formalisms?
Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou +1
Recent work on the interpretability of deep neural language models has concluded that many properties of natural language syntax are encoded in their representational spaces. Howev…
A Systematic Comparison of Architectures for Document-Level Sentiment Classification
Jeremy Barnes, Vinit Ravishankar, Lilja Øvrelid +1
Documents are composed of smaller pieces - paragraphs, sentences, and tokens - that have complex relationships between one another. Sentiment classification models that take into a…
Probing Multilingual Sentence Representations With X-Probe
Vinit Ravishankar, Lilja Øvrelid, Erik Velldal
This paper extends the task of probing sentence representations for linguistic insight in a multilingual domain. In doing so, we make two contributions: first, we provide datasets…
What can we learn from Semantic Tagging?
Mostafa Abdou, Artur Kulmizev, Vinit Ravishankar +2
We investigate the effects of multi-task learning using the recently introduced task of semantic tagging. We employ semantic tagging as an auxiliary task for three different NLP ta…