895 citations · 2.1k across the 169 of their papers we have counts for
24 papers · 1 filter
A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters
Mengjie Zhao, Yi Zhu, Ehsan Shareghi +4
Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT. Despite its growing popularity, little to no…
Few-Shot Text Generation with Pattern-Exploiting Training
Timo Schick, Hinrich Schütze
Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion. Moreover, when combined wit…
Subword Sampling for Low Resource Word Alignment
Ehsaneddin Asgari, Masoud Jalili Sabet, Philipp Dufter +2
Annotation projection is an important area in NLP that can greatly contribute to creating language resources for low-resource languages. Word alignment plays a key role in this set…
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification
Timo Schick, Helmut Schmid, Hinrich Schütze
A recent approach for few-shot text classification is to convert textual inputs to cloze questions that contain some form of task description, process them with a pretrained langua…
TopicBERT for Energy Efficient Document Classification
Yatin Chaudhary, Pankaj Gupta, Khushbu Saxena +3
Prior research notes that BERT's computational cost grows quadratically with sequence length thus leading to longer training times, higher GPU memory constraints and carbon emissio…
Dynamic Contextualized Word Embeddings
Valentin Hofmann, Janet B. Pierrehumbert, Hinrich Schütze
Static word embeddings that represent words by a single vector cannot capture the variability of word meaning in different linguistic and extralinguistic contexts. Building on prio…
Identifying Automatically Generated Headlines using Transformers
Antonis Maronikolakis, Hinrich Schutze, Mark Stevenson
False information spread via the internet and social media influences public opinion and user activity, while generative models enable fake content to be generated faster and more…
It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Timo Schick, Hinrich Schütze
When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts…
Investigating Pretrained Language Models for Graph-to-Text Generation
Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze +1
Graph-to-text generation aims to generate fluent texts from graph-based data. In this paper, we investigate two recently proposed pretrained language models (PLMs) and analyze the…
Automatic Domain Adaptation Outperforms Manual Domain Adaptation for Predicting Financial Outcomes
Marina Sedinkina, Nikolas Breitkopf, Hinrich Schütze
In this paper, we automatically create sentiment dictionaries for predicting financial outcomes. We compare three approaches: (I) manual adaptation of the domain-general dictionary…
Neural Topic Modeling with Continual Lifelong Learning
Pankaj Gupta, Yatin Chaudhary, Thomas Runkler +1
Lifelong learning has recently attracted attention in building machine learning systems that continually accumulate and transfer knowledge to help future learning. Unsupervised top…
Explainable and Discourse Topic-aware Neural Language Understanding
Yatin Chaudhary, Hinrich Schütze, Pankaj Gupta
Marrying topic models and language models exposes language understanding to a broader source of document-level context beyond sentences via topics. While introducing topical semant…
Are Pretrained Language Models Symbolic Reasoners Over Knowledge?
Nora Kassner, Benno Krojer, Hinrich Schütze
How can pretrained language models (PLMs) learn factual knowledge from the training set? We investigate the two most important mechanisms: reasoning and memorization. Prior work ha…
Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphs
Martin Schmitt, Leonardo F. R. Ribeiro, Philipp Dufter +2
We present Graformer, a novel Transformer-based encoder-decoder architecture for graph-to-text generation. With our novel graph self-attention, the encoding of a node relies on all…
Unsupervised Embedding-based Detection of Lexical Semantic Changes
Ehsaneddin Asgari, Christoph Ringlstetter, Hinrich Schütze
This paper describes EmbLexChange, a system introduced by the "Life-Language" team for SemEval-2020 Task 1, on unsupervised detection of lexical-semantic changes. EmbLexChange is d…
BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA
Nora Kassner, Hinrich Schütze
Khandelwal et al. (2020) use a k-nearest-neighbor (kNN) component to improve language model performance. We show that this idea is beneficial for open-domain question answering (QA…
DagoBERT: Generating Derivational Morphology with a Pretrained Language Model
Valentin Hofmann, Janet B. Pierrehumbert, Hinrich Schütze
Can pretrained language models (PLMs) generate derivationally complex words? We present the first study investigating this question, taking BERT as the example PLM. We examine BERT…
Identifying Necessary Elements for BERT's Multilinguality
Philipp Dufter, Hinrich Schütze
It has been shown that multilingual BERT (mBERT) yields high quality multilingual representations and enables effective zero-shot transfer. This is surprising given that mBERT does…
Masking as an Efficient Alternative to Finetuning for Pretrained Language Models
Mengjie Zhao, Tao Lin, Fei Mi +2
We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning. E…
Quantifying the Contextualization of Word Representations with Semantic Class Probing
Mengjie Zhao, Philipp Dufter, Yadollah Yaghoobzadeh +1
Pretrained language models have achieved a new state of the art on many NLP tasks, but there are still many open questions about how and why they work so well. We investigate the c…
SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings
Masoud Jalili Sabet, Philipp Dufter, François Yvon +1
Word alignments are useful for tasks like statistical and neural machine translation (NMT) and cross-lingual annotation projection. Statistical word aligners perform well, as do me…
Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA
Nina Poerner, Ulli Waltinger, Hinrich Schütze
Domain adaptation of Pretrained Language Models (PTLMs) is typically achieved by unsupervised pretraining on target-domain text. While successful, this approach is expensive in ter…
Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference
Timo Schick, Hinrich Schütze
Some NLP tasks can be solved in a fully unsupervised fashion by providing a pretrained language model with "task descriptions" in natural language (e.g., Radford et al., 2019). Whi…
Multipurpose Intelligent Process Automation via Conversational Assistant
Alena Moiseeva, Dietrich Trautmann, Michael Heimann +1
Intelligent Process Automation (IPA) is an emerging technology with a primary goal to assist the knowledge worker by taking care of repetitive, routine and low-cognitive tasks. Con…