25 citations · 179 across the 54 of their papers we have counts for
14 papers · 1 filter
Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis
Andor Diera, Ansgar Scherp
Understanding whether large language models (LLMs) capture structured meaning requires examining how they represent concept relationships. In this work, we study three models of in…
CRAWLDoc: A Dataset for Robust Ranking of Bibliographic Documents
Fabian Karl, Ansgar Scherp
Publication databases rely on accurate metadata extraction from diverse web sources, yet variations in web layouts and data formats present challenges for metadata providers. This…
Efficient Continual Learning for Small Language Models with a Discrete Key-Value Bottleneck
Andor Diera, Lukas Galke, Fabian Karl +1
Continual learning remains a challenge across various natural language processing (NLP) tasks, as models updated with new training data often risk catastrophic forgetting of previo…
Isotropy Matters: Soft-ZCA Whitening of Embeddings for Semantic Code Search
Andor Diera, Lukas Galke, Ansgar Scherp
Low isotropy in an embedding space impairs performance on tasks involving semantic inference. Our study investigates the impact of isotropy on semantic code search performance and…
Your Extreme Multi-label Classifier is Secretly a Hierarchical Text Classifier for Free
Nerijus Bertalis, Paul Granse, Ferhat Gül +6
Assigning a set of labels to a given text is a classification problem with many real-world applications, such as recommender systems. Two separate research streams address this iss…
GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding
Andor Diera, Abdelhalim Dahou, Lukas Galke +3
Language models can serve as a valuable tool for software developers to increase productivity. Large generative models can be used for code generation and code completion, while sm…
Fine-Tuning Language Models for Scientific Writing Support
Justin Mücke, Daria Waldow, Luise Metzger +4
We support scientific writers in determining whether a written sentence is scientific, to which section it belongs, and suggest paraphrasings to improve the sentence. Firstly, we p…
Memorization of Named Entities in Fine-tuned BERT Models
Andor Diera, Nicolas Lell, Aygul Garifullina +1
Privacy preserving deep learning is an emerging field in machine learning that aims to mitigate the privacy risks in the use of deep neural networks. One such risk is training data…
Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasets
Fabian Karl, Ansgar Scherp
Short text classification is a crucial and challenging aspect of Natural Language Processing. For this reason, there are numerous highly specialized short text classifiers. However…
Are We Really Making Much Progress in Text Classification? A Comparative Review
Lukas Galke, Ansgar Scherp, Andor Diera +5
We survey the literature on single-label, multi-label, and hierarchical text classification and provide a quantitative comparison of methods categorized into bag-of-words, sequence…
General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddings
Lukas Galke, Isabelle Cuber, Christoph Meyer +3
Large pretrained language models (PreLMs) are revolutionizing natural language processing across all benchmarks. However, their sheer size is prohibitive for small laboratories or…
Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP
Lukas Galke, Ansgar Scherp
Graph neural networks have triggered a resurgence of graph-based text classification methods, defining today's state of the art. We show that a wide multi-layer perceptron (MLP) us…
Analysis of GraphSum's Attention Weights to Improve the Explainability of Multi-Document Summarization
M. Lautaro Hickmann, Fabian Wurzberger, Megi Hoxhalli +3
Modern multi-document summarization (MDS) methods are based on transformer architectures. They generate state of the art summaries, but lack explainability. We focus on graph-based…
CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model
Florian Mai, Lukas Galke, Ansgar Scherp
Continuous Bag of Words (CBOW) is a powerful text embedding method. Due to its strong capabilities to encode word content, CBOW embeddings perform well on a wide range of downstrea…