15 citations · 54 across the 15 of their papers we have counts for
15 papers
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett +94
Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heter…
SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing
Luca Foppiano, Sotaro Takeshita, Pedro Ortiz Suarez +6
SciLaD is a novel, large-scale dataset of scientific language constructed entirely using open-source frameworks and publicly available data sources. It comprises a curated English…
Table Understanding and (Multimodal) LLMs: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data
Ekaterina Borisova, Fabio Barth, Nils Feldhus +5
Tables are among the most widely used tools for representing structured data in research, business, medicine, and education. Although LLMs demonstrate strong performance in downstr…
MMTEB: Massive Multilingual Text Embedding Benchmark
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua +83
Text embeddings are typically evaluated on a limited set of tasks, which are constrained by language, domain, and task diversity. To address these limitations and provide a more co…
Reward Modeling with Weak Supervision for Language Models
Ben Hauptvogel, Malte Ostendorff, Georg Rehm +1
Recent advancements in large language models (LLMs) have led to their increased application across various tasks, with reinforcement learning from human feedback (RLHF) being a cru…
Data Processing for the OpenGPT-X Model Family
Nicolo' Brandizzi, Hammam Abdelwahab, Anirban Bhowmick +19
This paper presents a comprehensive overview of the data preparation pipeline developed for the OpenGPT-X project, a large-scale initiative aimed at creating open and high-performa…
Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs
Mehdi Ali, Michael Fromm, Klaudia Thellmann +38
We present two multilingual LLMs, Teuken 7B-base and Teuken 7B-instruct, designed to embrace Europe's linguistic diversity by supporting all 24 official languages of the European U…
Symmetric Dot-Product Attention for Efficient Training of BERT Language Models
Martin Courtois, Malte Ostendorff, Leonhard Hennig +1
Initially introduced as a machine translation model, the Transformer architecture has now become the foundation for modern deep learning architecture, with applications in a wide r…
Investigating Gender Bias in Turkish Language Models
Orhun Mersin Caglidil, Malte Ostendorff, Georg Rehm
Language models are trained mostly on Web data, which often contains social stereotypes and biases that the models can inherit. This has potentially negative consequences, as model…
Tokenizer Choice For LLM Training: Negligible or Crucial?
Mehdi Ali, Michael Fromm, Klaudia Thellmann +18
The recent success of Large Language Models (LLMs) has been predominantly driven by curating the training dataset composition, scaling of model architectures and dataset sizes and…
AspectCSE: Sentence Embeddings for Aspect-based Semantic Textual Similarity Using Contrastive Learning and Structured Knowledge
Tim Schopf, Emanuel Gerber, Malte Ostendorff +1
Generic sentence embeddings provide a coarse-grained approximation of semantic textual similarity but ignore specific aspects that make texts similar. Conversely, aspect-based sent…
Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning
Malte Ostendorff, Georg Rehm
Most Transformer language models are primarily pretrained on English text, limiting their use for other languages. As the model sizes grow, the performance gap between English and…
Specialized Document Embeddings for Aspect-based Similarity of Research Papers
Malte Ostendorff, Till Blume, Terry Ruas +2
Document embeddings and similarity measures underpin content-based recommender systems, whereby a document is commonly represented as a single generic embedding. However, similarit…
Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings
Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein +2
Learning scientific document representations can be substantially improved through contrastive learning objectives, where the challenge lies in creating positive and negative train…
A Qualitative Evaluation of User Preference for Link-based vs. Text-based Recommendations of Wikipedia Articles
Malte Ostendorff, Corinna Breitinger, Bela Gipp
Literature recommendation systems (LRS) assist readers in the discovery of relevant content from the overwhelming amount of literature available. Despite the widespread adoption of…