8 papers
Task- and dataset-specific information in protein language models
Roman Joeres, Ilya Senatorov, Anastasia Kolchina +2
Protein language models (PLMs) have transferred the latest advances from natural language processing to computational biology. These models, trained on large corpora of protein seq…
Probing Chemical Language Models: Effects of Pre-training and Fine-tuning
Anna Karnysheva, Dietrich Klakow, Ji-Ung Lee
Chemical language models (CLMs) are trained with linearized representations such as SMILES, yet it remains unclear which chemically meaningful substructures they encode. To foster…
From Monolingual to Multilingual: Evaluating Mamba for ASR in South African Languages
Jesujoba O. Alabi, Julian Herreilers, Badr M. Abdullah +1
Recent advances in automatic speech recognition (ASR) have explored different sequence models, including Conformer-based models and newer state space models such as Mamba. Although…
Chemical Language Models for Natural Products: A State-Space Model Approach
Ho-Hsuan Wang, Afnan Sultan, Andrea Volkamer +1
Language models are widely used in chemistry for molecular property prediction and small-molecule generation, yet Natural Products (NPs) remain underexplored despite their importan…
It's Not a Walk in the Park! Challenges of Idiom Translation in Speech-to-text Systems
Iuliia Zaitova, Badr M. Abdullah, Wei Xue +3
Idioms are defined as a group of words with a figurative meaning not deducible from their individual components. Although modern machine translation systems have made remarkable pr…
Voice Conversion Improves Cross-Domain Robustness for Spoken Arabic Dialect Identification
Badr M. Abdullah, Matthew Baas, Bernd Möbius +1
Arabic dialect identification (ADI) systems are essential for large-scale data collection pipelines that enable the development of inclusive speech technologies for Arabic language…
Attention on Multiword Expressions: A Multilingual Study of BERT-based Models with Regard to Idiomaticity and Microsyntax
Iuliia Zaitova, Vitalii Hirak, Badr M. Abdullah +3
This study analyzes the attention patterns of fine-tuned encoder-only models based on the BERT architecture (BERT-based models) towards two distinct types of Multiword Expressions…
Transformers for molecular property prediction: Domain adaptation efficiently improves performance
Afnan Sultan, Max Rausch-Dupont, Shahrukh Khan +3
Over the past six years, molecular transformer models have become key tools in drug discovery. Most existing models are pre-trained on large, unlabeled datasets such as ZINC or ChE…