7 papers
How Post-Training Shapes Biological Reasoning Models
Lukas Fesser, Hanlin Zhang, Michelle M. Li +5
Scientific reasoning models for biology combine language models with foundation models trained on multimodal biological data, including DNA, RNA, and proteins. These models are bui…
A Unifying View of Attention Sinks: Two Algorithms, Two Solutions
Lukas Fesser, Mozes Jacobs, Thomas Fel +2
When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signatur…
Evaluating Relational Reasoning in LLMs with REL
Lukas Fesser, Yasha Ektefaie, Ada Fang +2
Relational reasoning is the ability to infer relations that jointly bind multiple entities, attributes, or variables. This ability is central to scientific reasoning, but existing…
Multimodal Medical Code Tokenizer
Xiaorui Su, Shvat Messica, Yepeng Huang +5
Foundation models trained on patient electronic health records (EHRs) require tokenizing medical data into sequences of discrete vocabulary items. Existing tokenizers treat medical…
Performance Heterogeneity in Graph Neural Networks: Lessons for Architecture Design and Preprocessing
Lukas Fesser, Melanie Weber
Graph Neural Networks have emerged as the most popular architecture for graph-level learning, including graph classification and regression tasks, which frequently arise in areas s…
Enhancing the Utility of Higher-Order Information in Relational Learning
Raphael Pellegrin, Lukas Fesser, Melanie Weber
Higher-order information is crucial for relational learning in many domains where relationships extend beyond pairwise interactions. Hypergraphs provide a natural framework for mod…