4 papers
Asymmetric Within-Document Predictive Learning for Scientific Document Representation
You Zuo, Éric de la Clergerie, Benoît Sagot
We study predictive pretraining for scientific document representation using the discourse structure of papers. We propose SciJEPA, a citation-free framework that learns through as…
Sparse Coverage: Semantic Center Representations for Patent Prior-Art Retrieval
You Zuo, Kim Gerdes, Éric de la Clergerie +1
Patent prior-art retrieval is a recall-oriented search task over long and highly structured technical documents. Dense retrieval improves semantic matching, but single-vector repre…
OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining
Rian Touchent, Éric de la Clergerie
We present OntoBook, a method that converts medical ontology structure into pretraining signal for encoder language models. Our approach has three stages: random walks through onto…
A Causal Language Modeling Detour Improves Encoder Continued Pretraining
Rian Touchent, Eric de la Clergerie
When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily switching to Causal Language M…