457 citations · 462 across the 9 of their papers we have counts for
4 papers · 1 filter
VANDAM: Viewing a nucleotide sequence with DNA molecular priors
Jeremy Levy, Ariel Larey, Yury Nahshan +18
Contemporary Genomic Foundation Models (GFMs) rely on a DNA-as-a-string paradigm that employs masked token prediction objectives for pretraining. However, this abstraction does not…
ProteinJEPA: Latent prediction improves protein language model pretraining
Dan Ofer, Dafna Shahaf, Michal Linial
Protein language models are trained primarily with masked language modeling (MLM), which predicts masked amino-acid identities. Joint-embedding predictive architectures (JEPA) inst…
ProtSent: Protein Sentence Transformers
Dan Ofer, Oriel Perets, Michal Linial +1
Protein language models produce rich per-residue representations, but the sequence embeddings obtained by mean-pooling them are never trained to place related proteins near one ano…
JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures
Ariel Larey, Elay Dahan, Amit Bleiweiss +15
Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP) to learn the "Laws of Nature". While effective at capturing local s…