284 citations · 1.3k across the 37 of their papers we have counts for
14 papers · 1 filter
Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
Bettina Messmer, Vinko Sabolčec, Martin Jaggi
Dataset curation has become a basis for strong large language model (LLM) performance. While various rule-based filtering heuristics exist for English and multilingual datasets, mo…
DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
Matteo Pagliardini, Amirkeivan Mohtashami, Francois Fleuret +1
The transformer architecture by Vaswani et al. (2017) is now ubiquitous across application domains, from natural language processing to speech processing and image understanding. W…
MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
Zeming Chen, Alejandro Hernández Cano, Angelika Romanou +17
Large language models (LLMs) can potentially democratize access to medical knowledge. While many efforts have been made to harness and improve LLMs' medical knowledge and reasoning…
Landmark Attention: Random-Access Infinite Context Length for Transformers
Amirkeivan Mohtashami, Martin Jaggi
While Transformers have shown remarkable success in natural language processing, their attention mechanism's large memory requirements have limited their ability to handle longer c…
SKILL: Structured Knowledge Infusion for Large Language Models
Fedor Moiseev, Zhe Dong, Enrique Alfonseca +1
Large language models (LLMs) have demonstrated human-level performance on a vast spectrum of natural language tasks. However, it is largely unexplored whether they can better inter…
Obtaining Better Static Word Embeddings Using Contextual Embedding Models
Prakhar Gupta, Martin Jaggi
The advent of contextual word embeddings -- representations of words which incorporate semantic and syntactic information from their context -- has led to tremendous improvements o…