200 citations · 295 across the 12 of their papers we have counts for
13 papers
mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
Marc Marone, Orion Weller, William Fleshman +3
Encoder-only languages models are frequently used for a variety of standard machine learning tasks, including classification and retrieval. However, there has been a lack of recent…
Seq vs Seq: An Open Suite of Paired Encoders and Decoders
Orion Weller, Kathryn Ricci, Marc Marone +3
The large language model (LLM) community focuses almost exclusively on decoder-only language models, since they are easier to use for text generation. However, a large subset of th…
Certified Mitigation of Worst-Case LLM Copyright Infringement
Jingyu Zhang, Jiacan Yu, Marc Marone +2
The exposure of large language models (LLMs) to copyrighted material during pre-training raises concerns about unintentional copyright infringement post deployment. This has driven…
AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees
William Fleshman, Aleem Khan, Marc Marone +1
Large language models (LLMs) are increasingly capable of completing knowledge intensive tasks by recalling information from a static pretraining corpus. Here we are concerned with…
Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data
Jingyu Zhang, Marc Marone, Tianjian Li +2
To trust the fluent generations of large language models (LLMs), humans must be able to verify their correctness against trusted, external sources. Recent efforts, such as providin…
Dated Data: Tracing Knowledge Cutoffs in Large Language Models
Jeffrey Cheng, Marc Marone, Orion Weller +3
Released Large Language Models (LLMs) are often paired with a claimed knowledge cutoff date, or the dates at which training data was gathered. Such information is crucial for appli…
StarCoder 2 and The Stack v2: The Next Generation
Anton Lozhkov, Raymond Li, Loubna Ben Allal +63
The BigCode project, an open-scientific collaboration focused on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder2. In partnership wi…
"According to ...": Prompting Language Models Improves Quoting from Pre-Training Data
Orion Weller, Marc Marone, Nathaniel Weir +3
Large Language Models (LLMs) may hallucinate and generate fake information, despite pre-training on factual data. Inspired by the journalistic device of "according to sources", we…
StarCoder: may the source be with you!
Raymond Li, Loubna Ben Allal, Yangtian Zi +64
The BigCode community, an open-scientific collaboration working on the responsible development of Large Language Models for Code (Code LLMs), introduces StarCoder and StarCoderBase…
Data Portraits: Recording Foundation Model Training Data
Marc Marone, Benjamin Van Durme
Foundation models are trained on increasingly immense and opaque datasets. Even while these models are now key in AI system building, it can be difficult to answer the straightforw…
Pretrained Models for Multilingual Federated Learning
Orion Weller, Marc Marone, Vladimir Braverman +2
Since the advent of Federated Learning (FL), research has applied these methods to natural language processing (NLP) tasks. Despite a plethora of papers in FL for NLP, no previous…
Everything Is All It Takes: A Multipronged Strategy for Zero-Shot Cross-Lingual Information Extraction
Mahsa Yarmohammadi, Shijie Wu, Marc Marone +10
Zero-shot cross-lingual information extraction (IE) describes the construction of an IE model for some target language, given existing annotations exclusively in some other languag…
Character Eyes: Seeing Language through Character-Level Taggers
Yuval Pinter, Marc Marone, Jacob Eisenstein
Character-level models have been used extensively in recent years in NLP tasks as both supplements and replacements for closed-vocabulary token-level word representations. In one p…