507 citations · 619 across the 13 of their papers we have counts for
18 papers
Muse Glimmer 30B: An On-Device Agentic Model
Arash Ebrahim Abtahizadeh, Muna Aghamelu, Adil Ahmed +327
We introduce Muse Glimmer, an open-source 30B parameter causal language model with perception encoder, purpose-built for autonomous agentic tasks on consumer hardware. Muse Glimmer…
Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4
Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in ter…
Compute Optimal Tokenization
Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer +6
Scaling laws enable the optimal selection of data amount and language model size, yet the impact of the data unit, the token, on this relationship remains underexplored. In this wo…
Fast Byte Latent Transformer
Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5
Recent byte-level language models (LMs) match the performance of token-level models without relying on subword vocabularies, yet their utility is limited by slow, byte-by-byte auto…
Byte Latent Transformer: Patches Scale Better Than Tokens
Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez +11
We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant imp…
Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling
Margaret Li, Weijia Shi, Artidoro Pagnoni +2
RLHF-aligned LMs have shown unprecedented ability on both benchmarks and long-form text generation, yet they struggle with one foundational task: next-token prediction. As RLHF mod…
QLoRA: Efficient Finetuning of Quantized LLMs
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman +1
We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning t…
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization
Artidoro Pagnoni, Alexander R. Fabbri, Wojciech Kryściński +1
In long document controllable summarization, where labeled data is scarce, pretrained models struggle to adapt to the task and effectively respond to user queries. In this paper, w…
EvEntS ReaLM: Event Reasoning of Entity States via Language Models
Evangelia Spiliopoulou, Artidoro Pagnoni, Yonatan Bisk +1
This paper investigates models of event implications. Specifically, how well models predict entity state-changes, by targeting their understanding of physical attributes. Nominally…
Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
Artidoro Pagnoni, Vidhisha Balachandran, Yulia Tsvetkov
Modern summarization models generate highly fluent but often factually unreliable outputs. This motivated a surge of metrics attempting to measure the factuality of automatically g…
StructSum: Summarization via Structured Representations
Vidhisha Balachandran, Artidoro Pagnoni, Jay Yoon Lee +3
Abstractive text summarization aims at compressing the information of a long source document into a rephrased, condensed summary. Despite advances in modeling techniques, abstracti…
Definition Frames: Using Definitions for Hybrid Concept Representations
Evangelia Spiliopoulou, Artidoro Pagnoni, Eduard Hovy
Advances in word representations have shown tremendous improvements in downstream NLP tasks, but lack semantic interpretability. In this paper, we introduce Definition Frames (DF),…
Making Classical Machine Learning Pipelines Differentiable: A Neural Translation Approach
Gyeong-In Yu, Saeed Amizadeh, Sehoon Kim +4
Classical Machine Learning (ML) pipelines often comprise of multiple ML models where models, within a pipeline, are trained in isolation. Conversely, when training neural network m…
Machine Learning at Microsoft with ML .NET
Zeeshan Ahmed, Saeed Amizadeh, Mikhail Bilenko +31
Machine Learning is transitioning from an art and science into a technology available to every developer. In the near future, every application on every platform will incorporate t…
Analyzing Branch-and-Bound Algorithms for the Multiprocessor Scheduling Problem
Thomas Lively, William Long, Artidoro Pagnoni
The Multiprocessor Scheduling Problem (MSP) is an NP-Complete problem with significant applications in computer and operations systems. We provide a survey of the wide array of pol…
PAC Learning Guarantees Under Covariate Shift
Artidoro Pagnoni, Stefan Gramatovici, Samuel Liu
We consider the Domain Adaptation problem, also known as the covariate shift problem, where the distributions that generate the training and test data differ while retaining the sa…
Conditional Variational Autoencoder for Neural Machine Translation
Artidoro Pagnoni, Kevin Liu, Shangyan Li
We explore the performance of latent variable models for conditional text generation in the context of neural machine translation (NMT). Similar to Zhang et al., we augment the enc…
Taint Tracking for WebAssembly
Aron Szanto, Timothy Tamm, Artidoro Pagnoni
WebAssembly seeks to provide an alternative to running large and untrusted binaries within web browsers by implementing a portable, performant, and secure bytecode format for nativ…