64 citations · 156 across the 18 of their papers we have counts for
19 papers
Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization
Chong Wang, Nan Du, Tom Gunter +8
Efficient large-scale inference of transformer-based large language models (LLMs) remains a fundamental systems challenge, frequently requiring multi-GPU parallelism to meet string…
Reusing Pre-Training Data at Test Time is a Compute Multiplier
Alex Fang, Thomas Voice, Ruoming Pang +2
Large language models learn from their vast pre-training corpora, gaining the ability to solve an ever increasing variety of tasks; yet although researchers work to improve these d…
Apple Intelligence Foundation Language Models: Tech Report 2025
Ethan Li, Anders Boesen Lindbo Larsen, Chen Zhang +395
We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: i a 3B-parameter on-device model opti…
Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?
Arduin Findeis, Floris Weers, Guoli Yin +3
Pairwise preferences over model responses are widely collected to evaluate and provide feedback to large language models (LLMs). Given two alternative model responses to the same i…
Language Models Improve When Pretraining Data Matches Target Tasks
David Mizrahi, Anders Boesen Lindbo Larsen, Jesse Allardice +7
Every data selection method inherently has a target. In practice, these targets often emerge implicitly through benchmark-driven iteration: researchers develop selection strategies…
AXLearn: Modular, Hardware-Agnostic Large Model Training
Mark Lee, Chang Lan, Tom Gunter +34
AXLearn is a production system which facilitates scalable and high-performance training of large deep learning models. Compared to other state-of-art deep learning systems, AXLearn…
Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality
Alex Fang, Hadi Pouransari, Matt Jordan +4
Data filtering has become a powerful tool for improving model performance while reducing computational cost. However, as large language model compute budgets continue to grow, the…
Apple Intelligence Foundation Language Models
Tom Gunter, Zirui Wang, Chong Wang +152
We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large serv…
Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training
Xianzhi Du, Tom Gunter, Xiang Kong +5
Mixture-of-Experts (MoE) enjoys performance gain by increasing model capacity while keeping computation cost constant. When comparing MoE to dense models, prior work typically adop…
Large Language Model-guided Document Selection
Xiang Kong, Tom Gunter, Ruoming Pang
Large Language Model (LLM) pre-training exhausts an ever growing compute budget, yet recent research has demonstrated that careful document selection enables comparable model quali…
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier +29
In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. T…
Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts
Erik Daxberger, Floris Weers, Bowen Zhang +7
Sparse Mixture-of-Experts models (MoEs) have recently gained popularity due to their ability to decouple model size from inference efficiency by only activating a small subset of t…
STAIR: Learning Sparse Text and Image Representation in Grounded Tokens
Chen Chen, Bowen Zhang, Liangliang Cao +7
Image and text retrieval is one of the foundational tasks in the vision and language domain with multiple real-world applications. State-of-the-art approaches, e.g. CLIP, ALIGN, re…
Masked Autoencoding Does Not Help Natural Language Supervision at Scale
Floris Weers, Vaishaal Shankar, Angelos Katharopoulos +2
Self supervision and natural language supervision have emerged as two exciting ways to train general purpose image encoders which excel at a variety of downstream tasks. Recent wor…
Unknowable Manipulators: Social Network Curator Algorithms
Samuel Albanie, Hillary Shakespeare, Tom Gunter
For a social networking service to acquire and retain users, it must find ways to keep them engaged. By accurately gauging their preferences, it is able to serve them with the subs…
Blitzkriging: Kronecker-structured Stochastic Gaussian Processes
Thomas Nickson, Tom Gunter, Chris Lloyd +2
We present Blitzkriging, a new approach to fast inference for Gaussian processes, applicable to regression, optimisation and classification. State-of-the-art (stochastic) inference…
Sampling for Inference in Probabilistic Models with Fast Bayesian Quadrature
Tom Gunter, Michael A. Osborne, Roman Garnett +2
We propose a novel sampling framework for inference in probabilistic models: an active learning approach that converges more quickly (in wall-clock time) than Markov chain Monte Ca…
Variational Inference for Gaussian Process Modulated Poisson Processes
Chris Lloyd, Tom Gunter, Michael A. Osborne +1
We present the first fully variational Bayesian inference scheme for continuous Gaussian-process-modulated Poisson processes. Such point processes are used in a variety of domains,…
Efficient Bayesian Nonparametric Modelling of Structured Point Processes
Tom Gunter, Chris Lloyd, Michael A. Osborne +1
This paper presents a Bayesian generative model for dependent Cox point processes, alongside an efficient inference scheme which scales as if the point processes were modelled inde…