11 citations · 16 across the 14 of their papers we have counts for
14 papers
Parallel Track Transformers: Enabling Fast GPU Inference with Reduced Synchronization
Chong Wang, Nan Du, Tom Gunter +8
Efficient large-scale inference of transformer-based large language models (LLMs) remains a fundamental systems challenge, frequently requiring multi-GPU parallelism to meet string…
Task-Aware Multi-Expert Architecture For Lifelong Deep Learning
Jianyu Wang, Jacob Nean-Hua Sheikh, Cat P. Le +1
Lifelong deep learning (LDL) trains neural networks to learn sequentially across tasks while preserving prior knowledge. We propose Task-Aware Multi-Expert (TAME), a continual lear…
RLAX: Large-Scale, Distributed Reinforcement Learning for Large Language Models on TPUs
Runlong Zhou, Lefan Zhang, Shang-Chen Wu +29
Reinforcement learning (RL) has emerged as the de-facto paradigm for improving the reasoning capabilities of large language models (LLMs). We have developed RLAX, a scalable RL fra…
Apple Intelligence Foundation Language Models: Tech Report 2025
Ethan Li, Anders Boesen Lindbo Larsen, Chen Zhang +395
We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: i a 3B-parameter on-device model opti…
AXLearn: Modular, Hardware-Agnostic Large Model Training
Mark Lee, Chang Lan, Tom Gunter +34
AXLearn is a production system which facilitates scalable and high-performance training of large deep learning models. Compared to other state-of-art deep learning systems, AXLearn…
Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Jianyu Wang, Zhiqiang Hu, Lidong Bing
We propose a novel prompt design paradigm that challenges conventional wisdom in large language model (LLM) prompting. While conventional wisdom prioritizes well-crafted instructio…
Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
Ajay Jaiswal, Jianyu Wang, Yixiao Li +6
Sparsely activated Mixture-of-Experts (SMoE) has shown promise in scaling up the learning capacity of neural networks. However, vanilla SMoEs have issues such as expert redundancy…
IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
Yixiao Li, Xianzhi Du, Ajay Jaiswal +4
Recent advancements in large language models have intensified the need for efficient and deployable models within limited inference budgets. Structured pruning pipelines have shown…
Instruction-Following Pruning for Large Language Models
Bairu Hou, Qibin Chen, Jianyu Wang +6
With the rapid scaling of large language models (LLMs), structured pruning has become a widely used technique to learn efficient, smaller models from larger ones, delivering superi…
Apple Intelligence Foundation Language Models
Tom Gunter, Zirui Wang, Chong Wang +152
We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large serv…
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier +29
In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. T…
Momentum Approximation in Asynchronous Private Federated Learning
Tao Yu, Congzheng Song, Jianyu Wang +1
Asynchronous protocols have been shown to improve the scalability of federated learning (FL) with a massive number of clients. Meanwhile, momentum-based methods can achieve the bes…
Wasserstein Nonnegative Tensor Factorization with Manifold Regularization
Jianyu Wang, Linruize Tang
Nonnegative tensor factorization (NTF) has become an important tool for feature extraction and part-based representation with preserved intrinsic structure information from nonnega…
FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descent
Ziyao Wang, Jianyu Wang, Ang Li
The theoretical landscape of federated learning (FL) undergoes rapid evolution, but its practical application encounters a series of intricate challenges, and hyperparameter optimi…