7 papers
Task-Aware Multi-Expert Architecture For Lifelong Deep Learning
Jianyu Wang, Jacob Nean-Hua Sheikh, Cat P. Le +1
Lifelong deep learning (LDL) trains neural networks to learn sequentially across tasks while preserving prior knowledge. We propose Task-Aware Multi-Expert (TAME), a continual lear…
RLAX: Large-Scale, Distributed Reinforcement Learning for Large Language Models on TPUs
Runlong Zhou, Lefan Zhang, Shang-Chen Wu +29
Reinforcement learning (RL) has emerged as the de-facto paradigm for improving the reasoning capabilities of large language models (LLMs). We have developed RLAX, a scalable RL fra…
Apple Intelligence Foundation Language Models: Tech Report 2025
Ethan Li, Anders Boesen Lindbo Larsen, Chen Zhang +395
We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: i a 3B-parameter on-device model opti…
Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
Jianyu Wang, Zhiqiang Hu, Lidong Bing
We propose a novel prompt design paradigm that challenges conventional wisdom in large language model (LLM) prompting. While conventional wisdom prioritizes well-crafted instructio…
Finding Fantastic Experts in MoEs: A Unified Study for Expert Dropping Strategies and Observations
Ajay Jaiswal, Jianyu Wang, Yixiao Li +6
Sparsely activated Mixture-of-Experts (SMoE) has shown promise in scaling up the learning capacity of neural networks. However, vanilla SMoEs have issues such as expert redundancy…
IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
Yixiao Li, Xianzhi Du, Ajay Jaiswal +4
Recent advancements in large language models have intensified the need for efficient and deployable models within limited inference budgets. Structured pruning pipelines have shown…