12 citations · 53 across the 37 of their papers we have counts for
6 papers · 2 filters
TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training
Michael Menezes, Barbara Su, Xinze Feng +3
We introduce TwIST, a distributed training framework for efficient large language model (LLM) sparsification. TwIST trains multiple subnetworks in parallel, periodically aggregates…
Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts
Fangshuo Liao, Anastasios Kyrillidis
Mixture-of-Experts (MoE) architectures have emerged as a cornerstone of modern AI systems. In particular, MoEs route inputs dynamically to specialized experts whose outputs are agg…
ADAPT: Lightweight, Long-Range Machine Learning Force Fields Without Graphs
Evan Dramko, Yihuang Xiong, Yizhi Zhu +4
Point defects play a central role in driving the properties of materials. First-principles methods are widely used to compute defect energetics and structures, including at scale f…
One Rank at a Time: Cascading Error Dynamics in Sequential Learning
Mahtab Alizadeh Vandchali, Fangshuo, Liao +1
Sequential learning -- where complex tasks are broken down into simpler, hierarchical components -- has emerged as a paradigm in AI. This paper views sequential learning through th…
Exploring How LLMs Capture and Represent Domain-Specific Knowledge
Mirian Hipolito Garcia, Camille Couturier, Daniel Madrigal Diaz +5
We study whether Large Language Models (LLMs) inherently capture domain-specific nuances in natural language. Our experiments probe the domain sensitivity of LLMs by examining thei…
Provable Model-Parallel Distributed Principal Component Analysis with Parallel Deflation
Fangshuo Liao, Wenyi Su, Anastasios Kyrillidis
We study a distributed Principal Component Analysis (PCA) framework where each worker targets a distinct eigenvector and refines its solution by updating from intermediate solution…