3 citations · 8 across the 9 of their papers we have counts for
9 papers
L: Large Lookup Layers
Albert Tseng, Christopher De Sa
Modern sparse language models typically achieve sparsity through Mixture-of-Experts (MoE) layers, which dynamically route tokens to dense MLP "experts." However, dynamic hard routi…
Model-Preserving Adaptive Rounding
Albert Tseng, Zhaofeng Sun, Christopher De Sa
The goal of quantization is to produce a compressed model whose output distribution is as close to the original model's as possible. To do this tractably, most quantization algorit…
Training LLMs with MXFP4
Albert Tseng, Tao Yu, Youngsuk Park
Low precision (LP) datatypes such as MXFP4 can accelerate matrix multiplications (GEMMs) and reduce training costs. However, directly using MXFP4 instead of BF16 during training si…
QTIP: Quantization with Trellises and Incoherence Processing
Albert Tseng, Qingyao Sun, David Hou +1
Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing weights to low-precision datatypes. Since LLM inference is usually memory-bound, PTQ methods can…
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
Albert Tseng, Jerry Chee, Qingyao Sun +2
Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing their weights to low-precision. In this work, we introduce QuIP#, a weight-only PTQ method that…
Coneheads: Hierarchy Aware Attention
Albert Tseng, Tao Yu, Toni J. B. Liu +1
Attention networks such as transformers have achieved state-of-the-art performance in many domains. These networks rely heavily on the dot product attention operator, which compute…
Shadow Cones: A Generalized Framework for Partial Order Embeddings
Tao Yu, Toni J. B. Liu, Albert Tseng +1
Hyperbolic space has proven to be well-suited for capturing hierarchical relations in data, such as trees and directed acyclic graphs. Prior work introduced the concept of entailme…
Automatic Synthesis of Diverse Weak Supervision Sources for Behavior Analysis
Albert Tseng, Jennifer J. Sun, Yisong Yue
Obtaining annotations for large training sets is expensive, especially in settings where domain knowledge is required, such as behavior analysis. Weak supervision has been studied…
Learning Calibratable Policies using Programmatic Style-Consistency
Eric Zhan, Albert Tseng, Yisong Yue +2
We study the problem of controllable generation of long-term sequential behaviors, where the goal is to calibrate to multiple behavior styles simultaneously. In contrast to the wel…