13 citations · 15 across the 9 of their papers we have counts for
10 papers
Unified Lookup-Table Inference with Signed-Digit K/V Caches for Ternary LLMs
Ziang Duan, Jiajun Wu, Zetian Chen +8
Ternary LLMs make their weight-dominated projections compact and efficient, but attention remains a mismatch: its K/V cache is created online and is typically processed by a separa…
Anon: Extrapolating Adaptivity Beyond SGD and Adam
Yiheng Zhang, Kaiyan Zhao, Shaowu Wu +5
Adaptive optimizers such as Adam and non-adaptive methods like SGD exhibit distinct generalization capabilities across different architectures. Prior tunable optimizers attempt to…
Shiva-DiT: Residual-Based Differentiable Top- Selection for Efficient Diffusion Transformers
Jiaji Zhang, Hailiang Zhao, Jiaju Wu +3
Diffusion Transformers (DiTs) are costly at high resolution because self-attention scales quadratically with token sequence length. Existing pruning methods do not jointly provide…
HVAdam: A Full-Dimension Adaptive Optimizer
Yiheng Zhang, Shaowu Wu, Yuanzhuo Xu +4
Adaptive optimizers such as Adam have achieved great success in training large-scale models like large language models and diffusion models. However, they often generalize worse th…
SegQuant: A Semantics-Aware and Generalizable Quantization Framework for Diffusion Models
Jiaji Zhang, Ruichao Sun, Hailiang Zhao +7
Diffusion models have demonstrated exceptional generative capabilities but are computationally intensive, posing significant challenges for deployment in resource-constrained or la…
Flow to the Mode: Mode-Seeking Diffusion Autoencoders for State-of-the-Art Image Tokenization
Kyle Sargent, Kyle Hsu, Justin Johnson +2
Since the advent of popular visual generation frameworks like VQGAN and latent diffusion models, state-of-the-art image generation systems have generally been two-stage systems tha…
TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecture
Jiajun Wu, Mo Song, Jingmin Zhao +3
Modern transformer-based deep neural networks present unique technical challenges for effective acceleration in real-world applications. Apart from the vast amount of linear operat…
Range, not Independence, Drives Modularity in Biologically Inspired Representations
Will Dorrell, Kyle Hsu, Luke Hollingsworth +6
Why do biological and artificial neurons sometimes modularise, each encoding a single meaningful variable, and sometimes entangle their representation of many variables? In this wo…
Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning
Kyle Hsu, Jubayer Ibn Hamid, Kaylee Burns +2
Inductive biases are crucial in disentangled representation learning for narrowing down an underspecified solution set. In this work, we consider endowing a neural network autoenco…
DyBit: Dynamic Bit-Precision Numbers for Efficient Quantized Neural Network Inference
Jiajun Zhou, Jiajun Wu, Yizhao Gao +7
To accelerate the inference of deep neural networks (DNNs), quantization with low-bitwidth numbers is actively researched. A prominent challenge is to quantize the DNN models into…