4 papers · 1 filter
Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers
Sayak Chakrabarti, Toniann Pitassi, Josh Alman
Quantization reduces the numerical precision of Transformer computations and is widely used to accelerate inference, yet its effect on expressivity remains poorly characterized. We…
Poly-attention: a general scheme for higher-order self-attention
Sayak Chakrabarti, Toniann Pitassi, Josh Alman
The self-attention mechanism, at the heart of the Transformer model, is able to effectively model pairwise interactions between tokens. However, numerous recent works have shown th…
Replay Can Provably Increase Forgetting
Yasaman Mahdaviyeh, James Lucas, Mengye Ren +3
Continual learning seeks to enable machine learning systems to solve an increasing corpus of tasks sequentially. A critical challenge for continual learning is forgetting, where th…
Improving Predictor Reliability with Selective Recalibration
Thomas P. Zollo, Zhun Deng, Jake C. Snell +2
A reliable deep learning system should be able to accurately express its confidence with respect to its predictions, a quality known as calibration. One of the most effective ways…