8 papers
Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination
Peng Wang, Xiao Li, Can Yaras +4
Over the past decade, deep learning has proven to be a highly effective tool for learning meaningful features from raw data. However, it remains an open question how deep networks…
Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention
Xiao Li, Chengruidong Zhang, Hao Luo +15
Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active corr…
Data-Constrained Language Model Pretraining: Improved Regularization and Scaling Laws
Zhiwei Xu, Shihao Wu, Hanseul Cho +2
Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant data and a single pass over t…
Lightweight Transformer for EEG Classification via Balanced Signed Graph Algorithm Unrolling
Junyi Yao, Parham Eftekhar, Gene Cheung +3
Samples of brain signals collected by EEG sensors have inherent anti-correlations that are well modeled by negative edges in a finite graph. To differentiate epilepsy patients from…
RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithm
Yongyi Yang, Jianyang Gao, Wei Hu
Post-training Quantization (PTQ) has become a widely used technique for improving inference efficiency of large language models (LLMs). However, existing PTQ methods generally suff…
Swing-by Dynamics in Concept Learning and Compositional Generalization
Yongyi Yang, Core Francisco Park, Ekdeep Singh Lubana +3
Prior work has shown that text-conditioned diffusion models can learn to identify and manipulate primitive concepts underlying a compositional data-generating process, enabling gen…