5 papers
Neuromem: A Granular Decomposition of the Streaming Lifecycle in External Memory for LLMs
Ruicheng Zhang, Xinyi Li, Tianyi Xu +3
Most evaluations of External Memory Module assume a static setting: memory is built offline and queried at a fixed state. In practice, memory is streaming: new facts arrive continu…
OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
Shaobo Wang, Xuan Ouyang, Tianyi Xu +9
As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall, pre-training is shifting from more tokens to better tokens. However, existing methods either…
SITA: Learning Speaker-Invariant and Tone-Aware Speech Representations for Low-Resource Tonal Languages
Tianyi Xu, Xuan Ouyang, Binwei Yao +4
Tonal low-resource languages are widely spoken yet remain underserved by modern speech technology. A key challenge is learning representations that are robust to nuisance variation…
Large Language Model Compression via the Nested Activation-Aware Decomposition
Jun Lu, Tianyi Xu, Bill Ding +2
In this paper, we tackle the critical challenge of compressing large language models (LLMs) to facilitate their practical deployment and broader adoption. We introduce a novel post…
SNAP: Stopping Catastrophic Forgetting in Hebbian Learning with Sigmoidal Neuronal Adaptive Plasticity
Tianyi Xu, Patrick Zheng, Shiyan Liu +2
Artificial Neural Networks (ANNs) suffer from catastrophic forgetting, where the learning of new tasks causes the catastrophic forgetting of old tasks. Existing Machine Learning (M…