8 papers
SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference
Yuqi Pan, Jinghao Zhuang, Yupeng Feng +16
Scaling context length is reshaping large-model development, yet full-attention Transformers suffer from prohibitive computation and inference bottlenecks at long sequences. A key…
PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers
Yibo Zhong, Haoxiang Jiang, Lincan Li +5
Fine-tuning large pre-trained foundation models often yields excellent downstream performance but is prohibitively expensive when updating all parameters. Parameter-efficient fine-…
ENA: Efficient N-dimensional Attention
Yibo Zhong
Efficient modeling of long sequences of high-order data requires a more efficient architecture than Transformer. In this paper, we investigate two key aspects of extending linear r…
Low-Rank Interconnected Adaptation across Layers
Yibo Zhong, Jinman Zhao, Yao Zhou
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method that learns weight updates for pretrained weights through low-rank adapters…
UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models
Xueyan Zhang, Jinman Zhao, Zhifei Yang +4
This paper introduces Uniform Orthogonal Reinitialization Adaptation (UORA), a novel parameter-efficient fine-tuning (PEFT) approach for Large Language Models (LLMs). UORA achieves…
Building Machine Learning Challenges for Anomaly Detection in Science
Elizabeth G. Campolongo, Yuan-Tang Chou, Ekaterina Govorkova +148
Scientific discoveries are often made by finding a pattern or object that was not predicted by the known rules of science. Oftentimes, these anomalous events or objects that do not…