4 papers · 1 filter
BitSparse: Compact Activation Storage for ReLU Feed-Forward Networks
Max Zhu, Bohang Sun, Pietro Lio +1
Reducing memory use in large language model (LLM) training is an ongoing challenge. Activation checkpointing saves memory by recomputation, while lossy compression changes the stor…
LCT: Losslessly Compressed Tensors with GPU friendly execution for Training
Max Zhu, Bohang Sun, Pietro Lio +1
Lossless tensor compression could reduce memory usage during training, but training requires tensors to be repeatedly re-encoded. We introduce Losslessly Compressed Tensors (LCT),…
SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inference
Yuqi Pan, Jinghao Zhuang, Yupeng Feng +16
Scaling context length is reshaping large-model development, yet full-attention Transformers suffer from prohibitive computation and inference bottlenecks at long sequences. A key…
SpikingBrain: Spiking Brain-inspired Large Models
Yuqi Pan, Yupeng Feng, Jinghao Zhuang +16
Mainstream Transformer-based large language models face major efficiency bottlenecks: training computation scales quadratically with sequence length, and inference memory grows lin…