4 papers
FG-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
Pingwei Sun, Yuxuan Hu, Jianchao Tan +6
Linear attention mechanisms have emerged as promising alternatives to softmax attention, offering linear-time complexity during inference. Recent advances such as Gated DeltaNet (G…
Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
Shuchen Zhu, Rizhen Hu, Mingze Wang +4
Pre-training Large Language Models requires immense computational resources, making optimizer efficiency essential. The optimization landscape is highly anisotropic, with loss redu…
MISA: Memory-Efficient LLMs Optimization with Module-wise Importance Sampling
Yuxi Liu, Renjia Deng, Yutong He +3
The substantial memory demands of pre-training and fine-tuning large language models (LLMs) require memory-efficient optimization algorithms. One promising approach is layer-wise o…
SimDiff: Simpler Yet Better Diffusion Model for Time Series Point Forecasting
Hang Ding, Xue Wang, Tian Zhou +1
Diffusion models have recently shown promise in time series forecasting, particularly for probabilistic predictions. However, they often fail to achieve state-of-the-art point esti…