1 citations · 1 across the 8 of their papers we have counts for
4 papers · 1 filter
Preserving Updates in Low-Bit Recurrent State Quantization
Haotian Li, Liran Dong, Kaicheng Xiao +1
Linear attention and recurrent models compress past tokens into a fixed-size state that is updated at every step. However, storing these states in high precision incurs substantial…
RAM-Net: Linear-Time Sequence Modeling with Sparsely Addressable State
Kaicheng Xiao, Haotian Li, Liran Dong +1
Linear attention offers an efficient alternative to full attention with a fixed-size recurrent state. However, this state is shared by all tokens, so information from distinct toke…
Two Is Better Than One: Rotations Scale LoRAs
Hongcan Guo, Guoshun Nan, Yuan Yang +9
Scaling Low-Rank Adaptation (LoRA)-based Mixture-of-Experts (MoE) facilitates large language models (LLMs) to efficiently adapt to diverse tasks. However, traditional gating mechan…
Self-supervised Spatial-Temporal Learner for Precipitation Nowcasting
Haotian Li, Arno Siebes, Siamak Mehrkanoon
Nowcasting, the short-term prediction of weather, is essential for making timely and weather-dependent decisions. Specifically, precipitation nowcasting aims to predict precipitati…