1 citations · 1 across the 10 of their papers we have counts for
5 papers · 1 filter
Generic Triple-Latent Compression with Gated Associative Retrieval
Liu Xiao
We study generic triple-latent sequence models that maintain a running token state and compressed pair-memory pathway to capture higher-order token interactions without benchmark-s…
Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence
Liu Xiao
We study whether a structured recurrent state can serve as a compact associative backbone for language modeling while still supporting exact retrieval. We introduce UniMatrix, a Un…
DREAMSTATE: Diffusing States and Parameters for Recurrent Large Language Models
Liu Xiao
Modern Recurrent Neural Networks (RNNs), such as RWKV, are distinguished by their powerful short-range modeling capabilities and efficient fixed-size states, which constitute a cor…
WuNeng: Hybrid State with Attention
Liu Xiao, Li Zhiyuan, Lin Yueyu
The WuNeng architecture introduces a novel approach to enhancing the expressivity and power of large language models by integrating recurrent neural network (RNN)-based RWKV-7 with…
ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer
Lin Yueyu, Li Zhiyuan, Peter Yue +1
As is known, hybrid quadratic and subquadratic attention models in multi-head architectures have surpassed both Transformer and Linear RNN models , with these works primarily focus…