11 papers
Generic Triple-Latent Compression with Gated Associative Retrieval
Liu Xiao
We study generic triple-latent sequence models that maintain a running token state and compressed pair-memory pathway to capture higher-order token interactions without benchmark-s…
Associative-State Universal Transformers: Sparse Retrieval Meets Structured Recurrence
Liu Xiao
We study whether a structured recurrent state can serve as a compact associative backbone for language modeling while still supporting exact retrieval. We introduce UniMatrix, a Un…
Belief-State RWKV for Reinforcement Learning under Partial Observability
Liu Xiao
We propose a stronger formulation of RL on top of RWKV-style recurrent sequence models, in which the fixed-size recurrent state is explicitly interpreted as a belief state rather t…
DREAMSTATE: Diffusing States and Parameters for Recurrent Large Language Models
Liu Xiao
Modern Recurrent Neural Networks (RNNs), such as RWKV, are distinguished by their powerful short-range modeling capabilities and efficient fixed-size states, which constitute a cor…
WuNeng: Hybrid State with Attention
Liu Xiao, Li Zhiyuan, Lin Yueyu
The WuNeng architecture introduces a novel approach to enhancing the expressivity and power of large language models by integrating recurrent neural network (RNN)-based RWKV-7 with…
Cross-attention for State-based model RWKV-7
Liu Xiao, Li Zhiyuan, Lin Yueyu
We introduce CrossWKV, a novel cross-attention mechanism for the state-based RWKV-7 model, designed to enhance the expressive power of text-to-image generation. Leveraging RWKV-7's…