18 citations · 37 across the 11 of their papers we have counts for
12 papers
NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
NCP Team, Jiaqi Cao, Chiyu Chen +25
We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns thro…
Controlling Exploration-Exploitation in GFlowNets via Markov Chain Perspectives
Lin Chen, Samuel Drapeau, Fanghao Shao +5
Generative Flow Network (GFlowNet) objectives implicitly fix an equal mixing of forward and backward policies, potentially constraining the exploration-exploitation trade-off durin…
Towards Compressive and Scalable Recurrent Memory
Yunchong Song, Jushi Kai, Liming Lu +2
Transformers face a quadratic bottleneck in attention when scaling to long contexts. Recent approaches introduce recurrent memory to extend context beyond the current window, yet t…
Flow of Spans: Generalizing Language Models to Dynamic Span-Vocabulary via GFlowNets
Bo Xue, Yunchong Song, Fanghao Shao +5
Standard autoregressive language models generate text token-by-token from a fixed vocabulary, inducing a tree-structured state space when viewing token sampling as an action, which…
Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
Yuliang Liu, Yunchong Song, Yixuan Wang +6
We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, t…
FlowRL: Matching Reward Distributions for LLM Reasoning
Xuekai Zhu, Daixuan Cheng, Dinghuai Zhang +20
We propose FlowRL: matching the full reward distribution via flow balancing instead of maximizing rewards in large language model (LLM) reinforcement learning (RL). Recent advanced…
Cluster-wise Graph Transformer with Dual-granularity Kernelized Attention
Siyuan Huang, Yunchong Song, Jiayue Zhou +1
In the realm of graph learning, there is a category of methods that conceptualize graphs as hierarchical structures, utilizing node clustering to capture broader structural informa…
FLAME: Empowering Frozen LLMs for Knowledge Graph Completion
Bo Xue, Yi Xu, Bolei Ma +4
Traditional knowledge graph completion (KGC) methods rely solely on structural information and struggle with sparsity, while Large Language Models (LLMs) address these limitations…
Graph Parsing Networks
Yunchong Song, Siyuan Huang, Xinbing Wang +2
Graph pooling compresses graph information into a compact representation. State-of-the-art graph pooling methods follow a hierarchical approach, which reduces the graph size step-b…
GeoGalactica: A Scientific Large Language Model in Geoscience
Zhouhan Lin, Cheng Deng, Le Zhou +18
Large language models (LLMs) have achieved huge success for their general knowledge and ability to solve a wide spectrum of tasks in natural language processing (NLP). Due to their…
Tailoring Self-Attention for Graph via Rooted Subtrees
Siyuan Huang, Yunchong Song, Jiayue Zhou +1
Attention mechanisms have made significant strides in graph learning, yet they still exhibit notable limitations: local attention faces challenges in capturing long-range informati…
Ordered GNN: Ordering Message Passing to Deal with Heterophily and Over-smoothing
Yunchong Song, Chenghu Zhou, Xinbing Wang +1
Most graph neural networks follow the message passing mechanism. However, it faces the over-smoothing problem when multiple times of message passing is applied to a graph, causing…