2 citations · 3 across the 21 of their papers we have counts for
24 papers
HABIT-Bench: Evaluating Habit Induction from Longitudinal Weak Evidence in Agent Memory
Junming Zhang, Rubin Wei, Xu Cheng +7
In long-term interactions with large language model (LLM) agents, users reveal habits through their choices and actions. These habits capture implicit user information that agents…
NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
NCP Team, Jiaqi Cao, Chiyu Chen +25
We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongside NTP, the model learns thro…
Intern-S2-Preview: Scientific Agentic Foundation Model
Lei Bai, Jiaqi Cao, Chiyu Chen +121
Scientific discovery increasingly requires AI systems that can reason over scientific evidence of heterogeneous modalities, interact with scientific tools and environments, and sus…
Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
Rubin Wei, Jiaqi Cao, Jiarui Wang +4
Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduce…
MemSFT: Mitigating Alignment Tax with an External Parametric Memory
Jiarui Wang, Xiang Shi, Jiaqi Cao +8
Adapting Large Language Models (LLMs) to specialized domains often incurs an alignment tax, as fine-tuning on domain-specific tasks can cause catastrophic forgetting and substantia…
Information-Aware KV Cache Compression for Long Reasoning
Jushi Kai, Zhuiri Xiao, Alexandra Birch +1
Reasoning capability has advanced rapidly in large language models (LLMs), leading to an increasing size of key-value (KV) cache in both prefilling and decoding stages. Existing KV…
Depth-Attention: Cross-Layer Value Mixing for Language Models
Boyi Zeng, Yiqin Hao, Zitong Wang +7
Self-attention selects information freely across the sequence, but across depth, Transformers merely add each layer's output to the residual stream, so later layers cannot selectiv…
FlowLM: Few-Step Language Modeling via Diffusion-to-Flow Adaptation
Runzhe Zhang, Letian Chen, Wenpeng Zhang +2
We present FlowLM, a flow matching language model transformed from pre-trained diffusion language models via efficient fine-tuning. By re-aligning the curved sampling trajectories…
Mitigating Premature Discretization with Progressive Quantization for Robust Vector Tokenization
Wenhao Zhao, Qiran Zou, Zhouhan Lin +1
Vector Quantization (VQ) has become the cornerstone of tokenization for many multimodal Large Language Models and diffusion synthesis. However, existing VQ paradigms suffer from a…
Early Quantization Shrinks Codebook: A Simple Fix for Diversity-Preserving Tokenization
Wenhao Zhao, Qiran Zou, Rushi Shah +3
Vector quantization is a technique in machine learning that discretizes continuous representations into a set of discrete vectors. It is widely employed in tokenizing data represen…
VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
Yixuan Wang, Qingyu Shi, Jiayu Zhou +3
The growing context length of Large Language Models (LLMs) enlarges the Key-Value (KV) cache, limiting deployment in resource-limited environments. Prior training-free approaches f…
AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth
Shixiang Song, He Li, Zitong Wang +6
Test-time scaling via recurrent/iterative Transformers enables large language models to spend more computation at inference, but most pretrained recurrent LMs run a fixed number of…
PonderLM-3: Adaptive Token-Wise Pondering with Differentiable Masking
He Li, Feichen Song, Boyi Zeng +4
Test-time scaling has shown that allocating more additional computation at inference can improve generation quality, motivating a natural follow-up question: where should this comp…
CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
Junzhe Shen, Jieru Zhao, Ziwei He +1
We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled…
Controlling Exploration-Exploitation in GFlowNets via Markov Chain Perspectives
Lin Chen, Samuel Drapeau, Fanghao Shao +5
Generative Flow Network (GFlowNet) objectives implicitly fix an equal mixing of forward and backward policies, potentially constraining the exploration-exploitation trade-off durin…
Towards Compressive and Scalable Recurrent Memory
Yunchong Song, Jushi Kai, Liming Lu +2
Transformers face a quadratic bottleneck in attention when scaling to long contexts. Recent approaches introduce recurrent memory to extend context beyond the current window, yet t…
Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
Yuliang Liu, Yunchong Song, Yixuan Wang +6
We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, t…
Context-level Language Modeling by Learning Predictive Context Embeddings
Beiya Dai, Yuliang Liu, Daozheng Xue +6
We propose ContextLM, a framework that implicitly learns multi-token prediction by augmenting standard pretraining with an intrinsic next-context prediction objective. ContextLM bu…
FlowRL: Matching Reward Distributions for LLM Reasoning
Xuekai Zhu, Daixuan Cheng, Dinghuai Zhang +20
We propose FlowRL: matching the full reward distribution via flow balancing instead of maximizing rewards in large language model (LLM) reinforcement learning (RL). Recent advanced…
Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
Jiaqi Cao, Jiarui Wang, Rubin Wei +4
Large Language Models (LLMs) have shown strong abilities in general language tasks, yet adapting them to specific domains remains a challenge. Current method like Domain Adaptive P…
Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models
Huanyu Wang, Jushi Kai, Haoli Bai +4
Vision-Language Models (VLMs) incur substantial computational overhead and inference latency due to the large number of vision tokens introduced by high-resolution image and video…
MLP Memory: A Retriever-Pretrained Memory for Large Language Models
Rubin Wei, Jiaqi Cao, Jiarui Wang +4
Modern approaches to enhancing Large Language Models' factual accuracy and knowledge utilization face a fundamental trade-off: non-parametric retrieval-augmented generation (RAG) p…
How to Synthesize Text Data without Model Collapse?
Xuekai Zhu, Daixuan Cheng, Hengli Li +7
Model collapse in synthetic data indicates that iterative training on self-generated data leads to a gradual decline in performance. With the proliferation of AI models, synthetic…
Critical Data Size of Language Models from a Grokking Perspective
Xuekai Zhu, Yao Fu, Bowen Zhou +1
We explore the critical data size in language models, a threshold that marks a fundamental shift from quick memorization to slow generalization. We formalize the phase transition u…