2 citations · 3 across the 11 of their papers we have counts for
13 papers
Low-Bit N-Gram Embeddings for Language Models
Stephen Chung, Wenyu Du
Many recent open-weight language models use n-gram embedding tables to improve performance. These tables store learned representations of short token sequences, such as pairs or tr…
Autonomous Mathematical Discovery in an Open-World Multi-Agent Environment
Stephen Chung, Wenyu Du, William J. Wesley
We study autonomous mathematical discovery in the Station, an open-world multi-agent environment in which AI agents from different model families pursue a shared research goal with…
GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22
Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional cod…
Cut Your Losses! Learning to Prune Paths Early for Efficient Parallel Reasoning
Jiaxi Bi, Tongxu Luo, Wenyu Du +2
Parallel reasoning enhances Large Reasoning Models (LRMs) but incurs prohibitive costs due to futile paths caused by early errors. To mitigate this, path pruning at the prefix leve…
The Station: An Open-World Environment for AI-Driven Discovery
Stephen Chung, Wenyu Du
We introduce the STATION, an open-world multi-agent environment for autonomous scientific discovery. The Station simulates a complete scientific ecosystem, where agents can engage…
Learning from Peers in Reasoning Models
Tongxu Luo, Wenyu Du, Jiaxi Bi +5
Large Reasoning Models (LRMs) have the ability to self-correct even when they make mistakes in their reasoning paths. However, our study reveals that when the reasoning process sta…
Thinker: Learning to Think Fast and Slow
Stephen Chung, Wenyu Du, Jie Fu
Recent studies show that the reasoning capabilities of Large Language Models (LLMs) can be improved by applying Reinforcement Learning (RL) to question-answering (QA) tasks in area…
Learning from Failures in Multi-Attempt Reinforcement Learning
Stephen Chung, Wenyu Du, Jie Fu
Recent advancements in reinforcement learning (RL) for large language models (LLMs), exemplified by DeepSeek R1, have shown that even a simple question-answering task can substanti…
Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Tracking
Yifan Zhang, Wenyu Du, Dongming Jin +2
Chain-of-thought (CoT) significantly enhances the performance of large language models (LLMs) across a wide range of tasks, and prior research shows that CoT can theoretically incr…
Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysis
Xu Wang, Yan Hu, Wenyu Du +3
Fine-tuning significantly improves the performance of Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. This paper aims to provide an in-depth i…
Unlocking Continual Learning Abilities in Language Models
Wenyu Du, Shuang Cheng, Tongxu Luo +5
Language models (LMs) exhibit impressive performance and generalization capabilities. However, LMs struggle with the persistent challenge of catastrophic forgetting, which undermin…
Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
Wenyu Du, Tongxu Luo, Zihan Qiu +5
LLMs are computationally expensive to pre-train due to their large scale. Model growth emerges as a promising approach by leveraging smaller models to accelerate the training of la…
m2mKD: Module-to-Module Knowledge Distillation for Modular Transformers
Ka Man Lo, Yiming Liang, Wenyu Du +5
Modular neural architectures are gaining attention for their powerful generalization and efficient adaptation to new domains. However, training these models poses challenges due to…