1 citations · 1 across the 8 of their papers we have counts for
11 papers
State of Thought Enables Endogenous Reasoning
Zhiren Gong, Yikun Hou, Zihao Zeng +3
Test-time compute has emerged as a major approach to improving the capabilities of Large Language Models (LLMs). However, existing test-time reasoning paradigms rely heavily on ext…
CoCurve: Cross-Module Co-Pruning Curvature for Structured LLM Pruning
Zhiren Gong, Zihao Zeng, Zijie Wang +6
Resource-constrained deployment requires sustaining large language model (LLM) capabilities as models scale under fixed memory and computation budgets. Structured pruning advances…
Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuits
Zhiren Gong, He Lu, Zihao Zeng +6
Mechanistic interpretability seeks to explain transformer behavior through circuits: sets of internal components that causally support a behavior. However, self-repair creates a bl…
Done Is Better than Perfect: Unlocking Efficient Reasoning by Structured Multi-Turn Decomposition
Zihao Zeng, Xuyao Huang, Boxiu Li +2
Large Reasoning Models (LRMs) are criticized for the excessively lengthy Chain-of-Thought (CoT) to derive the final answer, suffering from high first-token and overall latency. Typ…
Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions
Siqi Kou, Qingyuan Tian, Hanwen Xu +2
Large language models (LLMs) have demonstrated remarkable reasoning capabilities in math and coding, often bolstered by post-training on the chain-of-thoughts (CoTs) generated by s…
RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability
Yichi Zhang, Zihao Zeng, Dongbai Li +3
Large Reasoning Models (LRMs), such as OpenAI o1 and DeepSeek-R1, have been rapidly progressing and achieving breakthrough performance on complex reasoning tasks such as mathematic…
SIFT: Grounding LLM Reasoning in Contexts via Stickers
Zihao Zeng, Xuyao Huang, Boxiu Li +1
This paper identifies the misinterpretation of the context can be a significant issue during the reasoning process of large language models, spanning from smaller models like Llama…
ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
Xin He, Shunkang Zhang, Kaijie Tang +8
Sparse Mixture-of-Experts (MoE) models can outperform dense large language models at similar computation by activating only a small set of experts per token. However, stacking many…
MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection
Bokai Lin, Zihao Zeng, Zipeng Xiao +5
KV cache has become a de facto technique for the inference of large language models (LLMs), where tensors of shape (layer number, head number, sequence length, feature dimension) a…
In-context KV-Cache Eviction for LLMs via Attention-Gate
Zihao Zeng, Bokai Lin, Tianqi Hou +2
The KV-Cache technique has become the standard for the inference of large language models (LLMs). Yet, it is widely criticized that KV-Cache can become a bottleneck of the LLM infe…
AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models
Zihao Zeng, Yibo Miao, Hongcheng Gao +2
Mixture of experts (MoE) has become the standard for constructing production-level large language models (LLMs) due to its promise to boost model capacity without causing significa…