6 citations · 21 across the 23 of their papers we have counts for
24 papers
Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Ermo Hua, Xiangyu Hong, Baiting Wu +11
As model parameters scale up, sparsely activated Mixture-of-Experts (MoE) approaches have become essential for model scaling. However, mainstream models currently adopt hierarchica…
Safin-1: Safety from Within through Memory-Native State Evolution
Ming Zhang, Kaisen Yang, Shu Yu +15
Long-horizon complex tasks require foundation models to accumulate information, maintain internal states, and adapt over extended interactions. Safety should be an intrinsic proper…
Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning
Kai Chen, Jifeng Ding, Ning Ding +44
We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve comp…
MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
Ming Zhang, Kaisen Yang, Shu Yu +6
Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length. This flexibility, however, incurs a quadratic comput…
PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memory
Dawei Liu, Haixu Song, Shuang Cheng +9
Long-context reasoning remains a critical bottleneck for large language models, as recent recurrent-memory approaches face two inherent challenges: sequential chunk-wise updates ca…
Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering
Junlin Yang, Che Jiang, Yu Fu +21
Recursive self-improvement (RSI) requires AI systems that improve the process of building AI (i.e., AI4AI); machine learning engineering (MLE) offers a concrete, executable testbed…
A Survey of Reinforcement Learning for Large Reasoning Models
Kaiyan Zhang, Yuxin Zuo, Bingxiang He +36
In this paper, we survey recent advances in Reinforcement Learning (RL) for reasoning with Large Language Models (LLMs). RL has achieved remarkable success in advancing the frontie…
Nirvana: A Specialized Generalist Model With Task-Aware Memory Mechanism
Yuhua Jiang, Shuang Cheng, Yihao Liu +7
Large Language Models (LLMs) excel at general language tasks but struggle in specialized domains. Specialized Generalist Models (SGMs) address this by preserving broad capabilities…
FlowRL: Matching Reward Distributions for LLM Reasoning
Xuekai Zhu, Daixuan Cheng, Dinghuai Zhang +20
We propose FlowRL: matching the full reward distribution via flow balancing instead of maximizing rewards in large language model (LLM) reinforcement learning (RL). Recent advanced…
Automating Exploratory Multiomics Research via Language Models
Shang Qu, Ning Ding, Linhai Xie +13
This paper introduces PROTEUS, a fully automated system that produces data-driven hypotheses from raw data files. We apply PROTEUS to clinical proteogenomics, a field where effecti…
Deep Unfolding with Kernel-based Quantization in MIMO Detection
Zeyi Ren, Jingreng Lei, Yichen Jin +5
The development of edge computing places critical demands on energy-efficient model deployment for multiple-input multiple-output (MIMO) detection tasks. Deploying deep unfolding m…
Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models
Xingtai Lv, Youbang Sun, Kaiyan Zhang +8
State Space Models (SSMs) have emerged as a promising alternative to the popular transformer-based models and have been increasingly gaining attention. Compared to transformers, SS…
Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
Xinwei Long, Zhiyuan Ma, Ermo Hua +3
Retrieval-augmented generation (RAG) has emerged to address the knowledge-intensive visual question answering (VQA) task. Current methods mainly employ separate retrieval and gener…
MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
Yuxin Zuo, Shang Qu, Yifei Li +6
We introduce MedXpertQA, a highly challenging and comprehensive benchmark to evaluate expert-level medical knowledge and advanced reasoning. MedXpertQA includes 4,460 questions spa…
Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization
Ermo Hua, Che Jiang, Xingtai Lv +7
Extending the context length of Language Models (LMs) by improving Rotary Position Embedding (RoPE) has become a trend. While prior works mainly address RoPE's limitations within a…
How to Synthesize Text Data without Model Collapse?
Xuekai Zhu, Daixuan Cheng, Hengli Li +7
Model collapse in synthetic data indicates that iterative training on self-generated data leads to a gradual decline in performance. With the proliferation of AI models, synthetic…
Automating Exploratory Proteomics Research via Language Models
Ning Ding, Shang Qu, Linhai Xie +13
With the development of artificial intelligence, its contribution to science is evolving from simulating a complex problem to automating entire research processes and producing nov…
Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
Xingtai Lv, Ning Ding, Kaiyan Zhang +3
Improving the effectiveness and efficiency of large language models (LLMs) simultaneously is a critical yet challenging research goal. In this paper, we find that low-rank pre-trai…
LFQA-E: Carefully Benchmarking Long-form QA Evaluation
Yuchen Fan, Chen Lin, Xin Zhong +11
Long-Form Question Answering (LFQA) involves generating comprehensive, paragraph-level responses to open-ended questions, which poses a significant challenge for evaluation due to…
Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
Biqing Qi, Kaiyan Zhang, Kai Tian +6
The rapid growth of biomedical knowledge has outpaced our ability to efficiently extract insights and generate novel hypotheses. Large language models (LLMs) have emerged as a prom…
Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding
Kaiyan Zhang, Jianyu Wang, Ning Ding +4
Large Language Models (LLMs) exhibit impressive capabilities across various applications but encounter substantial challenges such as high inference latency, considerable training…
UltraMedical: Building Specialized Generalists in Biomedicine
Kaiyan Zhang, Sihang Zeng, Ermo Hua +11
Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains and are moving towards more specialized areas. Recent advanced proprietary models such…
Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
Ermo Hua, Biqing Qi, Kaiyan Zhang +4
Supervised Fine-Tuning (SFT) and Preference Optimization (PO) are key processes for aligning Language Models (LMs) with human preferences post pre-training. While SFT excels in eff…
CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following
Kaiyan Zhang, Jianyu Wang, Ermo Hua +3
With the advancement of language models (LMs), their exposure to private data is increasingly inevitable, and their deployment (especially for smaller ones) on personal devices, su…