6 citations · 10 across the 21 of their papers we have counts for
14 papers · 1 filter
BitNet Text Embeddings
Zhen Li, Xin Huang, Liang Wang +8
LLM-based text embedders have substantially improved retrieval and semantic representation quality, but their deployment remains costly: large backbone models slow down embedding i…
On-Policy Context Distillation for Language Models
Tianzhu Ye, Li Dong, Xun Wu +2
Context distillation enables language models to internalize in-context knowledge into their parameters. In our work, we propose On-Policy Context Distillation (OPCD), a framework t…
Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
Di Zhang, Xun Wu, Shaohan Huang +9
Semi-structured N:M sparsity and low-bit quantization (e.g., 1.58-bit BitNet) are two promising approaches for improving the efficiency of large language models (LLMs), yet they ha…
Online Experiential Learning for Language Models
Tianzhu Ye, Li Dong, Qingxiu Dong +3
The prevailing paradigm for improving large language models relies on offline training with human annotations or simulated environments, leaving the rich experience accumulated dur…
Black-Box On-Policy Distillation of Large Language Models
Tianzhu Ye, Li Dong, Zewen Chi +3
Black-box distillation creates student large language models (LLMs) by learning from a proprietary teacher model's text outputs alone, without access to its internal logits or para…
Code Aesthetics with Agentic Reward Feedback
Bang Xiao, Lingjie Jiang, Shaohan Huang +5
Large Language Models (LLMs) have become valuable assistants for developers in code-related tasks. While LLMs excel at traditional programming tasks such as code generation and bug…
VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models
Lingjie Jiang, Shaohan Huang, Xun Wu +3
Multimodal large language models (MLLMs) have significantly advanced the integration of visual and textual understanding. However, their ability to generate code from multimodal in…
Geometric-Mean Policy Optimization
Yuzhong Zhao, Yue Liu, Junpeng Liu +9
Group Relative Policy Optimization (GRPO) has significantly enhanced the reasoning capability of large language models by optimizing the arithmetic mean of token-level rewards. Unf…
Think Only When You Need with Large Hybrid-Reasoning Models
Lingjie Jiang, Xun Wu, Shaohan Huang +7
Recent Large Reasoning Models (LRMs) have shown substantially improved reasoning capabilities over traditional Large Language Models (LLMs) by incorporating extended thinking proce…
Reward Reasoning Model
Jiaxin Guo, Zewen Chi, Li Dong +4
Reward models play a critical role in guiding large language models toward outputs that align with human expectations. However, an open challenge remains in effectively utilizing t…
MH-MoE: Multi-Head Mixture-of-Experts
Shaohan Huang, Xun Wu, Shuming Ma +1
Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by using the multi-head mechanism to collectively attend to information from various representation spaces…
Textual Aesthetics in Large Language Models
Lingjie Jiang, Shaohan Huang, Xun Wu +1
Image aesthetics is a crucial metric in the field of image generation. However, textual aesthetics has not been sufficiently explored. With the widespread application of large lang…
Multi-Head Mixture-of-Experts
Xun Wu, Shaohan Huang, Wenhui Wang +1
Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activat…
Mixture of LoRA Experts
Xun Wu, Shaohan Huang, Furu Wei
LoRA has gained widespread acceptance in the fine-tuning of large pre-trained models to cater to a diverse array of downstream tasks, showcasing notable effectiveness and efficienc…