4 citations · 14 across the 19 of their papers we have counts for
20 papers
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
Siyu Ding, Mingchuan Ma, Jiabo Tong +3
Recent NVFP4 pretraining work has primarily optimized Transformer linear projections, leaving persistent optimizer states, optimizer computation, and low-precision attention forwar…
FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Self-Verification
Haoqing Wang, Xingrun Xing, Wei Xia +3
Agentic vision-language models (VLMs), which interleave textual reasoning with explicit tool calls such as cropping and code-based image manipulation, have emerged as a compelling…
Trust Region On-Policy Distillation
Xingrun Xing, Haoqing Wang, Boyan Gao +2
On-Policy Distillation (OPD) is a fundamental technique for efficient post-training of large language models (LLMs), with broad applications in agent learning, multi-task enhanceme…
MemTrain: Self-Supervised Context Memory Training
Ziheng Li, Xingrun Xing, Haoqing Wang +2
Memory is an indispensable capability for long-horizon LLM agents, enabling them to preserve and utilize information accumulated across extended interactions. Existing memory-agent…
Spike-driven Large Language Model
Han Xu, Xuerui Qiu, Baiyu Chen +7
Current Large Language Models (LLMs) are primarily based on large-scale dense matrix multiplications. Inspired by the brain's information processing mechanism, we explore the funda…
PretrainZero: Reinforcement Active Pretraining
Xingrun Xing, Zhiyuan Fan, Jie Lou +3
Mimicking human behavior to actively learning from general experience and achieve artificial general intelligence has always been a human dream. Recent reinforcement learning (RL)…
MedREK: Retrieval-Based Editing for Medical LLMs with Key-Aware Prompts
Shujun Xia, Haokun Lin, Yichen Wu +9
LLMs hold great promise for healthcare applications, but the rapid evolution of medical knowledge and errors in training data often cause them to generate outdated or inaccurate in…
PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference
Hao Zhang, Mengsi Lyu, Zhuo Chen +3
Large Language Models (LLMs) demonstrate exceptional capabilities across various tasks, but their deployment is constrained by high computational and memory costs. Model pruning pr…
Position-Aware Depth Decay Decoding (): Boosting Large Language Model Inference Efficiency
Siqi Fan, Xuezhi Fang, Xingrun Xing +3
Due to the large number of parameters, the inference phase of Large Language Models (LLMs) is resource-intensive. Unlike traditional model compression, which needs retraining, rece…
EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Models
Xingrun Xing, Zheng Liu, Shitao Xiao +6
Modern large language models (LLMs) driven by scaling laws, achieve intelligence emergency in large model sizes. Recently, the increasing concerns about cloud costs, latency, and p…
Enhancing Generalization via Sharpness-Aware Trajectory Matching for Dataset Condensation
Boyan Gao, Bo Zhao, Shreyank N Gowda +4
Dataset condensation aims to synthesize datasets with a few representative samples that can effectively represent the original datasets. This enables efficient training and produce…
Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging
Yiming Ju, Ziyi Ni, Xingrun Xing +4
Supervised fine-tuning (SFT) is crucial for adapting Large Language Models (LLMs) to specific tasks. In this work, we demonstrate that the order of training data can lead to signif…
From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
Wanpeng Zhang, Zilong Xie, Yicheng Feng +4
Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities. We…
OmniGen: Unified Image Generation
Shitao Xiao, Yueze Wang, Junjie Zhou +7
The emergence of Large Language Models (LLMs) has unified language generation tasks and revolutionized human-machine interaction. However, in the realm of image generation, a unifi…
SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
Xingrun Xing, Boyan Gao, Zheng Zhang +5
Recent advancements in large language models (LLMs) with billions of parameters have improved performance in various applications, but their inference processes demand significant…
SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms
Xingrun Xing, Zheng Zhang, Ziyi Ni +6
Towards energy-efficient artificial intelligence similar to the human brain, the bio-inspired spiking neural networks (SNNs) have advantages of biological plausibility, event-drive…
LM-Cocktail: Resilient Tuning of Language Models via Model Merging
Shitao Xiao, Zheng Liu, Peitian Zhang +1
The pre-trained language models are continually fine-tuned to better support downstream applications. However, this operation may result in significant performance degeneration on…
BiPFT: Binary Pre-trained Foundation Transformer with Low-rank Estimation of Binarization Residual Polynomials
Xingrun Xing, Li Du, Xinyuan Wang +4
Pretrained foundation models offer substantial benefits for a wide range of downstream tasks, which can be one of the most potential techniques to access artificial general intelli…
Quantifying and Attributing the Hallucination of Large Language Models via Association Analysis
Li Du, Yequan Wang, Xingrun Xing +4
Although demonstrating superb performance on various NLP tasks, large language models (LLMs) still suffer from the hallucination problem, which threatens the reliability of LLMs. T…
KLoB: a Benchmark for Assessing Knowledge Locating Methods in Language Models
Yiming Ju, Xingrun Xing, Zhixiong Zeng
Recently, Locate-Then-Edit paradigm has emerged as one of the main approaches in changing factual knowledge stored in the Language models. However, there is a lack of research on w…