9 papers
Dual Attention Residuals
Xingda Yu, Yining Li, Xinzhang Liu +5
Recent work extends Transformer residual pathways along two complementary axes: historical retrieval selects information from earlier depths, whereas multi-stream methods maintain…
Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization
Mingkuan Zhao, Wentao Hu, Tianchen Huang +6
Hallucination in Large Language Models (LLMs), characterized by the generation of content inconsistent with contextual facts or logical constraints -- remains a persistent challeng…
Stop Rewarding Hallucinated Steps: Faithfulness-Aware Step-Level Reinforcement Learning for Small Reasoning Models
Shuo Nie, Hexuan Deng, Chao Wang +6
As large language models become smaller and more efficient, small reasoning models (SRMs) are crucial for enabling chain-of-thought (CoT) reasoning in resource-constrained settings…
Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations
Wentao Hu, Yanbo Zhai, Xiaohui Hu +6
Sparse Mixture-of-Experts (MoE) models have achieved remarkable scalability, yet they remain vulnerable to hallucinations, particularly when processing long-tail knowledge. We iden…
D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximation
Junlin Li, Shuangyong Song, Guodong Du +6
Supervised Fine-Tuning (SFT) accelerates taskspecific large language models (LLMs) development, but the resulting proliferation of finetuned models incurs substantial memory overhe…
UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignment
Hongyan Xie, Yikun Ban, Ruiyu Fang +6
Multi-objective alignment aims to align LLM responses with multiple human preference objectives. Among existing methods, guiding the generation of frozen LLMs through autoregressiv…