17 papers
LongGuard: Mechanistic Analysis and Training-Free Mitigation of Long-Context Failure in Safety Guardrails
Ziyang Chen, Xing Wu, Songlin Hu
Safety guardrails serve as the last line of defense against harmful inputs and outputs of large language models (LLMs), yet they are trained and evaluated almost exclusively on sho…
Compute Allocation in Bilinear Contrastive Representation Learning via Scaling Laws
Ziyan Chen, Zhongzhu Zhou, Ding-Xuan Zhou
Efficient reasoning systems increasingly rely on large contrastive representation models for multimodal retrieval, grounding, and decision-making. However, under a limited compute…
Can Broad Biomedical Knowledge be Contextualized into Scenario-Grounded Propositions?
Qingyuan Zeng, Ziyang Chen, Pengxiang Cai +5
Biomedical discovery often requires connecting broad biomedical knowledge with specific experimental or clinical data. Background knowledge suggests relevant mechanisms but is usua…
GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment
Minxuan Lv, Tiehua Mei, Tanlong Du +9
We present GoLongRL, a fully open-source, capability-oriented post-training recipe for long-context reinforcement learning with verifiable rewards (RLVR). Existing long-context RL…
OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization
Zhongzhu Zhou, Donglin Zhuang, Jisen Li +4
INT2 KV-cache quantization is attractive for long-context LLM serving, but it remains difficult to make both accurate and deployable. Simple rotations such as Hadamard transforms r…
Don't Retrain, Just Reuse: Recovering Dual-Target Molecules from Single-Target Diffusion Models
Qingyuan Zeng, Pengxiang Cai, Zixin Guan +4
Designing a single molecule that modulates two targets is a promising strategy for polypharmacology, but it remains substantially harder than standard single-target generation beca…
Scaling Laws for Dynamic Mini-Batch SGD in Sketched Linear Regression
Ziyan Chen, Zhongzhu Zhou, Ding-Xuan Zhou
Mini-batching is central to large-scale optimization, yet its role in statistical scaling laws remains limited. We study one-pass and multi-pass batch SGD for sketched linear regre…
PolicyLong: Towards On-Policy Context Extension
Junlong Jia, Jiang Zhou, Ziyang Chen +5
Extending LLM context windows is hindered by scarce high-quality long-context data. Recent methods synthesize data with genuine long-range dependencies via information-theoretic ve…
On-Policy Supervised Fine-Tuning for Efficient Reasoning
Anhao Zhao, Ziyang Chen, Junlong Tong +6
Large reasoning models (LRMs) are commonly trained with reinforcement learning (RL) to explore long chain-of-thought reasoning, achieving strong performance at high computational c…
From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models
Longwei Ding, Anhao Zhao, Fanghua Ye +2
Large language models (LLMs) are increasingly costly to deploy, motivating extensive research on model pruning. However, most existing studies focus on instruction-following LLMs,…
SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment
Ziyang Chen, Zhenxuan Huang, Yile Wang +3
Traditional sentence embedding methods employ token-level contrastive learning on non-generative pre-trained models. Recently, there have emerged embedding methods based on generat…
LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark
Ziyang Chen, Xing Wu, Junlong Jia +4
The rapid expansion of context length in large language models (LLMs) has outpaced existing evaluation benchmarks. Current long-context benchmarks often trade off scalability and r…
Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach
Chanwoo Park, Ziyang Chen, Asuman Ozdaglar +1
Large language models (LLMs) are increasingly deployed as "agents" for decision-making (DM) in interactive and dynamic environments. Yet, since they were not originally designed fo…
EntropyLong: Effective Long-Context Training via Predictive Uncertainty
Junlong Jia, Ziyang Chen, Xing Wu +5
Training long-context language models to capture long-range dependencies requires specialized data construction. Current approaches, such as generic text concatenation or heuristic…
LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMs
Junlong Jia, Xing Wu, Chaochen Gao +8
High-quality long-context data is essential for training large language models (LLMs) capable of processing extensive documents, yet existing synthesis approaches using relevance-b…
Libra: Large Chinese-based Safeguard for AI Content
Ziyang Chen, Huimu Yu, Xing Wu +2
Large language models (LLMs) excel in text understanding and generation but raise significant safety and ethical concerns in high-stakes applications. To mitigate these risks, we p…
Using Prompts to Guide Large Language Models in Imitating a Real Person's Language Style
Ziyang Chen, Stylios Moscholios
Large language models (LLMs), such as GPT series and Llama series have demonstrated strong capabilities in natural language processing, contextual understanding, and text generatio…