5 papers
TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching
Truong Nguyen, Tien-Phat Nguyen, Linh Ngo Van +3
Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even thou…
Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence
Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3
Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger l…
BSO: Safety Alignment Is Density Ratio Matching
Tien-Phat Nguyen, Truong Nguyen, Thin Nguyen +3
Aligning language models for both helpfulness and safety typically requires complex pipelines-separate reward and cost models, online reinforcement learning, and primal-dual update…
FAID: Fine-Grained AI-Generated Text Detection Using Multi-Task Auxiliary and Multi-Level Contrastive Learning
Minh Ngoc Ta, Dong Cao Van, Duc-Anh Hoang +6
The growing collaboration between humans and AI models in generative tasks has introduced new challenges in distinguishing between human-written, LLM-generated, and human-LLM colla…
CTPD: Cross Tokenizer Preference Distillation
Truong Nguyen, Phi Van Dat, Ngan Nguyen +3
While knowledge distillation has seen widespread use in pre-training and instruction tuning, its application to aligning language models with human preferences remains underexplore…