8 papers
Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection
Debarshi Kundu, Swaroop Ghosh, Vasant Honavar
Data-adaptive sparse attention masks substantially outperform fixed patterns (e.g., BigBird and Longformer) and can even exceed dense attention on long sequences. Existing adaptive…
Restoring the Sweet Spot: Pass-Rate Weighted Self-Distillation for LLM Reasoning
Zehao Liu, Yuanpu Cao, Jinghui Chen +1
Self-Distillation Policy Optimization (SDPO) provides dense token-level credit assignment for reinforcement learning with large language models by leveraging the model's own feedba…
Projection-Free Transformers via Gaussian Kernel Attention
Debarshi Kundu, Archisman Ghosh, Swaroop Ghosh +1
Self-attention in Transformers is typically implemented as , where , , and are learned linear projections of the input…
Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning
Abhishek Dalvi, Vasant Honavar
Large unimodal foundation models for vision and language encode rich semantic structures, yet aligning them typically requires computationally intensive multimodal fine-tuning. Suc…
Simple Denoising Diffusion Language Models
Huaisheng Zhu, Zhengyu Chen, Shijie Zhou +8
Recent Uniform State Diffusion Models (USDMs), initialized from a uniform prior, offer the promise of fast text generation due to their inherent self-correction ability compared to…
On a Connection Between Imitation Learning and RLHF
Teng Xiao, Yige Yuan, Mingxiao Li +2
This work studies the alignment of large language models with preference data from an imitation learning perspective. We establish a close theoretical connection between reinforcem…