7 papers
ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling
Yiwen Chen, Joshua Ainslie, Krzysztof Choromanski +4
Rotary Position Embedding (RoPE) has been widely adopted in transformer-based large language models. However, its log-linear frequency schedule, originally designed to produce long…
Mixture of Geodesic Factor Analyzers on Riemannian Homogeneous Spaces
Hengchao Chen, Yuanyao Tan, Chao Huang +2
This paper introduces Mixtures of Geodesic Factor Analyzers (MGFA) on Riemannian homogeneous spaces. MGFA uses a geodesic factor model within each mixture component, providing grea…
Distributional Matching for Vector Quantization: A Unified Theoretical and Empirical Framework
Xianghong Fang, Litao Guo, Hengchao Chen +8
The effectiveness of modern visual representation learning and autoregressive models critically depends on vector quantization (VQ), which discretizes continuous feature representa…
CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings
Liuyin Yang, Qiang Sun, Bob Van Dyck +2
Intracranial electrocorticography (ECoG) offers high signal-to-noise access to cortical activity for brain-computer interfaces, yet limited per-patient data has led most prior work…
dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models
Zhengyan Wan, Yidong Ouyang, Panwen Hu +1
Discrete flow models (DFMs) are a class of flexible generative models for generating discrete data, and diffusion large language models (dLLMs) can be viewed as a special case with…
Enhancing Vector Quantization with Distributional Matching: A Theoretical and Empirical Study
Xianghong Fang, Litao Guo, Hengchao Chen +8
The success of autoregressive models largely depends on the effectiveness of vector quantization, a technique that discretizes continuous features by mapping them to the nearest co…
Decentralized Online Riemannian Optimization with Dynamic Environments
Hengchao Chen, Qiang Sun
This paper develops the first decentralized online Riemannian optimization algorithm on Hadamard manifolds. Our algorithm, the decentralized projected Riemannian gradient descent,…