5 papers
ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling
Yiwen Chen, Joshua Ainslie, Krzysztof Choromanski +4
Rotary Position Embedding (RoPE) has been widely adopted in transformer-based large language models. However, its log-linear frequency schedule, originally designed to produce long…
Mixture of Geodesic Factor Analyzers on Riemannian Homogeneous Spaces
Hengchao Chen, Yuanyao Tan, Chao Huang +2
This paper introduces Mixtures of Geodesic Factor Analyzers (MGFA) on Riemannian homogeneous spaces. MGFA uses a geodesic factor model within each mixture component, providing grea…
Distributional Matching for Vector Quantization: A Unified Theoretical and Empirical Framework
Xianghong Fang, Litao Guo, Hengchao Chen +8
The effectiveness of modern visual representation learning and autoregressive models critically depends on vector quantization (VQ), which discretizes continuous feature representa…
Enhancing Vector Quantization with Distributional Matching: A Theoretical and Empirical Study
Xianghong Fang, Litao Guo, Hengchao Chen +8
The success of autoregressive models largely depends on the effectiveness of vector quantization, a technique that discretizes continuous features by mapping them to the nearest co…
Decentralized Online Riemannian Optimization with Dynamic Environments
Hengchao Chen, Qiang Sun
This paper develops the first decentralized online Riemannian optimization algorithm on Hadamard manifolds. Our algorithm, the decentralized projected Riemannian gradient descent,…