17 citations · 27 across the 8 of their papers we have counts for
8 papers
Existence of a Hall ray in higher dimensional Lagrange spectra
Yitwah Cheung, Zhijing Wendy Wang, Ruichong Zhang
Given any norm on , we prove that the Lagrange spectrum for -vectors contains a Hall ray. Equivalently, every sufficiently small o…
Matrix-Driven Identification and Reconstruction of LLM Weight Homology
Ruichong Zhang, Daniel Goldstein
We propose Matrix-Driven Identification and Reconstruction (MDIR), a SOTA large language model homology method that accurately detects weight correspondences between models and pro…
DeepMath-Creative: A Benchmark for Evaluating Mathematical Creativity of Large Language Models
Xiaoyang Chen, Xinan Dai, Yu Du +28
To advance the mathematical proficiency of large language models (LLMs), the DeepMath team has launched an open-source initiative aimed at developing an open mathematical LLM and s…
RWKV-7 "Goose" with Expressive Dynamic State Evolution
Bo Peng, Ruichong Zhang, Daniel Goldstein +15
We present RWKV-7 "Goose", a new sequence modeling architecture with constant memory usage and constant inference time per token. Despite being trained on dramatically fewer tokens…
On General 2-dimensional Lattice Spectra: Closedness, Hall's Ray, and Examples
Ruichong Zhang
The Lagrange and Markov spectra have been studied since late 19th century, concerning badly approximable real numbers. The Mordell-Gruber spectrum has been studied since 1936, conc…
Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
Bo Peng, Daniel Goldstein, Quentin Anthony +27
We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) architecture. Our architectural design advancements include multi-headed matrix-value…
Cramer Type Distances for Learning Gaussian Mixture Models by Gradient Descent
Ruichong Zhang
The learning of Gaussian Mixture Models (also referred to simply as GMMs) plays an important role in machine learning. Known for their expressiveness and interpretability, Gaussian…
RWKV: Reinventing RNNs for the Transformer Era
Bo Peng, Eric Alcaide, Quentin Anthony +31
Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence leng…