15 citations · 16 across the 4 of their papers we have counts for
7 papers
AdamPM: Memory-Efficient LLM Optimization via Moment Pooling
Xiaowei Ji, Lishan Xu, Jianing An +3
Adam is widely used for training large language models (LLMs) owing to its strong optimization performance and training stability, but its element-wise estimation for the first and…
Activation-Adjacent Layer Normalization: Effects on Approximation Power
Yunhao Ni, Yuxin Guo, Jie Luo +2
Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore…
Kolmogorov-Arnold Networks: Curse of Dimensionality and Structural Bias
Yunhao Ni, Yuxin Guo, Jie Luo +2
Kolmogorov-Arnold networks (KANs) have a dimension-independent approximation exponent when the target admits a sufficiently smooth representation by univariate edge functions. We a…
Clustering Properties of Self-Supervised Learning
Xi Weng, Jianing An, Xudong Ma +5
Self-supervised learning (SSL) methods via joint embedding architectures have proven remarkably effective at capturing semantically rich representations with strong clustering prop…
On the Nonlinearity of Layer Normalization
Yunhao Ni, Yuxin Guo, Junlong Jia +1
Layer normalization (LN) is a ubiquitous technique in deep learning but our theoretical understanding to it remains elusive. This paper investigates a new theoretical direction for…
TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models
Junlong Jia, Ying Hu, Xi Weng +8
We present TinyLLaVA Factory, an open-source modular codebase for small-scale large multimodal models (LMMs) with a focus on simplicity of code implementations, extensibility of ne…
TinyLLaVA: A Framework of Small-scale Large Multimodal Models
Baichuan Zhou, Ying Hu, Xi Weng +5
We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects o…