15 citations · 16 across the 5 of their papers we have counts for
8 papers
AdamPM: Memory-Efficient LLM Optimization via Moment Pooling
Xiaowei Ji, Lishan Xu, Jianing An +3
Adam is widely used for training large language models (LLMs) owing to its strong optimization performance and training stability, but its element-wise estimation for the first and…
Activation-Adjacent Layer Normalization: Effects on Approximation Power
Yunhao Ni, Yuxin Guo, Jie Luo +2
Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore…
Kolmogorov-Arnold Networks: Curse of Dimensionality and Structural Bias
Yunhao Ni, Yuxin Guo, Jie Luo +2
Kolmogorov-Arnold networks (KANs) have a dimension-independent approximation exponent when the target admits a sufficiently smooth representation by univariate edge functions. We a…
Enjoy Your Layer Normalization with the Computational Efficiency of RMSNorm
Yuxin Guo, Yihao Yue, Yunhao Ni +4
Layer normalization (LN) is a fundamental component in modern deep learning, but its per-sample centering and scaling introduce non-negligible inference overhead. RMSNorm improves…
VoQA: Visual-only Question Answering
Jianing An, Luyang Jiang, Jie Luo +2
Visual understanding requires interpreting both natural scenes and the textual information that appears within them, motivating tasks such as Visual Question Answering (VQA). Howev…
Clustering Properties of Self-Supervised Learning
Xi Weng, Jianing An, Xudong Ma +5
Self-supervised learning (SSL) methods via joint embedding architectures have proven remarkably effective at capturing semantically rich representations with strong clustering prop…
TinyLLaVA Factory: A Modularized Codebase for Small-scale Large Multimodal Models
Junlong Jia, Ying Hu, Xi Weng +8
We present TinyLLaVA Factory, an open-source modular codebase for small-scale large multimodal models (LMMs) with a focus on simplicity of code implementations, extensibility of ne…
TinyLLaVA: A Framework of Small-scale Large Multimodal Models
Baichuan Zhou, Ying Hu, Xi Weng +5
We present the TinyLLaVA framework that provides a unified perspective in designing and analyzing the small-scale Large Multimodal Models (LMMs). We empirically study the effects o…