5 papers
Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers
Tianyi Li, Zhiqiang Shen
Linear mode connectivity (LMC) provides a promising foundation for understanding and merging independently trained neural networks, but existing methods typically optimize the inte…
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
Shengkun Tang, Zekun Wang, Bo Zheng +7
Structured pruning and knowledge distillation (KD) are typical techniques for compressing large language models, but it remains unclear how they should be applied at pretraining sc…
On the Cultural Anachronism and Temporal Reasoning in Vision Language Models
Mukul Ranjan, Prince Jha, Khushboo Kumari +1
Vision-Language Models (VLMs) are increasingly applied to cultural heritage materials, from digital archives to educational platforms. This work identifies a fundamental issue in h…
Initializing Models with Larger Ones
Zhiqiu Xu, Yanjie Chen, Kirill Vishniakov +5
Weight initialization plays an important role in neural network training. Widely used initialization methods are proposed and evaluated for networks that are trained from scratch.…
ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy
Kirill Vishniakov, Zhiqiang Shen, Zhuang Liu
Modern computer vision offers a great variety of models to practitioners, and selecting a model from multiple options for specific applications can be challenging. Conventionally,…