4 papers
On the Spectral Flattening of Quantized Embeddings
Junlin Huang, Wenyi Fang, Zhenheng Tang +5
Training Large Language Models (LLMs) at ultra-low precision is critically impeded by instability rooted in the conflict between discrete quantization constraints and the intrinsic…
Probing neutrino-night-quark effective scalar interactions from neutrino masses
Feng-Zhi Chen, Junlin Huang, Fanrong Xu
In this work, we use neutrino masses as a probe of the neutrino-light-quark effective scalar interactions. It is found that neutrinos can acquire masses not only from the usual lig…
DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization
Zhenheng Tang, Zichen Tang, Junlin Huang +7
The growth of large language models (LLMs) increases challenges of accelerating distributed training across multiple GPUs in different data centers. Moreover, concerns about data p…
Spatio-Temporal Field Neural Networks for Air Quality Inference
Yutong Feng, Qiongyan Wang, Yutong Xia +3
The air quality inference problem aims to utilize historical data from a limited number of observation sites to infer the air quality index at an unknown location. Considering the…