4 papers
MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
Dennis Liu, Zijie Yan, Xin Yao +15
Mixture of Experts (MoE) models enhance neural network scalability by dynamically selecting relevant experts per input token, enabling larger model sizes while maintaining manageab…
DeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Surveys
Guo-Biao Zhang, Ding-Yuan Liu, Da-Yi Wu +5
The rapid development of automated survey generation technology has made it increasingly important to establish a comprehensive benchmark to evaluate the quality of generated surve…
Beyond Label Attention: Transparency in Language Models for Automated Medical Coding via Dictionary Learning
John Wu, David Wu, Jimeng Sun
Medical coding, the translation of unstructured clinical text into standardized medical codes, is a crucial but time-consuming healthcare practice. Though large language models (LL…
DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction
John Wu, David Wu, Jimeng Sun
Predicting high-dimensional or extreme multilabels, such as in medical coding, requires both accuracy and interpretability. Existing works often rely on local interpretability meth…