4 papers
RegMix-D: Dynamic Data Mixing via Proxy Training Trajectories
Kaiyan Zhao, Zhongtao Miao, Akiko Aizawa +1
Data mixture selection is critical for Large Language Model pretraining. Existing methods such as RegMix select a single static mixture by fitting a regression model on small-scale…
NeoAMT: Neologism-Aware Agentic Machine Translation with Reinforcement Learning
Zhongtao Miao, Kaiyan Zhao, Masaaki Nagata +1
Neologism-aware machine translation aims to translate source sentences containing neologisms into target languages. This field remains underexplored compared with general machine t…
Direct Quantized Training of Language Models with Stochastic Rounding
Kaiyan Zhao, Tsuguchika Tabaru, Kenichi Kobayashi +3
Although recent quantized Large Language Models (LLMs), such as BitNet, have paved the way for significant reduction in memory usage during deployment with binary or ternary weight…
Improving Multimodal Contrastive Learning of Sentence Embeddings with Object-Phrase Alignment
Kaiyan Zhao, Zhongtao Miao, Yoshimasa Tsuruoka
Multimodal sentence embedding models typically leverage image-caption pairs in addition to textual data during training. However, such pairs often contain noise, including redundan…