3 papers
cs.LG2025
Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
Feiyang Kang, Newsha Ardalani, Michael Kuchnik +7
Training data plays a crucial role in Large Language Models (LLM) scaling, yet high quality data is of limited supply. Synthetic data techniques offer a potential path toward sides…
cs.CL2025
Structure-Aware Fill-in-the-Middle Pretraining for Code
Linyuan Gong, Alvin Cheung, Mostafa Elhoushi +1
Fill-in-the-Middle (FIM) is a common pretraining method for code LLMs, where models complete code segments given surrounding context. However, existing LLMs treat code as plain tex…
cs.LG2025
CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization
Irene Wang, Newsha Ardalani, Mostafa Elhoushi +6
Machine learning solutions are rapidly adopted to enable a variety of key use cases, from conversational AI assistants to scientific discovery. This growing adoption is expected to…