424 citations · 855 across the 20 of their papers we have counts for
21 papers
To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training
Meghana Madhyastha, Daniel Haziza, Jesse Cai +3
Trainings of Large Language Models are generally bottlenecked by matrix multiplications. In the Transformer architecture, a large portion of these operations happens in the Feed Fo…
Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
Feiyang Kang, Newsha Ardalani, Michael Kuchnik +7
Training data plays a crucial role in Large Language Models (LLM) scaling, yet high quality data is of limited supply. Synthetic data techniques offer a potential path toward sides…
Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
Feiyang Kang, Michael Kuchnik, Karthik Padthe +4
In post-training for reasoning Large Language Models (LLMs), the current state of practice trains LLMs in two independent stages: Supervised Fine-Tuning (SFT) and Reinforcement Lea…
Composer: A Search Framework for Hybrid Neural Architecture Design
Bilge Acun, Prasoon Sinha, Newsha Ardalani +7
Hybrid model architectures that combine computational primitives (e.g., Attention, MLP) in different ratios have shown promising performance beyond Transformers. Some studies have…
CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization
Irene Wang, Newsha Ardalani, Mostafa Elhoushi +6
Machine learning solutions are rapidly adopted to enable a variety of key use cases, from conversational AI assistants to scientific discovery. This growing adoption is expected to…
Text Quality-Based Pruning for Efficient Training of Language Models
Vasu Sharma, Karthik Padthe, Newsha Ardalani +8
In recent times training Language Models (LMs) have relied on computationally heavy training over massive datasets which makes this training process extremely laborious. In this pa…
Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Data
Yu Yang, Aaditya K. Singh, Mostafa Elhoushi +7
Code datasets, often collected from diverse and uncontrolled sources such as GitHub, potentially suffer from quality issues, thereby affecting the performance and training efficien…
Data Acquisition: A New Frontier in Data-centric AI
Lingjiao Chen, Bilge Acun, Newsha Ardalani +8
As Machine Learning (ML) systems continue to grow, the demand for relevant and comprehensive datasets becomes imperative. There is limited study on the challenges of data acquisiti…
MAD Max Beyond Single-Node: Enabling Large Machine Learning Model Acceleration on Distributed Systems
Samuel Hsia, Alicia Golden, Bilge Acun +5
Training and deploying large-scale machine learning models is time-consuming, requires significant distributed computing infrastructures, and incurs high operational costs. Our ana…
Sieve: Multimodal Dataset Pruning Using Image Captioning Models
Anas Mahmoud, Mostafa Elhoushi, Amro Abbas +4
Vision-Language Models (VLMs) are pretrained on large, diverse, and noisy web-crawled datasets. This underscores the critical need for dataset pruning, as the quality of these data…
Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference
Haiyang Huang, Newsha Ardalani, Anna Sun +6
Mixture-of-Experts (MoE) models have gained popularity in achieving state-of-the-art performance in a wide range of tasks in computer vision and natural language processing. They e…
MP-Rec: Hardware-Software Co-Design to Enable Multi-Path Recommendation
Samuel Hsia, Udit Gupta, Bilge Acun +5
Deep learning recommendation systems serve personalized content under diverse tail-latency targets and input-query loads. In order to do so, state-of-the-art recommendation models…
DeepFlow: A Cross-Stack Pathfinding Framework for Distributed AI Systems
Newsha Ardalani, Saptadeep Pal, Puneet Gupta
Over the past decade, machine learning model complexity has grown at an extraordinary rate, as has the scale of the systems training such large models. However there is an alarming…
Understanding Scaling Laws for Recommendation Models
Newsha Ardalani, Carole-Jean Wu, Zeliang Chen +2
Scale has been a major driving force in improving machine learning performance, and understanding scaling laws is essential for strategic planning for a sustainable model quality p…
DataPerf: Benchmarks for Data-Centric AI Development
Mark Mazumder, Colby Banbury, Xiaozhe Yao +42
Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithf…
Time Dependency, Data Flow, and Competitive Advantage
Ehsan Valavi, Joel Hestness, Marco Iansiti +3
Data is fundamental to machine learning-based products and services and is considered strategic due to its externalities for businesses, governments, non-profits, and more generall…
Time and the Value of Data
Ehsan Valavi, Joel Hestness, Newsha Ardalani +1
Managers often believe that collecting more data will continually improve the accuracy of their machine learning models. However, we argue in this paper that when data lose relevan…
Sustainable AI: Environmental Implications, Challenges and Opportunities
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta +22
This paper explores the environmental impact of the super-linear growth trends for AI from a holistic perspective, spanning Data, Algorithms, and System Hardware. We characterize t…
Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
Joel Hestness, Newsha Ardalani, Greg Diamos
Deep learning (DL) research yields accuracy and product improvements from both model architecture changes and scale: larger data sets and models, and more computation. For hardware…
A Static Analysis-based Cross-Architecture Performance Prediction Using Machine Learning
Newsha Ardalani, Urmish Thakker, Aws Albarghouthi +1
Porting code from CPU to GPU is costly and time-consuming; Unless much time is invested in development and optimization, it is not obvious, a priori, how much speed-up is achievabl…
Deep Learning Scaling is Predictable, Empirically
Joel Hestness, Sharan Narang, Newsha Ardalani +6
Deep learning (DL) creates impactful advances following a virtuous recipe: model architecture search, creating large training data sets, and scaling computation. It is widely belie…