10 citations · 18 across the 15 of their papers we have counts for
9 papers · 1 filter
Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler
Yikang Shen, Matthew Stallone, Mayank Mishra +6
Finding the optimal learning rate for language model pretraining is a challenging task. This is not only because there is a complicated correlation between learning rate, batch siz…
Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models
Sonam Gupta, Yatin Nandwani, Asaf Yehudai +4
Fine-tuning Large Language Models (LLMs) on specific datasets is a common practice to improve performance on target tasks. However, this performance gain often leads to overfitting…
Scaling Granite Code Models to 128K Context
Matt Stallone, Vaibhav Saxena, Leonid Karlinsky +19
This paper introduces long-context Granite code models that support effective context windows of up to 128K tokens. Our solution for scaling context length of Granite 3B/8B code mo…
The infrastructure powering IBM's Gen AI model development
Talia Gershon, Seetharami Seelam, Brian Belgodere +143
AI Infrastructure plays a key role in the speed and cost-competitiveness of developing and deploying advanced AI models. The current demand for powerful AI infrastructure for model…
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
William Brandon, Mayank Mishra, Aniruddha Nrusimha +2
Key-value (KV) caching plays an essential role in accelerating decoding for transformer-based autoregressive large language models (LLMs). However, the amount of memory required to…
Granite Code Models: A Family of Open Foundation Models for Code Intelligence
Mayank Mishra, Matt Stallone, Gaoyuan Zhang +43
Large Language Models (LLMs) trained on code are revolutionizing the software development process. Increasingly, code LLMs are being integrated into software development environmen…
Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
Bowen Pan, Yikang Shen, Haokun Liu +5
Mixture-of-Experts (MoE) language models can reduce computational costs by 2-4 compared to dense models without sacrificing performance, making them more efficient in compu…
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
Aniruddha Nrusimha, Mayank Mishra, Naigang Wang +3
We consider the problem of accurate quantization for language models, where both the weights and activations are uniformly quantized to 4 bits per parameter, the lowest bitwidth fo…
BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback
Gaurav Pandey, Yatin Nandwani, Tahira Naseem +6
Distribution matching methods for language model alignment such as Generation with Distributional Control (GDC) and Distributional Policy Gradient (DPG) have not received the same…