Showing cs.CLShow all
3 papers · 1 filter
cs.CL2025
Learning to Focus: Focal Attention for Selective and Scalable Transformers
Dhananjay Ram, Wei Xia, Stefano Soatto
Attention is a core component of transformer architecture, whether encoder-only, decoder-only, or encoder-decoder model. However, the standard softmax attention often produces nois…
cs.CL2024
HLAT: High-quality Large Language Model Pre-trained on AWS Trainium
Haozheng Fan, Hao Zhou, Guangtai Huang +6
Getting large language models (LLMs) to perform well on the downstream tasks requires pre-training over trillions of tokens. This typically demands a large number of powerful compu…
cs.CL2024
DEM: Distribution Edited Model for Training with Mixed Data Distributions
Dhananjay Ram, Aditya Rawal, Momchil Hardalov +2
Training with mixed data distributions is a common and important part of creating multi-task and instruction-following models. The diversity of the data distributions and cost of j…