10 citations · 26 across the 15 of their papers we have counts for
4 papers · 1 filter
PaTH Attention: Position Encoding via Accumulating Householder Transformations
Songlin Yang, Yikang Shen, Kaiyue Wen +5
The attention mechanism is a core primitive in modern large language models (LLMs) and AI more broadly. Since attention by itself is permutation-invariant, position encoding is ess…
Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler
Yikang Shen, Matthew Stallone, Mayank Mishra +6
Finding the optimal learning rate for language model pretraining is a challenging task. This is not only because there is a complicated correlation between learning rate, batch siz…
API Pack: A Massive Multi-Programming Language Dataset for API Call Generation
Zhen Guo, Adriana Meza Soria, Wei Sun +2
We introduce API Pack, a massive multi-programming language dataset containing over one million instruction-API calls for improving the API call generation capabilities of large la…
Structured Code Representations Enable Data-Efficient Adaptation of Code Language Models
Mayank Agarwal, Yikang Shen, Bailin Wang +2
Current language models tailored for code tasks often adopt the pre-training-then-fine-tuning paradigm from natural language processing, modeling source code as plain text. This ap…