5 papers
KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta
Gang Liao, Hongsen Qin, Ying Wang +36
Making deep learning recommendation model (DLRM) training and inference fast and efficient is important. However, this presents three key system challenges - model architecture div…
Reflections and New Directions for Human-Centered Large Language Models
Caleb Ziems, Dora Zhao, Rose E. Wang +55
Large Language Models (LLMs) are increasingly shaping the private and professional lives of users, with numerous applications in business, education, finance, healthcare, law, and…
Scaling Context Requires Rethinking Attention
Carles Gelada, Jacob Buckman, Sean Zhang +1
We argue that neither transformers nor sub-quadratic architectures are well suited to training at long sequence lengths: the cost of processing the context is too expensive in the…
Conformal Transformations for Symmetric Power Transformers
Saurabh Kumar, Jacob Buckman, Carles Gelada +1
Transformers with linear attention offer significant computational advantages over softmax-based transformers but often suffer from degraded performance. The symmetric power (sympo…
Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers
Melanie Subbiah, Sean Zhang, Lydia B. Chilton +1
We evaluate recent Large Language Models (LLMs) on the challenging task of summarizing short stories, which can be lengthy, and include nuanced subtext or scrambled timelines. Impo…