Efficient Continual Pre-training by Mitigating the Stability Gap

Yiduo Guo, Jie Fu, Huishuai Zhang +2

CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

Junyan Li, Delin Chen, Yining Hong +4

Ordered Memory

Yikang Shen, Shawn Tan, Arian Hosseini +3

Self-organized Hierarchical Softmax

Yikang Shen, Shawn Tan, Chrisopher Pal +1

Hyper-Decision Transformer for Efficient Online Policy Adaptation

Mengdi Xu, Yuchen Lu, Yikang Shen +3

LaMAGIC: Language-Model-based Topology Generation for Analog Integrated Circuits

Chen-Chia Chang, Yikang Shen, Shaoze Fan +5

Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models

Xinyi Wang, Shawn Tan, Shenbo Xu +4

GraphText: Graph Reasoning in Text Space

Jianan Zhao, Le Zhuo, Yikang Shen +5

Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners

Zitian Chen, Yikang Shen, Mingyu Ding +4

Planning with Large Language Models for Code Generation

Shun Zhang, Zhenfang Chen, Yikang Shen +3

Neural Language Modeling by Jointly Learning Syntax and Lexicon

Yikang Shen, Zhouhan Lin, Chin-Wei Huang +1

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models

Bowen Pan, Yikang Shen, Haokun Liu +5

Transformer-Patcher: One Mistake worth One Neuron

Zeyu Huang, Yikang Shen, Xiaofeng Zhang +3

TextPSG: Panoptic Scene Graph Generation from Textual Descriptions

Chengyang Zhao, Yikang Shen, Zhenfang Chen +2

The infrastructure powering IBM's Gen AI model development

Talia Gershon, Seetharami Seelam, Brian Belgodere +143

Investigating Biases in Textual Entailment Datasets

Shawn Tan, Yikang Shen, Chin-wei Huang +1

Generating Contradictory, Neutral, and Entailing Sentences

Yikang Shen, Shawn Tan, Chin-Wei Huang +1

Synthetic Data RL: Task Definition Is All You Need

Yiduo Guo, Zhen Guo, Chuanwei Huang +5

Structured Code Representations Enable Data-Efficient Adaptation of Code Language Models

Mayank Agarwal, Yikang Shen, Bailin Wang +2

Aligning Large Multimodal Models with Factually Augmented RLHF

Zhiqing Sun, Sheng Shen, Shengcao Cao +9

Parallelizing Linear Transformers with the Delta Rule over Sequence Length

Songlin Yang, Bailin Wang, Yu Zhang +2

Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization

Wei Liu, Anweshit Panda, Ujwal Pandey +6

An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training

Zitian Chen, Mingyu Ding, Yikang Shen +4

FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference

Aniruddha Nrusimha, William Brandon, Mayank Mishra +4

The Consensus Game: Language Model Generation via Equilibrium Search

Athul Paul Jacob, Yikang Shen, Gabriele Farina +1

Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

Zhiqing Sun, Longhui Yu, Yikang Shen +4

LaMAGIC2: Advanced Circuit Formulations for Language Model-Based Analog Topology Generation

Chen-Chia Chang, Wan-Hsuan Lin, Yikang Shen +2

Getting the Agent to Wait

Maryam Saeedi, Yikang Shen, Ali Shourideh

Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks

Yikang Shen, Shawn Tan, Alessandro Sordoni +1

Scaling Granite Code Models to 128K Context

Matt Stallone, Vaibhav Saxena, Leonid Karlinsky +19

Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention

Mingyu Ding, Yikang Shen, Lijie Fan +5

Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach

Wenyu Du, Zhouhan Lin, Yikang Shen +3

StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling

Yikang Shen, Yi Tay, Che Zheng +3

Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler

Yikang Shen, Matthew Stallone, Mayank Mishra +6

Straight to the Tree: Constituency Parsing with Neural Syntactic Distance

Yikang Shen, Zhouhan Lin, Athul Paul Jacob +3

Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle

Yikang Shen, Shawn Tan, Alessandro Sordoni +2

PaTH Attention: Position Encoding via Accumulating Householder Transformations

Songlin Yang, Yikang Shen, Kaiyue Wen +5

Learning Task Decomposition with Ordered Memory Policy Network

Yuchen Lu, Yikang Shen, Siyuan Zhou +3

Scattered Mixture-of-Experts Implementation

Shawn Tan, Yikang Shen, Rameswar Panda +1

JetMoE: Reaching Llama2 Performance with 0.1M Dollars

Yikang Shen, Zhen Guo, Tianle Cai +1

EXPLORE: Exploration with Guided Search for Analog Topology Generation using Language Models

Guanglei Zhou, Chen-Chia Chang, Yikang Shen +5

The paper introduces EXPLORE, a framework that combines transformer-based language model decoding with simulator-guided Monte Carlo Tree Search to improve the generation of analog…

Prompting Decision Transformer for Few-Shot Policy Generalization

Mengdi Xu, Yikang Shen, Shun Zhang +4

DISCA: A Digital In-memory Stochastic Computing Architecture Using A Compressed Bent-Pyramid Format

Shady Agwa, Yikang Shen, Shiwei Wang +1

Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study

Shawn Tan, Songlin Yang, Aaron Courville +2

Sparse Universal Transformer

Shawn Tan, Yikang Shen, Zhenfang Chen +2

API Pack: A Massive Multi-Programming Language Dataset for API Call Generation

Zhen Guo, Adriana Meza Soria, Wei Sun +2

Recursive Top-Down Production for Sentence Generation with Latent Trees

Shawn Tan, Yikang Shen, Timothy J. O'Donnell +2

Word Embedding based Correlation Model for Question/Answer Matching

Yikang Shen, Wenge Rong, Nan Jiang +3

Adaptive Online Replanning with Diffusion Models

Siyuan Zhou, Yilun Du, Shun Zhang +5

Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

Wenyu Du, Tongxu Luo, Zihan Qiu +5

ModuleFormer: Modularity Emerges from Mixture-of-Experts

Yikang Shen, Zheyu Zhang, Tianyou Cao +3

See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning

Zhenfang Chen, Qinhong Zhou, Yikang Shen +3

Syntactic Inductive Biases for Deep Learning Methods

Yikang Shen

FlexAttention for Efficient High-Resolution Vision-Language Models

Junyan Li, Delin Chen, Tianle Cai +5

Granite Code Models: A Family of Open Foundation Models for Code Intelligence

Mayank Mishra, Matt Stallone, Gaoyuan Zhang +43

Autonomous Tree-search Ability of Large Language Models

Zheyu Zhang, Zhuorui Ye, Yikang Shen +1

Mixture of Attention Heads: Selecting Attention Heads Per Token

Xiaofeng Zhang, Yikang Shen, Zeyu Huang +3

Octo-planner: On-device Language Model for Planner-Action Agents

Wei Chen, Zhiyuan Li, Zhen Guo +1

Gated Linear Attention Transformers with Hardware-Efficient Training

Songlin Yang, Bailin Wang, Yikang Shen +2

Diversity Measurement and Subset Selection for Instruction Tuning Datasets

Peiqi Wang, Yikang Shen, Zhen Guo +4

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

Shun Zhang, Zhenfang Chen, Sunli Chen +3

Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

Zhiqing Sun, Yikang Shen, Qinhong Zhou +5

SALMON: Self-Alignment with Instructable Reward Models

Zhiqing Sun, Yikang Shen, Hongxin Zhang +5

BanditSum: Extractive Summarization as a Contextual Bandit

Yue Dong, Yikang Shen, Eric Crawford +2

Long Range Arena: A Benchmark for Efficient Transformers

Yi Tay, Mostafa Dehghani, Samira Abnar +7