195 citations · 350 across the 23 of their papers we have counts for
23 papers
Sparse Universal Transformer
Shawn Tan, Yikang Shen, Zhenfang Chen +2
The Universal Transformer (UT) is a variant of the Transformer that shares parameters across its layers. Empirical evidence shows that UTs have better compositional generalization…
TextPSG: Panoptic Scene Graph Generation from Textual Descriptions
Chengyang Zhao, Yikang Shen, Zhenfang Chen +2
Panoptic Scene Graph has recently been proposed for comprehensive scene understanding. However, previous works adopt a fully-supervised learning manner, requiring large amounts of…
SALMON: Self-Alignment with Instructable Reward Models
Zhiqing Sun, Yikang Shen, Hongxin Zhang +5
Supervised Fine-Tuning (SFT) on response demonstrations combined with Reinforcement Learning from Human Feedback (RLHF) constitutes a powerful paradigm for aligning LLM-based AI ag…
Aligning Large Multimodal Models with Factually Augmented RLHF
Zhiqing Sun, Sheng Shen, Shengcao Cao +9
Large Multimodal Models (LMM) are built across modalities and the misalignment between two modalities can result in "hallucination", generating textual outputs that are not grounde…
An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training
Zitian Chen, Mingyu Ding, Yikang Shen +4
We present a model that can perform multiple vision tasks and can be adapted to other downstream tasks efficiently. Despite considerable progress in multi-task learning, most effor…
ModuleFormer: Modularity Emerges from Mixture-of-Experts
Yikang Shen, Zheyu Zhang, Tianyou Cao +3
Large Language Models (LLMs) have achieved remarkable results. However, existing models are expensive to train and deploy, and it is also difficult to expand their knowledge beyond…
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
Zhiqing Sun, Yikang Shen, Qinhong Zhou +5
Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to alig…
Hyper-Decision Transformer for Efficient Online Policy Adaptation
Mengdi Xu, Yuchen Lu, Yikang Shen +3
Decision Transformers (DT) have demonstrated strong performances in offline reinforcement learning settings, but quickly adapting to unseen novel tasks remains challenging. To addr…
Visual Dependency Transformers: Dependency Tree Emerges from Reversed Attention
Mingyu Ding, Yikang Shen, Lijie Fan +5
Humans possess a versatile mechanism for extracting structured representations of our visual world. When looking at an image, we can decompose the scene into entities and their par…
Planning with Large Language Models for Code Generation
Shun Zhang, Zhenfang Chen, Yikang Shen +3
Existing large language model-based code generation pipelines typically use beam search or sampling algorithms during the decoding process. Although the programs they generate achi…
Transformer-Patcher: One Mistake worth One Neuron
Zeyu Huang, Yikang Shen, Xiaofeng Zhang +3
Large Transformer-based Pretrained Language Models (PLMs) dominate almost all Natural Language Processing (NLP) tasks. Nevertheless, they still make mistakes from time to time. For…
See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning
Zhenfang Chen, Qinhong Zhou, Yikang Shen +3
Large pre-trained vision and language models have demonstrated remarkable capacities for various tasks. However, solving the knowledge-based visual reasoning tasks remains challeng…
Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners
Zitian Chen, Yikang Shen, Mingyu Ding +4
Optimization in multi-task learning (MTL) is more challenging than single-task learning (STL), as the gradient from different tasks can be contradictory. When tasks are related, it…
Mixture of Attention Heads: Selecting Attention Heads Per Token
Xiaofeng Zhang, Yikang Shen, Zeyu Huang +3
Mixture-of-Experts (MoE) networks have been proposed as an efficient way to scale up model capacity and implement conditional computing. However, the study of MoE components mostly…
Prompting Decision Transformer for Few-Shot Policy Generalization
Mengdi Xu, Yikang Shen, Shun Zhang +4
Humans can leverage prior experience and learn novel tasks from a handful of demonstrations. In contrast to offline meta-reinforcement learning, which aims to achieve quick adaptat…
Syntactic Inductive Biases for Deep Learning Methods
Yikang Shen
In this thesis, we try to build a connection between the two schools by introducing syntactic inductive biases for deep learning models. We propose two families of inductive biases…
Learning Task Decomposition with Ordered Memory Policy Network
Yuchen Lu, Yikang Shen, Siyuan Zhou +3
Many complex real-world tasks are composed of several levels of sub-tasks. Humans leverage these hierarchical structures to accelerate the learning process and achieve better gener…
StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling
Yikang Shen, Yi Tay, Che Zheng +3
There are two major classes of natural language grammar -- the dependency grammar that models one-to-one correspondences between words and the constituency grammar that models the…
Long Range Arena: A Benchmark for Efficient Transformers
Yi Tay, Mostafa Dehghani, Samira Abnar +7
Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transf…
Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle
Yikang Shen, Shawn Tan, Alessandro Sordoni +2
Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization problems and over-parametrization. In the pres…
Recursive Top-Down Production for Sentence Generation with Latent Trees
Shawn Tan, Yikang Shen, Timothy J. O'Donnell +2
We model the recursive production property of context-free grammars for natural and synthetic languages. To this end, we present a dynamic programming algorithm that marginalises o…
Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach
Wenyu Du, Zhouhan Lin, Yikang Shen +3
It is commonly believed that knowledge of syntactic structure should improve language modeling. However, effectively and computationally efficiently incorporating syntactic structu…
Ordered Memory
Yikang Shen, Shawn Tan, Arian Hosseini +3
Stack-augmented recurrent neural networks (RNNs) have been of interest to the deep learning community for some time. However, the difficulty of training memory models remains a pro…