8 papers
Revisiting the Shape Convention of Transformer Language Models
Feng-Ting Liao, Meng-Hsi Chen, Guan-Ting Yi +1
Dense Transformer language models have largely adhered to one consistent architectural shape: each layer consists of an attention module followed by a feed-forward network (FFN) wi…
Rethinking the shape convention of an MLP
Meng-Hsi Chen, Yu-Ang Lee, Feng-Ting Liao +1
Multi-layer perceptrons (MLPs) conventionally follow a narrow-wide-narrow design where skip connections operate at the input/output dimensions while processing occurs in expanded h…
RISE: Reasoning Enhancement via Iterative Self-Exploration in Multi-hop Question Answering
Bolei He, Xinran He, Mengke Chen +3
Large Language Models (LLMs) excel in many areas but continue to face challenges with complex reasoning tasks, such as Multi-Hop Question Answering (MHQA). MHQA requires integratin…
Latent Flow Transformer
Yen-Chen Wu, Feng-Ting Liao, Meng-Hsi Chen +3
Transformers, the standard implementation for large language models (LLMs), typically consist of tens to hundreds of discrete layers. While more layers can lead to better performan…
The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Models
Zhivar Sourati, Farzan Karimi-Malekabadi, Meltem Ozcan +7
Language is far more than a communication tool. A wealth of information - including but not limited to the identities, psychological states, and social contexts of its users - can…
On the Effectiveness of Large Language Models in Domain-Specific Code Generation
Xiaodong Gu, Meng Chen, Yalan Lin +6
Large language models (LLMs) such as ChatGPT have shown remarkable capabilities in code generation. Despite significant achievements, they rely on enormous training data to acquire…