18 citations · 56 across the 18 of their papers we have counts for
4 papers · 1 filter
MossNet: Mixture of State-Space Experts is a Multi-Head Attention
Shikhar Tuli, James Seale Smith, Haris Jeelani +5
Large language models (LLMs) have significantly advanced generative applications in natural language processing (NLP). Recent trends in model architectures revolve around efficient…
Your contrastive learning problem is secretly a distribution alignment problem
Zihao Chen, Chi-Heng Lin, Ran Liu +2
Despite the success of contrastive learning (CL) in vision and language, its theoretical foundations and mechanisms for building representations remain poorly understood. In this w…
FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing
James Seale Smith, Chi-Heng Lin, Shikhar Tuli +5
The rapid proliferation of large language models (LLMs) in natural language processing (NLP) has created a critical need for techniques that enable efficient deployment on memory-c…
ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
Shangqian Gao, Ting Hua, Reza Shirkavand +10
Large Language Models (LLMs) have demonstrated remarkable abilities in tackling a wide range of complex tasks. However, their huge computational and memory costs raise significant…