From the 1 of 9 linked papers with an AI index.
6 papers · 1 filter
Convolution for Large Language Models
Yuchuan Tian, Yingte Shu, Wei He +7
Large language models (LLMs) largely rely on Transformers, where self-attention provides global token interaction but does not explicitly encode the locality of natural language. W…
VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reuse
Ying Nie, Kai Han, Hongguang Li +5
The rapid scaling of Large Language Models (LLMs) has achieved remarkable performance, but it also leads to prohibitive memory costs. Existing parameter-efficient approaches such a…
Nexus: Higher-Order Attention Mechanisms in Transformers
Hanting Chen, Chong Zhu, Kai Han +6
Transformers have achieved significant success across various domains, relying on self-attention to capture dependencies. However, the standard first-order attention mechanism is o…
PanGu- Pro:Rethinking Optimization and Architecture for Tiny Language Models
Yehui Tang, Kai Han, Fangcheng Liu +7
The power of large language models (LLMs) has been demonstrated through numerous data and computing resources. However, the application of language models on mobile devices is faci…
Unshackling Context Length: An Efficient Selective Attention Approach through Query-Key Compression
Haoyu Wang, Tong Teng, Tianyu Guo +4
Handling long-context sequences efficiently remains a significant challenge in large language models (LLMs). Existing methods for token selection in sequence extrapolation either e…
CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models
Ying Nie, Binwei Yan, Tianyu Guo +9
Large language models (LLMs) have achieved remarkable performance on various NLP tasks, yet their potential in more challenging and domain-specific task, such as finance, has not b…