9 papers
Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Xiang Hu, Xinyu Wei, Hao Gu +10
Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse atten…
RePo: Language Models with Context Re-Positioning
Huayang Li, Tianyu Zhao, Deng Cai +1
In-context learning is fundamental to modern Large Language Models (LLMs); however, prevailing architectures impose a rigid and fixed contextual structure by assigning linear or co…
SeqPE: Transformer with Sequential Position Encoding
Huayang Li, Yahui Liu, Hongyu Sun +5
Since self-attention layers in Transformers are permutation invariant by design, positional encodings must be explicitly incorporated to enable spatial understanding. However, fixe…
GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation
Zhanyu Wang, Longyue Wang, Zhen Zhao +7
While the recent advances in Multimodal Large Language Models (MLLMs) constitute a significant leap forward in the field, these models are predominantly confined to the realm of in…
ALR: A Retrieve-then-Reason Framework for Long-context Question Answering
Huayang Li, Pat Verga, Priyanka Sen +5
The context window of large language models (LLMs) has been extended significantly in recent years. However, while the context length that the LLM can process has grown, the capabi…
Cross-lingual Contextualized Phrase Retrieval
Huayang Li, Deng Cai, Zhi Qu +4
Phrase-level dense retrieval has shown many appealing characteristics in downstream NLP tasks by leveraging the fine-grained information that phrases offer. In our work, we propose…