4 citations · 5 across the 11 of their papers we have counts for
5 papers · 1 filter
ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition
Xindian Ma, Rundong Kong, Peng Zhang +2
LoRA has become a universal Parameter-Efficient Fine-Tuning (PEFT) technique that equips Large Language Models (LLMs) to adapt quickly to new tasks. However, when these models are…
3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding
Xindian Ma, Wenyuan Liu, Peng Zhang +1
Inspired by the Bloch Sphere representation, we propose a novel rotary position encoding on a three-dimensional sphere, named 3D Rotary Position Encoding (3D-RPE). 3D-RPE is an adv…
TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language Modeling
Shuai Zhang, Peng Zhang, Xindian Ma +3
Transformer has been widely-used in many Natural Language Processing (NLP) tasks and the scaled dot-product attention between tokens is a core module of Transformer. This attention…
A Tensorized Transformer for Language Modeling
Xindian Ma, Peng Zhang, Shuai Zhang +4
Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, h…
A Generalized Language Model in Tensor Space
Lipeng Zhang, Peng Zhang, Xindian Ma +3
In the literature, tensors have been effectively used for capturing the context information in language models. However, the existing methods usually adopt relatively-low order ten…