2 papers
cs.LG2024
SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression
Jing Zhang, Shuzhen Sun, Peng Zhang +5
Transformer-based large language models exhibit groundbreaking capabilities, but their storage and computational costs are prohibitively high, limiting their application in resourc…
cs.CL2024
3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding
Xindian Ma, Wenyuan Liu, Peng Zhang +1
Inspired by the Bloch Sphere representation, we propose a novel rotary position encoding on a three-dimensional sphere, named 3D Rotary Position Encoding (3D-RPE). 3D-RPE is an adv…