4 papers
POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation
Zeju Qiu, Lixin Liu, Adrian Weller +2
Efficient and stable training of large language models (LLMs) remains a core challenge in modern machine learning systems. To address this challenge, Reparameterized Orthogonal Equ…
Rigidity-Aware Geometric Pretraining for Protein Design and Conformational Ensembles
Zhanghan Ni, Yanjing Li, Zeju Qiu +4
Generative models have recently advanced protein design by learning the statistical regularities of natural structures. However, current approaches face three ke…
Reparameterized LLM Training via Orthogonal Equivalence Transformation
Zeju Qiu, Simon Buchholz, Tim Z. Xiao +3
While large language models (LLMs) are driving the rapid advancement of artificial intelligence, effectively and reliably training these large models remains one of the field's mos…
Orthogonal Finetuning Made Scalable
Zeju Qiu, Weiyang Liu, Adrian Weller +1
Orthogonal finetuning (OFT) offers highly parameter-efficient adaptation while preventing catastrophic forgetting, but its high runtime and memory demands limit practical deploymen…